pysys.writer.ci

Writers for recording test results to Continuous Integration providers.

These writers only generate output if the environment variables associated with their CI system are set.

If you wish to run PySys from a CI provider that’s not listed here, for basic functionality all you need to do is run the tests with the --ci option. Many CI providers (including Jenkins) will be able to read the results of PySys tests if you configure them to look in the results directory generated by pysys.writer.outcomes.JUnitXMLResultsWriter. It may also be possible to configure some providers to automatically pick up archived test output from artifacts from the pysys.writer.testoutput.TestOutputArchiveWriter directory.

If you want to implement additional functionality for your CI provider, subclass pysys.writer.api.BaseRecordResultsWriter (using the classes in this module as an example), and consider contributing your implementation back to the PySys project with a pull request.

GitHubActionsCIWriter

class pysys.writer.ci.GitHubActionsCIWriter(logfile=None, **kwargs)

Bases: pysys.writer.api.BaseRecordResultsWriter, pysys.writer.api.TestOutcomeSummaryGenerator, pysys.writer.api.ArtifactPublisher

Writer for GitHub(R) Actions.

Produces annotations summarizing failures, adds grouping/folding of detailed test output, and sets step output variables for any published artifacts (e.g. performance .csv files, archived test output etc) which can be used to upload the artifacts when present. Step output variables for published artifacts are named ‘artifact_CATEGORY’ (for more details on artifact categories see pysys.writer.api.ArtifactPublisher.publishArtifact).

For example, if the step is given the id pysys then the directory containing archived test output for failures can be uploaded using {{ steps.pysys.outputs.artifact_TestOutputArchiveDir }}.

Be sure to include a unique run id (e.g. outdir) for each OS/job in the name of any uploaded artifacts so that they do not overwrite each other when uploaded.

See the PySys sample projects on GitHub(R) for a workflow file you can copy into your own project to enable running your PySys tests with GitHub(R) Actions.

Only enabled when running under GitHub Actions (specifically, if the GITHUB_ACTIONS=true environment variable is set).

cleanup(**kwargs)

Called after all tests have finished executing (or been cancelled).

This is where file headers can be written, and open handles should be closed.

Parameters

kwargs – Additional keyword arguments may be added in a future release.

failureSummaryAnnotations = True

Configures whether a single annotation is added with a summary of the number of failures and the outcome and reason for all failures.

failureTestLogAnnotations = True

Configures whether annotations are added with the (run.log) log output for each of the first few test failures.

isEnabled(**kwargs)

Determines whether this writer can be used in the current environment.

If set to False then after construction none of the other methods (including setup)) will be called.

Parameters

record – True if the user ran PySys with the --record flag, indicating that test results should be recorded.

Returns

For record writers, the default to enable only if record==True, but individual writers can use different criteria if desired, e.g. writers for logging output to a CI system may enable themselves based on environment variables indicating that system is present, even if record is not specified explicitly.

maxAnnotations = 10

Configures the maximum number of annotations generated by this invocation of PySys, to cope with GitHub limits.

This ensure we don’t use up our entire allocation of annotations leaving no space for annotations from other tools. Being aware of this limit also allows to us add a warning to the end of the last one to make clear no more annotations will be shown even if there are more warnings.

processResult(testObj, cycle=0, testTime=0, testStart=0, runLogOutput='', **kwargs)

Called when each test has completed.

This method is always invoked under a lock that prevents multiple concurrent invocations so additional locking is not usually necessary.

Parameters

• testObj (pysys.basetest.BaseTest) – Reference to an instance of a pysys.basetest.BaseTest class. The writer can extract data from this object but should not store a reference to it. The testObj.descriptor.id indicates the test that ran.

• cycle (int) – The cycle number. These start from 0, so please add 1 to this value before using.

• testTime (float) – Duration of the test in seconds as a floating point number.

• testStart (float) – The time when the test started.

• runLogOutput (str) – The logging output written to the console/run.log, as a unicode character string. This string will include ANSI escape codes if colored output is enabled; if desired these can be removed using pysys.utils.logutils.stripANSIEscapeCodes().

• kwargs – Additional keyword arguments may be added in future releases.

publishArtifact(path, category, **kwargs)

Called when a file or directory artifact has been written and is ready to be published (e.g. by another writer).

Parameters

• path (str) – Absolute path of the file or directory, using forward slashes as the path separator.

• category (str) – A string identifying what kind of artifact this is, e.g. “TestOutputArchive” and “TestOutputArchiveDir” (from pysys.writer.testoutput.TestOutputArchiveWriter) or “CSVPerformanceReport” (from pysys.utils.perfreporter.CSVPerformanceReporter). If you create your own category, be sure to add an org/company name prefix to avoid clashes.

setup(numTests=0, cycles=1, xargs=None, threads=0, testoutdir='', runner=None, **kwargs)

Called before any tests begin.

Before this method is called, for each property “PROP” specified for this writer in the project configuration file, the configured value will be assigned to self.PROP.

Parameters

• numTests – The total number of tests (cycles*testids) to be executed

• cycles – The number of cycles.

• xargs – The runner’s xargs

• threads – The number of threads used for running tests.

• testoutdir – The output directory used for this test run (equal to runner.outsubdir), an identifying string which often contains the platform, or when there are multiple test runs on the same machine may be used to distinguish between them. This is usually a relative path but may be an absolute path.

• runner – The runner instance that owns this writer. The default implementation of this methods sets the self.runner attribute to this value.

• kwargs – Additional keyword arguments may be added in a future release.

TravisCIWriter

class pysys.writer.ci.TravisCIWriter(logfile=None, **kwargs)

Bases: pysys.writer.api.BaseRecordResultsWriter

Writer for Travis CI(R).

Provides folding of test log details, and correct coloring of console output for Travis.

Only enabled when running under Travis (specifically, if the TRAVIS=true environment variable is set).

cleanup(**kwargs)

Called after all tests have finished executing (or been cancelled).

This is where file headers can be written, and open handles should be closed.

Parameters

kwargs – Additional keyword arguments may be added in a future release.

isEnabled(**kwargs)

Determines whether this writer can be used in the current environment.

If set to False then after construction none of the other methods (including setup)) will be called.

Parameters

record – True if the user ran PySys with the --record flag, indicating that test results should be recorded.

Returns

For record writers, the default to enable only if record==True, but individual writers can use different criteria if desired, e.g. writers for logging output to a CI system may enable themselves based on environment variables indicating that system is present, even if record is not specified explicitly.

processResult(testObj, cycle=0, testTime=0, testStart=0, runLogOutput='', **kwargs)

Called when each test has completed.

This method is always invoked under a lock that prevents multiple concurrent invocations so additional locking is not usually necessary.

Parameters

• testObj (pysys.basetest.BaseTest) – Reference to an instance of a pysys.basetest.BaseTest class. The writer can extract data from this object but should not store a reference to it. The testObj.descriptor.id indicates the test that ran.

• cycle (int) – The cycle number. These start from 0, so please add 1 to this value before using.

• testTime (float) – Duration of the test in seconds as a floating point number.

• testStart (float) – The time when the test started.

• runLogOutput (str) – The logging output written to the console/run.log, as a unicode character string. This string will include ANSI escape codes if colored output is enabled; if desired these can be removed using pysys.utils.logutils.stripANSIEscapeCodes().

• kwargs – Additional keyword arguments may be added in future releases.

setup(numTests=0, cycles=1, xargs=None, threads=0, testoutdir='', runner=None, **kwargs)

Called before any tests begin.

Before this method is called, for each property “PROP” specified for this writer in the project configuration file, the configured value will be assigned to self.PROP.

Parameters

• numTests – The total number of tests (cycles*testids) to be executed

• cycles – The number of cycles.

• xargs – The runner’s xargs

• threads – The number of threads used for running tests.

• testoutdir – The output directory used for this test run (equal to runner.outsubdir), an identifying string which often contains the platform, or when there are multiple test runs on the same machine may be used to distinguish between them. This is usually a relative path but may be an absolute path.

• runner – The runner instance that owns this writer. The default implementation of this methods sets the self.runner attribute to this value.

• kwargs – Additional keyword arguments may be added in a future release.