pysys.writer.testoutput

Writers that process the contents of test output directories, archiving or collecting output files, or aggregating code coverage data.

TestOutputArchiveWriter

class pysys.writer.testoutput.TestOutputArchiveWriter(logfile=None, **kwargs)[source]

Bases: pysys.writer.api.BaseRecordResultsWriter

Writer that creates zip archives of each failed test’s output directory, producing artifacts that could be uploaded to a CI system or file share to allow the failures to be analysed.

This writer is enabled when running with --record. If using this writer in conjunction with a CI writer that publishes the generated archives, be sure to include this writer first in the list of writers in your project configuration.

Publishes artifacts with category name “TestOutputArchive” and the directory (unless there are no archives) as “TestOutputArchiveDir” for any enabled pysys.writer.api.ArtifactPublisher writers.

New in version 1.6.0.

The following properties can be set in the project configuration for this writer:

_archiveTestOutputDir(id, outputDir, **kwargs)[source]

Creates an archive for the specified test, unless doing so would violate the configured limits (e.g. maxArchives).

Parameters

  • id (str) – The testId (plus a cycle suffix if it’s a multi-cycle run).

  • outputDir (str) – The path of the test output dir.

_newArchive(id, **kwargs)[source]

Creates and opens a new archive file for the specified id.

Returns

(str path, filehandle) The path will include an appropriate extension for this archive type. The filehandle must have the same API as Python’s ZipFile class.

archiveAtEndOfRun = True

By default all archives are created at the end of the run once all tests have finished executing. This avoids I/O contention with execution of tests, and also selection of the tests to generated archives to be done in a deterministic (but pseudo-random) fashion rather than just taking the first N failures.

Alternatively you can this property to false if you wish to create archives during the test run as each failure occurs.

cleanup(**kwargs)[source]

Called after all tests have finished executing (or been cancelled).

This is where file headers can be written, and open handles should be closed.

Parameters

kwargs – Additional keyword arguments may be added in a future release.

destDir = '__pysys_output_archives.${outDirName}/'

The directory to write the archives to, as an absolute path, or relative to the testRootDir (or –outdir if specified).

This directory will be deleted at the start of the run if it already exists.

Project ${...} properties can be used in the path.

fileExcludesRegex = ''

A regular expression indicating test output paths that will be excluded from archiving, for example large temporary files that are not useful for diagnosing problems.

For example ".*/MyTest_001/.*/mybigfile.*[.]tmp".

The expression is matched against the path of each output file relative to the test root dir, using forward slashes as the path separator. Multiple paths can be specified using “(path1|path2)” syntax.

fileIncludesRegex = ''

A regular expression indicating test output paths that will be included in the archive. This can be used to archive just some particular files. Note that for use cases such as collecting graphs and code coverage files generated by a test run, the collect-test-output feature is usually a better fit than using this writer.

The expression is matched against the path of each output file relative to the test root dir, using forward slashes as the path separator. Multiple paths can be specified using “(path1|path2)” syntax.

includeNonFailureOutcomes = 'REQUIRES INSPECTION'

In addition to failure outcomes, any outcomes listed here (as comma-separated display names) will be archived.

maxArchiveSizeMB = 200.0

The (approximate) limit on the size each individual test archive.

maxArchives = 50

The maximum number of archives to create.

maxTotalSizeMB = 1024.0

The (approximate) limit on the total size of all archives.

processResult(testObj, cycle=0, testTime=0, testStart=0, runLogOutput='', **kwargs)[source]

Called when each test has completed.

This method is always invoked under a lock that prevents multiple concurrent invocations so additional locking is not usually necessary.

Parameters

  • testObj (pysys.basetest.BaseTest) – Reference to an instance of a pysys.basetest.BaseTest class. The writer can extract data from this object but should not store a reference to it. The testObj.descriptor.id indicates the test that ran.

  • cycle (int) – The cycle number. These start from 0, so please add 1 to this value before using.

  • testTime (float) – Duration of the test in seconds as a floating point number.

  • testStart (float) – The time when the test started.

  • runLogOutput (str) – The logging output written to the console/run.log, as a unicode character string. This string will include ANSI escape codes if colored output is enabled; if desired these can be removed using pysys.utils.logutils.stripANSIEscapeCodes().

  • kwargs – Additional keyword arguments may be added in future releases.

setup(numTests=0, cycles=1, xargs=None, threads=0, testoutdir='', runner=None, **kwargs)[source]

Called before any tests begin.

Before this method is called, for each property “PROP” specified for this writer in the project configuration file, the configured value will be assigned to self.PROP.

Parameters

  • numTests – The total number of tests (cycles*testids) to be executed

  • cycles – The number of cycles.

  • xargs – The runner’s xargs

  • threads – The number of threads used for running tests.

  • testoutdir – The output directory used for this test run (equal to runner.outsubdir), an identifying string which often contains the platform, or when there are multiple test runs on the same machine may be used to distinguish between them. This is usually a relative path but may be an absolute path.

  • runner – The runner instance that owns this writer. The default implementation of this methods sets the self.runner attribute to this value.

  • kwargs – Additional keyword arguments may be added in a future release.

shouldArchive(testObj, **kwargs)[source]

Decides whether this test is eligible for archiving of its output.

The default implementation archives only tests that have a failure outcome, or are listed in includeNonFailureOutcomes, but this can be customized if needed by subclasses.

Parameters

testObj (pysys.basetest.BaseTest) – The test object under consideration.

Return bool

True if this test’s output can be archived.

CollectTestOutputWriter

class pysys.writer.testoutput.CollectTestOutputWriter(logfile=None, **kwargs)[source]

Bases: pysys.writer.api.BaseRecordResultsWriter, pysys.writer.api.TestOutputVisitor

Writer that collects files matching a specified pattern from the output directory after each test, and puts them in a single directory or archive - for example code coverage files or performance graphs.

This writer can be used as-is or as a base class for writers that need to collect files during test execution then do something with them during cleanup, for example generate a code coverage report.

Empty files are ignored.

This writer is always enabled.

New in version 1.6.0.

The following properties can be set in the project configuration for this writer:

archiveAndPublish()[source]

Generate an archive of the destDir (if configured) and publish artifacts (if configured).

Called by default as part of cleanup().

cleanup(**kwargs)[source]

Called after all tests have finished executing (or been cancelled).

This is where file headers can be written, and open handles should be closed.

Parameters

kwargs – Additional keyword arguments may be added in a future release.

destArchive = ''

Optional filename of a .zip archive to generate with the contents of the destDir.

If an absolute path is specified it is evaluated relative to the destDir.

Project ${...} properties can be used in the path.

destDir = ''

The directory in which the files will be collected, as an absolute path, or relative to the testRootDir (or –outdir if specified).

This directory will be deleted at the start of the run if it already exists.

Project ${...} properties can be used in the path.

fileExcludesRegex = ''

A regular expression indicating test output paths that will be excluded from collection.

For example ".*/MyTest_001/.*/mybigfile.*[.]tmp".

The expression is matched against the path as described for fileIncludesRegex.

fileIncludesRegex = ''

A regular expression indicating the test output paths that will be collected. This can be used to archive just some particular files. This is required.

The expression is matched against the final characters of each output file’s path (with the test root dir stripped off), using forward slashes as the path separator. Multiple paths can be specified using “(path1|path2)” syntax.

isEnabled(record=False, **kwargs)[source]

Determines whether this writer can be used in the current environment.

If set to False then after construction none of the other methods (including setup)) will be called.

Parameters

record – True if the user ran PySys with the --record flag, indicating that test results should be recorded.

Returns

For record writers, the default to enable only if record==True, but individual writers can use different criteria if desired, e.g. writers for logging output to a CI system may enable themselves based on environment variables indicating that system is present, even if record is not specified explicitly.

outputPattern = '@TESTID@.@FILENAME@.@UNIQUE@.@FILENAME_EXT@'

A string indicating the file (and optionally subdirectory name) to use when writing each collected file to the destDir.

In addition to any standard ${...} property variables from the project configuration, the output pattern can contain these @...@ substitutions:

  • @FILENAME@ is the original base filename with directory and extension removed, to which you can add prefixes or suffixes as desired.

  • .@FILENAME_EXT@ is the filename extension, such that the original filename is @FILENAME@.@FILENAME_EXT@ (note the dot prefix is mandatory here, and will be replaced with empty string is there is no extension).

  • @TESTID@ is replaced by the identifier of the test that generated the output file (including mode suffix if present), which may be useful for tracking where each one came from.

  • @UNIQUE@ is replaced by a number that ensures the file does not clash with any other collected output file from another test. The @UNIQUE@ substitution variable is mandatory.

publishArtifactArchiveCategory = ''

If specified the destArchive file (if any) will be published as an artifact using the specified category name.

publishArtifactDirCategory = ''

If specified, the output directory will be published as an artifact using the specified category name, e.g. MyCodeCoverageDir.

setup(numTests=0, cycles=1, xargs=None, threads=0, testoutdir='', runner=None, **kwargs)[source]

Called before any tests begin.

Before this method is called, for each property “PROP” specified for this writer in the project configuration file, the configured value will be assigned to self.PROP.

Parameters

  • numTests – The total number of tests (cycles*testids) to be executed

  • cycles – The number of cycles.

  • xargs – The runner’s xargs

  • threads – The number of threads used for running tests.

  • testoutdir – The output directory used for this test run (equal to runner.outsubdir), an identifying string which often contains the platform, or when there are multiple test runs on the same machine may be used to distinguish between them. This is usually a relative path but may be an absolute path.

  • runner – The runner instance that owns this writer. The default implementation of this methods sets the self.runner attribute to this value.

  • kwargs – Additional keyword arguments may be added in a future release.

visitTestOutputFile(testObj, path, **kwargs)[source]

Called after execution of each test (and before purging of files) for each file found in the output directory.

Parameters

  • testObj (pysys.basetest.BaseTest) – The test object, which can be used to find the outcome etc.

  • str – The absolute, normalize path to the output file (will be a ?long path safe path on windows).

Return bool

Return True if this visitor has handled this file fully (e.g. by deleting it) and it should not be passed on to any other registered visitors.

PythonCoverageWriter

class pysys.writer.testoutput.PythonCoverageWriter(logfile=None, **kwargs)[source]

Bases: pysys.writer.testoutput.CollectTestOutputWriter

Writer that collects Python code coverage files in a single directory and writes a coverage report during runner cleanup. Requires the “coverage.py” library to be installed.

To enable this, run with -XcodeCoverage (or -XpythonCoverage) and configure the destDir plugin property in pysysproject.xml (e.g. to __coverage_python.${outDirName}).

If coverage is generated, the directory containing all coverage files is published as an artifact named “PythonCoverageDir”. Optionally an archive of this directory can be generated by setting the destArchive property (see CollectTestOutputWriter), and published as “PythonCoverageArchive”.

Note that to maintain compatibility with pre-1.6.0 projects, a PythonCoverageWriter instance will be _automatically_ added to the project if the pythonCoverageDir project property is set and none is explicitly configured; but the automatic addition is deprecated so you should explicity add this writer to your project if you need it.

New in version 1.6.0.

The following property can be set in the project configuration for this writer (and see also CollectTestOutputWriter for inherited properties such as destArchive):

cleanup(**kwargs)[source]

Called after all tests have finished executing (or been cancelled).

This is where file headers can be written, and open handles should be closed.

Parameters

kwargs – Additional keyword arguments may be added in a future release.

isEnabled(record=False, **kwargs)[source]

Determines whether this writer can be used in the current environment.

If set to False then after construction none of the other methods (including setup)) will be called.

Parameters

record – True if the user ran PySys with the --record flag, indicating that test results should be recorded.

Returns

For record writers, the default to enable only if record==True, but individual writers can use different criteria if desired, e.g. writers for logging output to a CI system may enable themselves based on environment variables indicating that system is present, even if record is not specified explicitly.

pythonCoverageArgs = ''

A string of command line arguments used to customize the coverage run and coverage html commands. Use “…” double quotes around any arguments that contain spaces.

For example:

<property name="pythonCoverageArgs" value="--rcfile=${testRootDir}/python_coveragerc"/>