pysys.writer

Writers are configurable plug-ins that record test outcomes (typically on disk, on the console, or to your CI tooling).

This module contains the BaseResultsWriter abstract class which defines the writer API, as well as several sample implementations.

Output writers are responsible for summarising test results as each test completes, or at the end when all tests has completed. There are currently three distinct types of writers, namely ‘Record’, ‘Progress’, and ‘Summary’, each of which generates output at different stages of a run:

• BaseRecordResultsWriter: Record writers output the outcome of a specific test after completion of that test, to allow runtime auditing of the test output, e.g. into text file, a database, or to the console in a format that can be read by your Continuous Integration (CI) tooling. Several record writers are distributed with the PySys framework, such as the JUnitXMLResultsWriter and ci.TravisCIWriter. By default, record writers are enabled only when the --record flag is given to the PySys launcher, though some writers may enable/disable themselves under different conditions, by overriding the BaseResultsWriter.isEnabled method.

• BaseProgressResultsWriter: Progress writers output a summary of the test progress after completion of each test, to give an indication of how far and how well the run is progressing. A single implementation of a progress writer is distributed with the PySys framework, namely the ConsoleProgressResultsWriter, which details the percentage of tests selected to be run and that have executed, and a summary of the recent test failures. Progress writers should extend the BaseProgressResultsWriter and are enabled when the --progress flag is given to the PySys launcher, or when PYSYS_PROGRESS=true is set in the local environment.

• BaseSummaryResultsWriter: Summary writers output an overall summary of the status at the end of a test run. A single implementation of a summary writer is distributed with the PySys framework, namely the ConsoleSummaryResultsWriter, which details the overall test run outcome and lists any tests that did not pass. Summary writers are always enabled regardless of the flags given to the PySys launcher.

Project configuration of the writers is through the PySys project XML file using the <writer> tag. Multiple writers may be configured and their individual properties set through the nested <property> tag. Writer properties are set as attributes to the class through the setattr() function. Custom (site specific) modules can be created and configured by users of the PySys framework (e.g. to output test results into a relational database etc), though they must adhere to the interface demonstrated by the implementations demonstrated here. If no progress writers are explicitly configured in the PySys project XML file, an instance of ConsoleProgressResultsWriter is used. If no summary writer is explicitly configured in the PySys project XML file, an instance of ConsoleSummaryResultsWriter is used.

The writers are instantiated and invoked by the pysys.baserunner.BaseRunner class instance. This calls the class constructors of all configured test writers, and then the setup (prior to executing the set of tests), processResult (process a test result), and cleanup (upon completion of the execution of all tests). The **kwargs method parameter is used for variable argument passing in the interface methods to allow modification of the PySys framework without breaking writer implementations already in existence.

pysys.writer.ci

Contains writers for recording test results to Continuous Integration providers.

BaseResultsWriter

class pysys.writer.BaseResultsWriter(logfile=None, **kwargs)

Bases: object

Base class for all writers that get notified as and when test results are available.

Parameters

• logfile (str) – Optional configuration property specifying a file to store output in. Does not apply to all writers, can be ignored if not needed.

• kwargs – Additional keyword arguments may be added in a future release.

cleanup(**kwargs)

Called after all tests have finished executing (or been cancelled).

This is where file headers can be written, and open handles should be closed.

Parameters

kwargs – Additional keyword arguments may be added in a future release.

isEnabled(record=False, **kwargs)

Determines whether this writer can be used in the current environment.

If set to False then after construction none of the other methods (including setup)) will be called.

Parameters

record – True if the user ran PySys with the --record flag, indicating that test results should be recorded.

Returns

For record writers, the default to enable only if record==True, but individual writers can use different criteria if desired, e.g. writers for logging output to a CI system may enable themselves based on environment variables indicating that system is present, even if record is not specified explicitly.

processResult(testObj, cycle=0, testTime=0, testStart=0, runLogOutput='', **kwargs)

Called when each test has completed.

This method is always invoked under a lock that prevents multiple concurrent invocations so additional locking is not usually necessary.

Parameters

• testObj (pysys.basetest.BaseTest) – Reference to an instance of a pysys.basetest.BaseTest class. The writer can extract data from this object but should not store a reference to it. The testObj.descriptor.id indicates the test that ran.

• cycle (int) – The cycle number. These start from 0, so please add 1 to this value before using.

• testTime (float) – Duration of the test in seconds as a floating point number.

• testStart (float) – The time when the test started.

• runLogOutput (str) – The logging output written to run.log, as a unicode character string.

• kwargs – Additional keyword arguments may be added in future releases.

processTestStarting(testObj, cycle=- 1, **kwargs)

Called when a test is just about to begin executing.

Note on thread-safety: unlike the other methods on this interface, this is usually executed on a worker thread, so any data structures accessed in this method and others on this class must be synchronized if performing non-atomic operations.

Parameters

• testObj – Reference to an instance of a pysys.basetest.BaseTest class. The writer can extract data from this object but should not store a reference to it. The testObj.descriptor.id indicates the test that ran.

• cycle – The cycle number. These start from 0, so please add 1 to this value before using.

• kwargs – Additional keyword arguments may be added in a future release.

setup(numTests=0, cycles=1, xargs=None, threads=0, testoutdir='', runner=None, **kwargs)

Called before any tests begin.

Before this method is called, for each property “PROP” specified for this writer in the project configuration file, the configured value will be assigned to self.PROP.

Parameters

• numTests – The total number of tests (cycles*testids) to be executed

• cycles – The number of cycles.

• xargs – The runner’s xargs

• threads – The number of threads used for running tests.

• testoutdir – The output directory used for this test run (equal to runner.outsubdir), an identifying string which often contains the platform, or when there are multiple test runs on the same machine may be used to distinguish between them. This is usually a relative path but may be an absolute path.

• runner – The runner instance that owns this writer.

• kwargs – Additional keyword arguments may be added in a future release.

BaseRecordResultsWriter

class pysys.writer.BaseRecordResultsWriter(logfile=None, **kwargs)

Bases: pysys.writer.BaseResultsWriter

Base class for writers that record the results of tests, and are enabled only when the --record flag is specified.

For compatibility reasons writers that do not subclass BaseSummaryResultsWriter or BaseProgressResultsWriter are treated as “record” writers even if they do not inherit from this class.

BaseSummaryResultsWriter

class pysys.writer.BaseSummaryResultsWriter(logfile=None, **kwargs)

Bases: pysys.writer.BaseResultsWriter

Base class for writers that display a summary of test results.

Summary writers are always enabled (regardless of whether --progress or --record are specified). If no “summary” writers are configured, a default ConsoleSummaryResultsWriter instance will be added automatically.

Summary writers are invoked after all other writers, ensuring that their output will be displayed after output from any other writer types.

isEnabled(record=False, **kwargs)

Determines whether this writer can be used in the current environment.

If set to False then after construction none of the other methods (including setup)) will be called.

Parameters

record – True if the user ran PySys with the --record flag, indicating that test results should be recorded.

Returns

For record writers, the default to enable only if record==True, but individual writers can use different criteria if desired, e.g. writers for logging output to a CI system may enable themselves based on environment variables indicating that system is present, even if record is not specified explicitly.

BaseProgressResultsWriter

class pysys.writer.BaseProgressResultsWriter(logfile=None, **kwargs)

Bases: pysys.writer.BaseResultsWriter

Base class for writers that display progress information while tests are running.

Progress writers are only enabled if the --progress flag is specified.

isEnabled(record=False, **kwargs)

Determines whether this writer can be used in the current environment.

If set to False then after construction none of the other methods (including setup)) will be called.

Parameters

record – True if the user ran PySys with the --record flag, indicating that test results should be recorded.

Returns

For record writers, the default to enable only if record==True, but individual writers can use different criteria if desired, e.g. writers for logging output to a CI system may enable themselves based on environment variables indicating that system is present, even if record is not specified explicitly.

TextResultsWriter

class pysys.writer.TextResultsWriter(logfile, **kwargs)

Bases: pysys.writer.BaseRecordResultsWriter

Class to log results to logfile in text format.

Writing of the test summary file defaults to the working directory. This can be be overridden in the PySys project file using the nested <property> tag on the <writer> tag.

Variables

outputDir (str) – Path to output directory to write the test summary files

cleanup(**kwargs)

Implementation of the cleanup method.

Flushes and closes the file handle to the logfile.

Parameters

kwargs – Variable argument list

processResult(testObj, **kwargs)

Implementation of the processResult method.

Writes the test id and outcome to the logfile.

Parameters

• testObj – Reference to an instance of a pysys.basetest.BaseTest class

• kwargs – Variable argument list

setup(**kwargs)

Implementation of the setup method.

Creates the file handle to the logfile and logs initial details of the date, platform and test host.

Parameters

kwargs – Variable argument list

XMLResultsWriter

class pysys.writer.XMLResultsWriter(logfile, **kwargs)

Bases: pysys.writer.BaseRecordResultsWriter

Class to log results to logfile in XML format.

The class creates a DOM document to represent the test output results and writes the DOM to the logfile using toprettyxml(). The outputDir, stylesheet, useFileURL attributes of the class can be over-ridden in the PySys project file using the nested <property> tag on the <writer> tag.

Variables

• outputDir (str) – Path to output directory to write the test summary files

• stylesheet (str) – Path to the XSL stylesheet

• useFileURL (str) – Indicates if full file URLs are to be used for local resource references

cleanup(**kwargs)

Implementation of the cleanup method.

Updates the test run status in the DOM, and re-writes to logfile.

Parameters

kwargs – Variable argument list

processResult(testObj, **kwargs)

Implementation of the processResult method.

Adds the results node to the DOM and re-writes to logfile.

Parameters

• testObj – Reference to an instance of a pysys.basetest.BaseTest class

• kwargs – Variable argument list

setup(**kwargs)

Implementation of the setup method.

Creates the DOM for the test output summary and writes to logfile.

Parameters

kwargs – Variable argument list

JUnitXMLResultsWriter

class pysys.writer.JUnitXMLResultsWriter(**kwargs)

Bases: pysys.writer.BaseRecordResultsWriter

Class to log test results in Apache Ant JUnit XML format (one output file per test per cycle).

Variables

outputDir (str) – Path to output directory to write the test summary files

cleanup(**kwargs)

Implementation of the cleanup method.

Parameters

kwargs – Variable argument list

processResult(testObj, **kwargs)

Implementation of the processResult method.

Creates a test summary file in the Apache Ant JUnit XML format.

Parameters

• testObj – Reference to an instance of a pysys.basetest.BaseTest class

• kwargs – Variable argument list

purgeDirectory(dir, delTop=False)

Deprecated

Use pysys.utils.fileutils.deletedir instead.

setup(**kwargs)

Implementation of the setup method.

Creates the output directory for the writing of the test summary files.

Parameters

kwargs – Variable argument list

CSVResultsWriter

class pysys.writer.CSVResultsWriter(logfile, **kwargs)

Bases: pysys.writer.BaseRecordResultsWriter

Class to log results to logfile in CSV format.

Writing of the test summary file defaults to the working directory. This can be be over-ridden in the PySys project file using the nested <property> tag on the <writer> tag. The CSV column output is in the form;

id, title, cycle, startTime, duration, outcome

Variables

outputDir (str) – Path to output directory to write the test summary files

cleanup(**kwargs)

Implementation of the cleanup method.

Flushes and closes the file handle to the logfile.

Parameters

kwargs – Variable argument list

processResult(testObj, **kwargs)

Implementation of the processResult method.

Writes the test id and outcome to the logfile.

Parameters

• testObj – Reference to an instance of a pysys.basetest.BaseTest class

• kwargs – Variable argument list

setup(**kwargs)

Implementation of the setup method.

Creates the file handle to the logfile and logs initial details of the date, platform and test host.

Parameters

kwargs – Variable argument list

ConsoleSummaryResultsWriter

class pysys.writer.ConsoleSummaryResultsWriter(**kwargs)

Bases: pysys.writer.BaseSummaryResultsWriter

Default summary writer that is used to list a summary of the test results at the end of execution.

cleanup(**kwargs)

Called after all tests have finished executing (or been cancelled).

This is where file headers can be written, and open handles should be closed.

Parameters

kwargs – Additional keyword arguments may be added in a future release.

processResult(testObj, cycle=- 1, testTime=- 1, testStart=- 1, **kwargs)

Called when each test has completed.

This method is always invoked under a lock that prevents multiple concurrent invocations so additional locking is not usually necessary.

Parameters

• testObj (pysys.basetest.BaseTest) – Reference to an instance of a pysys.basetest.BaseTest class. The writer can extract data from this object but should not store a reference to it. The testObj.descriptor.id indicates the test that ran.

• cycle (int) – The cycle number. These start from 0, so please add 1 to this value before using.

• testTime (float) – Duration of the test in seconds as a floating point number.

• testStart (float) – The time when the test started.

• runLogOutput (str) – The logging output written to run.log, as a unicode character string.

• kwargs – Additional keyword arguments may be added in future releases.

setup(cycles=0, threads=0, **kwargs)

Called before any tests begin.

Before this method is called, for each property “PROP” specified for this writer in the project configuration file, the configured value will be assigned to self.PROP.

Parameters

• numTests – The total number of tests (cycles*testids) to be executed

• cycles – The number of cycles.

• xargs – The runner’s xargs

• threads – The number of threads used for running tests.

• testoutdir – The output directory used for this test run (equal to runner.outsubdir), an identifying string which often contains the platform, or when there are multiple test runs on the same machine may be used to distinguish between them. This is usually a relative path but may be an absolute path.

• runner – The runner instance that owns this writer.

• kwargs – Additional keyword arguments may be added in a future release.

ConsoleProgressResultsWriter

class pysys.writer.ConsoleProgressResultsWriter(**kwargs)

Bases: pysys.writer.BaseProgressResultsWriter

Default progress writer that logs a summary of progress so far to the console, after each test completes.

processResult(testObj, cycle=- 1, **kwargs)

Called when each test has completed.

This method is always invoked under a lock that prevents multiple concurrent invocations so additional locking is not usually necessary.

Parameters

• testObj (pysys.basetest.BaseTest) – Reference to an instance of a pysys.basetest.BaseTest class. The writer can extract data from this object but should not store a reference to it. The testObj.descriptor.id indicates the test that ran.

• cycle (int) – The cycle number. These start from 0, so please add 1 to this value before using.

• testTime (float) – Duration of the test in seconds as a floating point number.

• testStart (float) – The time when the test started.

• runLogOutput (str) – The logging output written to run.log, as a unicode character string.

• kwargs – Additional keyword arguments may be added in future releases.

processTestStarting(testObj, cycle=- 1, **kwargs)

Called when a test is just about to begin executing.

Note on thread-safety: unlike the other methods on this interface, this is usually executed on a worker thread, so any data structures accessed in this method and others on this class must be synchronized if performing non-atomic operations.

Parameters

• testObj – Reference to an instance of a pysys.basetest.BaseTest class. The writer can extract data from this object but should not store a reference to it. The testObj.descriptor.id indicates the test that ran.

• cycle – The cycle number. These start from 0, so please add 1 to this value before using.

• kwargs – Additional keyword arguments may be added in a future release.

setup(cycles=- 1, numTests=- 1, threads=- 1, **kwargs)

Called before any tests begin.

Before this method is called, for each property “PROP” specified for this writer in the project configuration file, the configured value will be assigned to self.PROP.

Parameters

• numTests – The total number of tests (cycles*testids) to be executed

• cycles – The number of cycles.

• xargs – The runner’s xargs

• threads – The number of threads used for running tests.

• testoutdir – The output directory used for this test run (equal to runner.outsubdir), an identifying string which often contains the platform, or when there are multiple test runs on the same machine may be used to distinguish between them. This is usually a relative path but may be an absolute path.

• runner – The runner instance that owns this writer.

• kwargs – Additional keyword arguments may be added in a future release.