pysys.writer.api¶
API for creating new writers.
Writers are responsible for summarising test results or processing test output as each test completes, or at the end when all tests has completed.
The most common type of writer is the standard ‘Record’ writer, but there are also ‘Progress’, and ‘Summary’ writers which do their things at different stages of the test run:
BaseRecordResultsWriter
: Record writers record the outcome or output of tests, allowing runtime auditing of the test output, e.g. into text file, a database, or to the console in a format that can be read by your Continuous Integration (CI) tooling, or collection/archiving of test output files.Several record writers are distributed with the PySys framework, such as the
pysys.writer.outcomes.JUnitXMLResultsWriter
andpysys.writer.ci.GitHubActionsCIWriter
. By default, record writers are enabled only when the--record
flag is given to the PySys launcher, though some writers may enable/disable themselves under different conditions, by overriding theBaseResultsWriter.isEnabled
method.
BaseProgressResultsWriter
: Progress writers output a summary of the test progress after completion of each test, to give an indication of how far and how well the run is progressing. A single implementation of a progress writer is distributed with the PySys framework, namely thepysys.writer.console.ConsoleProgressResultsWriter
, which details the percentage of tests selected to be run and that have executed, and a summary of the recent test failures. Progress writers should extend theBaseProgressResultsWriter
and are enabled when the--progress
flag is given to the PySys launcher, or whenPYSYS_PROGRESS=true
is set in the local environment.
BaseSummaryResultsWriter
: Summary writers output an overall summary of the status at the end of a test run. A single implementation of a summary writer is distributed with the PySys framework, namely thepysys.writer.console.ConsoleSummaryResultsWriter
, which details the overall test run outcome and lists any tests that did not pass. Summary writers are always enabled regardless of the flags given to the PySys launcher.
(See also pysys.perf.api
whcih is used for writing performance results, using a similar but slightly
different API).
Project configuration of the writers is through the PySys project XML file using the <writer>
tag. Multiple
writers may be configured and their individual properties set through the nested <property>
tag. Writer
properties are set as attributes to the writer instance just before setup is called, with automatic conversion of
type to match the default value if specified as a static attribute on the class.
The writers are instantiated and invoked by the pysys.baserunner.BaseRunner
class instance. This calls the class
constructors of all configured test writers, and then the setup (prior to executing the set of tests), processResult
(process a test result), and cleanup (upon completion of the execution of all tests). The **kwargs
method parameter
should always be included in the signature of each method, to allow for future additions to PySys without
breaking existing writer implementations.
Writers that generate output files/directories should by default put that output under either the
runner.output
directory, or (for increased prominence) the runner.output+'/..'
directory (which is typically testRootDir
unless an absolute --outdir
path was provided) .
A prefix of double underscore __pysys
is recommended to distinguish dynamically created directories
(ignored by version control) from the testcase directories (checked into version control).
Writer authors may wish to make use of these helpers:
replaceIllegalXMLCharacters
(unicodeString[, ...])Utility function that takes a unicode character string and replaces all characters that are not permitted to appear in an XML document.
Remove any ANSI escape sequences (for example to set console color) present in the specified string.
Writes the specified bytes or (preferably) unicode character string to stdout, avoiding any redirection to loggers performed by PySys, and performing replacements if needed based on the characters supported by the stdout encoding, and with support for output coloring using escape sequences (if enabled in PySys).
Formatter supporting colored output to a console.
BaseResultsWriter¶
- class pysys.writer.api.BaseResultsWriter(logfile=None, **kwargs)[source]¶
Bases:
object
Base class for all writers that get notified as and when test results are available.
Writer can additionally subclass
ArtifactPublisher
to be notified of artifacts produced by other writers that they wish to publish, orTestOutputVisitor
to be notified of each file in the test output directory. If you are implementing a writer that needs a textual summary of the test outcomes, you can addTestOutcomeSummaryGenerator
as a superclass to get this functionality.- Parameters
logfile (str) – Optional configuration property specifying a file to store output in. Does not apply to all writers, can be ignored if not needed.
kwargs – Additional keyword arguments may be added in a future release.
- isEnabled(record=False, **kwargs)[source]¶
Determines whether this writer can be used in the current environment.
If set to False then after construction none of the other methods (including
setup
)) will be called.- Parameters
record – True if the user ran PySys with the
--record
flag, indicating that test results should be recorded.- Returns
For record writers, the default to enable only if record==True, but individual writers can use different criteria if desired, e.g. writers for logging output to a CI system may enable themselves based on environment variables indicating that system is present, even if record is not specified explicitly.
- setup(numTests=0, cycles=1, xargs=None, threads=0, testoutdir='', runner=None, **kwargs)[source]¶
Called before any tests begin.
Before this method is called, for each property “PROP” specified for this writer in the project configuration file, the configured value will be assigned to
self.PROP
.- Parameters
numTests – The total number of tests (cycles*testids) to be executed
cycles – The number of cycles.
xargs – The runner’s xargs
threads – The number of threads used for running tests.
testoutdir – The output directory used for this test run (equal to
runner.outsubdir
), an identifying string which often contains the platform, or when there are multiple test runs on the same machine may be used to distinguish between them. This is usually a relative path but may be an absolute path.runner – The runner instance that owns this writer. The default implementation of this methods sets the
self.runner
attribute to this value.kwargs – Additional keyword arguments may be added in a future release.
- cleanup(**kwargs)[source]¶
Called after all tests have finished executing (or been cancelled).
This is where file headers can be written, and open handles should be closed.
- Parameters
kwargs – Additional keyword arguments may be added in a future release.
- processResult(testObj, cycle=0, testTime=0, testStart=0, runLogOutput='', **kwargs)[source]¶
Called when each test has completed.
This method is always invoked under a lock that prevents multiple concurrent invocations so additional locking is not usually necessary.
- Parameters
testObj (pysys.basetest.BaseTest) – Reference to an instance of a
pysys.basetest.BaseTest
class. The writer can extract data from this object but should not store a reference to it. ThetestObj.descriptor.id
indicates the test that ran.cycle (int) – The cycle number. These start from 0, so please add 1 to this value before using.
testTime (float) – Duration of the test in seconds as a floating point number.
testStart (float) – The time when the test started.
runLogOutput (str) – The logging output written to the console/run.log, as a unicode character string. This string will include ANSI escape codes if colored output is enabled; if desired these can be removed using
pysys.utils.logutils.stripANSIEscapeCodes()
.kwargs – Additional keyword arguments may be added in future releases.
- processTestStarting(testObj, cycle=- 1, **kwargs)[source]¶
Called when a test is just about to begin executing.
Note on thread-safety: unlike the other methods on this interface, this is usually executed on a worker thread, so any data structures accessed in this method and others on this class must be synchronized if performing non-atomic operations.
- Parameters
testObj – Reference to an instance of a
pysys.basetest.BaseTest
class. The writer can extract data from this object but should not store a reference to it. The testObj.descriptor.id indicates the test that ran.cycle – The cycle number. These start from 0, so please add 1 to this value before using.
kwargs – Additional keyword arguments may be added in a future release.
BaseRecordResultsWriter¶
- class pysys.writer.api.BaseRecordResultsWriter(logfile=None, **kwargs)[source]¶
Bases:
pysys.writer.api.BaseResultsWriter
Base class for writers that record the results of tests, and are enabled only when the
--record
flag is specified.For compatibility reasons writers that do not subclass BaseSummaryResultsWriter or BaseProgressResultsWriter are treated as “record” writers even if they do not inherit from this class.
BaseSummaryResultsWriter¶
- class pysys.writer.api.BaseSummaryResultsWriter(logfile=None, **kwargs)[source]¶
Bases:
pysys.writer.api.BaseResultsWriter
Base class for writers that display a summary of test results.
Summary writers are always enabled (regardless of whether
--progress
or--record
are specified). If no “summary” writers are configured, a default ConsoleSummaryResultsWriter instance will be added automatically.Summary writers are invoked after all other writers, ensuring that their output will be displayed after output from any other writer types.
BaseProgressResultsWriter¶
- class pysys.writer.api.BaseProgressResultsWriter(logfile=None, **kwargs)[source]¶
Bases:
pysys.writer.api.BaseResultsWriter
Base class for writers that display progress information while tests are running.
Progress writers are only enabled if the
--progress
flag is specified.
ArtifactPublisher¶
- class pysys.writer.api.ArtifactPublisher[source]¶
Bases:
object
Interface implemented by writers that implement publishing of file/directory artifacts.
For example, a writer for a CI provider that supports artifact uploading can subclass this interface to be notified when another writer (or performance reporter) produces an artifact. If implementing this interface, remember that the order each writer’s cleanup() is called is the same as the order the writers appear in the project configuration file, so if your writer relies on output published from another’s cleanup you may need to document this, or code your writer such that it doesn’t care what order the cleanup() methods are called.
To publish an artifact to all registered writers, call
pysys.baserunner.BaseRunner.publishArtifact()
.It is possible to restrict artifact publishing to just the categories you care about by setting the project property
publishArtifactCategoryIncludeRegex
which (if specified) must match the category name in order for writers to be notified.New in version 1.6.0.
- publishArtifact(path, category, **kwargs)[source]¶
Called when a file or directory artifact has been written and is ready to be published (e.g. by another writer).
- Parameters
path (str) – Absolute path of the file or directory, using forward slashes as the path separator.
category (str) – A string identifying what kind of artifact this is, e.g. “TestOutputArchive” and “TestOutputArchiveDir” (from
pysys.writer.testoutput.TestOutputArchiveWriter
) or “CSVPerformanceReport” (frompysys.perf.reporters.CSVPerformanceReporter
). If you create your own category, be sure to add an org/company name prefix to avoid clashes.
TestOutputVisitor¶
- class pysys.writer.api.TestOutputVisitor[source]¶
Bases:
object
Interface implemented by writers that wish to be notified of each file in the test output directory.
Implementing this interface is a lot more efficient than explicitly walking the directory tree. Note that in the interests of performance empty (zero byte) files are ignored.
New in version 1.6.0.
- visitTestOutputFile(testObj, path, **kwargs)[source]¶
Called after execution of each test (and before purging of files) for each file found in the output directory.
- Parameters
testObj (pysys.basetest.BaseTest) – The test object, which can be used to find the outcome etc.
str – The absolute, normalize path to the output file (will be a \?long path safe path on windows).
- Return bool
Return True if this visitor has handled this file fully (e.g. by deleting it) and it should not be passed on to any other registered visitors.
TestOutcomeSummaryGenerator¶
- class pysys.writer.api.TestOutcomeSummaryGenerator(logfile=None, **kwargs)[source]¶
Bases:
pysys.writer.api.BaseResultsWriter
Mix-in helper class that can be inherited by any writer to allow (configurable) generation of a textual summary of the test outcomes.
If subclasses provide their own implementation of
setup
andprocessResult
they must ensure this class’s methods of those names are also called. Then the summary can be obtained fromlogSummary
orgetSummaryText
, typically in the writer’scleanup
method.- showOutcomeReason = True¶
Configures whether the summary includes the reason for each failure.
- showOutputDir = True¶
Configures whether the summary includes the (relative) path to the output directory for each failure.
- showTestDir = True¶
Configures whether the summary includes the (relative) path to the test directory for each failure, unless the output dir is displayed and the test dir is a parent of it. This is useful if you run tests with an absolute –outdir.
- showTestTitle = False¶
Configures whether the summary includes the test title for each failure.
- showOutcomeStats = True¶
Configures whether the summary includes a count of the number of each outcomes.
- showDuration = False¶
Configures whether the summary includes the total duration of all tests.
- showRunDetails = False¶
Configures whether the summary includes the
runDetails
from thepysys.baserunner.BaseRunner
, such asoutDirName
andhostname
.
- showInspectSummary = True¶
Configures whether the summary includes a summary of INSPECT outcomes (if any).
- showNotVerifiedSummary = True¶
Configures whether the summary includes a summary of NOTVERIFIED outcomes (if any).
- showTestIdList = False¶
Configures whether the summary includes a short list of the failing test ids in a form that’s easy to paste onto the command line to re-run the failed tests.
- getSummaryText(**kwargs)[source]¶
Get the textual summary as a single string (with no coloring).
To customize what is included in the summary (rather than letting it be user-configurable), use the keyword arguments as for
logSummary
.- Return str
The summary as a string.
- logSummary(log, showDuration=None, showOutcomeStats=None, showTestIdList=None, showFailureSummary=True, showRunDetails=None, **kwargs)[source]¶
Writes a textual summary using the specified log function, with colored output if enabled.
The keyword arguments can be used to disable sections of the output (overriding the settings) if needed by the caller.
- Parameters
log (Callable[format,args,kwargs=]) – The function to call for each line of the summary (e.g. log.critical). The message is obtained with
format % args
, and color information is available from theextra=
keyword argument.
replaceIllegalXMLCharacters¶
- pysys.writer.api.replaceIllegalXMLCharacters(unicodeString, replaceWith='?')[source]¶
Utility function that takes a unicode character string and replaces all characters that are not permitted to appear in an XML document.
The XML specification says Char ::= #x9 | #xA | #xD | [#x20-#xD7FF] | [#xE000-#xFFFD] | [#x10000-#x10FFFF]
Currently Python’s XML libraries do not perform checking or removal of such characters, so if this function is not used then they may generate XML documents that no XML API (including Python’s) can read.
See https://bugs.python.org/issue5166
- Parameters
unicodeString – a unicode character string (not a byte string). Since most XML documents are encoded in utf-8, typical usage would be to decode the UTF-8 bytes into characters before calling this function and then re-encode as UTF-8 again afterwards.
replaceWith – the unicode character string to replace each illegal character with.