The BaseTest Class¶
In PySys each test is a class which contains the methods needed to execute your test logic and then to validate the results against the expected behaviour to produce a final “outcome” for the test.
All test classes inherit (directly or indirectly) from the BaseTest
class, which provides everything you need to
create your own tests.
Implementing your test class¶
Each PySys test class must implement an execute
method containing the main body of the test, and
(usually) a validate
method containing the assertions that check for the expected result.
Sometimes you may also need to provide setup
and/or cleanup
functionality:
|
Contains setup actions to be executed before the test is executed. |
|
The method tests implement to perform the test execution steps. |
|
The method tests implement to perform assertions that check for the expected behaviour. |
|
Registers a function that will be called as part of the |
Taken together, the PySys setup() -> execute() -> validate()
methods correspond to the common
‘Arrange, Act, Assert’ testing paradigm.
At the end of each test, PySys automatically terminates all processes it
started, in the cleanup
method. If any additional custom cleanup steps are required, these can be added by calling
addCleanupFunction
.
NB: Do not ever override the __init__
constructor of a test class; instead use setup
for any initialization,
which ensures orderly cleanup if an exception occurs during the setup process.
Test configuration¶
The input/output directories and other essential information about this test object and its parent project are
can be accessed via instance attributes on self
:
self.input
(str): Full path to the input directory of the testcase, which is where test-specific resources such as configuration files are stored. The directory is the main<testDir>
for projects created from PySys 2.0 onwards, or<testdir>/Input
for projects created earlier. The path can be customized in the pysysdirconfig at a project, directory (or even testcase) level.Here is an example of copying a configuration file template from the input directory:
self.copy(self.input+'/serverconfig.xml', self.output, mappers=[lambda line: line.replace('@PORT@', str(myport))])
When storing large text files in the input directory it is recommended to use compression; see
pysys.basetest.BaseTest.unpackArchive
for details of how to decompress input files.Do not use the
self.input
field as a way to locate other resources that are in directories above Input/ (since this will fail when the input directory is empty and using a version control system (e.g. git) that doesn’t commit empty directories). For other test resource locations the recommended approach is to useself.project.testRootDir
, define a specific project property for the directory (if widely used), or if that’s not practical then useself.descriptor.testDir
to get the test directory.self.output
(str): Full path to the output directory of the testcase, which is the only location to which the test should write output files, temporary files and logs. The output directory is automatically created and cleaned of existing content before the test runs to ensure separate runs do not influence each other. There are separate output directories when a test is run for multiple cycles or in multiple modes. The directory is usually<testdir>/Output/<outdir>
but the path can be customized in the testcase descriptor and the--outdir
command line argument.self.reference
(str): Full path to the reference directory of the testcase, which is where reference comparison files are stored (typically for use withassertDiff
). The directory is usually<testdir>/Reference
but the path can be customized in the testcase descriptor.self.log
(logging.Logger): The PythonLogger
instance that should be used to record progress and status information. For example:self.log.info("Starting myserver on port %d", serverport)
self.mode
(pysys.config.descriptor.TestMode
): The user-defined mode this test object is running. Tests can use this to modify how the test executed based upon the mode, for example to allow the test to run against either a mock or a real database. TestMode subclasses str so you can include this in situation where you need the name of the mode such as when recording performance result, and you can also use the.params
attribute to access any parameters defined on the mode.self.testCycle
(int): The cycle in which this test is running. Numbering starts from 1 in a multi-cycle test run. The special value of 0 is used to indicate that this is not part of a multi-cycle run.self.descriptor
(pysys.config.descriptor.TestDescriptor
): The descriptor contains information about this testcase such as the test id, groups and test type, and typically comes from the test’spysystest.*
file.self.runner
(pysys.baserunner.BaseRunner
): A reference to the singleton runner instance that’s responsible for orchestrating all the tests in this project. The runner is where any cross-test state can be held, for example if you want to have multiple tests share use of a single server, VM, or other resource.self.project
(pysys.config.project.Project
): A reference to the singleton project instance containing the configuration of this PySys test project as defined bypysysproject.xml
. The project can be used to access information such as the project properties which are shared across all tests (e.g. for hosts and credentials).self.testPlugins
(list[object]): A list of any configured test plugin instances. This allows plugins to access the functionality of other plugins if needed (for example looking them up by type in this list).self.disableCoverage
(bool): Set this to True to request that no code coverage is generated for this test (even when code coverage has been enabled for the PySys run), for example because this is a time-sensitive or performance-measuring test. Typically this would be set in an individual testcase, or in thesetup
method of aBaseTest
subclass based on groups in theself.descriptor
that indicate what kind of test this is.self.disablePerformanceReporting
(bool): Set this to True to request that no performance figures are recorded. For example you might dynamically set this to True when a test is using options such as profiling or code coverage that would make the results meaningless.
Additional variables that affect only the behaviour of a single method are documented in the associated method.
There is also a field for each test plugin listed in the project configuration. Plugins provide additional
functionality such as methods for starting and working with a particular language or tool. A test plugin is
just a class with a method setup(self, testobj)
(and no constructor arguments), that provides methods and
fields for use by tests. Each test plugin listed in the project configuration
with <test-plugin classname="..." alias="..."/>
is instantiated for each
BaseTest
instance, and can be accessed using self.<alias>
on the test object. If you are using a third party
PySys test plugin, consult the documentation for the third party test plugin class to find out what methods and fields
are available using self.<alias>.*
.
If you wish to support test parameters that can be overridden on the command line using -Xkey=value
, just add a
static variable just after the `class MyClass(BaseTest):
line containing the default value, and access it using
self.key
. If a new value for that key is specified with -Xkey=value
, that value will be set as an attribute by
the BaseTest constructor, with automatic conversion from string to the correct type if the default value is a
bool/int/float.
Assertions and outcomes¶
|
Perform a validation by using a regular expression to extract a "value" from a text file and then check the extracted value using an |
|
Perform a validation by checking for the presence or absence of a regular expression in the specified text file. |
|
Perform a validation assert on the count of lines in a text file matching a specific regular expression. |
|
Perform a validation by comparing the contents of two text files, typically a file from the output directory against a file in the |
|
Performs equality/range tests or any general-purpose validation by evaluating a Python |
|
Perform a validation assert on a regular expression occurring in the last line of a text file. |
|
Perform a validation assert on a list of regular expressions occurring in specified order in a text file. |
|
Perform a validation that the specified file or directory path exists (or does not exist). |
|
Raise an AbortException with the specified outcome and reason. |
|
Raise an AbortException that will set the test outcome to SKIPPED and ensure that the rest of the execute() and validate() methods do not execute. |
|
Add a validation outcome (and optionally a reason string) to the validation list. |
Get the final outcome for this test, based on the precedence order defined in |
|
Get the reason string for the current overall outcome (if specified). |
|
Get the location in the Python source file where this location was added. |
|
|
Reports a new performance number to the configured performance reporters, with an associated unique string key that identifies it for comparison purposes. |
PySys has a library of assertion methods that are great for typical system testing validation tasks such as checking log messages, diff’ing output files against a reference, and for more complex cases, evaluating arbitrary Python expressions. All these methods are designed to give a really clear outcome reason if the assertion fails. Note that by default assertion failures do not abort the test, so all the validation statements will be executed even if some early ones fail.
For example:
def validate(self):
self.assertGrep('myserver.log', expr=' (ERROR|FATAL|WARN) .*', contains=False)
self.assertThatGrep('myserver.log', r'Successfully authenticated user "([^"]*)"',
"value == expected", expected='myuser')
self.assertThat('actualNumberOfLogFiles == expected', actualNumberOfLogFiles__eval="len(glob.glob(self.output+'/myserver*.log'))", expected=3)
The available test outcomes are listed in pysys.constants.OUTCOMES
.
There are some deprecated methods which we do not recommend using: assertEval
, assertTrue
, assertFalse
(assertThat
should be used instead of these).
Processes¶
|
Start a process running in the foreground or background, and return the |
|
Start a Python process with the specified arguments. |
|
Allocate a free TCP port which can be used for starting a server on this machine. |
|
Allocate unique filenames of the form |
|
Create a new customized dictionary of environment variables suitable for passing to |
|
Create a new dictionary of environment variables, suitable for passing to |
|
Send a signal to a running process. |
|
Stops the specified process, if it is currently running. |
|
Write binary data to the stdin of a process. |
|
Wait for any running background processes to terminate, then check that all background processes completed with the expected exit status. |
|
Start a background thread to monitor process statistics such as memory and CPU usage. |
|
Request a process monitor to stop. |
The most important part of a test’s execute
method is starting the process(es) that you are testing. Always use
the startProcess
method for this (or a wrapper such as startPython
) rather than os.system
or subprocess
,
to ensure that PySys can cleanup all test processes at the end of the test.
This class also provides methods to dynamically allocate a free TCP server port, and to monitor the memory/CPU usage of the processes it starts.
Waiting¶
PySys provides a number of methods that can be used in your execute
method to wait for operations to complete,
of which the most commonly used is waitForGrep
.
|
Wait for a regular expression line to be seen (one or more times) in a text file in the output directory (waitForGrep was formerly known as |
|
Wait for a file to exist on disk. |
|
Wait for any running background processes to terminate, then check that all background processes completed with the expected exit status. |
|
Wait for a background process to terminate. |
|
Wait until it is possible to establish a socket connection to a server running on the specified local or remote port. |
|
Wait for a specified period of time, and log a message to indicate this is happening. |
|
Sleeps the current thread for the specified number of seconds, between polling/checking for some condition to be met. |
Do not use wait
unless there is no alternative, since it makes tests both slower and more fragile than they ought to be.
Files¶
The helper methods for common file I/O operations are:
|
Copy a directory or a single text or binary file, optionally tranforming the contents by filtering each line through a list of mapping functions. |
|
Unpacks the specified file(s) from an archive to a directory. |
|
Create a directory, with recursive creation of any parent directories. |
|
Recursively delete the specified directory. |
|
Delete the specified file, with optional retries and ignoring of errors. |
|
Returns the first occurrence of a regular expression in the specified file, or raises an exception if not found. |
|
Returns the first occurrence of a regular expression in the specified file, or None if not found. |
|
Returns a list of all the occurrences of a regular expression in the specified file. |
|
Logs some or all of the lines from the specified file. |
|
Writes the specified characters to a file in the output directory. |
|
Specifies what encoding should be used to read or write the specified text file. |
To get the configured input/output directories for this test see Test configuration.
For additional file/directory helper functions such as support for loading JSON and properties files,
see the pysys.utils.fileutils
module.
Miscellaneous¶
|
Compares two alphanumeric dotted version strings to see which is more recent. |
|
Logs the differences between two values in a human-friendly way, on multiple lines (as displayed by |
Temporarily stops logging for the current thread. |
|
|
Start a new background thread that will invoke the specified |
|
Execute the Python doctests that exist in the specified python file; adds a FAILED outcome if any do not pass. |
|
Start the manual tester, which provides a UI to guide a human through the tests needed to implement this testcase. |
Stop the manual tester if running. |
|
|
Wait for the manual tester to be stopped via user interaction. |
|
Get a True/False indicating whether the specified attribute is set on this object (typically as a result of specifying -X on the command line), or else from the project configuration. |
All BaseTest members¶
- class pysys.basetest.BaseTest(descriptor, outsubdir, runner)[source]¶
Bases:
pysys.process.user.ProcessUser
BaseTest is the base class of every individual PySys test class, and contains the methods needed to execute your test logic and then to validate the results against the expected behaviour.
Apart from the
addOutcome
method this class is not thread-safe, so if you need to access it from multiple threads be sure to add your own locking around use of its fields and methods, including any cleanup functions.- setup()[source]¶
Contains setup actions to be executed before the test is executed.
The
setup
method may be overridden by individual test classes, or (more commonly) in a customBaseTest
subclass that provides common functionality for multiple individual tests. However before implementing a customBaseTest
subclass with its ownsetup()
method, consider whether the PySys concept of test plugins would meet your needs.If you do override this method, be sure to call
super(BASETEST_CLASS_HERE, self).setup()
to allow the setup commands from the base test to run.If
setup
throws an exception, thecleanup
method will still be called, to clean up any resources that were already allocated.
- execute()[source]¶
The method tests implement to perform the test execution steps.
- Raises
NotImplementedError – If this method was not implemented yet.
- validate()[source]¶
The method tests implement to perform assertions that check for the expected behaviour.
In some cases all of the required assertions (e.g. checking that a process ran without error etc) will have been performed in the
execute
method and sovalidate
will be empty. However, where possible it is recommended to put assertions into thevalidate
method for clarity, and so that thepysys run --validateOnly
option can be used during test development.
- cleanup()[source]¶
Contains cleanup actions to be executed after the test’s
execute
andvalidate
methods have completed.The cleanup method automatically stops all processes that are still still running (assuming they were started with
startProcess
). It also stops all process monitors running in separate threads, and any instances of the manual tester user interface.If any custom cleanup is required, use
addCleanupFunction
instead of overriding this method.
- addResource(resource)[source]¶
Add a resource which is owned by the test and therefore gets its
__del__
method called when the test is cleaned up.- Deprecated
Please use
addCleanupFunction
instead of this function.
- startProcessMonitor(process, interval=5, file=None, handlers=[], **pmargs)[source]¶
Start a background thread to monitor process statistics such as memory and CPU usage.
All process monitors are automatically stopped on completion of the test by
BaseTest.cleanup
, but you may also wish to explicitly stop your process monitors by callingpysys.process.monitor.BaseProcessMonitor.stop
before you begin shutting down processes at the end of a test to avoid unwanted spikes and noise in the last few samples of the data.You can specify a
file
and/or a list ofhandlers
. If you usefile
, a defaultpysys.process.monitor.ProcessMonitorTextFileHandler
instance is created to produce tab-delimited lines with default columns specified bypysys.process.monitor.ProcessMonitorTextFileHandler.DEFAULT_COLUMNS
. If you wish to customize this for an individual test create your ownProcessMonitorTextFileHandler
instance and pass it to handlers instead. Additional default columns may be added in future releases.- Parameters
process – The process handle returned from the
startProcess
method.interval – The polling interval in seconds between collection of monitoring statistics.
file –
The name of a tab separated values (.tsv) file to write to, for example ‘monitor-myprocess.tsv’.
A default
pysys.process.monitor.ProcessMonitorTextFileHandler
instance is created if this parameter is specified, with default columns frompysys.process.monitor.ProcessMonitorTextFileHandler.DEFAULT_COLUMNS
.handlers – A list of
pysys.process.monitor.BaseProcessMonitorHandler
instances (such aspysys.process.monitor.ProcessMonitorTextFileHandler
), which will process monitoring data every polling interval. This can be used for recording results (for example in a file) or for dynamically analysing them and reporting problems.pmargs – Keyword arguments to allow advanced parameterization of the process monitor class, which will be passed to its constructor. It is an error to specify any parameters not supported by the process monitor class on each platform.
- Returns
An object representing the process monitor (
pysys.process.monitor.BaseProcessMonitor
).- Return type
- stopProcessMonitor(monitor)[source]¶
Request a process monitor to stop.
This method is deprecated - just call
pysys.process.monitor.BaseProcessMonitor.stop
directly instead.Waits for the monitor to fully stop if possible, but does not throw an exception if it fails.
All process monitors are automatically stopped and joined during cleanup, however you may wish to explicitly stop your process monitors before you begin shutting down processes at the end of a test to avoid unwanted spikes and noise in the last few samples of the data.
- Parameters
monitor – The process monitor handle returned from the
startProcessMonitor
method
- startBackgroundThread(name, target, kwargsForTarget={})[source]¶
Start a new background thread that will invoke the specified
target
function.The target function will be invoked with the specified keyword arguments, preceded by the special keyword arguments
stopping
andlog
. Thestopping
argument is a Pythonthreading.Event
instance that can be used to detect when the thread has been requested to terminate. It is recommended to use this event instead oftime.sleep
to avoid waiting when the thread is meant to be finishing.Example usage:
class PySysTest(BaseTest): def dosomething(self, stopping, log, param1, pollingInterval): log.debug('Message from my thread') while not stopping.is_set(): # ... do stuff here # sleep for pollingInterval, waking up if requested to stop; # (hint: ensure this wait time is small to retain # responsiveness to Ctrl+C interrupts) if stopping.wait(pollingInterval): return def execute(self): t = self.startBackgroundThread('DoSomething1', self.dosomething, {'param1':True, 'pollingInterval':1.0}) ... t.stop() # requests thread to stop but doesn't wait for it to stop t.join()
Note that
BaseTest
is not thread-safe (apart fromaddOutcome
,startProcess
and the reading of fields likeself.output
that don’t change) so if you need to use its fields or methods from background threads, be sure to add your own locking to the foreground and background threads in your test, including any custom cleanup functions.The BaseTest will stop and join all running background threads at the beginning of cleanup. If a thread doesn’t stop within the expected timeout period a
constants.TIMEDOUT
outcome will be appended. If a thread’starget
function raises an Exception then aconstants.BLOCKED
outcome will be appended during cleanup or when it is joined.- Parameters
name – A name for this thread that concisely describes its purpose. Should be unique within this test/owner instance. A prefix indicating the test/owner will be added to the provided name.
target – The function or instance method that will be executed on the background thread. The function must accept a keyword argument named
stopping
in addition to whichever keyword arguments are specified inkwargsForTarget
.kwargsForTarget – A dictionary of keyword arguments that will be passed to the target function.
- Returns
A
pysys.utils.threadutils.BackgroundThread
instance wrapping the newly started thread.- Return type
- startManualTester(file, filedir=None, state=11, timeout=1800)[source]¶
Start the manual tester, which provides a UI to guide a human through the tests needed to implement this testcase.
The manual tester user interface (UI) is used to describe a series of manual steps to be performed to execute and validate a test. Only a single instance of the UI can be running at any given time, and can be run either in the
FOREGROUND
(method will not return until the UI is closed or the timeout occurs) or in theBACKGROUND
(method will return straight away so automated actions may be performed concurrently). Should the UI be terminated due to expiry of the timeout, aTIMEDOUT
outcome will be added to the outcome list. The UI can be stopped via thestopManualTester
method. An instance of the UI not explicitly stopped within a test will automatically be stopped via thecleanup
method of the BaseTest.- Parameters
file – The name of the manual test xml input file
filedir – The directory containing the manual test xml input file (defaults to the output subdirectory)
state – Start the manual tester either in the
FOREGROUND
orBACKGROUND
(defaults toFOREGROUND
)timeout – The timeout period after which to termintate a manual tester running in the
FOREGROUND
- waitManualTester(timeout=1800)[source]¶
Wait for the manual tester to be stopped via user interaction.
- wait(interval)[source]¶
Wait for a specified period of time, and log a message to indicate this is happening.
Tests that rely on waiting for arbitrary times usually take longer to execute than necessary, and are fragile if the timings or machine load changes, so wherever possible use a method like
waitForGrep
to wait for something specific instead.See also
pollWait
which should be used when performing repeated polling to wait for a condition without logging.- Parameters
interval – The time interval in seconds to wait.
- assertPathExists(path, exists=True, abortOnError=False)[source]¶
Perform a validation that the specified file or directory path exists (or does not exist).
- Parameters
path – The path to be checked. This can be an absolute path or relative to the testcase output directory.
exists – True if the path is asserted to exist, False if it should not.
abortOnError – Set to True to make the test immediately abort if the assertion fails.
- Returns
True if the assertion succeeds, False if a failure outcome was appended.
- assertEval(evalstring, abortOnError=False, **formatparams)[source]¶
Perform a validation by substituting named
{}
placeholder values into a Python expression such as{expected}=={actual}
or4 <= {actual} <= 10
.- Deprecated:
Deprecated since 1.5.1 in favour of
assertThat
which is now significantly more powerful and should be used for new tests.
Example use:
self.assertEval('os.path.getsize({filename}) > {origFileSize}', filename=self.output+'/file.txt', origFileSize=1000)
- Parameters
evalstring –
a string that will be formatted using
.format(...)
with the specified parameters, and result in failure outcome if not true.Parameters should be specified using {name} syntax, and quoting is not required as string values are automatically escaped using repr. e.g. ‘
os.path.size({filename}) > {origFileSize}'
.Do not use an f-string instead of explicitly passing formatparams, as with an f-string this method will not know the names of the substituted parameters which makes the intention of the assertion harder to understand from looking at the test output.
The global environment used for evaluation includes the
os.path
,math
,sys
,re
,json
, andlocale
standard Python modules, as well as thepysys
module and the contents of thepysys.constants
module, e.g.IS_WINDOWS
, and also the BaseTest’sself
variable.formatparams –
Named parameters for the format string, which can be of any type. Use descriptive names for the parameters to produce an assertion message that makes it really clear what is being checked.
String parameters will be automatically passed through
repr()
before being formatted, so there is no need to perform additional quoting or escaping of strings.abortOnError – Set to True to make the test immediately abort if the assertion fails. Unless abortOnError=True this method only throws an exception if the format string is invalid; failure to execute the eval(…) results in a BLOCKED outcome but no exception.
- Returns
True if the assertion succeeds, False if a failure outcome was appended.
- assertThat(conditionstring, *positional_arguments, **kwargs)[source]¶
Performs equality/range tests or any general-purpose validation by evaluating a Python
eval()
expression in the context of some named values.This method is designed to produce very clear and informative logging and failure reasons if the assertion is unsuccessful (using the
logValueDiff
method).Example usage:
# Equality comparison of an 'actual' vs 'expected' message from when our server started; # note the use of a descriptive name for the 'actualXXX=' keyword to produce a nice clear message if it fails self.assertThat("actualStartupMessage == expected", expected='Started successfully', actualStartupMessage=msg) # This produces the self-describing log messages like: # Assert that (actualStartupMessage == expected) with expected='Started successfully', actualStartupMessage='Started unsuccessfully' ... passed # Always design tests to give clear messages if there's a failure. Here's an example of adding an extra # parameter (fromLogFile) that's not used in the condition string, to indicate which server we're testing here self.assertThat('actualUser == expected', expected='myuser', actualUser=user, fromLogFile='server1.log') # Any valid Python expression is permitted (not only equality testing): self.assertThat("actualStartupMessage.endswith('successfully')", actualStartupMessage=msg) self.assertThat("re.match(expected, actualStartupMessage)", expected=".* successfully", actualStartupMessage=msg) self.assertThat("(0 <= actualValue < max) and type(actualValue)!=float", actualValue=v, max=100) # Use ``is`` for comparisons to True/False/None as in Python ``==``/``!=`` don't always do what you'd # expect for these types. self.assertThat("actualValue is not None", actualValue=v)
This method is powerful enough for almost any validation that the other assert methods don’t handle, and by enforcing the discipline of naming values it generates self-describing log messages and outcome reasons to make it really obvious what is going on. For best results, make sure that your keyword= parameters have clear and unique names so it’s obvious how each assertThat() differs from the other ones, and ensure that all values you’re going to want to see are included as one of the named parameter values (rather than buried deep inside the main conditionstring).
The condition string is just a Python expression, which will be passed to
eval()
and can use any of thekeyword=
argument values passed to this method (but not the caller’s local variables). The evaluation is performed in a namespace that also includes the currentBaseTest
instance (self
), some standard Python modules (os.path
,math
,sys
,re
,json
, andlocale
), thepysys
module, and the contents of thepysys.constants
module, e.g.IS_WINDOWS
. If necessary, symbols for additional modules can be imported dynamically usingimport_module()
. For example:self.assertThat("IS_WINDOWS or re.match(expected, actual)", actual="foo", expected="f.*") self.assertThat("import_module('tarfile').is_tarfile(self.output+file) is False", file='/foo.zip')
Sometimes the differences between assertThat expressions are hard to describe in the parameter names themselves, and for these cases you can get self-describing behaviour with a parameter ending in the suffix
__eval
whose value is itself a Python expression to be evaluated, using any local variable in the namespace of the calling code, for example:myDataStructure = ... self.assertThat("actual == expected", actual__eval="myDataStructure['item1'][-1].getId()", expected="foo") self.assertThat("actual == expected", actual__eval="myDataStructure['item2'][-1].getId()", expected="bar") self.assertThat("actual == expected", actual__eval="myDataStructure['item3'][-1].getId()", expected="baz") # Produces self-describing log messages like this: # Assert that (actual == expected) with actual (myDataStructure['item1'][-1].getId()) ='foo', expected='foo' ... passed # Assert that (actual == expected) with actual (myDataStructure['item2'][-1].getId()) ='bar', expected='bar' ... passed # Assert that (actual == expected) with actual (myDataStructure['item3'][-1].getId()) ='baZaar', expected='baz' ... failed # actual: 'baZaar' # expected: 'baz' # ^
As shown above, when (at least) two named parameters are provided and the condition string is a simple comparison using exactly two of the parameters, additional lines are logged if the assertion fails, showing at what point the two arguments differ (based on finding the longest common substring). So it’s a good idea to include both the actual and expected value as named parameters rather than as literals inside the condition string.
Changed in version 1.5.1: The ability to pass named keyword= parameters was added in 1.5.1 (prior to that this method was deprecated).
- Parameters
conditionstring (str) –
A string containing Python code that will be evaluated using
eval()
as a boolean expression, for exampleactualXXX == expected
, where XXX is a brief description of what value is being tested.It’s best to put expected values into a separate named parameter (rather than using literals inside the conditionstring), since this will produce more informative messages if there is a failure.
Do not be tempted to use a Python f-string here, as that would deprive PySys of the opportunity to provide a self-describing message and outcome reason.
**kwargs – All additional keyword arguments are treated as values which will be made available when evaluating the condition string. Any keyword ending in the special suffix
__eval
will be treated as a Python expression string (rather than a string literal) and will be be evaluated in a namespace containing the local variables of the calling code and (on Python 3.6+) any preceding named parameters.*positional_arguments – (deprecated) Unnamed positional arguments will be substituted into the condition string using the old
%
format string mechanism, before it is evaluated. This feature is deprecated as it provides poor diagnostic information, requires string parameters to be explicitly escaped usingrepr()
, and only permits stringifiable data structures to be used by the conditionstring. Instead use namedkeyword=
in all new tests.failureOutcome=FAILED – The outcome that will be added if the condition fails. For example you could set this to
pysys.constants.BADPERF
for a performance assertion. Added in PySys v2.1.abortOnError=False – Set to True to make the test immediately abort if the assertion fails. By default this method produces a BLOCKED output but does not throw if the eval(…) cannot be executed.
assertMessage='' (str) – A high-level description of what the assertion is achieving, used in log messages and the outcome reason. For example “Check the startup logging”. Alternatively, to add some additional information to the default message message (e.g. which file/server it pertains to etc), just add the info as a string value with an extra keyword argument e.g.
server="server1"
.
- Returns
True if the assertion succeeds, False if a failure outcome was appended (and abortOnError=False).
- logValueDiff(actual=None, expected=None, logFunction=None, **namedvalues)[source]¶
Logs the differences between two values in a human-friendly way, on multiple lines (as displayed by
assertThat
).Special handling is provided for common types such as strings, lists and dicts (note that this isn’t intended for use with data structures whose string representation is too big to display on the console).
- Parameters
actual (obj) – The actual value. Alternatively, this parameter can be ignored and a value with a different key provided as a keyword argument instead.
expected (obj) – The baseline/expected value from which a diff will be computed to the actual value. Alternatively, this parameter can be ignored and a value with a different key provided as a keyword argument instead.
logFunction (function) – By default each line is logged at INFO level using
log.info()
, but an alternative log function with the same signature can be provided if desired.
- assertTrue(expr, abortOnError=False, assertMessage=None)[source]¶
Perform a validation assert on the supplied expression evaluating to true.
- Deprecated
Use
assertThat
instead of this method, which produces clearer messages if the assertion fails.
If the supplied expression evaluates to true a
PASSED
outcome is added to the outcome list. Should the expression evaluate to false, aFAILED
outcome is added.- Parameters
expr – The expression, as a boolean, to check for the True | False value
abortOnError – Set to True to make the test immediately abort if the assertion fails.
assertMessage – Overrides the string used to describe this assertion in log messages and the outcome reason.
- Returns
True if the assertion succeeds, False if a failure outcome was appended.
- assertFalse(expr, abortOnError=False, assertMessage=None)[source]¶
Perform a validation assert on the supplied expression evaluating to false.
- Deprecated
Use
assertThat
instead of this method, which produces clearer messages if the assertion fails.
If the supplied expression evaluates to false a
PASSED
outcome is added to the outcome list. Should the expression evaluate to true, aFAILED
outcome is added.- Parameters
expr – The expression to check for the true | false value
abortOnError – Set to True to make the test immediately abort if the assertion fails.
assertMessage – Overrides the string used to describe this assertion in log messages and the outcome reason.
- Returns
True if the assertion succeeds, False if a failure outcome was appended.
- assertDiff(file1, file2=None, filedir1=None, filedir2=None, ignores=[], sort=False, replace=[], includes=[], encoding=None, abortOnError=False, assertMessage=None, stripWhitespace=None)[source]¶
Perform a validation by comparing the contents of two text files, typically a file from the output directory against a file in the
<testdir>/Reference/
directory containing the expected output.Differences in line ending are always ignored, and depending on the value of
stripWhitespace
leading and trailing whitespace may also be ignored.The files can be pre-processed prior to the comparison to either ignore particular lines, sort their constituent lines, replace matches to regular expressions in a line with an alternate value, or to only include particular lines. However, it is often easier to instead use
copy
to perform the transformations (e.g. stripping out timestamps, finding lines of interest etc) and then separately call assertDiff on the file generated by copy. This makes it easier to generate a suitable reference file and to diagnose test failures. For example:self.assertDiff(self.copy('myfile.txt', 'myfile-processed.txt', mappers=[RegexReplace(RegexReplace.DATETIME_REGEX, '<timestamp>')])))
The above example shows a very compact form of assertDiff, which uses the fact that copy() returns the path of the destination file, and that there is no need to specify assertDiff’s file2 (reference) parameter if it’s the same basename as the first argument (just located in a different directory). In practice it’s very convenient to use the same basename for both the reference file and the output file it’s compared to.
Should the files after pre-processing be equivalent a
PASSED
outcome is added to the test outcome list, otherwise aFAILED
outcome is added.If you have any serialized dictionary/map data structures in the comparison files, or any lists that come from a directory listing, be really careful to ensure there is deterministic sorting of the keys/filenames, as by default the ordering is often subject to unpredictable changes when upgrading tools or switching between OSes, and you don’t want to have to update lots of testcases every time the sorting changes. If you are able to get the data into a Python data structure (e.g. by serializing from JSON), Python’s
json.dump(..., sort_keys=True)
can be a convenient way to produce a predictable order for dictionary keys.If you have a large set of test reference files which need to be updated after a behaviour or output formatting change, you can use a special command line option which makes
assertDiff
overwrite the reference files with a copy of the actual comparison file in cases where the diff would otherwise fail. Use this feature with caution, since it overwrites reference files with no backup. In particular, make sure you have committed all reference files to version control before running the command, and the afterwards be sure to carefully check the resulting diff to make sure the changes were as expected before committing. To use this feature, run:pysys.py run -XautoUpdateAssertDiffReferences
- Parameters
file1 (str) – The actual (or first) file to be compared; can be an absolute path or relative to the test output directory.
file2 (str) – The expected/reference (or second) file to be compared; can be an absolute path or relative to the Reference directory. The default is for file2 to be the same basename as file1 (but located in the Reference/ directory).
filedir1 (str) – The dirname of the first file (defaults to the testcase output subdirectory)
filedir2 (str) – The dirname of the second file (defaults to the testcase reference directory)
ignores (list[str]) – A list of regular expressions used to denote lines in the files which should be ignored
sort (bool) – Boolean flag to indicate if the lines in the files should be sorted prior to the comparison
list[(regexp:str,repl:str),...]replace – List of tuples of the form (‘regexpr’, ‘replacement’). For each regular expression in the list, any occurences in the files are replaced with the replacement value prior to the comparison being carried out. This is often useful to replace timestamps in logfiles etc.
stripWhitespace (bool) – If True, every line has leading and trailing whitespace stripped before comparison, which means indentation differences and whether the file ends with a blank line do not affect the outcome. If the value is
None
, delegates to the value of the project propertydefaultAssertDiffStripWhitespace
(which is True for old projects, but recommended to be False for new projects).includes (list[str]) – A list of regular expressions used to denote lines in the files which should be used in the comparison. Only lines which match an expression in the list are used for the comparison.
encoding (str) – The encoding to use to open the file. The default value is None which indicates that the decision will be delegated to the
getDefaultFileEncoding()
method.abortOnError (bool) – Set to True to make the test immediately abort if the assertion fails.
assertMessage (str) – An additional high-level description of what this assertion is checking, e.g. “Assert message contents match expected values”. Used in log messages and the outcome reason.
- Returns
True if the assertion succeeds, False if a failure outcome was appended.
- assertThatGrep(file, grepRegex, conditionstring='value == expected', encoding=None, reFlags=0, mappers=[], **kwargsForAssertThat)[source]¶
Perform a validation by using a regular expression to extract a “value” from a text file and then check the extracted value using an
assertThat
conditionstring.For example:
# This is the typical case - "value" is assigned to the first (...) regex group, and keyword parameters # (e.g. "expected=") are used to validate that the "value" is correct self.assertThatGrep('myserver.log', r'Successfully authenticated user "([^"]*)"', "value == expected", expected='myuser') # In cases where you need multiple regex groups for matching purpose, name the one containing the value using (?P<value>...) self.assertThatGrep('myserver.log', r'Successfully authenticated user "([^"]*)" in (?P<value>[^ ]+) seconds', "0.0 <= float(value) <= 60.0")
This method is implemented using
grep
andassertThat
, so see those methods for more detailed information on the parameters.New in version 1.6.0.
- Parameters
file – The name or relative/absolute path of the file to be searched.
grepRegex (str) – The regular expression to use for extracting the value of interest from the file. Typically this will use a
(...)
regular expression group to identify the part of the expression containing the value; alternatively a single(?P<value>...)
named group may be used.conditionstring (str) –
A string containing Python code that will be evaluated using
eval()
to validate that “value” is correct. For examplevalue == expected
.It’s best to put expected values into a separate named parameter (rather than using literals inside the conditionstring), since this will produce more informative messages if there is a failure.
Do not be tempted to use a Python f-string here, as that would deprive PySys of the opportunity to provide a self-describing message and outcome reason.
mappers (List[callable[str]->str]) –
A list of filter functions that will be used to pre-process each line from the file (returning None if the line is to be filtered out). This provides a very powerful capability for filtering the file, for example
pysys.mappers.IncludeLinesBetween
provides the ability to filter in/out sections of a file.Do not share mapper instances across multiple tests or threads as this can cause race conditions.
encoding (str) – The encoding to use to open the file. The default value is None which indicates that the decision will be delegated to the
getDefaultFileEncoding()
method.reFlags (int) –
Zero or more flags controlling how the behaviour of regular expression matching, combined together using the
|
operator, for examplereFlags=re.VERBOSE | re.IGNORECASE
.For details see the
re
module in the Python standard library. Note thatre.MULTILINE
cannot be used because expressions are matched against one line at a time.abortOnError (bool) – Set to True to make the test immediately abort if the assertion fails. By default this method produces a BLOCKED output but does not throw if the eval(…) cannot be executed.
assertMessage='' (str) – A high-level description of what the assertion is achieving, used in log messages and the outcome reason. For example “Check the startup logging”. Alternatively, to add some additional information to the default message message (e.g. which file/server it pertains to etc), just add the info as a string value with an extra keyword argument e.g.
server="server1"
.**kwargs – All additional keyword arguments are treated as values which will be made available when evaluating the condition string. Any keyword ending in the special suffix
__eval
will be treated as a Python expression string (rather than a string literal) and will be be evaluated in a namespace containing the local variables of the calling code and (on Python 3.6+) any preceding named parameters.
- Returns
True if the assertion succeeds, False if a failure outcome was appended (and abortOnError=False).
- assertGrep(file, _expr=None, _unused=None, contains=True, ignores=None, literal=False, encoding=None, abortOnError=False, assertMessage=None, reFlags=0, mappers=[], expr='', filedir=None)[source]¶
Perform a validation by checking for the presence or absence of a regular expression in the specified text file.
Note that if your goal is to check that a value in the file matches some criteria, it is better to use
assertThatGrep
instead of this function, as assertThatGrep can produce better messages on failure, and also allows for more powerful matching using a full Python expression (e.g. numeric range checks, pre-processing strings to normalize case, etc). If you need to extract a string for further processing without updating the test outcome, consider usinggrep
instead.The assertGrep method is good for checking in a log to confirm that something happened, or to check that there are no error messages. When contains=True it adds a
PASSED
outcome if found or aFAILED
outcome if not found (except when where are named groups in the expression in which caseBLOCKED
is used to indicate that the return value is not valid). When contains=False this is inverted so aPASSED
outcome is added if not found andFAILED
if found.For example:
self.assertGrep('myserver.log', expr=r' ERROR .*', contains=False) # If error messages may be accompanied by stack traces you can use a mapper to join them into the same line # so if your test fails, the outcome reason includes all the information. This also allows ignoring errors # based on the stack trace: self.assertGrep('myserver.log', expr=r' (ERROR|FATAL) .*', contains=False, mappers=[pysys.mappers.JoinLines.JavaStackTrace()], ignores=['Caused by: java.lang.RuntimeError: My expected exception']) # In Python 3+, f-Strings can be used to substitute in parameters, including in-line escaping of regex literals: self.assertGrep('myserver.log', expr=f'Successfully authenticated user "{re.escape(username)}" in .* seconds[.]') # If you need to use \ characters use a raw r'...' string to avoid the need for Python \ escaping in # addition to regex escaping. Square brackets are often the clearest way to escape regular expression # characters such as \ . and () self.assertGrep('myserver.log', expr=r'c:[\]Foo[\]bar[.]txt') # The IncludeLinesBetween mapper is very useful if you want to grep within a subset of the lines: self.assertGrep('myserver.log', expr=r'MyClass', mappers=[ pysys.mappers.IncludeLinesBetween('Error message.* - stack trace is:', stopBefore='^$'), ])
The behaviour of the regular expression can be controlled using
reFlags=
. For example, to perform case-insensitive matching and to use Python’s verbose regular expression syntax which permits whitespace and comments:self.assertGrep('myserver.log', reFlags=re.VERBOSE | re.IGNORECASE, expr=r\""" in\ \d + # the integral part \. # the decimal point \d * # some fractional digits \ seconds\. # in verbose regex mode we escape spaces with a slash \""")
Remember to escape regular expression special characters such as
.
,(
,[
,{
and\
if you want them to be treated as literal values. If you have a regular expression string with backslashes, it’s best to use a ‘raw’ Python string so that you don’t need to double-escape them, e.g.self.assertGrep(..., expr=r'c:\\Foo\\filename\.txt')
.If you want to search for a string that needs lots of regex escaping, a nice trick is to use a substitution string (containing only A-Z chars) for the regex special characters and pass everything else through re.escape:
self.assertGrep('myserver.log', expr=re.escape(r'A"string[with \lots*] of crazy characters e.g. VALUE.').replace('VALUE', '(.*)'))
Changed in version 1.5.1: The return value and reFlags were added in 1.5.1.
- Parameters
file – The name or relative/absolute path of the file to be searched.
expr (str) –
The regular expression to check for in the file (or a string literal if literal=True), for example
" ERROR .*"
.Remember to escape regular expression special characters such as
.
,(
,[
,{
and\
if you want them to be treated as literal values.If you wish to do something with the text inside the match you can use the
re
named group syntax(?P<groupName>...)
to specify a name for parts of the regular expression.For contains=False matches, you should end the expr with
*
if you wish to include just the matching text in the outcome failure reason. If contains=False and expr does not end with a*
then the entire matching line will be included in the outcome failure reason.contains (bool) – Boolean flag to specify if the expression should or should not be seen in the file.
ignores (list[str]) – Optional list of regular expressions that will be ignored when reading the file. Ignore expressions are applied after any mappers.
mappers (List[callable[str]->str]) –
A list of filter functions that will be used to pre-process each line from the file (returning None if the line is to be filtered out). This provides a very powerful capability for filtering the file, for example
pysys.mappers.IncludeLinesBetween
provides the ability to filter in/out sections of a file andpysys.mappers.JoinLines
can combine related error lines such as stack trace to provide all the information in the test outcome reason.Mappers must always preserve the final
\n
of each line (if present).Do not share mapper instances across multiple tests or threads as this can cause race conditions.
Added in PySys 1.6.0.
literal (bool) – By default expr is treated as a regex, but set this to True to pass in a string literal instead.
encoding (str) – The encoding to use to open the file. The default value is None which indicates that the decision will be delegated to the
getDefaultFileEncoding()
method.abortOnError (bool) – Set to True to make the test immediately abort if the assertion fails.
assertMessage (str) – An additional high-level description of what this assertion is checking, e.g. “Check for expected error message”. Used in log messages and the outcome reason.
filedir (str) – The directory of the file (defaults to the testcase output subdirectory); this is deprecated, as it’s simpler to just include the directory in the file parameter.
reFlags (int) –
Zero or more flags controlling how the behaviour of regular expression matching, combined together using the
|
operator, for examplereFlags=re.VERBOSE | re.IGNORECASE
.For details see the
re
module in the Python standard library. Note thatre.MULTILINE
cannot be used because expressions are matched against one line at a time. Added in PySys 1.5.1.
- Returns
The
re.Match
object, or None if there was no match (note the return value is not affected by the contains=True/False parameter).However if the expr contains any
(?P<groupName>...)
named groups, then a dict is returned containingdict(groupName: str, matchValue: str or None)
(or an empty{}
dict if there is no match) which allows the assertGrep result to be passed toassertThat
for further checking (typically unpacked using the**
operator; see example above).
- assertLastGrep(file, _expr='', _unused=None, contains=True, ignores=[], includes=[], encoding=None, abortOnError=False, assertMessage=None, reFlags=0, expr='', filedir=None)[source]¶
Perform a validation assert on a regular expression occurring in the last line of a text file.
Rather than using this method, use
grepAll
withassertThat
for better error messages and more maintainable tests, e.g. you can extract the last line of a file easily withself.grepAll(file, '.')[0]
.When the
contains
input argument is set to true, this method will add aPASSED
outcome to the test outcome list if the supplied regular expression is seen in the file; otherwise aFAILED
outcome is added. Shouldcontains
be set to false, aPASSED
outcome will only be added should the regular expression not be seen in the file.- Parameters
file – The basename of the file used in the grep
filedir – The dirname of the file (defaults to the testcase output subdirectory)
expr – The regular expression to check for in the last line of the file
contains – Boolean flag to denote if the expression should or should not be seen in the file
ignores – A list of regular expressions used to denote lines in the file which should be ignored
includes – A list of regular expressions used to denote lines in the file which should be used in the assertion.#
encoding – The encoding to use to open the file. The default value is None which indicates that the decision will be delegated to the
getDefaultFileEncoding()
method.abortOnError – Set to True to make the test immediately abort if the assertion fails.
assertMessage (str) – An additional high-level description of what this assertion is checking, e.g. “Assert error message is as expected”. Used in log messages and the outcome reason.
reFlags (int) –
Zero or more flags controlling how the behaviour of regular expression matching, combined together using the
|
operator, for examplereFlags=re.VERBOSE | re.IGNORECASE
.For details see the
re
module in the Python standard library. Note thatre.MULTILINE
cannot be used because expressions are matched against one line at a time. Added in PySys 1.5.1.
- Returns
The
re.Match
object, or None if there was no match (note the return value is not affected by the contains=True/False parameter).However if the expr contains any
(?P<groupName>...)
named groups, then a dict is returned containingdict(groupName: str, matchValue: str or None)
(or an empty{}
dict if there is no match) which allows the result to be passed toassertThat
for further checking (typically unpacked using the**
operator; seeassertGrep
for a similar example).
- assertOrderedGrep(file, _exprList=[], _unused=None, contains=True, encoding=None, abortOnError=False, assertMessage=None, reFlags=0, exprList=[], filedir=None)[source]¶
Perform a validation assert on a list of regular expressions occurring in specified order in a text file.
When the
contains
input argument is set to true, this method will append aPASSED
outcome to the test outcome list if the supplied regular expressions in theexprList
are seen in the file in the order they appear in the list; otherwise aFAILED
outcome is added. Shouldcontains
be set to false, aPASSED
outcome will only be added should the regular expressions not be seen in the file in the order they appear in the list.Warning: while this assertion method can be very convenient for checking the order of a small number of expressions, it becomes unwieldy when the number of expressions grows beyond a handful, and this is definitely not the best tool for the job if what you’re doing is really about checking that a subset of data from an output file matches expectations. For that use case, it’s better to do a filtered
copy()
of the file to remove prefixes (e.g. timestamps) and lines that are not important, and then useassertDiff
to check the extracted text matches expectations. This approach makes it easier to write tests, and crucially makes it much easier to figure out what went wrong if they fail.Similarly, if you need to check that an expression appears between two other lines (e.g. for a start/end of section) this method will not give you a reliable way to do that if there’s a chance the section markers could appear more than once in the file, so instead use the filtered
copy()
approach described above, or useassertGrep
with apysys.mappers.IncludeLinesBetween
mapper.- Parameters
file – The basename of the file used in the ordered grep
filedir – The dirname of the file (defaults to the testcase output subdirectory)
exprList – A list of regular expressions which should occur in the file in the order they appear in the list
contains – Boolean flag to denote if the expressions should or should not be seen in the file in the order specified
encoding – The encoding to use to open the file. The default value is None which indicates that the decision will be delegated to the
getDefaultFileEncoding()
method.abortOnError – Set to True to make the test immediately abort if the assertion fails.
assertMessage (str) – An additional high-level description of what this assertion is checking, e.g. “Assert error messages are as expected”. Used in log messages and the outcome reason.
reFlags (int) –
Zero or more flags controlling how the behaviour of regular expression matching, combined together using the
|
operator, for examplereFlags=re.VERBOSE | re.IGNORECASE
.For details see the
re
module in the Python standard library. Note thatre.MULTILINE
cannot be used because expressions are matched against one line at a time. Added in PySys 1.5.1.
- Returns
True if the assertion succeeds, False if a failure outcome was appended.
- assertLineCount(file, _expr='', _unused=None, condition='>=1', ignores=None, encoding=None, abortOnError=False, assertMessage=None, reFlags=0, expr='', filedir=None, mappers=[])[source]¶
Perform a validation assert on the count of lines in a text file matching a specific regular expression.
This method will add a
PASSED
outcome to the outcome list if the number of lines in the file matching the specified regular expression evaluate to true when evaluated against the suppliedcondition
.- Parameters
file – The basename of the file used in the line count
filedir – The dirname of the file (defaults to the testcase output subdirectory)
expr – The regular expression string used to match a line of the file
condition – The condition to be met for the number of lines matching the regular expression
ignores – A list of regular expressions that will cause lines to be excluded from the count. Ignore expressions are applied after any mappers.
encoding – The encoding to use to open the file. The default value is None which indicates that the decision will be delegated to the
getDefaultFileEncoding()
method.mappers (List[callable[str]->str]) –
A list of filter functions that will be used to pre-process each line from the file (returning None if the line is to be filtered out). This provides a very powerful capability for filtering the file, for example
pysys.mappers.IncludeLinesBetween
provides the ability to filter in/out sections of a file andpysys.mappers.JoinLines
can combine related error lines such as stack trace to provide all the information in the test outcome reason.Mappers must always preserve the final
\n
of each line (if present).Do not share mapper instances across multiple tests or threads as this can cause race conditions.
Added in PySys 2.0.
abortOnError – Set to True to make the test immediately abort if the assertion fails.
assertMessage (str) – An additional high-level description of what this assertion is checking, e.g. “Assert all messages received”. Used in log messages and the outcome reason.
reFlags (int) –
Zero or more flags controlling how the behaviour of regular expression matching, combined together using the
|
operator, for examplereFlags=re.VERBOSE | re.IGNORECASE
.For details see the
re
module in the Python standard library. Note thatre.MULTILINE
cannot be used because expressions are matched against one line at a time. Added in PySys 1.5.1.
- Returns
True if the assertion succeeds, False if a failure outcome was appended.
- reportPerformanceResult(value, resultKey, unit, toleranceStdDevs=None, resultDetails=None)[source]¶
Reports a new performance number to the configured performance reporters, with an associated unique string key that identifies it for comparison purposes.
Where possible it is better to report the rate at which an operation can be performed (e.g. throughput) rather than the total time taken, since this allows the number of iterations to be increased without affecting historical comparisons. For example:
self.reportPerformanceResult(int(iterations)/float(calctime), 'Fibonacci sequence calculation rate', '/s'))
If your test runs in multiple modes, make sure you include some information about the mode in the resultKey:
self.reportPerformanceResult(int(iterations)/float(calctime), 'Fibonacci sequence calculation rate using %s' % self.mode, '/s', resultDetails=[('mode',self.mode)])
While use of standard units such as ‘/s’, ‘s’ or ‘ns’ (nano-seconds) is recommended, custom units can be provided when needed using
pysys.perf.api.PerformanceUnit
:self.reportPerformanceResult(int(iterations)/float(calctime)/1000, 'Fibonacci sequence calculation rate using %s with different units' % self.mode, unit=PerformanceUnit('kilo_fibonacci/s', biggerIsBetter=True))
If the current test is executed with unusual options (e.g. enabling a profiler or code coverage) that would invalidate performance numbers, you can set
self.disablePerformanceReporting = True
to preventreportPerformanceResult
calls from doing anything.- Parameters
value – The numeric value to be reported. If a str is provided, it will be converted to a float.
resultKey – A unique string that fully identifies what was measured, which will be used to compare results from different test runs. For example “HTTP transport message sending throughput using with 3 connections in SSL mode”. The resultKey must be unique across all test cases and modes. It should be fully self-describing (without the need to look up extra information such as the associated testId). Do not include the test id or units in the resultKey string. It must be stable across different runs, so cannot contain process identifiers, date/times or other numbers that will vary. If possible resultKeys should be written so that related results will be together when all performance results are sorted by resultKey, which usually means putting general information near the start of the string and specifics (throughput/latency, sending/receiving) towards the end of the string. It should be as concise as possible (given the above).
unit – Identifies the unit the value is measured in, including whether bigger numbers are better or worse (used to determine improvement or regression). Must be an instance of
pysys.perf.api.PerformanceUnit
. In most cases, usepysys.perf.api.PerformanceUnit.SECONDS
(e.g. for latency) orpysys.perf.api.PerformanceUnit.PER_SECOND
(e.g. for throughput); the string literals ‘s’ and ‘/s’ can be used as a shorthand for those PerformanceUnit instances.toleranceStdDevs – (optional) A float that indicates how many standard deviations away from the mean a result needs to be to be considered a regression.
resultDetails – (optional) A dictionary of detailed information about this specific result and/or test that should be recorded together with the result, for example detailed information about what mode or versions the test is measuring. Note this is result-specific, unlike the global “run details” shared across all tests in this PySys execution, which can be customized with a runner plugin. If no resultDetails are specified explicitly then parameters from the test’s mode will be used if present.
- getDefaultFileEncoding(file, **xargs)[source]¶
Specifies what encoding should be used to read or write the specified text file. The default implementation for BaseTest delegates to the runner, which in turn gets its defaults from the
pysyproject.xml
configuration.See
pysys.process.user.ProcessUser.getDefaultFileEncoding
for more details.
- abort(outcome, outcomeReason, callRecord=None)¶
Raise an AbortException with the specified outcome and reason.
See also
skipTest
.- Parameters
outcome – The outcome, which will override any existing outcomes previously recorded.
outcomeReason – A string summarizing the reason for the outcome.
- addCleanupFunction(fn, ignoreErrors=False)¶
Registers a function that will be called as part of the
cleanup
of this object.Cleanup functions should have no arguments, and are invoked in reverse order with the most recently added first (LIFO), and before the automatic termination of any remaining processes associated with this object.
Typical cleanup tasks are to cleanly shutdown processes (which is sometimes necessary to obtain code coverage information), and to (attempt to) delete large files/directories created by the test:
self.addCleanupFunction(lambda: self.cleanlyShutdownMyProcess(params)) self.addCleanupFunction(lambda: self.deleteDir('my-large-dir'), ignoreErrors=True) self.addCleanupFunction(lambda: self.deleteFile('my-large-file.log'), ignoreErrors=True)
- Parameters
fn (Callable[]) – The cleanup function.
ignoreErrors (bool) – By default, errors from cleanup functions will result in a test failure; set this to True to log them but not produce a test failure. This parameter was added in 1.6.0.
- addOutcome(outcome, outcomeReason='', printReason=True, abortOnError=False, callRecord=None, override=False)¶
Add a validation outcome (and optionally a reason string) to the validation list.
The method provides the ability to add a validation outcome to the internal data structure storing the list of validation outcomes. Multiple validations may be performed, the current supported validation outcomes of which are described in Assertions and outcomes.
The outcomes are considered to have a precedence order, as defined by the order of the outcomes listed above. Thus a
pysys.constants.BLOCKED
outcome has a higher precedence than apysys.constants.PASSED
outcome. The outcomes are defined inpysys.constants
.This method is thread-safe.
Although this method exists on all subclasses of
pysys.process.user.ProcessUser
, in practice onlypysys.basetest.BaseTest
subclasses actually do anything with the resulting outcome.- Parameters
outcome (pysys.constants.Outcome) – The outcome to add, e.g.
pysys.constants.FAILED
.outcomeReason (str) – A string summarizing the reason for the outcome, for example “Grep on x.log contains ‘ERROR: server failed’”.
printReason (bool) – If True the specified outcomeReason will be printed
abortOnError (bool) – If true abort the test on any error outcome. This should usually be set to False for assertions, or the configured
self.defaultAbortOnError
setting (typically True) for operations that involve waiting.callRecord (list[str]) – An array of strings of the form path:lineno indicating the call stack that lead to this outcome. This will be appended to the log output for better test triage.
override (bool) – Remove any existing test outcomes when adding this one, ensuring that this outcome is the one and only one reported even if an existing outcome has higher precedence.
- allocateUniqueStdOutErr(processKey)¶
Allocate unique filenames of the form
processKey[.n].out/.err
which can be used for thestartProcess
stdouterr
parameter.The first time this is called it will return names like
('outdir/myprocess.out', 'outdir/myprocess.err')
, the second time it will return('outdir/myprocess.1.out', 'outdir/myprocess.1.err')
, then('outdir/myprocess.2.out', 'outdir/myprocess.2.err')
etc.- Parameters
processKey (str) – A user-defined identifier that will form the prefix onto which
[.n].out
is appended- Returns
A STDOUTERR_TUPLE named tuple of (stdout, stderr), where each is an absolute path.
- Return type
- static compareVersions(v1, v2)¶
Compares two alphanumeric dotted version strings to see which is more recent.
Example usage:
if self.compareVersions(thisversion, '1.2.alpha-3') > 0: ... # thisversion is newer than 1.2.alpha-3
The comparison algorithm ignores case, and normalizes separators ./-/_ so that
'1.alpha2'=='1Alpha2'
. Any string components are compared lexicographically with other strings, and compared to numbers strings are always considered greater.>>> ProcessUser.compareVersions('10-alpha5.dev10', '10alpha-5-dEv_10') == 0 # normalization of case and separators True
>>> ProcessUser.compareVersions(b'1....alpha.2', u'1Alpha2') == 0 # ascii byte and unicode strings both supported True
>>> ProcessUser.compareVersions('1.2.0', '1.2') 0
>>> ProcessUser.compareVersions('1.02', '1.2') 0
>>> ProcessUser().compareVersions('1.2.3', '1.2') > 0 True
>>> ProcessUser.compareVersions('1.2', '1.2.3') -1
>>> ProcessUser.compareVersions('10.2', '1.2') 1
>>> ProcessUser.compareVersions('1.2.text', '1.2.0') # letters are > numbers 1
>>> ProcessUser.compareVersions('1.2.text', '1.2') # letters are > numbers 1
>>> ProcessUser.compareVersions('10.2alpha1', '10.2alpha') 1
>>> ProcessUser.compareVersions('10.2dev', '10.2alpha') # letters are compared lexicographically 1
>>> ProcessUser.compareVersions('', '') 0
>>> ProcessUser.compareVersions('1', '') 1
- Parameters
v1 – A string containing a version number, with any number of components.
v2 – A string containing a version number, with any number of components.
- Returns
an integer > 0 if v1>v2, an integer < 0 if v1<v2, or 0 if they are semantically the same.
- copy(src, dest, mappers=[], encoding=None, symlinks=False, ignoreIf=None, skipMappersIf=None, overwrite=None)¶
Copy a directory or a single text or binary file, optionally tranforming the contents by filtering each line through a list of mapping functions.
If any
pysys.mappers
are provided, the file is copied in text mode and each mapper is given the chance to modify or omit each line, or even reorder the lines of the file. If no mappers are provided, the file is copied in binary mode.For example:
self.copy('output-raw.txt', 'output-processed.txt', encoding='utf-8', mappers=[ lambda line: None if ('Timestamp: ' in line) else line, lambda line: line.replace('foo', 'bar'), pysys.mappers.IncludeLinesBetween('Error message .*:', stopBefore='^$'), pysys.mappers.RegexReplace(pysys.mappers.RegexReplace.DATETIME_REGEX, '<timestamp>'), ])
In addition to the file contents the attributes such as modification time and executable permission will be copied where possible.
This function is useful for creating a modified version of an output file that’s more suitable for later validation steps such as diff-ing, and also for copying required files from the input to the output directory.
It can also be used for copying a whole directory, similar to
shutil.copytree
but with the advantages of support for long paths on Windows, better error safety, and that relative paths are evaluated relative to the self.output directory (which is both convenient, and safer than shutil’s evaluation relative to the current working directory).For example:
self.copy('src.txt', 'dest.txt') # copies to outputdir/dest.txt self.copy('src.txt', self.output) # copies to outputdir/src.txt, since self.output is an existing directory self.copy('src.txt', 'foo/') # copies to outputdir/foo/src.txt since destination ends with a slash self.copy('srcdirname', 'foo/') # copies to outputdir/foo/srcdirname since destination ends with a slash
For more information about pre-defined mappers, see
pysys.mappers
. Custom mappers can be specified as simple functions or lambdas, however for advanced use cases you can additionally providemapper.fileStarted([self,] srcPath, destPath, srcFile, destFile)
and/ormapper.fileFinished(...)
methods to allow stateful operations, or to perform extra read/write operations before lines are read/written. For example:class CustomLineMapper(object): def fileStarted(self, srcPath, destPath, srcFile, destFile): self.src = os.path.basename(srcPath) def __call__(self, line): return '"'+self.src+'": '+line def fileFinished(self, srcPath, destPath, srcFile, destFile): destFile.write('\n' + 'footer added by CustomLineMapper') self.copy('src.txt', 'dest.txt', mappers=[CustomLineMapper()])
Changed in version 1.6.0: Ability to copy directories was added, along with the
overwrite=
,symlinks=
,ignoreIf=
andskipMappersIf=
arguments.- Parameters
src (str) – The source filename or directory, which can be an absolute path, or a path relative to the
self.output
directory. Usesrc=self.input+'/myfile'
if you wish to copy a file from the test input directory.dest (str) –
The destination file or directory name, which can be an absolute path, or a path relative to the
self.output
directory. Destination file(s) are overwritten if the dest already exists.The dest and src can be the same for file copies (but not directory copies).
As a convenience to avoid repeating the same text in the src and destination, if the dest ends with a slash, or the src is a file and the dest is an existing directory, the dest is taken as a parent directory into which the src will be copied in retaining its current name.
It is best to avoid copies where the src dir already contains the dest (which would be recursive) such as copying the test dir (possibly configured as
self.input
) to destinationself.output
, however if this is attempted PySys will log a warning and copy everything else except the recursive part.overwrite (bool) – If True, source files will be allowed to overwrite destination files, if False an exception will be raised if a destination file already exists. By default overwrite=None which means it’s enabled for single file copy() but disabled for directory copies.
mappers (List[callable[str]->str]) –
A list of filter functions that will be applied, in order, to map each line from source to destination. Each function accepts a string for the current line as input and returns either a string to write or None if the line is to be omitted. Any
None
items in the mappers list will be ignored. Mappers must always preserve the final\n
of each line (if present).See
pysys.mappers
for some useful predefined mappers such aspysys.mappers.IncludeLinesBetween
,pysys.mappers.RegexReplace
andpysys.mappers.SortLines
.If present the
mapper.fileStarted(...)
and/ormapper.fileFinished(...)
methods will be called on each mapper in the list at the start and end of each file; see above for an example.Do not share mapper instances across multiple tests or threads as this can cause race conditions.
encoding (str) – The encoding to use to open the file (only used if mappers are provided; if not, it is opened in binary mode). The default value is None which indicates that the decision will be delegated to the
getDefaultFileEncoding()
method.symlinks (bool) – Set to True if symbolic links in the source tree should be represented as symbolic links in the destination (rather than just being copied).
ignoreIf (callable[str]->bool) – A callable that accepts a source path and returns True if this file/directory should be omitted from a directory copy. For example:
ignoreIf=lambda src: src.endswith(('.tmp', '.log')))
skipMappersIf (callable[str]->bool) – A callable that accepts a source path and returns True if this file should be copied in binary mode (as if no mappers had been specified). For example:
skipMappersIf=lambda src: not src.endswith(('.xml', '.properties')))
- Return str
the absolute path of the destination file.
- createEnvirons(overrides=None, addToLibPath=[], addToExePath=[], command=None, **kwargs)¶
Create a new customized dictionary of environment variables suitable for passing to
startProcess()
’senvirons=
argument.As a starting point, this method uses the value returned by
getDefaultEnvirons()
for thiscommand
. See the documentation on that method for more details. If you don’t care about minimizing the risk of your local environment affecting the test processes you start, just useenvirons=os.environ
to allow child processes to inherit the entire parent environment instead of using this method.- Parameters
overrides – A dictionary of environment variables whose values will be used instead of any existing values. You can use
os.getenv('VARNAME','')
if you need to pass selected variables from the current process as part of the overrides list. If the value is set to None then any variable of this name will be deleted. Use unicode strings if possible (byte strings will be converted depending on the platform). A list of dictionaries can be specified, in which case the latest will override the earlier if there are any conflicts.addToLibPath – A path or list of paths to be prepended to the default value for the environment variable used to load libraries (or the value specified in overrides, if any), i.e.
[DY]LD_LIBRARY_PATH
on Unix orPATH
on Windows. This is usually more convenient than adding it directly tooverrides
.addToExePath – A path or list of paths to be prepended to the default value for the environment variable used to locate executables (or the value specified in overrides, if any), i.e.
PATH
on both Unix and Windows. This is usually more convenient than adding it directly tooverrides
.command – If known, the full path of the executable for which a default environment is being created (passed to
getDefaultEnvirons
).kwargs – Overrides of this method should pass any additional kwargs down to the super implementation, to allow for future extensions.
- Returns
A new dictionary containing the environment variables.
- deleteDir(path, **kwargs)¶
Recursively delete the specified directory.
Does nothing if it does not exist. Raises an exception if the deletion fails.
- Parameters
path – The path to be deleted. This can be an absolute path or relative to the testcase output directory.
kwargs – Any additional arguments such as
retries
andignore_errors
are passed topysys.utils.fileutils.deletedir()
.
- deleteFile(path, **kwargs)¶
Delete the specified file, with optional retries and ignoring of errors.
Does nothing if it does not exist. Raises an exception if the deletion fails.
- Parameters
path – The path to be deleted. This can be an absolute path or relative to the testcase output directory.
kwargs – Any additional arguments such as
retries
andignore_errors
are passed topysys.utils.fileutils.deletefile()
.
- disableLogging()¶
Temporarily stops logging for the current thread.
For example:
with self.disableLogging(): self.startProcess(...)
Note that this method will do nothing if the log level if pysys is run with
-vDEBUG
or-vdisabledLogging=DEBUG
.New in version 1.6.0.
- getBoolProperty(propertyName, default=False)¶
Get a True/False indicating whether the specified attribute is set on this object (typically as a result of specifying -X on the command line), or else from the project configuration.
See also
pysys.baserunner.getXArg()
andpysys.config.project.Project.getProperty()
.- Parameters
propertyName – The name of a property set on the command line or project configuration.
- getDefaultEnvirons(command=None, **kwargs)¶
Create a new dictionary of environment variables, suitable for passing to
startProcess()
, with a minimal clean set of environment variables for this platform, unaffected (as much as possible) by the environment that the tests are being run under.This environment contains a minimal PATH/LD_LIBRARY_PATH but does not attempt to replicate the full set of default environment variables on each OS, and in particular it does not include any that identify the current username or home area. Additional environment variables can be added as needed with
createEnvirons
overrides. If you don’t care about minimizing the risk of your local environment affecting the test processes you start, just useenvirons=os.environ
to allow child processes to inherit the entire parent environment.The
createEnvirons()
andstartProcess()
methods use this as the basis for creating a new set of default environment variables.If needed this method can be overridden in subclasses to add common environment variables for every process invoked by startProcess, for example to enable options such as code coverage for Java/Python/etc. This is also a good place to customize behaviour for different operating systems.
Some features of this method can be configured by setting project properties:
defaultEnvirons.ENV_KEY
: if any properties with this prefix are defined, an environment variable with the ENV_KEY is set by this method (unless the property value is empty). For example, to set a default JVM heap size for all processes with theJAVA_TOOL_OPTIONS
environment variable you could setdefaultEnvirons.JAVA_TOOL_OPTIONS = -Xmx512M
.defaultEnvironsDefaultLang
: if set to a value such asen_US.UTF-8
the specified value is set for the LANG= variable on Unix; otherwise, the LANG variable is not set (which might result in use of the legacy POSIX/C encoding).defaultEnvironsTempDir
: if set the expression will be passed to Pythoneval()
and used to set the OS-specific temp directory environment variables. A typical value isself.output
.defaultEnvironsLegacyMode
: set to true to enable compatibility mode which keeps the behaviour the same as PySys v1.1, 1.2 and 1.3, namely using a completely empty default environment on Unix, and a copy of the entire parent environment on Windows. This is not recommended unless you have a lot of legacy tests that cannot easily be changed to only set minimal required environment variables usingcreateEnvirons()
.
- Parameters
command –
If known, the full path of the executable for which a default environment is being created (when called from
startProcess
this is always set). This allows default environment variables to be customized for different process types e.g. Java, Python, etc.When using
command=sys.executable
to launch another copy of the current Python executable, extra items from this process’s path environment variables are added to the returned dictionary so that it can start correctly. On Unix-based systems this includes copying all of the load library path environment variable from the parent process.kwargs – Overrides of this method should pass any additional kwargs down to the super implementation, to allow for future extensions.
- Returns
A new dictionary containing the environment variables.
- getExprFromFile(path, expr, groups=[1], returnAll=False, returnNoneIfMissing=False, encoding=None, reFlags=0, mappers=[])¶
Searches for a regular expression in the specified file, and returns it.
Use of this function is discouraged - consider using
grep
/grepOrNone
/grepAll
instead.If the regex contains unnamed groups using
(expr)
syntax, the specified group is returned. If the expression is not found, an exception is raised, unless returnAll=True or returnNoneIfMissing=True. For example:myKey = self.getExprFromFile('test.txt', r'myKey="(.*)"') # on a file containing 'myKey="foobar"' would return "foobar" err = self.getExprFromFile('test.txt', r'ERROR .*') # on a file containing 'ERROR It went wrong' would return "that entire string"
If you have a complex expression with multiple values to extract, it is usually clearer to use
(?P<groupName>...)
named groups rather than unnamed groups referenced by index. This produces a dictionary:authInfo = self.getExprFromFile('myserver.log', expr=r'Successfully authenticated user "(?P<username>[^"]*)" in (?P<authSecs>[^ ]+) seconds\.')) allAuthList = self.getExprFromFile('myserver.log', expr=r'Successfully authenticated user "(?P<username>[^"]*)" in (?P<authSecs>[^ ]+) seconds\.', returnAll=True))
See also
pysys.basetest.BaseTest.assertThatGrep
which should be used when instead of just finding out what’s in the file you want to assert that a specific expression is matched. The documentation for assertGrep also provides some helpful examples of regular expressions that could also be applied to this method, and tips for dealing with escaping in regular expressions.Changed in version 1.6.0: Support for named groups was added in 1.6.0.
- Parameters
path (str) – file to search (located in the output dir unless an absolute path is specified)
expr (str) –
the regular expression, optionally containing the regex group operator
(...)
Remember to escape regular expression special characters such as
.
,(
,[
,{
and\
if you want them to be treated as literal values. If you have a string with regex backslashes, it’s best to use a ‘raw’ Python string so that you don’t need to double-escape them, e.g.expr=r'function[(]"str", 123[.]4, (\d+), .*[)]'
.groups (List[int]) – which numeric regex group numbers (as indicated by brackets in the regex) should be returned; default is
[1]
meaning the first group. If more than one group is specified, the result will be a tuple of group values, otherwise the result will be the value of the group at the specified index as a str. This parameter is ignored if the regular expression contains any(?P<groupName>...)
named groups.returnAll (bool) – returns a list containing all matching lines if True, the first matching line otherwise.
returnNoneIfMissing (bool) – True to return None instead of raising an exception if the regex is not found in the file (not needed when returnAll is used).
encoding (str) – The encoding to use to open the file. The default value is None which indicates that the decision will be delegated to the
getDefaultFileEncoding()
method.mappers (List[callable[str]->str]) –
A list of filter functions that will be used to pre-process each line from the file (returning None if the line is to be filtered out). This provides a very powerful capability for filtering the file, for example
pysys.mappers.IncludeLinesBetween
provides the ability to filter in/out sections of a file.Do not share mapper instances across multiple tests or threads as this can cause race conditions.
Added in PySys 1.6.0.
reFlags (int) –
Zero or more flags controlling how the behaviour of regular expression matching, combined together using the
|
operator, for examplereFlags=re.VERBOSE | re.IGNORECASE
.For details see the
re
module in the Python standard library. Note thatre.MULTILINE
cannot be used because expressions are matched against one line at a time. Added in PySys 1.5.1.
- Returns
For a regular expression with one unnamed group, the match value is a str; if there are multiple unnamed numeric groups it is List[str] (with values corresponding to the groups= argument); if it contains any
(?P<groupName>...)
named groups a dict[str,str] is returned where the keys are the groupNames.If returnAll=True, the return value is a list of all the match values, with types as above.
- getInstanceCount(displayName)¶
(Deprecated) Return the number of processes started within the testcase matching the supplied displayName.
- Deprecated
The recommended way to allocate unique names is now
allocateUniqueStdOutErr
The ProcessUser class maintains a reference count of processes started within the class instance via the
startProcess()
method. The reference count is maintained against a logical name for the process, which is thedisplayName
used in the method call tostartProcess()
, or the basename of the command if no displayName was supplied. The method returns the number of processes started with the supplied logical name, or 0 if no processes have been started.- Parameters
displayName – The process display name
- Returns
The number of processes started matching the command basename
- Return type
int
- getNextAvailableTCPPort(hosts=['', 'localhost'], socketAddressFamily=AddressFamily.AF_INET)¶
Allocate a free TCP port which can be used for starting a server on this machine.
The port is taken from the pool of available server (non-ephemeral) ports on this machine, and will not be available for use by any other code in the current PySys process until this object’s
cleanup
method is called to return it to the pool of available ports. For advanced options such as port exclusions seepysys.utils.allocport
.To allocate an IPv4 port for use only on this host:
port = self.getNextAvailableTCPPort(hosts=['localhost'])
Changed in version 1.5.1: Added hosts and socketAddressFamily parameters.
- Parameters
hosts (list(Str)) –
A list of the host names or IP addresses to check when establishing that a potential allocated port isn’t already in use by a process outside the PySys framework. By default we check
""
(which corresponds toINADDR_ANY
and depending on the OS means either one or all non-localhost IPv4 addresses) and alsolocalhost
.Many machines have multiple network cards each with its own host IP address, and typically you’ll only be using one of them in your test, most commonly
localhost
. If you do know which host/IP you’ll actually be using, just specify that directly to save time, and avoid needlessly opening remote ports on hosts you’re not using. A list of available host addresses can be found fromsocket.getaddrinfo('', None)
.socketAddressFamily – The socket address family e.g. IPv4 vs IPv6. See Python’s
socket
module for details.
- getOutcome()¶
Get the final outcome for this test, based on the precedence order defined in
pysys.constants.OUTCOMES
.To find out whether this test has failed:
if self.getOutcome().isFailure(): ...
- Return pysys.constants.Outcome
The overall outcome. Use
%s
orstr()
to convert to a display name.
- getOutcomeLocation()¶
Get the location in the Python source file where this location was added.
New in version 1.6.0.
- Return (str,str)
The absolute filename, and the line number. Returns (None,None) if not known.
- getOutcomeReason()¶
Get the reason string for the current overall outcome (if specified).
- Returns
The overall test outcome reason or ‘’ if not specified
- Return type
string
- grep(path, expr, encoding=None, reFlags=0, mappers=[], **kwargs)¶
Returns the first occurrence of a regular expression in the specified file, or raises an exception if not found.
See also
grepOrNone
andgrepAll
or no-error-on-missing and return-all behaviour.If you want to use a grep to set the outcome of the test, use
pysys.basetest.BaseTest.assertThatGrep
orpysys.basetest.BaseTest.assertGrep
instead. The documentation for assertGrep also provides some helpful examples of regular expressions that could also be used with this method, and tips for escaping in regular expressions.If you have a complex expression with multiple values to extract, you can use
(?P<groupName>...)
named groups in which case a dictionary is returned providing access to the individual elements:authInfoDict = self.grep('myserver.log', expr=r'Successfully authenticated user "(?P<username>[^"]*)" in (?P<authSecs>[^ ]+) seconds\.'))
For extracting a single value you can use an unnamed group using
(expr)
syntax, in which case that group is returned:myKey = self.grep('test.txt', r'myKey="(.*)"') # on a file containing 'myKey="foobar"' would return "foobar"
- Parameters
path (str) – file to search (located in the output dir unless an absolute path is specified)
expr (str) –
the regular expression, optionally containing named groups.
Remember to escape regular expression special characters such as
.
,(
,[
,{
and\
if you want them to be treated as literal values. If you have a string with regex backslashes, it’s best to use a ‘raw’ Python string so that you don’t need to double-escape them, e.g.expr=r'function[(]"str", 123[.]4, (\d+), .*[)]'
.encoding (str) – The encoding to use to open the file. The default value is None which indicates that the decision will be delegated to the
getDefaultFileEncoding()
method.mappers (List[callable[str]->str]) –
A list of filter functions that will be used to pre-process each line from the file (returning None if the line is to be filtered out). This provides a very powerful capability for filtering the file, for example
pysys.mappers.IncludeLinesBetween
provides the ability to filter in/out sections of a file.Do not share mapper instances across multiple tests or threads as this can cause race conditions.
Added in PySys 1.6.0.
reFlags (int) –
Zero or more flags controlling how the behaviour of regular expression matching, combined together using the
|
operator, for examplereFlags=re.VERBOSE | re.IGNORECASE
.For details see the
re
module in the Python standard library. Note thatre.MULTILINE
cannot be used because expressions are matched against one line at a time. Added in PySys 1.5.1.
- Returns
A str containing the matching expression, or if the expr contains any
(?P<groupName>...)
named groups a dict[str,str] is returned where the keys are the groupNames.
- grepAll(path, expr, encoding=None, reFlags=0, mappers=[], **kwargs)¶
Returns a list of all the occurrences of a regular expression in the specified file.
See also
grep
andgrepOrNone
for return-first-only behaviour.If you have a complex expression with multiple values to extract, you can use
(?P<groupName>...)
named groups in which case each item in the returned list is a dictionary is returned providing access to the individual elements:authInfoDictList = self.grepAll('myserver.log', expr=r'Successfully authenticated user "(?P<username>[^"]*)" in (?P<authSecs>[^ ]+) seconds\.'))
For extracting a single value you can use an unnamed group using
(expr)
syntax, in which case that group is returned:myKey = self.grepAll('test.txt', r'myKey="(.*)"') # on a file containing 'myKey="foobar"' would return ["foobar"]
- Parameters
path (str) – file to search (located in the output dir unless an absolute path is specified)
expr (str) –
the regular expression, optionally containing named groups.
Remember to escape regular expression special characters such as
.
,(
,[
,{
and\
if you want them to be treated as literal values. If you have a string with regex backslashes, it’s best to use a ‘raw’ Python string so that you don’t need to double-escape them, e.g.expr=r'function[(]"str", 123[.]4, (\d+), .*[)]'
.encoding (str) – The encoding to use to open the file. The default value is None which indicates that the decision will be delegated to the
getDefaultFileEncoding()
method.mappers (List[callable[str]->str]) –
A list of filter functions that will be used to pre-process each line from the file (returning None if the line is to be filtered out). This provides a very powerful capability for filtering the file, for example
pysys.mappers.IncludeLinesBetween
provides the ability to filter in/out sections of a file.Do not share mapper instances across multiple tests or threads as this can cause race conditions.
Added in PySys 1.6.0.
reFlags (int) –
Zero or more flags controlling how the behaviour of regular expression matching, combined together using the
|
operator, for examplereFlags=re.VERBOSE | re.IGNORECASE
.For details see the
re
module in the Python standard library. Note thatre.MULTILINE
cannot be used because expressions are matched against one line at a time. Added in PySys 1.5.1.
- Returns
A list where each item is a str containing the matching expression, or if the expr contains any
(?P<groupName>...)
named groups each item is a dict[str,str] where the keys are the groupNames.
- grepOrNone(path, expr, encoding=None, reFlags=0, mappers=[], **kwargs)¶
Returns the first occurrence of a regular expression in the specified file, or None if not found.
See also
grep
andgrepAll
for error-on-missing and return-all behaviour.If you want to use a grep to set the outcome of the test, use
pysys.basetest.BaseTest.assertThatGrep
orpysys.basetest.BaseTest.assertGrep
instead. The documentation for assertGrep also provides some helpful examples of regular expressions that could also be used with this method, and tips for escaping in regular expressions.If you have a complex expression with multiple values to extract, you can use
(?P<groupName>...)
named groups in which case a dictionary is returned providing access to the individual elements:authInfoDict = self.grepOrNone('myserver.log', expr=r'Successfully authenticated user "(?P<username>[^"]*)" in (?P<authSecs>[^ ]+) seconds\.') ) or {'username':'myuser', 'authSecs': '0.0'}
For extracting a single value you can use an unnamed group using
(expr)
syntax, in which case that group is returned:myKey = self.grepOrNone('test.txt', r'myKey="(.*)"') or 'mydefault' # on a file containing 'myKey="foobar"' would return "foobar"
- Parameters
path (str) – file to search (located in the output dir unless an absolute path is specified)
expr (str) –
the regular expression, optionally containing named groups.
Remember to escape regular expression special characters such as
.
,(
,[
,{
and\
if you want them to be treated as literal values. If you have a string with regex backslashes, it’s best to use a ‘raw’ Python string so that you don’t need to double-escape them, e.g.expr=r'function[(]"str", 123[.]4, (\d+), .*[)]'
.encoding (str) – The encoding to use to open the file. The default value is None which indicates that the decision will be delegated to the
getDefaultFileEncoding()
method.mappers (List[callable[str]->str]) –
A list of filter functions that will be used to pre-process each line from the file (returning None if the line is to be filtered out). This provides a very powerful capability for filtering the file, for example
pysys.mappers.IncludeLinesBetween
provides the ability to filter in/out sections of a file.Do not share mapper instances across multiple tests or threads as this can cause race conditions.
Added in PySys 1.6.0.
reFlags (int) –
Zero or more flags controlling how the behaviour of regular expression matching, combined together using the
|
operator, for examplereFlags=re.VERBOSE | re.IGNORECASE
.For details see the
re
module in the Python standard library. Note thatre.MULTILINE
cannot be used because expressions are matched against one line at a time. Added in PySys 1.5.1.
- Returns
A str containing the matching expression, None if there are no matches, or if the expr contains any
(?P<groupName>...)
named groups a dict[str,str] is returned where the keys are the groupNames.
- logFileContents(path, includes=None, excludes=None, maxLines=20, tail=False, encoding=None, logFunction=None, reFlags=0, stripWhitespace=True, mappers=[], color=True)¶
Logs some or all of the lines from the specified file.
If the file does not exist or cannot be opened, does nothing. The method is useful for providing key diagnostic information (e.g. error messages from tools executed by the test) directly in run.log, or to make test failures easier to triage quickly.
- Parameters
path (str) – May be an absolute, or relative to the test output directory.
includes (list[str]) – Optional list of regex strings. If specified, only matches of these regexes will be logged.
excludes (list[str]) –
Optional list of regex strings. If specified, no line containing these will be logged.
The variable
self.logFileContentsDefaultExcludes
(=[]
by default) is used when this method is called with the default argument ofexcludes=None
, and can be used to provide a global set of default exclusion lines shared by all your tests, which is particularly useful if some processes always log some unimportant text to stderr (or stdout) that would be distracting to log out.Added in PySys 1.6.0.
mappers (List[callable[str]->str]) –
A list of filter functions that will be used to pre-process each line from the file (returning None if the line is to be filtered out). This provides a very powerful capability for filtering the file, for example
pysys.mappers.IncludeLinesBetween
provides the ability to filter in/out sections of a file.Do not share mapper instances across multiple tests or threads as this can cause race conditions.
Added in PySys 2.0.
maxLines (int) – Upper limit on the number of lines from the file that will be logged. Set to zero for unlimited
tail (bool) – Prints the _last_ ‘maxLines’ in the file rather than the first ‘maxLines’.
encoding (str) – The encoding to use to open the file. The default value is None which indicates that the decision will be delegated to the
getDefaultFileEncoding()
method.logFunction (Callable[[line],None]) –
The function that will be used to log individual lines from the file. Usually this is
self.log.info(u' %s', line, extra=BaseLogFormatter.tag(LOG_FILE_CONTENTS))
but a custom implementation can be provided, for example to provide a different color usingpysys.utils.logutils.BaseLogFormatter.tag
.Added in PySys 1.5.1.
reFlags (int) –
Zero or more flags controlling how the behaviour of regular expression matching, combined together using the
|
operator, for examplereFlags=re.VERBOSE | re.IGNORECASE
.For details see the
re
module in the Python standard library. Note thatre.MULTILINE
cannot be used because expressions are matched against one line at a time. Added in PySys 1.5.1.stripWhitespace (bool) – By default blank lines are removed; set this to False to disable that behaviour. Added in PySys 2.0.
color (bool) – By default logged lines are colored blue to distinguish from the rest of the log contents. Set this to False to disable coloring. Added in PySys 2.1.
- Returns
True if anything was logged, False if not.
- mkdir(path)¶
Create a directory, with recursive creation of any parent directories.
This function does nothing (does not raise an except) if the directory already exists.
- Parameters
path – The path to be created. This can be an absolute path or relative to the testcase output directory.
- Returns
the absolute path of the new directory, to facilitate fluent-style method calling.
- pollWait(secs)¶
Sleeps the current thread for the specified number of seconds, between polling/checking for some condition to be met.
Unlike
pysys.basetest.BaseTest.wait
(which should be used for larger waits inside tests), pollWait does not log anything.Use this method instead of
time.sleep
as it provides PySys the chance to abort test execution early when requested.- Parameters
secs (float) – The time to sleep for, typically a few hundred milliseconds. Do not use this method for really long waits.
- pythonDocTest(pythonFile, pythonPath=None, output=None, environs=None, **kwargs)[source]¶
Execute the Python doctests that exist in the specified python file; adds a FAILED outcome if any do not pass.
- Parameters
pythonFile – the absolute path to a python file name.
pythonPath – a list of directories to be added to the PYTHONPATH.
output – the output file; if not specified, ‘%s-doctest.txt’ is used with the basename of the python file.
kwargs – extra arguments are passed to startProcess/startPython.
- signalProcess(process, signal, abortOnError=None)¶
Send a signal to a running process.
This method uses the
pysys.process.Process.signal
method to send a signal to a running process.Should the request to send the signal to the running process fail, a
BLOCKED
outcome will be added to the outcome list.- Parameters
process – The process handle returned from the
startProcess
methodsignal – The integer value of the signal to send
abortOnError – If True aborts the test with an exception on any error, if False just log it as a warning. (defaults to the defaultAbortOnError project setting)
- skipTest(outcomeReason, callRecord=None)¶
Raise an AbortException that will set the test outcome to SKIPPED and ensure that the rest of the execute() and validate() methods do not execute.
This is useful when a test should not be executed in the current mode or platform.
- Parameters
outcomeReason – A string summarizing the reason the test is being skipped, for example “Feature X is not supported on Windows”.
- startProcess(command, arguments, environs=None, workingDir=None, state=None, timeout=600, stdout=None, stderr=None, displayName=None, abortOnError=None, expectedExitStatus='==0', ignoreExitStatus=None, onError=None, quiet=False, stdouterr=None, background=False, info={}, processFactory=<class 'pysys.process.helper.ProcessImpl'>)¶
Start a process running in the foreground or background, and return the
pysys.process.Process
object.Typical use is:
myexecutable = self.startProcess('path_to_my_executable', arguments=['myoperation', 'arg1','arg2'], environs=self.createEnvirons(addToLibPath=['my_ld_lib_path']), # if a customized environment is needed stdouterr=self.allocateUniqueStdOutErr('myoperation'), # for stdout/err files, pick a suitable logical name for what it's doing background=True # or remove for default behaviour of executing in foreground )
The method allows spawning of new processes in a platform independent way. The command, arguments, environment and working directory to run the process in can all be specified in the arguments to the method, along with the filenames used for capturing the stdout and stderr of the process. Processes may be started in the foreground, in which case the method does not return until the process has completed or a time out occurs, or in the background in which case the method returns immediately to the caller returning a handle to the process to allow manipulation at a later stage, typically with
waitProcess
.All processes started in the background are automatically killed on completion of the test via the
cleanup()
destructor. To wait for background processes to complete duringexecute
and/or beforeverify
commences, callwaitForBackgroundProcesses()
. This is especially useful when you have a test that needs to execute lots of processes asynchronously, since having them all execute concurrently in the background and then callingwaitForBackgroundProcesses()
will be a lot quicker than executing them serially in the foreground.When starting a process that will listen on a server socket, use
getNextAvailableTCPPort
to allocate a free port before calling this method.Note that although is is possible to use this command to execute OS shell commands, that should only used for testing of shell scripts - other logic such as file system operations can be executed more easily and robustly using built-in Python (
os
module) or PySys (e.g.BaseTest.copy
) functions.Changed in version 1.6.0: Added onError parameter and default behaviour of logging stderr/out when there’s a failure. Added info parameter.
Changed in version 2.0: Added processFactory parameter.
- Parameters
command (str) – The path to the executable to be launched (should include the full path)
arguments (list[str]) – A list of arguments to pass to the command. Any non-string values in the list are converted to strings automatically.
environs (dict(str,str)) – A dictionary specifying the environment to run the process in. If a None or empty dictionary is passed,
getDefaultEnvirons
will be invoked to produce a suitable clean default environment for thiscommand
, containing a minimal set of variables. If you wish to specify a customized environment,createEnvirons()
is a great way to create it.workingDir (str) – The working directory for the process to run in (defaults to the testcase output subdirectory)
background (bool) – Set to True to start the process in the background. By default processes are started in the foreground, meaning execution of the test will continue only once the process has terminated.
state – Alternative way to set
background=True
. Run the process either in theFOREGROUND
orBACKGROUND
(defaults toFOREGROUND
). Setting state=BACKGROUND is equivalent to setting background=True; in new tests using background=True is the preferred way to do this.timeout (int) – The number of seconds after which to terminate processes running in the foreground. For processes that complete in a few seconds or less, it is best to avoid overriding this and stick with the default. However for long-running foreground processes it will be necessary to set a larger number, for example if running a soak test where the process needs to run for up to 2 hours you could set
timeout=2*60*60
.stdouterr (str) – The filename prefix to use for the stdout and stderr of the process (
out
/err
will be appended), or a tuple of (stdout,stderr) as returned fromallocateUniqueStdOutErr
. The stdouterr prefix is also used to form a default display name for the process if none is explicitly provided. The files are created relative to the test output directory. The filenames can be accessed from the returned process object using.stdout/err
frompysys.process.Process
.stdout (str) – The filename used to capture the stdout of the process. It is usually simpler to use
stdouterr
instead of this.stderr (str) – The filename used to capture the stderr of the process. It is usually simpler to use
stdouterr
instead of this.displayName (str) – Logical name of the process used for display in log messages, and the str(…) representation of the returned process object (defaults to a string generated from the stdouterr and/or the command).
abortOnError (bool) – If true abort the test on any error outcome (defaults to the defaultAbortOnError project setting)
expectedExitStatus (str) – The condition string used to determine whether the exit status/code returned by the process is correct. The default is ‘==0’, as an exit code of zero usually indicates success, but if you are expecting a non-zero exit status (for example because you are testing correct handling of a failure condition) this could be set to ‘!=0’ or a specific value such as ‘==5’.
ignoreExitStatus (bool) –
If False, a BLOCKED outcome is added if the process terminates with an exit code that doesn’t match expectedExitStatus (or if the command cannot be run at all). This can be set to True in cases where you do not care whether the command succeeds or fails, or wish to handle the exit status separately with more complicated logic.
The default value of ignoreExitStatus=None means the value will be taken from the project property defaultIgnoreExitStatus, which can be configured in the project XML (the recommended default property value is defaultIgnoreExitStatus=False), or is set to True for compatibility with older PySys releases if no project property is set.
onError (Callable[pysys.process.Process]->str) –
A function that will be called if the process times out or returns an unexpected exit status (unless ignoreExitStatus=True), before any abort exception is raised. This provides a convenient place to add logging of diagnostic information (perhaps using the stdout/err of the process) and/or extracting and returning an error message from the output, for example:
onError=lambda process: self.logFileContents(process.stderr, tail=True) or self.logFileContents(process.stdout, tail=True)
.If a string value is returned from it will be added to the failure reason, e.g.
onError=lambda process: self.logFileContents(process.stderr, tail=True) and self.grepOrNone(process.stderr, 'Error: (.*)')
.If no onError function is specified, the default is to log the last few lines of stderr (or if empty, stdout) when a process fails and abortOnError=True.
The
self.logFileContentsDefaultExcludes
variable can be used to add regular expressions to exclude unimportant lines of output such as standard startup lines (seelogFileContents
).quiet (bool) – If True, this method will not do any INFO or WARN level logging (only DEBUG level), unless a failure outcome is appended. This parameter can be useful to avoid filling up the log where it is necessary to repeatedly execute a command check for completion of some operation until it succeeds; in such cases you should usually set ignoreExitStatus=True as well since both success and failure exit statuses are valid.
info (dict[str,obj]) – A dictionary of user-defined information about this process that will be set as a field on the returned Process instance. This is useful for keeping track of things like server port numbers and log file paths.
processFactory (callable[kwargs]) –
A callable (such as a class constructor) that returns an instance or subclass of
pysys.process.helper.ProcessImpl
. This can be used either to provide a custom process subclass with extra features, or to make modifications to the arguments or environment that were specified by the code that invokedstartProcess()
.The signature must consist of a
**kwargs
parameter, the members of which will be populated by the parameters listed in the constructor ofpysys.process.Process
, and can be modified by the factory. For example:def myProcessFactory(**kwargs): kwargs['arguments'] = kwargs['arguments'][0]+['my_extra_arg']+kwargs['arguments'][1:] return pysys.process.helper.ProcessImpl(**kwargs)
- Returns
The process object.
- Return type
- startPython(arguments, disableCoverage=False, **kwargs)¶
Start a Python process with the specified arguments.
Uses the same Python process the tests are running under.
If PySys was run with the argument
-XcodeCoverage
or-XpythonCoverage
thenstartPython
will add the necessary arguments to enable generation of code coverage. Note that this requried the coverage.py library to be installed. If a project property calledpythonCoverageArgs
exists then its value will be added as (space-delimited) arguments to the coverage tool.- Parameters
arguments – The arguments to pass to the Python executable. Typically the first one be either the name of a Python script to execute, or
-m
followed by a module name.kwargs – See
startProcess
for detail on available arguments.disableCoverage – Disables code coverage for this specific process. Coverage can also be disabled by setting
self.disableCoverage==True
on this test instance.
- Returns
The process handle of the process.
- Return type
- stopProcess(process, abortOnError=None)¶
Stops the specified process, if it is currently running.
Does nothing if the process is not running.
This is equivalent to calling
pysys.process.Process.stop()
, except it also logs an info message when the process is stopped.- Parameters
process – The process handle returned from the
startProcess
methodabortOnError – If True abort the test on any error outcome (defaults to the defaultAbortOnError project setting), if False a failure to stop the process will just be logged as a warning.
- unpackArchive(archive, dest=None, autoCleanup=True)¶
Unpacks the specified file(s) from an archive to a directory. Supports archive format such as zip/tar.gz/gz/tar.xz/xz.
It is a good idea to store large textual Input/ assets (such as log files, which usually compress very well) as compressed archives to reduce disk space in your version control system.
By default this method will automatically delete the extracted file/dir during test cleanup so it doesn’t sit around on disk (or in CI uploaded failure archives) consuming space; if the file/dir is mutated by the test you may wish to disable this so you can manually inspect them by setting
autoCleanup=False
.For example:
unpacked = self.unpackArchive('mybigfile.log.xz') # do something with "unpacked"...
Note that
.xz
(for single files) and.tar.xz
(for multiple files) are recommended for optimal compression, and these (and.gz
) are significantly better than zip, which performs poorly when compressing multiple similar text files into one archive. Don’t use more than one single archive per testcase (if possible) to ensure you benefit from similarities between the various files.Files are decompressed in binary mode, so if you require platform-native line endings you should use
copy
to post-process them after decompressing.- Parameters
archive (str) – The path of the archive to unpack, by default from the test input directory. Alternatively you could provide an absolute path using
self.project.testRootDir
or similar if an archive is shared across multiple test cases.dest (str) – The directory in which the contents of the archive will be written; by default this is the test output directory for archive types that are always single-file, and a subdirectory named after the archive if not. This dir will be created if needed.
autoCleanup (bool) – Automatically deletes the unpackaged file/directory during test cleanup to save disk space (even if the test fails).
- Returns
The full path to the decompressed file or directory.
- waitForBackgroundProcesses(includes=[], excludes=[], timeout=600, abortOnError=None, checkExitStatus=True)¶
Wait for any running background processes to terminate, then check that all background processes completed with the expected exit status.
This can be useful for speeding up a test that needs to run many processes, by executing all its subprocesses in parallel in the background (and then waiting for completion) rather than one-by-one in the foreground.
Timeouts will result in a TIMEDOUT outcome and an exception unless the project property
defaultAbortOnError==False
is set.- Parameters
includes (list[pysys.process.Process]) – A list of processes to wait for, each returned by
startProcess()
. If none are specified, this method will wait for (and check the exit status of) all background processes.excludes (list[pysys.process.Process]) – A list of processes which are not expected to have terminated yet (this is only useful when not setting includes=[]).
timeout (int) – The total time in seconds to wait for all processes to have completed, for example
timeout=TIMEOUTS['WaitForProcess']
.abortOnError (bool) – If True aborts the test with an exception on any error, if False appends an outcome but does not raise an exception.
checkExitStatus (bool) – By default this method not only waits for completion but also checks the exit status of all (included) background processes (regardless of when they completed), but set this argument to False to disable checking that the exit status of the processes matches the
expectedExitStatus
specified in the call tostartProcess
(typically==0
). The last few lines of the stderr (or stdout) will be logged if the exit status is wrong.
- waitForFile(file, filedir=None, timeout=30, abortOnError=None)¶
Wait for a file to exist on disk.
This method blocks until a file is created on disk. This is useful for test timing where a component under test creates a file (e.g. for logging) indicating it has performed all initialisation actions and is ready for the test execution steps. If a file is not created on disk within the specified timeout interval, the method returns to the caller.
- Parameters
file – The basename of the file used to wait to be created
filedir – The dirname of the file (defaults to the testcase output subdirectory)
timeout – The timeout in seconds to wait for the file to be created
abortOnError – If true abort the test on any failure (defaults to the defaultAbortOnError project setting)
- waitForGrep(file, expr='', condition='>=1', timeout=60, poll=0.25, ignores=[], process=None, errorExpr=[], errorIf=None, abortOnError=None, encoding=None, detailMessage='', filedir=None, reFlags=0, mappers=[], quiet=False)¶
Wait for a regular expression line to be seen (one or more times) in a text file in the output directory (waitForGrep was formerly known as
waitForSignal
).This method provides some parameters that give helpful fail-fast behaviour with a descriptive outcome reason; use these whenever possible:
process=
to abort if success becomes impossible due to premature termination of the process that’s generating the outputerrorExpr=
to abort if an error message/expression is written to the file being greppederrorIf=
to abort if the specified lambda function returns an error string (which can be used if the error messages go to a different file than that being grepped
This will generate much clearer outcome reasons, which makes test failures easy to triage, and also avoids wasting time waiting for something that will never happen.
Example:
self.waitForGrep('myprocess.log', expr='INFO .*Started successfully', encoding='utf-8', process=myprocess, errorExpr=[' (ERROR|FATAL) ', 'Failed to start']) self.waitForGrep('myoutput.log', expr='My message', encoding='utf-8', process=myprocess, errorIf=lambda: self.grepOrNone('myprocess.err', ' ERROR .*'))
Note that waitForGrep fails the test if the expression is not found (unless abortOnError was set to False, which isn’t recommended), so there is no need to add duplication with an
assertGrep
to check for the same expression in your validation logic.You can extract information from the matched expression, optionally perform assertions on it, by using one or more
(?P<groupName>...)
named groups in the expression. A common pattern is to unpack the resulting dict using**kwargs
syntax and pass toBaseTest.assertThat
. For example:self.assertThat('username == expected', expected='myuser', **self.waitForGrep('myserver.log', expr=r'Successfully authenticated user "(?P<username>[^"]*)"'))
New in version 1.5.1.
- Parameters
file (str) – The path of the file to be searched. Usually this is a name/path relative to the
self.output
directory, but alternatively an absolute path can be specified.expr (str) – The regular expression to search for in the text file.
condition (str) – The condition to be met for the number of lines matching the regular expression; by default we wait until there is at least one occurrence.
timeout (int) – The number of seconds to wait for the regular expression before giving up and aborting the test with
pysys.constants.TIMEDOUT
(unless abortOnError=False in which case execution will continue).process (pysys.process.Process) – The process that is generating the specified file, to allow the wait to fail fast (instead of timing out) if the process dies before the expected signal appears. Can be None if the process is not known or is expected to terminate itself during this period.
errorExpr (list[str]) – Optional list of regular expressions, which if found in the file will cause waiting for the main expression to be aborted with a
pysys.constants.BLOCKED
outcome. This is useful to avoid waiting a long time for the expected expression when an ERROR is logged that means it will never happen, and also provides much clearer test failure messages in this case.errorIf (callable->str) –
A zero-arg function that returns False/None when there is no error, or a non-empty string if an error is detected which should cause us to abort looking for the grep expression. This function will be executed frequently (every
poll
seconds) so avoid doing anything time-consuming here unless you set a large polling interval. See above for an example.Added in PySys 1.6.0.
ignores (list[str]) – A list of regular expressions used to identify lines in the files which should be ignored when matching both
expr
anderrorExpr
.mappers (List[callable[str]->str]) –
A list of filter functions that will be used to pre-process each line from the file (returning None if the line is to be filtered out). This provides a very powerful capability for filtering the file, for example
pysys.mappers.IncludeLinesBetween
provides the ability to filter in/out sections of a file, andpysys.mappers.JoinLines
can be used to put exception stack traces onto the same line as the error message.Do not share mapper instances across multiple tests or threads as this can cause race conditions.
Added in PySys 1.6.0.
poll (float) – The time in seconds between to poll the file looking for the regular expression and to check against the condition
abortOnError (bool) – If True abort the test on any error outcome (defaults to the defaultAbortOnError project setting, which for a modern project will be True).
encoding (str) – The encoding to use to open the file and convert from bytes to characters. The default value is None which indicates that the decision will be delegated to the
getDefaultFileEncoding()
method.detailMessage (str) –
An extra string to add to the start of the message logged when waiting to provide extra information about the wait condition. e.g.
detailMessage='Wait for server startup: '
.Added in v1.5.1. From v2.1+ the detail string is added at the beginning not the end.
reFlags (int) –
Zero or more flags controlling how the behaviour of regular expression matching, combined together using the
|
operator, for examplereFlags=re.VERBOSE | re.IGNORECASE
.For details see the
re
module in the Python standard library. Note thatre.MULTILINE
cannot be used because expressions are matched against one line at a time. Added in PySys 1.5.1.filedir (str) – Can be used to provide a directory name to add to the beginning of the
file
parameter; however usually it is clearer just to specify that directory in thefile
.quiet (bool) – Set this to true to suppress INFO-level logging, which can be useful when you wish to print your own progress message in a more high-level fashion.
- Return list[re.Match]
Usually this returns a list of
re.Match
objects found for theexpr
, or an empty list if there was no match.If the expr contains any
(?P<groupName>...)
named groups, and assuming the condition is still the default of “>=1” (i.e. not trying to find multiple matches), then instead of a list, a dict is returned containingdict(groupName: str, matchValue: str or None)
(or an empty{}
dict if there is no match) which allows the result to be passed toassertThat
for further checking of the matched groups (typically unpacked using the**
operator; see example above).
- waitForSignal(file, filedir=None, expr='', **waitForGrepArgs)¶
Old alias for
waitForGrep
; please usewaitForGrep
in new tests.All parameters are the same, except that in waitForSignal the (rarely used)
filedir
argument can be specified as the 2nd positional argument (afterfile
and beforeexpr
) whereas in waitForGrep it can only be specified as afiledir=
keyword argument.
- waitForSocket(port, host='localhost', timeout=60, abortOnError=None, process=None, socketAddressFamily=AddressFamily.AF_INET)¶
Wait until it is possible to establish a socket connection to a server running on the specified local or remote port.
This method blocks until connection to a particular host:port pair can be established. This is useful for test timing where a component under test creates a socket for client server interaction - calling of this method ensures that on return of the method call the server process is running and a client is able to create connections to it. If a connection cannot be made within the specified timeout interval, the method returns to the caller, or aborts the test if abortOnError=True.
Changed in version 1.5.1: Added host and socketAddressFamily parameters.
- Parameters
port – The port value in the socket host:port pair
host – The host value in the socket host:port pair
timeout – The timeout in seconds to wait for connection to the socket
abortOnError – If true abort the test on any failure (defaults to the defaultAbortOnError project setting)
process – If a handle to a process is specified, the wait will abort if the process dies before the socket becomes available. It is recommended to set this wherever possible.
socketAddressFamily – The socket address family e.g. IPv4 vs IPv6. See Python’s
socket
module for details.
- waitProcess(process, timeout=600, abortOnError=None, checkExitStatus=False)¶
Wait for a background process to terminate.
For a convenient way to wait for all remaining background processes to complete, see
waitForBackgroundProcesses
.Timeouts will result in an exception and TIMEDOUT outcome unless the project property
defaultAbortOnError==False
is set.- Parameters
process (pysys.process.Process) – The process handle returned from the
startProcess
methodtimeout (int) – The timeout value in seconds to wait before returning, for example
timeout=TIMEOUTS['WaitForProcess']
.abortOnError (bool) – If True aborts the test with an exception on any error, if False just log it as a warning. (defaults to the defaultAbortOnError project setting)
checkExitStatus (bool) – By default this method does not check the exit status, but set this argument to True to check that the exit status matches the
expectedExitStatus
specified in the call tostartProcess
(typically==0
). Note that this argument is not affected by thedefaultIgnoreExitStatus
project property. The last few lines of the stderr (or stdout) will be logged if the exit status is wrong.
- Returns
The process’s
exitStatus
. This will be None if the process timed out and abortOnError is disabled.
- writeProcess(process, data, addNewLine=True)¶
Write binary data to the stdin of a process.
This method uses
pysys.process.Process.write
to write binary data to the stdin of a process. This wrapper around the write method of the process helper only adds checking of the process running status prior to the write being performed, and logging to the testcase run log to detail the write.- Parameters
process (pysys.process.Process) – The process handle returned from the
startProcess()
methoddata (bytes) – The data to write to the process stdin. As only binary data can be written to a process stdin, if a character string rather than a byte object is passed as the data, it will be automatically converted to a bytes object using the encoding given by PREFERRED_ENCODING.
addNewLine (bool) – True if a new line character is to be added to the end of the data string
- write_text(file, text, encoding=None)¶
Writes the specified characters to a file in the output directory.
- Parameters
file – The path of the file to write, either an absolute path or relative to the
self.output
directory.text –
The string to write to the file, with
\n
for newlines (do not useos.linesep
as the file will be opened in text mode so platform line separators will be added automatically).This must be a character string.
encoding – The encoding to use to open the file. The default value is None which indicates that the decision will be delegated to the
getDefaultFileEncoding()
method.
- Return str
Return the absolute path of the generated file.
- log¶
The logger instance that should be used to log from this class.
- project¶
The
pysys.config.project.Project
instance containing settings for this PySys project.
- lock¶
A recursive lock that can be used for protecting the fields of this instance from access by background threads, as needed.