pysys.config.descriptor¶

The TestDescriptor class holds metadata for each testcase (pysystest.*) or directory (pysysdirconfig.xml), and the DescriptorLoader class allows customization of the test discovery process.

IGNORED_PYSYSTEST_SUFFIXES¶

pysys.config.descriptor.IGNORED_PYSYSTEST_SUFFIXES = ('~', '.tmp', '.bak', '.swp', '.orig')¶

A tuple listing pysystest.* suffixes that will be ignored due to being temporary/backup/swap files for common editors and IDEs.

The list can be extended by setting the PYSYS_IGNORED_PYSYSTEST_SUFFIXES environment variable to a comma-separated list of additional extensions.

TestDescriptor¶

class pysys.config.descriptor.TestDescriptor(file, id, type='auto', state='runnable', title='', purpose='', groups=[], modes=[], classname='PySysTest', module='run.py', input='Input', output='Output', reference='Reference', traceability=[], executionOrderHint=0.0, skippedReason=None, authors=[], created=None, testDir=None, isDirConfig=False, userData=None)[source]¶

Bases: object

Descriptor metadata for an individual testcase (pysystest.*) or defaults for tests under a directory subtree (pysysdirconfig.xml); see Test Descriptors.

The DescriptorLoader class is responsible for determining the available descriptor instances.

Variables

~.file (str) – The absolute path of the testcase descriptor file.
~.testDir (str) – The absolute path of the test, which is used to convert any relative paths into absolute paths.
~.id (str) – The testcase identifier, or the id prefix if this is a directory config descriptor rather than a testcase descriptor. Includes a mode suffix if this is a multi-mode test.
~.idWithoutMode (str) – The raw testcase identifier with no mode suffix.
~.type (str) – The kind of test this is (auto or manual)
~.skippedReason (str) – If set to a non-empty string, indicates that this testcase is skipped and provides the reason. If this is set then the test is skipped regardless of the value of state.
~.state (str) – The state of the testcase (runnable, deprecated or skipped). This field is deprecated - we recommend using skippedReason instead, which provides a descriptive outcome to explain why.
~.title (str) – The one-line title summarizing this testcase.
~.purpose (str) – A detailed description of the purpose of the testcase.
~.groups (list[str]) – A list of the user defined groups the testcase belongs to.
~.modes (list[TestMode]) – A list of the user defined modes the testcase can be run in.
~.mode (TestMode) –
Specifies which of the possible modes this descriptor represents or None if the the descriptor has no modes. This field is only present after the raw descriptors have been expanded into multiple mode-specific descriptors. Note that after a descriptor is created from the on-disk file, the mode attribute is not set until the later phase when multi-mode descriptors are cloned and expanded based on the selected modes.

You can use descriptor.mode.params to get the parameter dictionary for this mode, and descriptor.mode.isPrimary to find out of this is a primary mode.
~.classname (str) – The Python classname to be executed for this testcase.
~.module (str) – The path to the python module containing the testcase class. Relative to testDir, or an absolute path. If not set, the class is looked up in the PYTHONPATH.
~.input (str) – The path to the input directory of the testcase. Relative to testDir, or an absolute path.
~.output (str) – The path to the output parent directory of the testcase. Relative to testDir, or an absolute path.
~.reference (str) – The path to the reference directory of the testcase. Relative to testDir, or an absolute path.
~.traceability (list) – A list of the requirements covered by the testcase, typically keywords or bug/story ids.
~.authors (list[str]) – A list of the names or user ids of people who contributed to the test.
~.created (str) – The date when the test was created in yyyy-mm-dd format.
~.executionOrderHint (float) – A float priority value used to determine the order in which testcases will be run; higher values are executed before low values. The default is 0.0.
~.isDirConfig (bool) – True if this is a directory configuration, or False if it’s a normal testcase.
~.userData (dict[str,obj]) – A Python dictionary that can be used for storing user-defined data in the descriptor. In a pysystest.xml, this can be populated by one or more user-data elements, e.g. <data><user-data name="key" value="val ${projectProperty}"</user-data></data>.

toDict()[source]¶: Converts this descriptor to an (ordered) dict suitable for serialization.

TestModesConfigHelper¶

class pysys.config.descriptor.TestModesConfigHelper(inheritedModes, project, testDir)[source]¶

Bases: object

A helper class that is passed to the lambda which defines test modes in a pysystest configuration. It provides access to the list of inherited modes, to project properties and also helper functions for combining multiple mode lists into one and for configuring a collection of modes as primary modes.

See User Guide for detailed information about what you can do with PySys modes.

Variables

~.inheritedModes (list[dict[str,obj]]) – A list of the inherited modes, each defined by a dictionary containing a mode key and any number of additional parameters.
~.constants – A reference to pysys.constants which can be used to access constants such as IS_WINDOWS for platform-dependent mode configuration.
~.pysys – A reference to the pysys module.
import_module (callable[str]) – A reference to the Python importlib.import_module function that can be used if you need to access functions from additional modules such as sys, re, etc.
~.project (pysys.config.project.Project) – The project configuration, from which you can read properties.
~.testDir (str) – The test directory, i.e. the directory containing the pysystest.* file where this modes configuration is defined. This is not available when defining modes in pysysdirconfig, but only when the mode configuration is directly in the pysystest.* file.

makeAllPrimary(modes)[source]¶

Modifies the specified list (or dict) of modes so that all of them have isPrimary=True.

By default only the first mode in the mode list is “primary”, so the test will only run in that one mode by default during local development (unless you supply a --modes or --ci argument). This is optimal when using modes to validate the same behaviour/conditions in different execution environments e.g. browsers/databases etc. However when using modes to validate different behaviours/conditions (e.g. testing out different command line options) using a single PySysTest class, then you should have all your modes as “primary” as you want all of them to execute by default in a quick local test run.

You would typically combine test-specific behaviour modes with any inherited execution environment modes like this:

lambda modes: modes.createModeCombinations(
        helper.inheritedModes,
        helper.makeAllPrimary(
                {
                        'Usage':        {'cmd': ['--help'], 'expectedExitStatus':'==0'}, 
                        'BadPort':      {'cmd': ['--port', '-1'],  'expectedExitStatus':'!=0'}, 
                        'MissingPort':  {'cmd': [],  'expectedExitStatus':'!=0'}, 
                }, 
)

Parameters: modes (list[dict[str,obj]]|dict[str,dict[str,obj]]) – A list or dict of modes to be made primary.
Returns: A list[dict[str,obj]] containing the modes, each with isPrimary set to true.

createModeCombinations(*dimensions)[source]¶

Generates a mode list containing all the combinations from each mode list passed into the function.

For example, you could combine a list of inherited modes (defined in parent directories’ pysysdirconfig.xml files), with a second dimension containing modes for each database you want to test with and a third dimension with modes for each web browser. The result would be a single (flat) list containing modes, with names and parameter dictionaries automatically merged together from each input dimension:

lambda helper: helper.createModeCombinations(
        helper.inheritedModes,
        {
                'MySQL':  {'db': 'MySQL',  'dbTimeoutSecs':60}, 
                'SQLite': {'db': 'SQLite', 'dbTimeoutSecs':120},
                'Mock':   {'db': 'Mock',   'dbTimeoutSecs':30},
        },

        # can use dict or list format for each mode list, whichever is more convenient: 
        [ 
                {'browser':'Chrome'}, # if mode is not explicitly specified it is auto-generated from the parameter(s)
                {'browser':'Firefox'},
        ]
        )

would generate a list of modes named:

MySQL_Chrome
MySQL_Firefox
SQLite_Chrome
SQLite_Firefox
Mock_Chrome
Mock_Firefox

NB: By default the first mode in each dimension is designated a primary mode (one that executes by default when no --modes or --ci argument is specified), but this can be overridden by setting 'isPrimary': True/False in the dict for any mode. When mode dimensions are combined, the primary modes are AND-ed together, i.e. any where all mode dimensions will be designated primary. So in the above case, since MySQL and Chrome are automatically set as primary modes, the MySQL_Chrome mode would be the (only) primary mode returned from this function. When using modes for different execution environments/browsers etc you probably want only the first (typically fastest/simplest/most informative) mode to be primary; on the other hand if using modes to re-use the same PySysTest logic for against various behavioural tests (different input files/args etc) you should usually set all of the modes to be primary so that all of them are executed in your test runs during local development.

A common use case is to combine inherited modes from the parent pysysdirconfigs with a list of modes specific to this test:

lambda helper: helper.createModeCombinations(
        helper.inheritedModes,

        helper.makeAllPrimary(
                {
                        'Usage':         {'cmd': ['--help'], 
                                'expectedExitStatus':'==0', 'expectedMessage':None}, 
                        'BadPort':       {'cmd': ['--port', '-1'],  
                                'expectedExitStatus':'!=0', 'expectedMessage':'Server failed: Invalid port number specified: -1'}, 
                        'SetPortTwice':  {'cmd': ['--port', '123', '--config', helper.testDir+'/myserverconfig.json'], 
                                'expectedExitStatus':'!=0', 'expectedMessage':'Server failed: Cannot specify port twice'}, 
                }), 
)

For simplicity this common case can be expressed with the more concise syntax:

__pysys_parameterized_test_modes__ = {
                'Usage':        {'cmd': ['--help'], 'expectedExitStatus':'==0'}, 
                'BadPort':      {'cmd': ['--port', '-1'],  'expectedExitStatus':'!=0'}, 
                'MissingPort':  {'cmd': [],  'expectedExitStatus':'!=0'}, 
        }

NB: For efficiency reasons, don’t use the createModeCombinations method in your configuration if you are just using the inherited modes unchanged.

Parameters: dimensions (list[dict[str,obj]]|dict[str,dict[str,obj]]) – Each argument passed to this function is a list of modes, each mode defined by a dict which may contain a mode key plus any number of parameters. Alternatively, each dimension can be a dict where the mode is the key and the value is a parameters dict.
Returns: A list[dict[str,obj]] containing the flattened list of modes consisting of all combinations of the passed in. Mode names and parameter dictionaries will be merged. For example: [{"mode":"MyMode", "param1":100}, {"mode": "myMode2", "param1":200}]. The returned list can be further manipulated using Python list comprehensions (e.g. to exclude certain combinations) if desired.

New in version 2.1.

combineModeDimensions(*dimensions)[source]¶: Old name for the createModeCombinations method.

TestMode¶

class pysys.config.descriptor.TestMode(s, params=None, isPrimary=False)[source]¶

Bases: str

Represents a mode that a test can run in in a TestDescriptor, and optionally a dict of parameters that define that mode.

To create one:

mode = TestMode('MyMode', {'param1': 'value1'})

See the mode parameter/field in TestDescriptor where this class is used.

This class is immutable, so create a new instance if you want to change something.

The mode values can be of any type as long as it is pickleable (see Python’s pickle module for more information). Standard Python types like strings, dicts etc are fine.

For convenience and compatibility, this TestMode subclasses a string holding the mode.

Variables

~.name (str) – The name of the mode as a string.
~.params (dict[str,obj]) – A dictionary of parameters associated with this mode. The parameters are available to the test (as self.mode.params) and also assigned as instance fields on the test class when it runs in this mode.
~.isPrimary (bool) – Indicates whether this is a primary mode, one that executes by default even when not explicity requested with a command line option such as --modes=ALL.

New in version 2.0.

DescriptorLoader¶

class pysys.config.descriptor.DescriptorLoader(project, **kwargs)[source]¶

Bases: object

This class is responsible for locating and loading all available testcase descriptors.

A custom DescriptorLoader subclass can be provided to customize the test discovery process, typically by overriding loadDescriptors and modifying the returned list of descriptors and configuring your pysysproject.xml with:

<descriptor-loader module="mypkg.custom" classname="CustomDescriptorLoader"/>

You could use this approach to add additional descriptor instances to represent non-PySys testcases found under the search directory, for example based on discovery of unit test classes.

Another key use case would be dynamically adding or changing descriptor settings such as the list of modes for each testcase or the executionOrderHint, perhaps based on a per-group basis. For example, you could modify descriptors in the “database-tests” group to have a dynamically generated list of modes identifying the possible database servers supported without having to hardcode that list into any descriptor files on disk, and allowing for different database modes on different platforms.

This class may use multi-threading to improve performance, so any extensions must be thread-safe.

Variables: ~.project (pysys.config.project.Project) – The pysys.config.project.Project instance.

loadDescriptors(dir, **kwargs)[source]¶

Find all descriptors located under the specified directory, and return them as a list.

Subclasses may change the returned descriptors and/or add additional instances of their own to the list after calling the super implementation:

descriptors = super(CustomDescriptorLoader, self).loadDescriptors(dir, **kwargs)
...
return descriptors

Parameters

dir – The parent directory to search for runnable tests.

Returns

List of pysys.config.descriptor.TestDescriptor objects which could be selected for execution.

If a test can be run in multiple modes there must be a single descriptor for it in the list returned from this method. Each multi-mode descriptor is later expanded out into separate mode-specific descriptors (at the same time as descriptor filtering based on command line arguments, and addition of project-level execution-order), before the final list is sorted and passed to pysys.baserunner.BaseRunner.

The order of the returned list is random, so the caller is responsible for sorting this list to ensure deterministic behaviour.

Return type

list

Raises

UserError – Raised if no testcases can be found.

_handleSubDirectory(dir, subdirs, files, descriptors, parentDirDefaults, **kwargs)[source]¶

Overrides the handling of each sub-directory found while walking the directory tree during loadDescriptors.

Can be used to add test descriptors, and/or add custom logic for preventing PySys searching a particular part of the directory tree perhaps based on the presence of specific files or subdirectories within it.

This method is called before directories containing pysysignore files are stripped out.

Parameters

dir (str) – The full path of the directory to be processed. On Windows, this will be a long-path safe unicode string.
subdirs (list[str]) – a list of the subdirectories under dir, which can be used to detect what kind of directory this is, and also can be modified by this method to prevent other loaders looking at subdirectories.
files (list[str]) – a list of the files under dir, which can be used to detect what kind of directory this is, and also can be modified by this method to prevent other loaders looking at them.
descriptors (list[TestDescriptor]) – A list of TestDescriptor items which this method can add to if desired.
parentDirDefaults (TestDescriptor) – A TestDescriptor containing defaults from the parent directory, or None if there are none. Test loaders may optionally merge some of this information with test-specific information when creating test descriptors.
kwargs (dict) – Reserved for future use. Pass this to the base class implementation when calling it.

Returns

If True, this part of the directory tree has been fully handled and PySys will not search under it any more. False to allow normal PySys handling of the directory to proceed.

_parseTestDescriptor(descriptorfile, parentDirDefaults=None, isDirConfig=False, **kwargs)[source]¶

Parses a single descriptor file (typically an XML file, or a file of another type containing XML) for a testcase or directory configuration and returns the resulting descriptor.

Parameters

descriptorfile – The absolute path of the descriptor file.
parentDirDefaults – A TestDescriptor instance containing defaults to inherit from the parent directory, or None if none was found.
isDirConfig – False for normal test descriptors, True for a directory configuration.

Returns

The TestDescriptor instance, or None if none should be added for this descriptor file. Note that subclasses may modify the contents of the returned instance.

Raises

UserError – If the descriptor is invalid and an error should be displayed to the user without any Python stacktrace. The exception message must contain the path of the descriptorfile.