Package pysys :: Module basetest :: Class BaseTest
[hide private]
[frames] | no frames]

Class BaseTest

source code

process.user.ProcessUser --+
                           |
                          BaseTest
Known Subclasses:

The base class for all PySys testcases.

BaseTest is the parent class of all PySys system testcases. The class provides utility functions for cross-platform process management and manipulation, test timing, and test validation. Any PySys testcase should inherit from the base test and provide an implementation of the abstract execute method defined in this class. Child classes can also overide the setup, cleanup and validate methods of the class to provide custom setup and cleanup actions for a particual test, and to perform all validation steps in a single method should this prove logically more simple.

Execution of a PySys testcase is performed through an instance of the pysys.baserunner.BaseRunner class, or a subclass thereof. The base runner instantiates an instance of the testcase, and then calls the setup, execute, validate and cleanup methods of the instance. All processes started during the test execution are reference counted within the base test, and terminated within the cleanup method.

Validation of the testcase is through the assert* methods. Execution of each method appends an outcome to an internal data structure thus building up a record of the individual validation outcomes. Several potential outcomes are supported by the PySys framework (SKIPPED, BLOCKED, DUMPEDCORE, TIMEDOUT, FAILED, NOTVERIFIED, and PASSED) and the overall outcome of the testcase is determined using a precedence order of the individual outcomes. All assert* methods support variable argument lists for common non-default parameters. Currently this only includes the assertMessage parameter, to override the default statement logged by the framework to stdout and the run log.

Instance Methods [hide private]
 
__init__(self, descriptor, outsubdir, runner)
Create an instance of the BaseTest class.
source code
 
setKeywordArgs(self, xargs)
Set the xargs as data attributes of the test class.
source code
 
addOutcome(self, outcome)
Add a test validation outcome to the validation list.
source code
integer
getOutcome(self)
Get the overall outcome of the test based on the precedence order.
source code
 
setup(self)
Setup method which may optionally be overridden to perform custom setup operations prior to test execution.
source code
 
execute(self)
Execute method which must be overridden to perform the test execution steps.
source code
 
validate(self)
Validate method which may optionally be overridden to group all validation steps.
source code
 
cleanup(self)
Cleanup method which performs cleanup actions after execution and validation of the test.
source code
 
addResource(self, resource)
Add a resource which is owned by the test and is therefore cleaned up (deleted) when the test is cleaned up
source code
handle
startProcess(self, command, arguments, environs=None, workingDir=None, state=11, timeout=None, stdout=None, stderr=None, displayName=None)
Start a process running in the foreground or background, and return the process handle.
source code
 
stopProcess(self, process)
Send a soft or hard kill to a running process to stop its execution.
source code
 
signalProcess(self, process, signal)
Send a signal to a running process (Unix only).
source code
 
waitProcess(self, process, timeout)
Wait for a process to terminate, return on termination or expiry of the timeout.
source code
handle
startProcessMonitor(self, process, interval, file, **kwargs)
Start a separate thread to log process statistics to logfile, and return a handle to the process monitor.
source code
 
stopProcessMonitor(self, monitor)
Stop a process monitor.
source code
 
startManualTester(self, file, filedir=None, state=11, timeout=1800)
Start the manual tester.
source code
 
stopManualTester(self)
Stop the manual tester if running.
source code
 
waitManualTester(self, timeout=1800)
Wait for the manual tester to be stopped via user interaction.
source code
 
wait(self, interval)
Wait for a specified period of time.
source code
 
assertTrue(self, expr, **xargs)
Perform a validation assert on the supplied expression evaluating to true.
source code
 
assertFalse(self, expr, **xargs)
Perform a validation assert on the supplied expression evaluating to false.
source code
 
assertDiff(self, file1, file2, filedir1=None, filedir2=None, ignores=[], sort=False, replace=[], includes=[], **xargs)
Perform a validation assert on the comparison of two input text files.
source code
 
assertGrep(self, file, filedir=None, expr='', contains=True, **xargs)
Perform a validation assert on a regular expression occurring in a text file.
source code
 
assertLastGrep(self, file, filedir=None, expr='', contains=True, ignores=[], includes=[], **xargs)
Perform a validation assert on a regular expression occurring in the last line of a text file.
source code
 
assertOrderedGrep(self, file, filedir=None, exprList=[], contains=True, **xargs)
Perform a validation assert on a list of regular expressions occurring in specified order in a text file.
source code
 
assertLineCount(self, file, filedir=None, expr='', condition='>=1', **xargs)
Perform a validation assert on the number of lines in a text file matching a specific regular expression.
source code
 
getNextAvailableTCPPort(self)
Allocate a TCP port which is available for a server to be started on.
source code
 
__assertMsg(self, xargs, default)
Return an assert statement requested to override the default value.
source code

Inherited from process.user.ProcessUser: __del__, __getattr__, getInstanceCount, waitForFile, waitForSignal, waitForSocket, writeProcess

Instance Variables [hide private]
string input
Location for input to any processes (defaults to current working directory)
logging.Logger log
Reference to the logger instance of this class
string mode
The user defined mode the test is running within.
string output
Location for output from any processes (defaults to current working directory)
Project project
Reference to the project details as set on the module load of the launching executable
string reference
Full path to the reference directory of the testcase.
Method Details [hide private]

__init__(self, descriptor, outsubdir, runner)
(Constructor)

source code 

Create an instance of the BaseTest class.

Parameters:
  • descriptor - The descriptor for the test giving all test details
  • outsubdir - The output subdirectory the test output will be written to
  • runner - Reference to the runner responsable for executing the testcase
Overrides: process.user.ProcessUser.__init__

setKeywordArgs(self, xargs)

source code 

Set the xargs as data attributes of the test class.

Values in the xargs dictionary are set as data attributes using the builtin setattr method. Thus an xargs dictionary of the form {'foo': 'bar'} will result in a data attribute of the form self.foo with value bar. This is used so that subclasses can define default values of data attributes, which can be overriden on instantiation e.g. using the -X options to the runTest.py launch executable.

Parameters:
  • xargs - A dictionary of the user defined extra arguments

addOutcome(self, outcome)

source code 

Add a test validation outcome to the validation list.

The method provides the ability to add a validation outcome to the internal data structure storing the list of test validation outcomes. In a single test run multiple validations may be performed. The currently supported validation outcomes are:

 SKIPPED:     An execution/validation step of the test was skipped (e.g. deliberately)
 BLOCKED:     An execution/validation step of the test could not be run (e.g. a missing resource)
 DUMPEDCORE:  A process started by the test produced a core file (unix only)
 TIMEDOUT:    An execution/validation step of the test timed out (e.g. process deadlock)
 FAILED:      A validation step of the test failed
 NOTVERIFIED: No validation steps were performed
 INSPECT:     A validation step of the test requires manual inspection
 PASSED:      A validation step of the test passed 

The outcomes are considered to have a precedence order, as defined by the order of the outcomes listed above. Thus a BLOCKED outcome has a higher precedence than a PASSED outcome. The outcomes are defined in pysys.constants.

Parameters:
  • outcome - The outcome to add

getOutcome(self)

source code 

Get the overall outcome of the test based on the precedence order.

The method returns the overal outcome of the test based on the outcomes stored in the internal data structure. The precedence order of the possible outcomes is used to determined the overall outcome of the test, e.g. if PASSED, BLOCKED and FAILED were recorded during the execution of the test, the overall outcome would be BLOCKED.

The method returns the integer value of the outcome as defined in pysys.constants. To convert this to a string representation use the LOOKUP dictionary i.e. LOOKUP[test.getOutcome()]

Returns: integer
The overall test outcome

execute(self)

source code 

Execute method which must be overridden to perform the test execution steps.

Raises:
  • NotImplementedError - Raised exeception should the method not be overridden

cleanup(self)

source code 

Cleanup method which performs cleanup actions after execution and validation of the test.

The cleanup method performs actions to stop all processes started in the background and not explicitly killed during the test execution. It also stops all process monitors running in seperate threads, and any instances of the manual tester user interface. Should a custom cleanup for a subclass be required, the BaseTest cleanup method should first be called. e.g. :

 class MyTest(BaseTest):
 
   def cleanup(self):
     # call base test cleanup first
     BaseTest.cleanup(self)
               
     # perform custom cleanup actions
     ...

startProcess(self, command, arguments, environs=None, workingDir=None, state=11, timeout=None, stdout=None, stderr=None, displayName=None)

source code 

Start a process running in the foreground or background, and return the process handle.

The method allows spawning of new processes in a platform independent way. The command, arguments, environment and working directory to run the process in can all be specified in the arguments to the method, along with the filenames used for capturing the stdout and stderr of the process. Processes may be started in the FOREGROUND, in which case the method does not return until the process has completed or a time out occurs, or in the BACKGROUND in which case the method returns immediately to the caller returning a handle to the process to allow manipulation at a later stage. All processes started in the BACKGROUND and not explicitly killed using the returned process handle are automatically killed on completion of the test via the cleanup method of the BaseTest.

This method uses the pysys.process.helper module to start the process. On failure conditions the method may append BLOCKED or TIMEDOUT outcomes to the test validation data structure when it was not possible to start the process (e.g. command does not exist etc), or the timeout period expired (indicating a potential deadlock or livelock in the process).

Parameters:
  • command - The command to start the process (should include the full path)
  • arguments - A list of arguments to pass to the command
  • environs - A dictionary of the environment to run the process in (defaults to clean environment)
  • workingDir - The working directory for the process to run in (defaults to the testcase output subdirectory)
  • state - Run the process either in the FOREGROUND or BACKGROUND (defaults to FOREGROUND)
  • timeout - The timeout period after which to termintate processes running in the FOREGROUND
  • stdout - The filename used to capture the stdout of the process
  • stderr - The filename user to capture the stderr of the process
  • displayName - Logical name of the process used for display and reference counting (defaults to the basename of the command)
Returns: handle
The process handle of the process (pysys.process.helper.ProcessWrapper)
Overrides: process.user.ProcessUser.startProcess

stopProcess(self, process)

source code 

Send a soft or hard kill to a running process to stop its execution.

This method uses the pysys.process.helper module to stop a running process. Should the request to stop the running process fail, a BLOCKED outcome will be added to the test outcome list.

Parameters:
  • process - The process handle returned from the startProcess method
Overrides: process.user.ProcessUser.stopProcess

signalProcess(self, process, signal)

source code 

Send a signal to a running process (Unix only).

This method uses the pysys.process.helper module to send a signal to a running process. Should the request to send the signal to the running process fail, a BLOCKED outcome will be added to the test outcome list.

Parameters:
  • process - The process handle returned from the startProcess method
  • signal - The integer value of the signal to send
Overrides: process.user.ProcessUser.signalProcess

waitProcess(self, process, timeout)

source code 

Wait for a process to terminate, return on termination or expiry of the timeout.

Parameters:
  • process - The process handle returned from the startProcess method
  • timeout - The timeout value in seconds to wait before returning
Overrides: process.user.ProcessUser.waitProcess

startProcessMonitor(self, process, interval, file, **kwargs)

source code 

Start a separate thread to log process statistics to logfile, and return a handle to the process monitor.

This method uses the pysys.process.monitor module to perform logging of the process statistics, starting the monitor as a seperate background thread. Should the request to log the statistics fail a BLOCKED outcome will be added to the test outcome list. All process monitors not explicitly stopped using the returned handle are automatically stopped on completion of the test via the cleanup method of the BaseTest.

Parameters:
  • process - The process handle returned from the startProcess method
  • interval - The interval in seconds between collecting and logging the process statistics
  • file - The full path to the filename used for logging the process statistics
  • kwargs - Keyword arguments to allow platform specific configurations
Returns: handle
A handle to the process monitor (pysys.process.monitor.ProcessMonitor)

stopProcessMonitor(self, monitor)

source code 

Stop a process monitor.

Parameters:

startManualTester(self, file, filedir=None, state=11, timeout=1800)

source code 

Start the manual tester.

The manual tester user interface (UI) is used to describe a series of manual steps to be performed to execute and validate a test. Only a single instance of the UI can be running at any given time, and can be run either in the FOREGROUND (method will not return until the UI is closed or the timeout occurs) or in the BACKGROUND (method will return straight away so automated actions may be performed concurrently). Should the UI be terminated due to expiry of the timeout, a TIMEDOUT outcome will be added to the outcome list. The UI can be stopped via the stopManualTester method. An instance of the UI not explicitly stopped within a test will automatically be stopped via the cleanup method of the BaseTest.

Parameters:
  • file - The name of the manual test xml input file (see pysys.xml.manual for details on the DTD)
  • filedir - The directory containing the manual test xml input file (defaults to the output subdirectory)
  • state - Start the manual tester either in the FOREGROUND or BACKGROUND (defaults to FOREGROUND)
  • timeout - The timeout period after which to termintate a manual tester running in the FOREGROUND

wait(self, interval)

source code 

Wait for a specified period of time.

Parameters:
  • interval - The time interval in seconds to wait

assertTrue(self, expr, **xargs)

source code 

Perform a validation assert on the supplied expression evaluating to true.

If the supplied expression evaluates to true a PASSED outcome is added to the outcome list. Should the expression evaluate to false, a FAILED outcome is added.

Parameters:
  • expr - The expression, as a string, to check for the true | false value
  • xargs - Variable argument list (see class description for supported parameters)

assertFalse(self, expr, **xargs)

source code 

Perform a validation assert on the supplied expression evaluating to false.

If the supplied expression evaluates to false a PASSED outcome is added to the outcome list. Should the expression evaluate to true, a FAILED outcome is added.

Parameters:
  • expr - The expression to check for the true | false value
  • xargs - Variable argument list (see class description for supported parameters)

assertDiff(self, file1, file2, filedir1=None, filedir2=None, ignores=[], sort=False, replace=[], includes=[], **xargs)

source code 

Perform a validation assert on the comparison of two input text files.

This method performs a file comparison on two input files. The files are pre-processed prior to the comparison to either ignore particular lines, sort their constituent lines, replace matches to regular expressions in a line with an alternate value, or to only include particular lines. Should the files after pre-processing be equivalent a PASSED outcome is added to the test outcome list, otherwise a FAILED outcome is added.

Parameters:
  • file1 - The basename of the first file used in the file comparison
  • file2 - The basename of the second file used in the file comparison
  • filedir1 - The dirname of the first file (defaults to the testcase output subdirectory)
  • filedir2 - The dirname of the second file (defaults to the testcase reference directory)
  • ignores - A list of regular expressions used to denote lines in the files which should be ignored
  • sort - Boolean flag to indicate if the lines in the files should be sorted prior to the comparison
  • replace - List of tuples of the form ('regexpr', 'replacement'). For each regular expression in the list, any occurences in the files is replaced with the replacement value prior to the comparison being carried out. This is often useful to replace timestamps in logfiles etc.
  • includes - A list of regular expressions used to denote lines in the files which should be used in the comparison. Only lines which match an expression in the list are used for the comparison
  • xargs - Variable argument list (see class description for supported parameters)

assertGrep(self, file, filedir=None, expr='', contains=True, **xargs)

source code 

Perform a validation assert on a regular expression occurring in a text file.

When the contains input argument is set to true, this method will add a PASSED outcome to the test outcome list if the supplied regular expression is seen in the file; otherwise a FAILED outcome is added. Should contains be set to false, a PASSED outcome will only be added should the regular expression not be seen in the file.

Parameters:
  • file - The basename of the file used in the grep
  • filedir - The dirname of the file (defaults to the testcase output subdirectory)
  • expr - The regular expression to check for in the file
  • contains - Boolean flag to denote if the expression should or should not be seen in the file
  • xargs - Variable argument list (see class description for supported parameters)

assertLastGrep(self, file, filedir=None, expr='', contains=True, ignores=[], includes=[], **xargs)

source code 

Perform a validation assert on a regular expression occurring in the last line of a text file.

When the contains input argument is set to true, this method will add a PASSED outcome to the test outcome list if the supplied regular expression is seen in the file; otherwise a FAILED outcome is added. Should contains be set to false, a PASSED outcome will only be added should the regular expression not be seen in the file.

Parameters:
  • file - The basename of the file used in the grep
  • filedir - The dirname of the file (defaults to the testcase output subdirectory)
  • expr - The regular expression to check for in the last line of the file
  • contains - Boolean flag to denote if the expression should or should not be seen in the file
  • ignores - A list of regular expressions used to denote lines in the file which should be ignored
  • includes - A list of regular expressions used to denote lines in the file which should be used in the assertion.
  • xargs - Variable argument list (see class description for supported parameters)

assertOrderedGrep(self, file, filedir=None, exprList=[], contains=True, **xargs)

source code 

Perform a validation assert on a list of regular expressions occurring in specified order in a text file.

When the contains input argument is set to true, this method will append a PASSED outcome to the test outcome list if the supplied regular expressions in the exprList are seen in the file in the order they appear in the list; otherwise a FAILED outcome is added. Should contains be set to false, a PASSED outcome will only be added should the regular expressions not be seen in the file in the order they appear in the list.

Parameters:
  • file - The basename of the file used in the ordered grep
  • filedir - The dirname of the file (defaults to the testcase output subdirectory)
  • exprList - A list of regular expressions which should occur in the file in the order they appear in the list
  • contains - Boolean flag to denote if the expressions should or should not be seen in the file in the order specified
  • xargs - Variable argument list (see class description for supported parameters)

assertLineCount(self, file, filedir=None, expr='', condition='>=1', **xargs)

source code 

Perform a validation assert on the number of lines in a text file matching a specific regular expression.

This method will add a PASSED outcome to the outcome list if the number of lines in the input file matching the specified regular expression evaluate to true when evaluated against the supplied condition.

Parameters:
  • file - The basename of the file used in the line count
  • filedir - The dirname of the file (defaults to the testcase output subdirectory)
  • expr - The regular expression used to match a line of the input file
  • condition - The condition to be met for the number of lines matching the regular expression
  • xargs - Variable argument list (see class description for supported parameters)

getNextAvailableTCPPort(self)

source code 

Allocate a TCP port which is available for a server to be started on. Take ownership of it for the duration of the test

__assertMsg(self, xargs, default)

source code 

Return an assert statement requested to override the default value.

Parameters:
  • xargs - Variable argument list to an assert method
  • default - Default assert statement to return if a parameter is not supplied

Instance Variable Details [hide private]

input

Location for input to any processes (defaults to current working directory)
Type:
string

mode

The user defined mode the test is running within. Subclasses can use this in conditional checks to modify the test execution based upon the mode.
Type:
string

output

Location for output from any processes (defaults to current working directory)
Type:
string

reference

Full path to the reference directory of the testcase. This is used both by the class and its subclasses to locate the default directory containing all reference data to the testcase, as defined in the testcase descriptor.
Type:
string