pysys.mappers¶

Mappers that filter or transform lines of input, for use with methods such as pysys.basetest.BaseTest.copy.

`RegexReplace`(regex, replacement)	Mapper that transforms lines by replacing all character sequences matching the specified regular expression.
`IncludeLinesBetween`([startAt, stopAfter, …])	Mapper that filters out all lines except those within a range of expressions.
`IncludeLinesMatching`(regex)	Mapper that filters lines by including only lines matching the specified regular expression.
`ExcludeLinesMatching`(regex)	Mapper that filters lines by excluding/ignoring lines matching the specified regular expression.

New in version 1.6.0.

RegexReplace¶

class pysys.mappers.RegexReplace(regex, replacement)[source]¶

Bases: object

Mapper that transforms lines by replacing all character sequences matching the specified regular expression.

For example:

self.copy('myfile.txt', 'myfile-processed.txt', mappers=[RegexReplace(RegexReplace.DATETIME_REGEX, '<timestamp>')])

Parameters

regex (str|compiled_regex) – The regular expression to search for.
replacement (str) – The string to replace it with. This can contain backslash references to groups in the regex; see re.sub() in the Python documentation for more information.

>>> RegexReplace(RegexReplace.DATETIME_REGEX, '<timestamp>')('Test string x=2020-07-15T19:22:34+00:00.')
'Test string x=<timestamp>.'

>>> RegexReplace(RegexReplace.DATETIME_REGEX, '<timestamp>')('Test string x=5/7/2020 19:22:34.1234.')
'Test string x=<timestamp>.'

>>> RegexReplace(RegexReplace.DATETIME_REGEX, '<timestamp>')('Test string x=20200715T192234Z.')
'Test string x=<timestamp>.'

>>> RegexReplace(RegexReplace.NUMBER_REGEX, '<number>')('Test string x=123.')
'Test string x=<number>.'

>>> RegexReplace(RegexReplace.NUMBER_REGEX, '<number>')('Test string x=-12.45e+10.')
'Test string x=<number>.'

DATETIME_REGEX = '(([0-9]{1,4}[/-][0-9]{1,2}[/-][0-9]{2,4}[ T]?)?[0-9]{1,2}:[0-9]{2}:[0-9]{2}([.][0-9]+|Z|[+-][0-9][0-9](:[0-9][0-9])?)?|[0-9]{8}T[0-9]{6}(Z|[+-][0-9][0-9]:)?)'¶: A regular expression that can be used to match timestamps in ISO 8601 format and other common alternatives such as: “2020-07-15T19:22:34+00:00”, “5/7/2020 19:22:34.1234”, “20200715T192234Z”

NUMBER_REGEX = '[+-]?[0-9]+([.][0-9]+)?([eE][-+]?[0-9]+)?'¶

Mapper that transforms lines, replacing all integer or floating point numbers with “<number>”.

This is useful for removing ids that would diff-ing files more difficult, if you only care about validating the non-numeric text.

IncludeLinesBetween¶

class pysys.mappers.IncludeLinesBetween(startAt=None, stopAfter=None, stopBefore=None)[source]¶

Bases: object

Mapper that filters out all lines except those within a range of expressions.

This is useful when a log file contains lots of data you don’t care about, in addition to some multi-line sequences that you want to extract (with pysys.basetest.BaseTest.copy) ready for pysys.basetest.BaseTest.assertDiff.

As this mapper is stateful, so not use a single instance of it in multiple tests (or multiple threads).

The following parameters can be either a callable/lambda that accepts an input line and returns a boolean, or a regular expression string to search for in the specified line.

Parameters

startAt (str|callable[str]->bool) – If it matches then the current line and subsequent lines are included (not filtered out). If not specified, lines from the start of the file onwards are matched.
stopAfter (str|callable[str]->bool) – If it matches then lines after the current one are filtered out (unless/until a line matching startAt is found). Includes the stop line.
stopBefore (str|callable[str]->bool) – If it matches then this line and lines after it are filtered out (unless/until a line matching startAt is found). Excludes the stop line.

>>> def test_IncludeLinesBetween(mapper, input): return ','.join(x for x in (mapper(line) for line in input.split(',')) if x is not None)
>>> test_IncludeLinesBetween( IncludeLinesBetween('start.*', 'stopafter.*'), 'a,start line,b,c,stopafter line,d,start line2,e')
'start line,b,c,stopafter line,start line2,e'

>>> test_IncludeLinesBetween( IncludeLinesBetween(startAt='start.*'), 'a,start line,b,c')
'start line,b,c'

>>> test_IncludeLinesBetween( IncludeLinesBetween(stopAfter='stopafter.*'), 'a,stopafter,b,c')
'a,stopafter'

>>> test_IncludeLinesBetween( IncludeLinesBetween(stopBefore='stopbefore.*'), 'a,b,stopbefore,c')
'a,b'

IncludeLinesMatching¶

class pysys.mappers.IncludeLinesMatching(regex)[source]¶

Bases: object

Mapper that filters lines by including only lines matching the specified regular expression.

Parameters: regex (str|compiled_regex) – The regular expression to match (use .* at the beginning to allow extra characters at the start of the line). Multiple expressions can be combined using (expr1|expr2) syntax.

>>> IncludeLinesMatching('Foo.*')('Foo bar')
'Foo bar'

>>> IncludeLinesMatching('bar.*')('Foo bar') is None
True

ExcludeLinesMatching¶

class pysys.mappers.ExcludeLinesMatching(regex)[source]¶

Bases: object

Mapper that filters lines by excluding/ignoring lines matching the specified regular expression.

Parameters: regex (str|compiled_regex) – The regular expression to match (use .* at the beginning to allow extra characters at the start of the line). Multiple expressions can be combined using (expr1|expr2) syntax.

>>> ExcludeLinesMatching('Foo.*')('Foo bar') is None
True

>>> ExcludeLinesMatching('bar.*')('Foo bar')
'Foo bar'