pysys.baserunner

The runner is responsible for orchestrating concurrent execution of the tests, and for setup/cleanup of any resources that are shared across multiple tests.

BaseRunner

class pysys.baserunner.BaseRunner(record, purge, cycle, mode, threads, outsubdir, descriptors, xargs)

Bases: pysys.process.user.ProcessUser

A single instance of the runner class is responsible for orchestrating concurrent execution of tests, and managing setup and cleanup of any resources that are shared across multiple testcases.

Selection of the tests (and modes) to be run is performed through the pysys.py run launch script, which locates and creates a set of pysys.config.descriptor.TestDescriptor objects based on the command line arguments supplied by the user, and passes it to the runner. After executing any custom setup logic the runner’s start method is responsible for iterating through the descriptor list and for each entry importing and creating an instance of the BaseTest subclass named in the descriptor. The runner deletes the contents of the test output directory (to remove any output from previous runs) then calls the test’s setup, execute, validate and cleanup methods. After each test is complete it performs cleanup of the output directory (removing all files but run.log if purge is enabled, or else just files that are empty), detects any core files produced by the test, and invokes any applicable writers to record the results of each test.

In most cases the best way to provide runner (i.e. cross-test) shared functionality by creating one or more runner plugins rather than subclassing BaseRunner. Runner plugins provide functionality such as starting of servers and virtual machines to be shared by all tests (not created per-test, for which you’d use a test plugin instead), or to execute extra logic when the runner is setup, and when it is cleaned up after all tests have completed.

At minimum, a runner plugin is just a class with a setup method:

def setup(self, runner):
        ...  
        self.addCleanupFunction(...) # do this if you need to execute code after tests have completed

… and no constructor (or at least no constructor arguments). Optionally it can have public methods and fields for use by testcases using self.runner.<plugin alias>.XXX, and for configuration it may have static fields for any plugin configuration properties. Static fields provides the default value (and hence the type) for each property, and then plugin.XXX is assigned the actual value before the plugin’s setup method is called. In addition to plugin properties, pysys run -Xkey=value command line overrides can be accessed using the runner’s getXArg() method.

Each runner plugin listed in the project configuration with <runner-plugin classname="..." alias="..."/> is instantiated once by the runner, and can be accessed using self.<alias> on the runner object (if an alias is provided). If you are using a third party PySys runner plugin, consult the documentation for the third party test plugin class to find out what methods and fields are available using runner.<alias>.*.

Although plugins are the recommended way to extend the runner, if needed BaseRunner itself can be subclassed, for example:

• override setup() if you need to provision resources (e.g. virtual machines, servers, user accounts, populating a database, etc) that must be shared by many testcases. The corresponding teardown should be implemented by calling addCleanupFunction().

• also override setup() if you want to customize the order or contents of the self.descriptors list of tests to be run.

• override testComplete() to customize how test output directories are cleaned up at the end of a test’s execution.

Do not override the __init__ constructor when creating a runner subclass; instead, add any initialization logic to your setup() method.

Variables

• outsubdir (str) – The --outdir for this test run, which gives the directory to be used for the output of each testcase. Typically a relative path, but can also be an absolute path. The basename of this (outDirName) is often used as an identifier for the current test run.

• output (str) –

  The full path of the output directory that this runner can use for storing global logs, persistent state for servers started in the runner setup method, and other data.

  By default the runner output directory is named based on the outsubdir and located either as a subdirectory of the testRootDir, or if under outsubdir (if it’s an absolute path); it can also be overridden using the pysysRunnerDirName project property.

  Writers and runner plugins (e.g. for code coverage reports) can either put their output under this output directory, or for increased prominence, add /.. which will put their output directly under the testDirRoot (unless an absolute --outdir path is specified in which case it will go there instead).

  Unlike test directories, the runner output directory is not automatically created or cleaned between runs, so if this is required the runner should do it be calling deleteDir() and mkdir().

• log (logging.Logger) – The Python Logger instance that should be used to record progress and status information.

• runDetails (dict[str,str]) –

  A dictionary of metadata about this test run that is included in performance summary reports and by some writers.

  The default contains a few standard values (currently these include outDirName, hostname and startTime), and additional items can be added by runner plugins - for example the build number of the application under test.

  Note that it is not permitted to try to change this dictionary after setup has completed.

• project (pysys.config.project.Project) – A reference to the singleton project instance containing the configuration of this PySys test project as defined by pysysproject.xml. The project can be used to access information such as the project properties which are shared across all tests (e.g. for hosts and credentials).

• record (bool) – Indicates if the test results should be recorded by the record writer(s), due to the --record command line argument being specified.

• purge (bool) – Indicates that all files other than run.log should be deleted from the output directory unless the test fails; this corresponds to the --purge command line argument.

• cycles (int) – The total number of times each test should be cycled; this corresponds to the --cycle command line argument. (added in PySys v2.1).

• cycle (int) – Old name, identical to cycles.

• mode (str) – No longer used.

• threads (int) – The number of worker threads to execute the requested testcases.

• descriptors (list[pysys.config.descriptor.TestDescriptor]) – A list of all the pysys.config.descriptor.TestDescriptor test descriptors that are selected for execution by the runner.

• xargs (dict(str,str|bool)) – A dictionary of additional -Xkey=value user-defined arguments. These are also set as data attributes on the class (but with automatic conversion to match the default value’s bool/int/float/list[str] type if a static variable of the same name exists on the class), and for the benefit of other classes such as runner plugins and writers that might want to define their own -X options, see the getXArg() method.

• validateOnly (bool) – True if the user has requested that instead of cleaning output directories and running each test, the validation for each test should be re-run on the previous output.

• startTime (float) – The time when the test run started (in seconds since the epoch).

• runnerPlugins (list[object]) – A list of any plugin instances configured for this runner. This allows plugins to access the functionality of other plugins if needed (for example looking them up by type in this list).

• self.runner (pysys.baserunner.BaseRunner) – Identical to self. Included so that you can write self.runner to get a reference to the runner whether self is a BaseTest object or already a BaseRunner object.

Additional variables that affect only the behaviour of a single method are documented in the associated method.

There is also a field for any runner plugins that were configured with an “alias” (see above).

runnerAbort(**kwargs)

Requests that the entire test run aborts as soon as possible.

Typically this is triggered by a process signal or a keyboard interruption, but it is also possible for a runner or test to call it directly in response to a fatal problem. It sets the ProcessUser.isRunnerAborting flag which is checked periodically by methods that poll or wait for long-running operations, and also calls handleRunnerAbort which in turn calls that method on tests that are currently executing to terminate any running processes.

It is not recommended to override this method, but subclasses may customize runner-level handling of aborts by overriding handleRunnerAbort.

New in version 2.2.

Returns

False if nothing was done because the runner was already aborting, otherwise True. Overriding subclass implementations may wish to perform their own actions only if True is returned.

handleRunnerAbort(**kwargs)

Called on a background thread to implement the aborting of the test run.

The default implementation calls ProcessUser.handleRunnerAbort on the tests that are currently executing to terminate any running processes.

New in version 2.2.

getXArg(key, default)

Get the value of a pysys run -Xkey=value argument, with conversion of the value to the required int/float/bool/list[str] type as needed (to match the type of the specified default value). The default value returned if no -X argument was provided for this key.

This method is useful for reading the -X arguments defined by runner plugins or writers.

New in version 1.6.0.

Parameters

• key (str) – The name of the -X argument.

• default (bool/int/float/list[str]/str) – The default value to return if the argument was not set on the command line. The type of the default parameter will be used to convert the property value from a string if it is provided (for list[str], comma-separated input is assumed). An exception will be raised if the value is non-empty but cannot be converted to the indicated type.

setup()

Setup method which may optionally be overridden to perform custom setup operations prior to execution of a set of testcases.

All runner plugins will be setup and instantiated before this method is executed.

Always ensure you call the super implementation of setup() before adding any custom logic, using super().setup().

testComplete(testObj, dir)

Called after a testcase’s completion (including finalization of the output and pysys.basetest.BaseTest.cleanup) to allow for post-completion tasks such as purging unwanted files from the output directory and giving writers a chance to visit the output and collect interesting files.

The default implementation removes all files with a zero file length in order to only include files with content of interest. Should self.purge be True, the purging will remove all files (excluding the run.log) on a PASSED outcome of the testcase in order to reduce the on-disk memory footprint when running a large number of tests.

See also isPurgableFile which can be used to customize how this method performs purging.

If you override this method, be sure to call the BaseRunner’s implementation afterwards inside a try...finally block. Do not put logic which could change the test outcome into this method; instead, use pysys.basetest.BaseTest.cleanup for anything which might affect the outcome.

This method is always invoked from a single thread, even in multi-threaded mode.

Parameters

• testObj – Reference to the pysys.basetest.BaseTest instance of the test just completed.

• dir – The absolute path of the test output directory to perform the purge on (testObj.output).

isPurgableFile(path)

Decides if the specified non-empty file should be purged (deleted) after a test passes when --purge is enabled.

By default this will return True, meaning that all files (other than the special case of run.log) will be purged.

This method is called by testComplete to provide runners with the ability to veto deletion of non-empty files that should always be left in a test’s output directory even when the test has passed, by returning False from this method.

Usually it is best to avoid customizing this method and instead use the collect-test-output project option to collect any required files (e.g code coverage, performance graphs etc), as collection happens before purging.

Parameters

path (str) – The absolute path of the file to be purged.

cycleComplete()

Cycle complete method which can optionally be overridden to perform custom operations between the repeated execution of a set of testcases.

The default implementation of this method does nothing. Note that providing an override for this method will result in disabling concurrent test execution across multiple cycles.

Warning

This method is deprecated and overriding it is strongly discouraged as that disables concurrent test execution across cycles. Instead, cleanup should be performed using either pysys.basetest.BaseTest.cleanup or testComplete.

start(printSummary=True)

Starts the execution of a set of testcases.

Do not override this method - instead, override setup and/or call addCleanupFunction to customize the behaviour of this runner.

The start method is the main method for executing the set of requested testcases. The set of testcases are executed a number of times determined by the self.cycle attribute. When executing a testcase all output from the execution is saved in the testcase output subdirectory; should self.cycle be set to more than 1, the output subdirectory is further split into cycle[n] directories to sandbox the output from each iteration.

Parameters

printSummary – Ignored, exists only for compatibility reasons. To provide a custom summary printing implementation, specify a BaseSummaryResultsWriter subclass in the <writers> section of your project XML file.

Returns

Use of this value is deprecated as of 1.3.0. This method returns a dictionary of testcase outcomes, and for compatibility reasons this will continue in the short term, but will be removed in a future release. Please ignore the return value of start() and use a custom BaseSummaryResultsWriter if you need to customize summarization of results.

processCoverageData()

Called during cleanup after execution of all tests has completed to allow processing of coverage data (if enabled), for example generating reports etc.

Deprecated

Instead of overriding this method, create a pysys.writer.testoutput.CollectTestOutputWriter subclass, and generate a coverage report in its cleanup method.

publishArtifact(path, category)

Notifies any interested pysys.writer.api.ArtifactPublisher writers about an artifact that they may wish to publish.

Called when a file or directory artifact is published (e.g. by another writer).

New in version 1.6.0.

Parameters

• path (str) – Absolute path of the file or directory. Where possible it is often useful to include the outDirName in the filename, so that artifacts from multiple test runs/platforms do not clash.

• category (str) – A string identifying what kind of artifact this is, e.g. “TestOutputArchive” and “TestOutputArchiveDir” (from pysys.writer.TestOutputArchiveWriter) or “CSVPerformanceReport” (from pysys.perf.reporters.CSVPerformanceReporter). If you create your own category, be sure to add an org/company name prefix to avoid clashes. Use alphanumeric characters and underscores only.

getInProgressTests()

Get a list of the test objects that are currently executing.

This method can be called from any thread, and it is safe to access testObj.descriptor and to call str() on the test object. However unless otherwise stated, avoid accessing other fields/methods of the test object since most are not designed to be safely used by other threads. The objects returned by this method will always have been fully constructed, but setup may not yet have been called, and if the test is finishing, cleanup may or may not have been called or be underway.

New in version 2.2.

Returns

A list of pysys.basetest.BaseTest test objects, sorted based on their string representation.

handleKbrdInt(prompt=True)

Deprecated - for internal use only.

logTestHeader(descriptor, cycle, **kwargs)

Logs the header for the specified descriptor before a test begin to execute, typically including the testId, title and (if applicable) cycle.

This method can be overridden if you wish to customize the information that is written to the run.log and console or how it is formatted.

abort(outcome, outcomeReason, callRecord=None)

Raise an AbortExecution exception with the specified outcome and reason, to abort the current test.

See also skipTest.

Parameters

• outcome –

  The outcome, which will NOT override any existing outcomes previously recorded, unless the abort outcome is a non-failure (such as SKIPPED). If you want overriding behaviour for failure outcomes, instead of this function use:

  self.addOutcome(outcome, outcomeReason, abortOnError=True, override=True)
  

• outcomeReason – A string summarizing the reason for the outcome.

Changed in version 2.2: Previous outcomes are no longer overridden when aborting.

addCleanupFunction(fn, ignoreErrors=False)

Registers a function that will be called as part of the cleanup of this object.

Cleanup functions should have no arguments, and are invoked in reverse order with the most recently added first (LIFO), and before the automatic termination of any remaining processes associated with this object.

Typical cleanup tasks are to cleanly shutdown processes (which is sometimes necessary to obtain code coverage information), and to (attempt to) delete large files/directories created by the test:

self.addCleanupFunction(lambda: self.cleanlyShutdownMyProcess(params))
self.addCleanupFunction(lambda: self.deleteDir('my-large-dir'), ignoreErrors=True)
self.addCleanupFunction(lambda: self.deleteFile('my-large-file.log'), ignoreErrors=True)

Parameters

• fn (Callable[]) – The cleanup function.

• ignoreErrors (bool) – By default, errors from cleanup functions will result in a test failure; set this to True to log them but not produce a test failure. This parameter was added in 1.6.0.

addOutcome(outcome, outcomeReason='', printReason=True, abortOnError=False, callRecord=None, override=False)

Add a validation outcome (and optionally a reason string) to the validation list.

The method provides the ability to add a validation outcome to the internal data structure storing the list of validation outcomes. Multiple validations may be performed, the current supported validation outcomes of which are described in Assertions and outcomes.

The outcomes are considered to have a precedence order, as defined by the order of the outcomes listed above. Thus a pysys.constants.BLOCKED outcome has a higher precedence than a pysys.constants.PASSED outcome. The outcomes are defined in pysys.constants.

This method is thread-safe.

Although this method exists on all subclasses of pysys.process.user.ProcessUser, in practice only pysys.basetest.BaseTest subclasses actually do anything with the resulting outcome.

Parameters

• outcome (pysys.constants.Outcome) – The outcome to add, e.g. pysys.constants.FAILED.

• outcomeReason (str) – A string summarizing the reason for the outcome, for example “Grep on x.log contains ‘ERROR: server failed’”.

• printReason (bool) – If True the specified outcomeReason will be printed

• abortOnError (bool) – If true abort the test on any error outcome. This should usually be set to False for assertions, or the configured self.defaultAbortOnError setting (typically True) for operations that involve waiting.

• callRecord (list[str]) – An array of strings of the form absolutepath:lineno indicating the call stack that lead to this outcome. This will be appended to the log output for better test triage.

• override (bool) – Remove any existing test outcomes when adding this one, ensuring that this outcome is the one and only one reported even if an existing outcome has higher precedence.

allocateUniqueStdOutErr(processKey)

Allocate unique filenames of the form processKey[.n].out/.err which can be used for the startProcess stdouterr parameter.

The first time this is called it will return names like ('outdir/myprocess.out', 'outdir/myprocess.err'), the second time it will return ('outdir/myprocess.1.out', 'outdir/myprocess.1.err'), then ('outdir/myprocess.2.out', 'outdir/myprocess.2.err') etc.

Parameters

processKey (str) – A user-defined identifier that will form the prefix onto which [.n].out is appended

Returns

A STDOUTERR_TUPLE named tuple of (stdout, stderr), where each is an absolute path.

Return type

STDOUTERR_TUPLE

cleanup()

Tear down function that frees resources managed by this object, for example terminating processes it has started.

Should be called exactly once by the owner of this object when is no longer needed.

Do not override this method, instead use addCleanupFunction.

static compareVersions(v1, v2)

Compares two alphanumeric dotted version strings to see which is more recent.

Example usage:

if self.compareVersions(thisversion, '1.2.alpha-3') > 0:
        ... # thisversion is newer than 1.2.alpha-3 

The comparison algorithm ignores case, and normalizes separators ./-/_ so that '1.alpha2'=='1Alpha2'. Any string components are compared lexicographically with other strings, and compared to numbers strings are always considered greater.

>>> ProcessUser.compareVersions('10-alpha5.dev10', '10alpha-5-dEv_10') == 0 # normalization of case and separators
True

>>> ProcessUser.compareVersions(b'1....alpha.2', u'1Alpha2') == 0 # ascii byte and unicode strings both supported
True

>>> ProcessUser.compareVersions('1.2.0', '1.2')
0

>>> ProcessUser.compareVersions('1.02', '1.2')
0

>>> ProcessUser().compareVersions('1.2.3', '1.2') > 0
True

>>> ProcessUser.compareVersions('1.2', '1.2.3')
-1

>>> ProcessUser.compareVersions('10.2', '1.2')
1

>>> ProcessUser.compareVersions('1.2.text', '1.2.0') # letters are > numbers
1

>>> ProcessUser.compareVersions('1.2.text', '1.2') # letters are > numbers 
1

>>> ProcessUser.compareVersions('10.2alpha1', '10.2alpha')
1

>>> ProcessUser.compareVersions('10.2dev', '10.2alpha') # letters are compared lexicographically
1

>>> ProcessUser.compareVersions('', '')
0

>>> ProcessUser.compareVersions('1', '')
1

Parameters

• v1 – A string containing a version number, with any number of components.

• v2 – A string containing a version number, with any number of components.

Returns

an integer > 0 if v1>v2, an integer < 0 if v1<v2, or 0 if they are semantically the same.

copy(src, dest, mappers=[], encoding=None, symlinks=False, ignoreIf=None, skipMappersIf=None, overwrite=None)

Copy a directory or a single text or binary file, optionally tranforming the contents by filtering each line through a list of mapping functions.

If any pysys.mappers are provided, the file is copied in text mode and each mapper is given the chance to modify or omit each line, or even reorder the lines of the file. If no mappers are provided, the file is copied in binary mode.

For example:

self.copy('output-raw.txt', 'output-processed.txt', encoding='utf-8', 
        mappers=[
                lambda line: None if ('Timestamp: ' in line) else line, 
                lambda line: line.replace('foo', 'bar'), 
                pysys.mappers.IncludeLinesBetween('Error message .*:', stopBefore='^$'),
                pysys.mappers.RegexReplace(pysys.mappers.RegexReplace.DATETIME_REGEX, '<timestamp>'),
        ])

In addition to the file contents the attributes such as modification time and executable permission will be copied where possible.

This function is useful for creating a modified version of an output file that’s more suitable for later validation steps such as diff-ing, and also for copying required files from the input to the output directory.

It can also be used for copying a whole directory, similar to shutil.copytree but with the advantages of support for long paths on Windows, better error safety, and that relative paths are evaluated relative to the self.output directory (which is both convenient, and safer than shutil’s evaluation relative to the current working directory).

For example:

self.copy('src.txt', 'dest.txt')  # copies to outputdir/dest.txt
self.copy('src.txt', self.output) # copies to outputdir/src.txt, since self.output is an existing directory
self.copy('src.txt', 'foo/')      # copies to outputdir/foo/src.txt since destination ends with a slash
self.copy('srcdirname', 'foo/')   # copies to outputdir/foo/srcdirname since destination ends with a slash

For more information about pre-defined mappers, see pysys.mappers. Custom mappers can be specified as simple functions or lambdas, however for advanced use cases you can additionally provide mapper.fileStarted([self,] srcPath, destPath, srcFile, destFile) and/or mapper.fileFinished(...) methods to allow stateful operations, or to perform extra read/write operations before lines are read/written. For example:

class CustomLineMapper(object):
        def fileStarted(self, srcPath, destPath, srcFile, destFile):
                self.src = os.path.basename(srcPath)

        def __call__(self, line):
                return '"'+self.src+'": '+line

        def fileFinished(self, srcPath, destPath, srcFile, destFile):
                destFile.write('\n' + 'footer added by CustomLineMapper')
self.copy('src.txt', 'dest.txt', mappers=[CustomLineMapper()])

Changed in version 1.6.0: Ability to copy directories was added, along with the overwrite=, symlinks=, ignoreIf= and skipMappersIf= arguments.

Parameters

• src (str) – The source filename or directory, which can be an absolute path, or a path relative to the self.output directory. Use src=self.input+'/myfile' if you wish to copy a file from the test input directory.

• dest (str) –

  The destination file or directory name, which can be an absolute path, or a path relative to the self.output directory. Destination file(s) are overwritten if the dest already exists.

  The dest and src can be the same for file copies (but not directory copies).

  As a convenience to avoid repeating the same text in the src and destination, if the dest ends with a slash, or the src is a file and the dest is an existing directory, the dest is taken as a parent directory into which the src will be copied in retaining its current name.

  It is best to avoid copies where the src dir already contains the dest (which would be recursive) such as copying the test dir (possibly configured as self.input) to destination self.output, however if this is attempted PySys will log a warning and copy everything else except the recursive part.

• overwrite (bool) – If True, source files will be allowed to overwrite destination files, if False an exception will be raised if a destination file already exists. By default overwrite=None which means it’s enabled for single file copy() but disabled for directory copies.

• mappers (List[callable[str]->str]) –

  A list of filter functions that will be applied, in order, to map each line from source to destination. Each function accepts a string for the current line as input and returns either a string to write or None if the line is to be omitted. Any None items in the mappers list will be ignored. Mappers must always preserve the final \n of each line (if present).

  See pysys.mappers for some useful predefined mappers such as pysys.mappers.IncludeLinesBetween, pysys.mappers.RegexReplace and pysys.mappers.SortLines.

  If present the mapper.fileStarted(...) and/or mapper.fileFinished(...) methods will be called on each mapper in the list at the start and end of each file; see above for an example.

  Do not share mapper instances across multiple tests or threads as this can cause race conditions.

• encoding (str) – The encoding to use to open the file (only used if mappers are provided; if not, it is opened in binary mode). The default value is None which indicates that the decision will be delegated to the getDefaultFileEncoding() method.

• symlinks (bool) – Set to True if symbolic links in the source tree should be represented as symbolic links in the destination (rather than just being copied).

• ignoreIf (callable[str]->bool) – A callable that accepts a source path and returns True if this file/directory should be omitted from a directory copy. For example: ignoreIf=lambda src: src.endswith(('.tmp', '.log')))

• skipMappersIf (callable[str]->bool) – A callable that accepts a source path and returns True if this file should be copied in binary mode (as if no mappers had been specified). For example: skipMappersIf=lambda src: not src.endswith(('.xml', '.properties')))

Return str

the absolute path of the destination file.

createEnvirons(overrides=None, addToLibPath=[], addToExePath=[], command=None, **kwargs)

Create a new customized dictionary of environment variables suitable for passing to startProcess()’s environs= argument.

As a starting point, this method uses the value returned by getDefaultEnvirons() for this command. See the documentation on that method for more details. If you don’t care about minimizing the risk of your local environment affecting the test processes you start, just use environs=os.environ to allow child processes to inherit the entire parent environment instead of using this method.

Parameters

• overrides – A dictionary of environment variables whose values will be used instead of any existing values. You can use os.getenv('VARNAME','') if you need to pass selected variables from the current process as part of the overrides list. If the value is set to None then any variable of this name will be deleted. Use unicode strings if possible (byte strings will be converted depending on the platform). A list of dictionaries can be specified, in which case the latest will override the earlier if there are any conflicts.

• addToLibPath – A path or list of paths to be prepended to the default value for the environment variable used to load libraries (or the value specified in overrides, if any), i.e. [DY]LD_LIBRARY_PATH on Unix or PATH on Windows. This is usually more convenient than adding it directly to overrides.

• addToExePath – A path or list of paths to be prepended to the default value for the environment variable used to locate executables (or the value specified in overrides, if any), i.e. PATH on both Unix and Windows. This is usually more convenient than adding it directly to overrides.

• command – If known, the full path of the executable for which a default environment is being created (passed to getDefaultEnvirons).

• kwargs – Overrides of this method should pass any additional kwargs down to the super implementation, to allow for future extensions.

Returns

A new dictionary containing the environment variables.

createThreadPoolExecutor(maxWorkers=None) → concurrent.futures.thread.ThreadPoolExecutor

Create a PySys-friendly instance of Python’s ThreadPoolExecutor, configured with support for PySys loggers, cleanup, and a recommended default number of workers.

During cleanup, the thread pool is shutdown, with any unstarted futures cancelled (requires Python 3.9+), and a wait until all in-progress futures have completed.

This is useful for tests that need to perform many latency-bound operations such as making HTTP requests. Do not use it for starting lots of processes in parallel and waiting for them, since that is more easily achieved using waitForBackgroundProcesses().

If you use submit (rather than map), then any exceptions during the submitted job will not be logged anywhere unless you you wait for the submitted job (or add an exception handler). Note that this class is not thread-safe (apart from addOutcome, startProcess and the reading of fields like self.output that don’t change) so if you need to use its fields or methods from background threads, be sure to add your own locking to the foreground and background threads in your test, including any custom cleanup functions.

Example usage:

tp = self.createThreadPoolExecutor()
results = tp.map(items, makeHTTPRequest)

# Or for more complex use cases:
submittedFutures = []
submittedFutures.append(tp.submit(...))
concurrent.futures.wait(submittedFutures)

New in version 2.2.

Parameters

maxWorkers (int) –

Overrides the maximum number of worker threads that can be created by this pool. Only override this if needed. The default maxWorkers is configured in self.threadPoolMaxWorkers and is currently set at 6 since a higher number might overload a machine (or Python, due to the GIL) if pools are used by many/all of the PySys workers/tests running in parallel. The default may change in future.

If overriding this value, use a small number of workers for Python-based logic that will hold the Python Global Interpreter Lock, or a larger/more scalable number of workers for heavily I/O-bound operations with little Python logic.

Returns

A concurrent.futures.ThreadPoolExecutor instance.

deleteDir(path, **kwargs)

Recursively delete the specified directory.

Does nothing if it does not exist. Raises an exception if the deletion fails.

Parameters

• path – The path to be deleted. This can be an absolute path or relative to the testcase output directory.

• kwargs – Any additional arguments such as retries and ignore_errors are passed to pysys.utils.fileutils.deletedir().

deleteFile(path, **kwargs)

Delete the specified file, with optional retries and ignoring of errors.

Does nothing if it does not exist. Raises an exception if the deletion fails.

Parameters

• path – The path to be deleted. This can be an absolute path or relative to the testcase output directory.

• kwargs – Any additional arguments such as retries and ignore_errors are passed to pysys.utils.fileutils.deletefile().

disableLogging()

Temporarily stops logging for the current thread.

For example:

with self.disableLogging():
        self.startProcess(...)

Note that this method will do nothing if the log level if pysys is run with -vDEBUG or -vdisabledLogging=DEBUG.

New in version 1.6.0.

getBoolProperty(propertyName, default=False)

Get a True/False indicating whether the specified attribute is set on this object (typically as a result of specifying -X on the command line), or else from the project configuration.

See also pysys.baserunner.getXArg() and pysys.config.project.Project.getProperty().

Parameters

propertyName – The name of a property set on the command line or project configuration.

getDefaultEnvirons(command=None, **kwargs)

Create a new dictionary of environment variables, suitable for passing to startProcess(), with a minimal clean set of environment variables for this platform, unaffected (as much as possible) by the environment that the tests are being run under.

This environment contains a minimal PATH/LD_LIBRARY_PATH but does not attempt to replicate the full set of default environment variables on each OS, and in particular it does not include any that identify the current username or home area. Additional environment variables can be added as needed with createEnvirons overrides. If you don’t care about minimizing the risk of your local environment affecting the test processes you start, just use environs=os.environ to allow child processes to inherit the entire parent environment.

The createEnvirons() and startProcess() methods use this as the basis for creating a new set of default environment variables.

If needed this method can be overridden in subclasses to add common environment variables for every process invoked by startProcess, for example to enable options such as code coverage for Java/Python/etc. This is also a good place to customize behaviour for different operating systems.

Some features of this method can be configured by setting project properties:

• defaultEnvirons.ENV_KEY: if any properties with this prefix are defined, an environment variable with the ENV_KEY is set by this method (unless the property value is empty). For example, to set a default JVM heap size for all processes with the JAVA_TOOL_OPTIONS environment variable you could set defaultEnvirons.JAVA_TOOL_OPTIONS = -Xmx512M.

• defaultEnvironsDefaultLang: if set to a value such as en_US.UTF-8 the specified value is set for the LANG= variable on Unix; otherwise, the LANG variable is not set (which might result in use of the legacy POSIX/C encoding).

• defaultEnvironsTempDir: if set the expression will be passed to Python eval() and used to set the OS-specific temp directory environment variables. A typical value is self.output.

• defaultEnvironsLegacyMode: set to true to enable compatibility mode which keeps the behaviour the same as PySys v1.1, 1.2 and 1.3, namely using a completely empty default environment on Unix, and a copy of the entire parent environment on Windows. This is not recommended unless you have a lot of legacy tests that cannot easily be changed to only set minimal required environment variables using createEnvirons().

Parameters

• command –

  If known, the full path of the executable for which a default environment is being created (when called from startProcess this is always set). This allows default environment variables to be customized for different process types e.g. Java, Python, etc.

  When using command=sys.executable to launch another copy of the current Python executable, extra items from this process’s path environment variables are added to the returned dictionary so that it can start correctly. On Unix-based systems this includes copying all of the load library path environment variable from the parent process.

• kwargs – Overrides of this method should pass any additional kwargs down to the super implementation, to allow for future extensions.

Returns

A new dictionary containing the environment variables.

getDefaultFileEncoding(file, **xargs)

Specifies what encoding should be used to read or write the specified text file.

This method is used to select the appropriate encoding whenever PySys needs to open a file, for example to wait for a signal, for a file-based assertion, or to write a file with replacements. Many methods allow the encoding to be overridden for just that call, but getDefaultFileEncoding exists to allow global defaults to be specified based on the filename.

For example, this method could be overridden to specify that utf-8 encoding is to be used for opening filenames ending in .xml, .json and .yaml.

The default implementation of this method uses pysysproject.xml configuration rules such as:

<default-file-encoding pattern="*.xml" encoding="utf-8"/>

A return value of None indicates default behaviour, which is to use the default OS encoding, as specified by pysys.constants.PREFERRED_ENCODING.

Parameters

• file – The filename to be read or written. This may be an absolute path or a relative path.

• xargs – Ensure that an **xargs argument is specified so that additional information can be passed to this method in future releases.

Returns

The encoding to use for this file, or None if default behaviour is to be used. For example, utf-8 or (for UTF-8 with a Byte Order Mark), utf-8-sig.

getExprFromFile(path, expr, groups=[1], returnAll=False, returnNoneIfMissing=False, mustExist=True, encoding=None, encodingReplaceOnError=False, reFlags=0, mappers=[])

Searches for a regular expression in the specified file, and returns it.

Use of this function is discouraged - consider using grep / grepOrNone / grepAll instead.

If the regex contains unnamed groups using (expr) syntax, the specified group is returned. If the expression is not found, an exception is raised, unless returnAll=True or returnNoneIfMissing=True. For example:

myKey = self.getExprFromFile('test.txt', r'myKey="(.*)"') # on a file containing 'myKey="foobar"' would return "foobar"
err = self.getExprFromFile('test.txt', r'ERROR .*') # on a file containing 'ERROR It went wrong' would return "that entire string"

If you have a complex expression with multiple values to extract, it is usually clearer to use (?P<groupName>...) named groups rather than unnamed groups referenced by index. This produces a dictionary:

authInfo = self.getExprFromFile('myserver.log', expr=r'Successfully authenticated user "(?P<username>[^"]*)" in (?P<authSecs>[^ ]+) seconds\.'))

allAuthList = self.getExprFromFile('myserver.log', expr=r'Successfully authenticated user "(?P<username>[^"]*)" in (?P<authSecs>[^ ]+) seconds\.', returnAll=True))

See also pysys.basetest.BaseTest.assertThatGrep which should be used when instead of just finding out what’s in the file you want to assert that a specific expression is matched. The documentation for assertGrep also provides some helpful examples of regular expressions that could also be applied to this method, and tips for dealing with escaping in regular expressions.

Changed in version 1.6.0: Support for named groups was added in 1.6.0.

Parameters

• path (str) – file to search (located in the output dir unless an absolute path is specified)

• expr (str) –

  the regular expression, optionally containing the regex group operator (...)

  Remember to escape regular expression special characters such as ., (, [, { and \ if you want them to be treated as literal values. If you have a string with regex backslashes, it’s best to use a ‘raw’ Python string so that you don’t need to double-escape them, e.g. expr=r'function[(]"str", 123[.]4, (\d+), .*[)]'.

• groups (List[int]) – which numeric regex group numbers (as indicated by brackets in the regex) should be returned; default is [1] meaning the first group. If more than one group is specified, the result will be a tuple of group values, otherwise the result will be the value of the group at the specified index as a str. This parameter is ignored if the regular expression contains any (?P<groupName>...) named groups.

• returnAll (bool) – returns a list containing all matching lines if True, the first matching line otherwise.

• returnNoneIfMissing (bool) – True to return None instead of raising an exception if the regex is not found in the file (not needed when returnAll is used).

• mustExist (bool) – Set to False to tolerate the file not existing and treat a missing file like an empty file.

• encoding (str) – The encoding to use to open the file. The default value is None which indicates that the decision will be delegated to the getDefaultFileEncoding() method.

• encodingReplaceOnError (bool) – Set to True to replace erroneous characters that are invalid in the expected encoding (with a backslash escape) rather than throwing an exception. Added in PySys 2.2.

• mappers (List[callable[str]->str]) –

  A list of filter functions that will be used to pre-process each line from the file (returning None if the line is to be filtered out). This provides a very powerful capability for filtering the file, for example pysys.mappers.IncludeLinesBetween provides the ability to filter in/out sections of a file.

  Do not share mapper instances across multiple tests or threads as this can cause race conditions.

  Added in PySys 1.6.0.

• reFlags (int) –

  Zero or more flags controlling how the behaviour of regular expression matching, combined together using the | operator, for example reFlags=re.VERBOSE | re.IGNORECASE.

  For details see the re module in the Python standard library. Note that re.MULTILINE cannot be used because expressions are matched against one line at a time. Added in PySys 1.5.1.

Returns

For a regular expression with one unnamed group, the match value is a str; if there are multiple unnamed numeric groups it is List[str] (with values corresponding to the groups= argument); if it contains any (?P<groupName>...) named groups a dict[str,str] is returned where the keys are the groupNames.

If returnAll=True, the return value is a list of all the match values, with types as above.

getInstanceCount(displayName)

(Deprecated) Return the number of processes started within the testcase matching the supplied displayName.

Deprecated

The recommended way to allocate unique names is now allocateUniqueStdOutErr

The ProcessUser class maintains a reference count of processes started within the class instance via the startProcess() method. The reference count is maintained against a logical name for the process, which is the displayName used in the method call to startProcess(), or the basename of the command if no displayName was supplied. The method returns the number of processes started with the supplied logical name, or 0 if no processes have been started.

Parameters

displayName – The process display name

Returns

The number of processes started matching the command basename

Return type

int

getNextAvailableTCPPort(hosts=['', 'localhost'], socketAddressFamily=<AddressFamily.AF_INET: 2>)

Allocate a free TCP port which can be used for starting a server on this machine.

The port is taken from the pool of available server (non-ephemeral) ports on this machine, and will not be available for use by any other code in the current PySys process until this object’s cleanup method is called to return it to the pool of available ports. For advanced options such as port exclusions see pysys.utils.allocport.

To allocate an IPv4 port for use only on this host:

port = self.getNextAvailableTCPPort(hosts=['localhost'])

Changed in version 1.5.1: Added hosts and socketAddressFamily parameters.

Parameters

• hosts (list(Str)) –

  A list of the host names or IP addresses to check when establishing that a potential allocated port isn’t already in use by a process outside the PySys framework. By default we check "" (which corresponds to INADDR_ANY and depending on the OS means either one or all non-localhost IPv4 addresses) and also localhost.

  Many machines have multiple network cards each with its own host IP address, and typically you’ll only be using one of them in your test, most commonly localhost. If you do know which host/IP you’ll actually be using, just specify that directly to save time, and avoid needlessly opening remote ports on hosts you’re not using. A list of available host addresses can be found from socket.getaddrinfo('', None).

• socketAddressFamily – The socket address family e.g. IPv4 vs IPv6. See Python’s socket module for details.

getOutcome()

Get the final outcome for this test, based on the precedence order defined in pysys.constants.OUTCOMES.

To find out whether this test has failed:

if self.getOutcome().isFailure():
        ...

Return pysys.constants.Outcome

The overall outcome. Use %s or str() to convert to a display name.

getOutcomeLocation()

Get the location in the Python source file where this outcome was added.

New in version 1.6.0.

Return (str,str)

The absolute filename, and the line number. Returns (None,None) if not known.

getOutcomeReason()

Get the reason string for the current overall outcome (if specified).

Returns

The overall test outcome reason or ‘’ if not specified

Return type

string

grep(path, expr, encoding=None, reFlags=0, mappers=[], **kwargs)

Returns the first occurrence of a regular expression in the specified file, or raises an exception if not found.

See also grepOrNone and grepAll or no-error-on-missing and return-all behaviour.

If you want to use a grep to set the outcome of the test, use pysys.basetest.BaseTest.assertThatGrep or pysys.basetest.BaseTest.assertGrep instead. The documentation for assertGrep also provides some helpful examples of regular expressions that could also be used with this method, and tips for escaping in regular expressions.

If you have a complex expression with multiple values to extract, you can use (?P<groupName>...) named groups in which case a dictionary is returned providing access to the individual elements:

authInfoDict = self.grep('myserver.log', expr=r'Successfully authenticated user "(?P<username>[^"]*)" in (?P<authSecs>[^ ]+) seconds\.'))

For extracting a single value you can use an unnamed group using (expr) syntax, in which case that group is returned:

myKey = self.grep('test.txt', r'myKey="(.*)"') # on a file containing 'myKey="foobar"' would return "foobar"

Parameters

• path (str) – file to search (located in the output dir unless an absolute path is specified)

• expr (str) –

  the regular expression, optionally containing named groups.

  Remember to escape regular expression special characters such as ., (, [, { and \ if you want them to be treated as literal values. If you have a string with regex backslashes, it’s best to use a ‘raw’ Python string so that you don’t need to double-escape them, e.g. expr=r'function[(]"str", 123[.]4, (\d+), .*[)]'.

• encoding (str) – The encoding to use to open the file. The default value is None which indicates that the decision will be delegated to the getDefaultFileEncoding() method.

• mappers (List[callable[str]->str]) –

  A list of filter functions that will be used to pre-process each line from the file (returning None if the line is to be filtered out). This provides a very powerful capability for filtering the file, for example pysys.mappers.IncludeLinesBetween provides the ability to filter in/out sections of a file.

  Do not share mapper instances across multiple tests or threads as this can cause race conditions.

  Added in PySys 1.6.0.

• reFlags (int) –

  Zero or more flags controlling how the behaviour of regular expression matching, combined together using the | operator, for example reFlags=re.VERBOSE | re.IGNORECASE.

  For details see the re module in the Python standard library. Note that re.MULTILINE cannot be used because expressions are matched against one line at a time. Added in PySys 1.5.1.

Returns

A str containing the matching expression, or if the expr contains any (?P<groupName>...) named groups a dict[str,str] is returned where the keys are the groupNames.

grepAll(path, expr, encoding=None, reFlags=0, mappers=[], mustExist=True, **kwargs)

Returns a list of all the occurrences of a regular expression in the specified file.

See also grep and grepOrNone for return-first-only behaviour.

If you have a complex expression with multiple values to extract, you can use (?P<groupName>...) named groups in which case each item in the returned list is a dictionary is returned providing access to the individual elements:

authInfoDictList = self.grepAll('myserver.log', expr=r'Successfully authenticated user "(?P<username>[^"]*)" in (?P<authSecs>[^ ]+) seconds\.'))

For extracting a single value you can use an unnamed group using (expr) syntax, in which case that group is returned:

myKey = self.grepAll('test.txt', r'myKey="(.*)"') # on a file containing 'myKey="foobar"' would return ["foobar"]

Parameters

• path (str) – file to search (located in the output dir unless an absolute path is specified)

• expr (str) –

  the regular expression, optionally containing named groups.

  Remember to escape regular expression special characters such as ., (, [, { and \ if you want them to be treated as literal values. If you have a string with regex backslashes, it’s best to use a ‘raw’ Python string so that you don’t need to double-escape them, e.g. expr=r'function[(]"str", 123[.]4, (\d+), .*[)]'.

• encoding (str) – The encoding to use to open the file. The default value is None which indicates that the decision will be delegated to the getDefaultFileEncoding() method.

• mappers (List[callable[str]->str]) –

  A list of filter functions that will be used to pre-process each line from the file (returning None if the line is to be filtered out). This provides a very powerful capability for filtering the file, for example pysys.mappers.IncludeLinesBetween provides the ability to filter in/out sections of a file.

  Do not share mapper instances across multiple tests or threads as this can cause race conditions.

  Added in PySys 1.6.0.

• mustExist (bool) – Set this to False to tolerate the file not existing and treat a missing file like an empty file. Added in PySys 2.2.

• reFlags (int) –

  Zero or more flags controlling how the behaviour of regular expression matching, combined together using the | operator, for example reFlags=re.VERBOSE | re.IGNORECASE.

  For details see the re module in the Python standard library. Note that re.MULTILINE cannot be used because expressions are matched against one line at a time. Added in PySys 1.5.1.

Returns

A list where each item is a str containing the matching expression, or if the expr contains any (?P<groupName>...) named groups each item is a dict[str,str] where the keys are the groupNames.

grepOrNone(path, expr, encoding=None, reFlags=0, mappers=[], mustExist=True, **kwargs)

Returns the first occurrence of a regular expression in the specified file, or None if not found.

See also grep and grepAll for error-on-missing and return-all behaviour.

If you want to use a grep to set the outcome of the test, use pysys.basetest.BaseTest.assertThatGrep or pysys.basetest.BaseTest.assertGrep instead. The documentation for assertGrep also provides some helpful examples of regular expressions that could also be used with this method, and tips for escaping in regular expressions.

If you have a complex expression with multiple values to extract, you can use (?P<groupName>...) named groups in which case a dictionary is returned providing access to the individual elements:

authInfoDict = self.grepOrNone('myserver.log', 
                expr=r'Successfully authenticated user "(?P<username>[^"]*)" in (?P<authSecs>[^ ]+) seconds\.')
        ) or {'username':'myuser', 'authSecs': '0.0'}

For extracting a single value you can use an unnamed group using (expr) syntax, in which case that group is returned:

myKey = self.grepOrNone('test.txt', r'myKey="(.*)"') or 'mydefault' # on a file containing 'myKey="foobar"' would return "foobar"

Parameters

• path (str) – file to search (located in the output dir unless an absolute path is specified)

• expr (str) –

  the regular expression, optionally containing named groups.

  Remember to escape regular expression special characters such as ., (, [, { and \ if you want them to be treated as literal values. If you have a string with regex backslashes, it’s best to use a ‘raw’ Python string so that you don’t need to double-escape them, e.g. expr=r'function[(]"str", 123[.]4, (\d+), .*[)]'.

• encoding (str) – The encoding to use to open the file. The default value is None which indicates that the decision will be delegated to the getDefaultFileEncoding() method.

• mappers (List[callable[str]->str]) –

  A list of filter functions that will be used to pre-process each line from the file (returning None if the line is to be filtered out). This provides a very powerful capability for filtering the file, for example pysys.mappers.IncludeLinesBetween provides the ability to filter in/out sections of a file.

  Do not share mapper instances across multiple tests or threads as this can cause race conditions.

  Added in PySys 1.6.0.

• mustExist (bool) – Set this to False to tolerate the file not existing and treat a missing file like an empty file. Added in PySys 2.2.

• reFlags (int) –

  Zero or more flags controlling how the behaviour of regular expression matching, combined together using the | operator, for example reFlags=re.VERBOSE | re.IGNORECASE.

  For details see the re module in the Python standard library. Note that re.MULTILINE cannot be used because expressions are matched against one line at a time. Added in PySys 1.5.1.

Returns

A str containing the matching expression, None if there are no matches, or if the expr contains any (?P<groupName>...) named groups a dict[str,str] is returned where the keys are the groupNames.

listDirContents(path, recurse=True)

Recursively scans the specified directory and returns a sorted list of the file/directory paths under it suitable for diffing.

The contents are returned in a normalized form suitable for diffing: relative to the scanned path, with forward slashes on all platforms, a trailing slash for directories, and sorted to ensure deterministic results. Symbolic links are not searched.

For example this can be used with pysys.basetest.BaseTest.assertDiff like this:

self.assertDiff(
        self.write_text('MyDir-contents.txt', '\\n'.join( self.listDirContents('MyDir') )))

Parameters

• path (str) – The path to search, either absolute or relative to the output directory.

• recurse (bool) – Set this to False to just include the specified directory but not any children.

Returns

A list of strings with the relative paths found, e.g. ["mysubdir/myfile.txt", "mysubdir/mysubsubdir/"].

New in version 2.2.

logFileContents(path, includes=None, excludes=None, maxLines=20, tail=False, encoding=None, logFunction=None, reFlags=0, stripWhitespace=True, mappers=[], color=True, message=None)

Logs some or all of the lines from the specified file.

If the file does not exist or cannot be opened, does nothing. The method is useful for providing key diagnostic information (e.g. error messages from tools executed by the test) directly in run.log, or to make test failures easier to triage quickly.

Parameters

• path (str) – May be an absolute, or relative to the test output directory.

• includes (list[str]) – Optional list of regex strings. If specified, only matches of these regexes will be logged.

• excludes (list[str]) –

  Optional list of regex strings. If specified, no line containing these will be logged.

  The variable self.logFileContentsDefaultExcludes (= [] by default) is used when this method is called with the default argument of excludes=None, and can be used to provide a global set of default exclusion lines shared by all your tests, which is particularly useful if some processes always log some unimportant text to stderr (or stdout) that would be distracting to log out.

  Added in PySys 1.6.0.

• mappers (List[callable[str]->str]) –

  A list of filter functions that will be used to pre-process each line from the file (returning None if the line is to be filtered out). This provides a very powerful capability for filtering the file, for example pysys.mappers.IncludeLinesBetween provides the ability to filter in/out sections of a file.

  Do not share mapper instances across multiple tests or threads as this can cause race conditions.

  Added in PySys 2.0.

• maxLines (int) – Upper limit on the number of lines from the file that will be logged. Set to zero for unlimited

• tail (bool) – Prints the _last_ ‘maxLines’ in the file rather than the first ‘maxLines’.

• encoding (str) –

  The encoding to use to open the file. The default value is None which indicates that the decision will be delegated to the getDefaultFileEncoding() method.

  Any character encoding errors will result in backslash escape sequences being logged instead.

• logFunction (Callable[[line],None]) –

  The function that will be used to log individual lines from the file. Usually this is self.log.info(u'  %s', line, extra=BaseLogFormatter.tag(LOG_FILE_CONTENTS)) but a custom implementation can be provided, for example to provide a different color using pysys.utils.logutils.BaseLogFormatter.tag.

  Added in PySys 1.5.1.

• reFlags (int) –

  Zero or more flags controlling how the behaviour of regular expression matching, combined together using the | operator, for example reFlags=re.VERBOSE | re.IGNORECASE.

  For details see the re module in the Python standard library. Note that re.MULTILINE cannot be used because expressions are matched against one line at a time. Added in PySys 1.5.1.

• stripWhitespace (bool) – By default blank lines are removed; set this to False to disable that behaviour. Added in PySys 2.0.

• color (bool) – By default logged lines are colored blue to distinguish from the rest of the log contents. Set this to False to disable coloring. Added in PySys 2.1.

• message (str) – The introductory message to log before the file content, with @PATH@ as a placeholder for the path. If not specified the default is equivalent to "Contents of @PATH@: ". Added in PySys 2.2

Returns

True if anything was logged, False if not.

mkdir(path)

Create a directory, with recursive creation of any parent directories.

This function does nothing (does not raise an except) if the directory already exists.

Parameters

path – The path to be created. This can be an absolute path or relative to the testcase output directory.

Returns

the absolute path of the new directory, to facilitate fluent-style method calling.

pollWait(secs)

Sleeps the current thread for the specified number of seconds, between polling/checking for some condition to be met.

Unlike pysys.basetest.BaseTest.wait (which should be used for larger waits inside tests), pollWait does not log anything.

Use this method instead of time.sleep as it provides PySys the chance to abort test execution early when requested, for example as a result of a keyboard interrupt or signal.

Parameters

secs (float) – The time to sleep for, typically a few hundred milliseconds. Do not use this method for really long waits. Cannot be negative.

signalProcess(process, signal, abortOnError=None)

Send a signal to a running process.

This method uses the pysys.process.Process.signal method to send a signal to a running process.

Should the request to send the signal to the running process fail, a BLOCKED outcome will be added to the outcome list.

Parameters

• process – The process handle returned from the startProcess method

• signal – The integer value of the signal to send

• abortOnError – If True aborts the test with an exception on any error, if False just log it as a warning. (defaults to the defaultAbortOnError project setting)

skipTest(outcomeReason, callRecord=None)

Raise an AbortException that will set the test outcome to SKIPPED and ensure that the rest of the execute() and validate() methods do not execute.

This is useful when a test should not be executed in the current mode or platform.

Parameters

outcomeReason – A string summarizing the reason the test is being skipped, for example “Feature X is not supported on Windows”.

startProcess(command, arguments, environs=None, workingDir=None, state=None, timeout=600, stdout=None, stderr=None, displayName=None, abortOnError=None, expectedExitStatus='==0', ignoreExitStatus=None, onError=None, quiet=False, stdouterr=None, background=False, info={}, processFactory=None)

Start a process running in the foreground or background, and return the pysys.process.Process object.

Typical use is:

myexecutable = self.startProcess('path_to_my_executable', 
        arguments=['myoperation', 'arg1','arg2'],
        environs=self.createEnvirons(addToLibPath=['my_ld_lib_path']), # if a customized environment is needed
        stdouterr=self.allocateUniqueStdOutErr('myoperation'), # for stdout/err files, pick a suitable logical name for what it's doing
        background=True # or remove for default behaviour of executing in foreground
        )

The method allows spawning of new processes in a platform independent way. The command, arguments, environment and working directory to run the process in can all be specified in the arguments to the method, along with the filenames used for capturing the stdout and stderr of the process. Processes may be started in the foreground, in which case the method does not return until the process has completed or a time out occurs, or in the background in which case the method returns immediately to the caller returning a handle to the process to allow manipulation at a later stage, typically with waitProcess.

All processes started in the background are automatically killed on completion of the test via the cleanup() destructor. To wait for background processes to complete during execute and/or before verify commences, call waitForBackgroundProcesses(). This is especially useful when you have a test that needs to execute lots of processes asynchronously, since having them all execute concurrently in the background and then calling waitForBackgroundProcesses() will be a lot quicker than executing them serially in the foreground.

When starting a process that will listen on a server socket, use getNextAvailableTCPPort to allocate a free port before calling this method.

Note that although is is possible to use this command to execute OS shell commands, that should only used for testing of shell scripts - other logic such as file system operations can be executed more easily and robustly using built-in Python (os module) or PySys (e.g. BaseTest.copy) functions.

Changed in version 1.6.0: Added onError parameter and default behaviour of logging stderr/out when there’s a failure. Added info parameter.

Changed in version 2.0: Added processFactory parameter.

Parameters

• command (str) – The path to the executable to be launched (should include the full path)

• arguments (list[str]) – A list of arguments to pass to the command. Any non-string values in the list are converted to strings automatically.

• environs (dict(str,str)) – A dictionary specifying the environment to run the process in. If a None or empty dictionary is passed, getDefaultEnvirons will be invoked to produce a suitable clean default environment for this command, containing a minimal set of variables. If you wish to specify a customized environment, createEnvirons() is a great way to create it.

• workingDir (str) – The working directory for the process to run in (defaults to the testcase output subdirectory)

• background (bool) – Set to True to start the process in the background. By default processes are started in the foreground, meaning execution of the test will continue only once the process has terminated.

• state – Alternative way to set background=True. Run the process either in the FOREGROUND or BACKGROUND (defaults to FOREGROUND). Setting state=BACKGROUND is equivalent to setting background=True; in new tests using background=True is the preferred way to do this.

• timeout (int) – The number of seconds after which to terminate processes running in the foreground. For processes that complete in a few seconds or less, it is best to avoid overriding this and stick with the default. However for long-running foreground processes it will be necessary to set a larger number, for example if running a soak test where the process needs to run for up to 2 hours you could set timeout=2*60*60.

• stdouterr (str) – The filename prefix to use for the stdout and stderr of the process (out/err will be appended), or a tuple of (stdout,stderr) as returned from allocateUniqueStdOutErr. The stdouterr prefix is also used to form a default display name for the process if none is explicitly provided. The files are created relative to the test output directory. The filenames can be accessed from the returned process object using .stdout/err from pysys.process.Process.

• stdout (str) – The filename used to capture the stdout of the process. It is usually simpler to use stdouterr instead of this.

• stderr (str) – The filename used to capture the stderr of the process. It is usually simpler to use stdouterr instead of this.

• displayName (str) – Logical name of the process used for display in log messages, and the str(…) representation of the returned process object (defaults to a string generated from the stdouterr and/or the command).

• abortOnError (bool) – If true abort the test on any error outcome (defaults to the defaultAbortOnError project setting)

• expectedExitStatus (str) – The condition string used to determine whether the exit status/code returned by the process is correct. The default is ‘==0’, as an exit code of zero usually indicates success, but if you are expecting a non-zero exit status (for example because you are testing correct handling of a failure condition) this could be set to ‘!=0’ or a specific value such as ‘==5’.

• ignoreExitStatus (bool) –

  If False, a BLOCKED outcome is added if the process terminates with an exit code that doesn’t match expectedExitStatus (or if the command cannot be run at all). This can be set to True in cases where you do not care whether the command succeeds or fails, or wish to handle the exit status separately with more complicated logic.

  The default value of ignoreExitStatus=None means the value will be taken from the project property defaultIgnoreExitStatus, which can be configured in the project XML (the recommended default property value is defaultIgnoreExitStatus=False), or is set to True for compatibility with older PySys releases if no project property is set.

• onError (Callable[pysys.process.Process]->str) –

  A function that will be called if the process times out or returns an unexpected exit status (unless ignoreExitStatus=True), before any abort exception is raised. This provides a convenient place to add logging of diagnostic information (perhaps using the stdout/err of the process) and/or extracting and returning an error message from the output, for example: onError=lambda process: self.logFileContents(process.stderr, tail=True) or self.logFileContents(process.stdout, tail=True).

  If a string value is returned from it will be added to the failure reason, e.g. onError=lambda process: self.logFileContents(process.stderr, tail=True) and self.grepOrNone(process.stderr, 'Error: (.*)').

  If no onError function is specified, the default is to log the last few lines of stderr (or if empty, stdout) when a process fails and abortOnError=True.

  The self.logFileContentsDefaultExcludes variable can be used to add regular expressions to exclude unimportant lines of output such as standard startup lines (see logFileContents).

• quiet (bool) – If True, this method will not do any INFO or WARN level logging (only DEBUG level), unless a failure outcome is appended. This parameter can be useful to avoid filling up the log where it is necessary to repeatedly execute a command check for completion of some operation until it succeeds; in such cases you should usually set ignoreExitStatus=True as well since both success and failure exit statuses are valid.

• info (dict[str,obj]) – A dictionary of user-defined information about this process that will be set as a field on the returned Process instance. This is useful for keeping track of things like server port numbers and log file paths.

• processFactory (callable[kwargs]) –

  A callable (such as a class constructor) that returns an instance or subclass of pysys.process.helper.ProcessImpl. This can be used either to provide a custom process subclass with extra features, or to make modifications to the arguments or environment that were specified by the code that invoked startProcess().

  The signature must consist of a **kwargs parameter, the members of which will be populated by the parameters listed in the constructor of pysys.process.Process, and can be modified by the factory. For example:

  def myProcessFactory(**kwargs):
          kwargs['arguments'] = kwargs['arguments'][0]+['my_extra_arg']+kwargs['arguments'][1:]
          return pysys.process.helper.ProcessImpl(**kwargs)
  

Returns

The process object.

Return type

pysys.process.Process

startPython(arguments, disableCoverage=False, **kwargs)

Start a Python process with the specified arguments.

Uses the same Python process the tests are running under.

If PySys was run with the argument -XcodeCoverage or -XpythonCoverage then startPython will add the necessary arguments to enable generation of code coverage. Note that this required the coverage.py library to be installed. If a project property called pythonCoverageArgs exists then its value will be added as (space-delimited) arguments to the coverage tool.

Parameters

• arguments – The arguments to pass to the Python executable. Typically the first one be either the name of a Python script to execute, or -m followed by a module name.

• kwargs – See startProcess for detail on available arguments.

• disableCoverage – Disables code coverage for this specific process. Coverage can also be disabled by setting self.disableCoverage==True on this test instance.

Returns

The process handle of the process.

Return type

pysys.process.Process

stopProcess(process, abortOnError=None)

Stops the specified process, if it is currently running.

Does nothing if the process is not running.

This is equivalent to calling pysys.process.Process.stop(), except it also logs an info message when the process is stopped.

Parameters

• process – The process handle returned from the startProcess method

• abortOnError – If True abort the test on any error outcome (defaults to the defaultAbortOnError project setting), if False a failure to stop the process will just be logged as a warning.

unpackArchive(archive, dest=None, autoCleanup=True)

Unpacks the specified file(s) from an archive to a directory. Supports archive format such as zip/tar.gz/gz/tar.xz/xz.

It is a good idea to store large textual Input/ assets (such as log files, which usually compress very well) as compressed archives to reduce disk space in your version control system.

By default this method will automatically delete the extracted file/dir during test cleanup so it doesn’t sit around on disk (or in CI uploaded failure archives) consuming space; if the file/dir is mutated by the test you may wish to disable this so you can manually inspect them by setting autoCleanup=False.

For example:

unpacked = self.unpackArchive('mybigfile.log.xz')
# do something with "unpacked"...

Note that .xz (for single files) and .tar.xz (for multiple files) are recommended for optimal compression, and these (and .gz) are significantly better than zip, which performs poorly when compressing multiple similar text files into one archive. Don’t use more than one single archive per testcase (if possible) to ensure you benefit from similarities between the various files.

Files are decompressed in binary mode, so if you require platform-native line endings you should use copy to post-process them after decompressing.

Parameters

• archive (str) – The path of the archive to unpack, by default from the test input directory. Alternatively you could provide an absolute path using self.project.testRootDir or similar if an archive is shared across multiple test cases.

• dest (str) – The directory in which the contents of the archive will be written; by default this is the test output directory for archive types that are always single-file, and a subdirectory named after the archive if not. This dir will be created if needed.

• autoCleanup (bool) – Automatically deletes the unpackaged file/directory during test cleanup to save disk space (even if the test fails).

Returns

The full path to the decompressed file or directory.

waitForBackgroundProcesses(includes=[], excludes=[], timeout=600, abortOnError=None, checkExitStatus=True)

Wait for any running background processes to terminate, then check that all background processes completed with the expected exit status.

This can be useful for speeding up a test that needs to run many processes, by executing all its subprocesses in parallel in the background (and then waiting for completion) rather than one-by-one in the foreground.

Timeouts will result in a TIMEDOUT outcome and an exception unless the project property defaultAbortOnError==False is set.

Parameters

• includes (list[pysys.process.Process]) – A list of processes to wait for, each returned by startProcess(). If none are specified, this method will wait for (and check the exit status of) all background processes.

• excludes (list[pysys.process.Process]) – A list of processes which are not expected to have terminated yet (this is only useful when not setting includes=[]).

• timeout (int) – The total time in seconds to wait for all processes to have completed, for example timeout=TIMEOUTS['WaitForProcess'].

• abortOnError (bool) – If True aborts the test with an exception on any error, if False appends an outcome but does not raise an exception.

• checkExitStatus (bool) – By default this method not only waits for completion but also checks the exit status of all (included) background processes (regardless of when they completed), but set this argument to False to disable checking that the exit status of the processes matches the expectedExitStatus specified in the call to startProcess (typically ==0). The last few lines of the stderr (or stdout) will be logged if the exit status is wrong.

waitForFile(file, filedir=None, timeout=30, abortOnError=None)

Wait for a file to exist on disk.

This method blocks until a file is created on disk. This is useful for test timing where a component under test creates a file (e.g. for logging) indicating it has performed all initialisation actions and is ready for the test execution steps. If a file is not created on disk within the specified timeout interval, the method returns to the caller.

Parameters

• file – The basename of the file used to wait to be created

• filedir – The dirname of the file (defaults to the testcase output subdirectory)

• timeout – The timeout in seconds to wait for the file to be created

• abortOnError – If true abort the test on any failure (defaults to the defaultAbortOnError project setting)

waitForGrep(file, expr='', condition='>=1', timeout=60, poll=0.25, ignores=[], process=None, errorExpr=[], errorIf=None, abortOnError=None, encoding=None, encodingReplaceOnError=False, detailMessage='', filedir=None, reFlags=0, mappers=[], quiet=False)

Wait for a regular expression line to be seen (one or more times) in a text file in the output directory (waitForGrep was formerly known as waitForSignal).

This method provides some parameters that give helpful fail-fast behaviour with a descriptive outcome reason; use these whenever possible:

• process= to abort if success becomes impossible due to premature termination of the process that’s generating the output

• errorExpr= to abort if an error message/expression is written to the file being grepped

• errorIf= to abort if the specified lambda function returns an error string (which can be used if the error messages go to a different file than that being grepped

This will generate much clearer outcome reasons, which makes test failures easy to triage, and also avoids wasting time waiting for something that will never happen.

Example:

self.waitForGrep('myprocess.log', 'INFO .*Started successfully', encoding='utf-8',
        process=myprocess, errorExpr=[' (ERROR|FATAL) ', 'Failed to start'])

self.waitForGrep('myoutput.log', 'My message', encoding='utf-8',
        process=myprocess, errorIf=lambda: self.grepOrNone('myprocess.err', ' ERROR .*'))

Note that waitForGrep fails the test if the expression is not found (unless abortOnError was set to False, which isn’t recommended), so there is no need to add duplication with an assertGrep to check for the same expression in your validation logic.

You can extract information from the matched expression, optionally perform assertions on it, by using one or more (?P<groupName>...) named groups in the expression. A common pattern is to unpack the resulting dict using **kwargs syntax and pass to BaseTest.assertThat. For example:

self.assertThat('username == expected', expected='myuser',
        **self.waitForGrep('myserver.log', r'Successfully authenticated user "(?P<username>[^"]*)"'))

If the file is large or contains long lines, it can take a long time for the regular expressions to be evaluated over each line. It is important to avoid the possibility that data is written to the file faster than Python can read it, which would lead to the PySys process running very slowly and likely a timeout. To help detect this situation, this method will log warnings if dangerously long lines are detected. If needed the threshold for these can be configured by setting the self.grepWarnIfLineLongerThan field. If reading from a file that has long lines, consider adding pysys.mappers.TruncateLongLines to the mappers list.

New in version 1.5.1.

Parameters

• file (str) – The path of the file to be searched. Usually this is a name/path relative to the self.output directory, but alternatively an absolute path can be specified.

• expr (str) – The regular expression to search for in the text file.

• condition (str) – The condition to be met for the number of lines matching the regular expression; by default we wait until there is at least one occurrence.

• timeout (int) – The number of seconds to wait for the regular expression before giving up and aborting the test with pysys.constants.TIMEDOUT (unless abortOnError=False in which case execution will continue).

• process (pysys.process.Process) – The process that is generating the specified file, to allow the wait to fail fast (instead of timing out) if the process dies before the expected signal appears. Can be None if the process is not known or is expected to terminate itself during this period.

• errorExpr (list[str]) – Optional list of regular expressions, which if found in the file will cause waiting for the main expression to be aborted with a pysys.constants.BLOCKED outcome. This is useful to avoid waiting a long time for the expected expression when an ERROR is logged that means it will never happen, and also provides much clearer test failure messages in this case.

• errorIf (callable->str) –

  A zero-arg function that returns False/None when there is no error, or a non-empty string if an error is detected which should cause us to abort looking for the grep expression. This function will be executed frequently (every poll seconds) so avoid doing anything time-consuming here unless you set a large polling interval. See above for an example.

  Added in PySys 1.6.0.

• ignores (list[str]) – A list of regular expressions used to identify lines in the files which should be ignored when matching both expr and errorExpr.

• mappers (List[callable[str]->str]) –

  A list of filter functions that will be used to pre-process each line from the file (returning None if the line is to be filtered out). This provides a very powerful capability for filtering the file, for example pysys.mappers.IncludeLinesBetween provides the ability to filter in/out sections of a file, and pysys.mappers.JoinLines can be used to put exception stack traces onto the same line as the error message.

  Do not share mapper instances across multiple tests or threads as this can cause race conditions.

  Added in PySys 1.6.0.

• poll (float) – The time in seconds between to poll the file looking for the regular expression and to check against the condition

• abortOnError (bool) – If True abort the test on any error outcome (defaults to the defaultAbortOnError project setting, which for a modern project will be True).

• encoding (str) – The encoding to use to open the file and convert from bytes to characters. The default value is None which indicates that the decision will be delegated to the getDefaultFileEncoding() method.

• encodingReplaceOnError (bool) – Set to True to replace erroneous characters that are invalid in the expected encoding (with a backslash escape) rather than throwing an exception. Added in PySys 2.2.

• detailMessage (str) –

  An extra string to add to the start of the message logged when waiting to provide extra information about the wait condition. e.g. detailMessage='Wait for server startup: '.

  Added in v1.5.1. From v2.1+ the detail string is added at the beginning not the end.

• reFlags (int) –

  Zero or more flags controlling how the behaviour of regular expression matching, combined together using the | operator, for example reFlags=re.VERBOSE | re.IGNORECASE.

  For details see the re module in the Python standard library. Note that re.MULTILINE cannot be used because expressions are matched against one line at a time. Added in PySys 1.5.1.

• filedir (str) – Can be used to provide a directory name to add to the beginning of the file parameter; however usually it is clearer just to specify that directory in the file.

• quiet (bool) – Set this to true to suppress INFO-level logging, which can be useful when you wish to print your own progress message in a more high-level fashion.

Return list[re.Match]

Usually this returns a list of re.Match objects found for the expr, or an empty list if there was no match.

If the expr contains any (?P<groupName>...) named groups, and assuming the condition is still the default of “>=1” (i.e. not trying to find multiple matches), then instead of a list, a dict is returned containing dict(groupName: str, matchValue: str or None) (or an empty {} dict if there is no match) which allows the result to be passed to assertThat for further checking of the matched groups (typically unpacked using the ** operator; see example above).

waitForSignal(file, filedir=None, expr='', **waitForGrepArgs)

Old alias for waitForGrep; please use waitForGrep in new tests.

All parameters are the same, except that in waitForSignal the (rarely used) filedir argument can be specified as the 2nd positional argument (after file and before expr) whereas in waitForGrep it can only be specified as a filedir= keyword argument.

waitForSocket(port, host='localhost', timeout=60, abortOnError=None, process=None, socketAddressFamily=<AddressFamily.AF_INET: 2>)

Wait until it is possible to establish a socket connection to a server running on the specified local or remote port.

This method blocks until connection to a particular host:port pair can be established. This is useful for test timing where a component under test creates a socket for client server interaction - calling of this method ensures that on return of the method call the server process is running and a client is able to create connections to it. If a connection cannot be made within the specified timeout interval, the method returns to the caller, or aborts the test if abortOnError=True.

Changed in version 1.5.1: Added host and socketAddressFamily parameters.

Parameters

• port – The port value in the socket host:port pair

• host – The host value in the socket host:port pair

• timeout – The timeout in seconds to wait for connection to the socket

• abortOnError – If true abort the test on any failure (defaults to the defaultAbortOnError project setting)

• process – If a handle to a process is specified, the wait will abort if the process dies before the socket becomes available. It is recommended to set this wherever possible.

• socketAddressFamily – The socket address family e.g. IPv4 vs IPv6. See Python’s socket module for details.

waitProcess(process, timeout=600, abortOnError=None, checkExitStatus=False)

Wait for a background process to terminate.

For a convenient way to wait for all remaining background processes to complete, see waitForBackgroundProcesses.

Timeouts will result in an exception and TIMEDOUT outcome unless the project property defaultAbortOnError==False is set.

Parameters

• process (pysys.process.Process) – The process handle returned from the startProcess method

• timeout (int) – The timeout value in seconds to wait before returning, for example timeout=TIMEOUTS['WaitForProcess'].

• abortOnError (bool) – If True aborts the test with an exception on any error, if False just log it as a warning. (defaults to the defaultAbortOnError project setting)

• checkExitStatus (bool) – By default this method does not check the exit status, but set this argument to True to check that the exit status matches the expectedExitStatus specified in the call to startProcess (typically ==0). Note that this argument is not affected by the defaultIgnoreExitStatus project property. The last few lines of the stderr (or stdout) will be logged if the exit status is wrong.

Returns

The process’s exitStatus. This will be None if the process timed out and abortOnError is disabled.

writeProcess(process, data, addNewLine=True)

Write binary data to the stdin of a process.

This method uses pysys.process.Process.write to write binary data to the stdin of a process. This wrapper around the write method of the process helper only adds checking of the process running status prior to the write being performed, and logging to the testcase run log to detail the write.

Parameters

• process (pysys.process.Process) – The process handle returned from the startProcess() method

• data (bytes) – The data to write to the process stdin. As only binary data can be written to a process stdin, if a character string rather than a byte object is passed as the data, it will be automatically converted to a bytes object using the encoding given by PREFERRED_ENCODING.

• addNewLine (bool) – True if a new line character is to be added to the end of the data string

write_text(file, text, encoding=None)

Writes the specified characters to a file in the output directory.

Parameters

• file – The path of the file to write, either an absolute path or relative to the self.output directory.

• text –

  The string to write to the file, with \n for newlines (do not use os.linesep as the file will be opened in text mode so platform line separators will be added automatically).

  This must be a character string.

• encoding – The encoding to use to open the file. The default value is None which indicates that the decision will be delegated to the getDefaultFileEncoding() method.

Return str

Return the absolute path of the generated file.