pysys.writer.testoutput

Writers that process the contents of test output directories by archiving or collecting output files.

TestOutputArchiveWriter

class pysys.writer.testoutput.TestOutputArchiveWriter(logfile=None, **kwargs)

Bases: pysys.writer.api.BaseRecordResultsWriter

Writer that creates zip of tar.gz/xz archives of each failed test’s output directory, producing artifacts that could be uploaded to a CI system or file share to allow the failures to be analysed.

This writer is enabled when running with --record. If using this writer in conjunction with a CI writer that publishes the generated archives, be sure to include this writer first in the list of writers in your project configuration.

Note that the zip file format typically generates much larger files than tar.xz (and tar.gz) so use the latter format where possible.

Publishes artifacts with category name “TestOutputArchive” and the directory (unless there are no archives) as “TestOutputArchiveDir” for any enabled pysys.writer.api.ArtifactPublisher writers.

New in version 1.6.0.

The following properties can be set in the project configuration for this writer:

destDir = '__pysys_output_archives.${outDirName}/'

The directory to write the archives to, as an absolute path, or relative to the testRootDir (or –outdir if specified).

This directory will be deleted at the start of the run if it already exists.

Project ${...} properties can be used in the path.

format = 'zip'

The archive type. Supported types are zip, tar.gz and tar.xz. The latter are often significantly smaller than zip files due to cross-file compression.

New in version 2.2.

maxTotalSizeMB = 1024.0

The (approximate) limit on the total size of all archives.

maxArchiveSizeMB = 200.0

The (approximate) limit on the size each individual test zip file, or of the total uncompressed size of the files if making a tar.* file.

maxArchives = 50

The maximum number of archives to create.

archiveAtEndOfRun = True

By default all archives are created at the end of the run once all tests have finished executing. This avoids I/O contention with execution of tests, and also selection of the tests to generated archives to be done in a deterministic (but pseudo-random) fashion rather than just taking the first N failures.

Alternatively you can this property to false if you wish to create archives during the test run as each failure occurs.

includeNonFailureOutcomes = 'REQUIRES INSPECTION'

In addition to failure outcomes, any outcomes listed here (as comma-separated display names) will be archived.

fileExcludesRegex = ''

A regular expression indicating test output paths that will be excluded from archiving, for example large temporary files that are not useful for diagnosing problems.

For example ".*/MyTest_001/.*/mybigfile.*[.]tmp".

The expression is matched against the path of each output file relative to the test root dir, using forward slashes as the path separator. Multiple paths can be specified using “(path1|path2)” syntax.

fileIncludesRegex = ''

A regular expression indicating test output paths that will be included in the archive. This can be used to archive just some particular files. Note that for use cases such as collecting graphs and code coverage files generated by a test run, the collect-test-output feature is usually a better fit than using this writer.

The expression is matched against the path of each output file relative to the test root dir, using forward slashes as the path separator. Multiple paths can be specified using “(path1|path2)” syntax.

shouldArchive(testObj, **kwargs)

Decides whether this test is eligible for archiving of its output.

The default implementation archives only tests that have a failure outcome, or are listed in includeNonFailureOutcomes, but this can be customized if needed by subclasses.

Parameters

testObj (pysys.basetest.BaseTest) – The test object under consideration.

Return bool

True if this test’s output can be archived.

_newArchive(id, **kwargs)

Creates and opens a new archive file for the specified id.

Returns

(str path, filehandle) The path will include an appropriate extension for this archive type. The filehandle must have the same API as Python’s ZipFile class.

_archiveTestOutputDir(id, outputDir, **kwargs)

Creates an archive for the specified test, unless doing so would violate the configured limits (e.g. maxArchives).

Parameters

• id (str) – The testId (plus a cycle suffix if it’s a multi-cycle run).

• outputDir (str) – The path of the test output dir.

CollectTestOutputWriter

class pysys.writer.testoutput.CollectTestOutputWriter(logfile=None, **kwargs)

Bases: pysys.writer.api.BaseRecordResultsWriter, pysys.writer.api.TestOutputVisitor

Writer that collects files matching a specified pattern from the output directory after each test, and puts them in a single directory or archive - for example code coverage files or performance graphs.

This writer can be used as-is or as a base class for writers that need to collect files during test execution then do something with them during cleanup, for example generate a code coverage report.

Empty files are ignored.

This writer is always enabled.

New in version 1.6.0.

The following properties can be set in the project configuration for this writer:

destDir = ''

The directory in which the files will be collected, as an absolute path, or relative to the testRootDir (or –outdir if specified).

This directory will be deleted at the start of the run if it already exists.

Project ${...} properties can be used in the path.

destArchive = ''

Optional filename of a .zip archive to generate with the contents of the destDir.

If a non-absolute path is specified it is evaluated relative to the destDir.

Project ${...} properties can be used in the path.

includeTestIf = ''

A Python lambda that will be evaluated at the end of a test to determine whether output from a given test should be collected.

For example code coverage collectors built on this class can include only unit tests or only tests that run in pull requests/CI (to ensure a stable baseline for coverage comparisons). This is useful if you wish to have multiple coverage writers to generate separate coverage reports for all correctness/integration tests versus seeing the coverage achieved in your unit tests (or a small set of smoke tests used in pull requests).

Note that this option only disables the collection/aggregation it does not do anything to actually disable the generation of the files, so do not use it for excluding performance/soak/reliability tests from code coverage. For that purpose set the disableCoverage group on the relevant tests (possibly using pysysdirconfig.xml at the directory level) or the set self.disableCoverage = True on the test object, which will prevent any coverage-related slowdown in the test execution.

For example:

<property name="includeTestIf">lambda testObj: 
        'unitTest' in testObj.descriptor.groups
        or testObj.project.getProperty('isLocalDeveloperTestRun',False)
</property>

The expression is evaluated using the pysys.utils.safeeval.safeEval function.

New in version 2.2.

fileIncludesRegex = ''

A regular expression indicating the test output paths that will be collected. This can be used to archive just some particular files. This is required.

The expression is matched against the final characters of each output file’s path (with the test root dir stripped off), using forward slashes as the path separator. Multiple paths can be specified using “(path1|path2)” syntax.

fileExcludesRegex = ''

A regular expression indicating test output paths that will be excluded from collection.

For example ".*/MyTest_001/.*/mybigfile.*[.]tmp".

The expression is matched against the path as described for fileIncludesRegex.

outputPattern = '@TESTID@.@FILENAME@.@UNIQUE@.@FILENAME_EXT@'

A string indicating the file (and optionally subdirectory name) to use when writing each collected file to the destDir.

In addition to any standard ${...} property variables from the project configuration, the output pattern can contain these @...@ substitutions:

• @FILENAME@ is the original base filename with directory and extension removed, to which you can add prefixes or suffixes as desired.

• .@FILENAME_EXT@ is the filename extension, such that the original filename is @FILENAME@.@FILENAME_EXT@ (note the dot prefix is mandatory here, and will be replaced with empty string is there is no extension).

• @TESTID@ is replaced by the identifier of the test that generated the output file (including mode suffix if present), which may be useful for tracking where each one came from.

• @UNIQUE@ is replaced by a number that ensures the file does not clash with any other collected output file from another test. The @UNIQUE@ substitution variable is mandatory.

publishArtifactDirCategory = ''

If specified, the output directory will be published as an artifact using the specified category name, e.g. MyCodeCoverageDir.

publishArtifactArchiveCategory = ''

If specified the destArchive file (if any) will be published as an artifact using the specified category name.

archiveAndPublish()

Generate an archive of the destDir (if configured) and publish artifacts (if configured).

Called by default as part of cleanup().