pysys.writer.api

API for creating new writers.

Writers are responsible for summarising test results or processing test output as each test completes, or at the end when all tests has completed.

The most common type of writer is the standard ‘Record’ writer, but there are also ‘Progress’, and ‘Summary’ writers which do their things at different stages of the test run:

  • BaseRecordResultsWriter: Record writers record the outcome or output of tests, allowing runtime auditing of the test output, e.g. into text file, a database, or to the console in a format that can be read by your Continuous Integration (CI) tooling, or collection/archiving of test output files.

    Several record writers are distributed with the PySys framework, such as the pysys.writer.outcomes.JUnitXMLResultsWriter and pysys.writer.ci.GitHubActionsCIWriter. By default, record writers are enabled only when the --record flag is given to the PySys launcher, though some writers may enable/disable themselves under different conditions, by overriding the BaseResultsWriter.isEnabled method.

  • BaseProgressResultsWriter: Progress writers output a summary of the test progress after completion of each test, to give an indication of how far and how well the run is progressing. A single implementation of a progress writer is distributed with the PySys framework, namely the pysys.writer.console.ConsoleProgressResultsWriter, which details the percentage of tests selected to be run and that have executed, and a summary of the recent test failures. Progress writers should extend the BaseProgressResultsWriter and are enabled when the --progress flag is given to the PySys launcher, or when PYSYS_PROGRESS=true is set in the local environment.

  • BaseSummaryResultsWriter: Summary writers output an overall summary of the status at the end of a test run. A single implementation of a summary writer is distributed with the PySys framework, namely the pysys.writer.console.ConsoleSummaryResultsWriter, which details the overall test run outcome and lists any tests that did not pass. Summary writers are always enabled regardless of the flags given to the PySys launcher.

(See also pysys.perf.api whcih is used for writing performance results, using a similar but slightly different API).

Project configuration of the writers is through the PySys project XML file using the <writer> tag. Multiple writers may be configured and their individual properties set through the nested <property> tag. Writer properties are set as attributes to the writer instance just before setup is called, with automatic conversion of type to match the default value if specified as a static attribute on the class.

The writers are instantiated and invoked by the pysys.baserunner.BaseRunner class instance. This calls the class constructors of all configured test writers, and then the setup (prior to executing the set of tests), processResult (process a test result), and cleanup (upon completion of the execution of all tests). The **kwargs method parameter should always be included in the signature of each method, to allow for future additions to PySys without breaking existing writer implementations.

Writers that generate output files/directories should by default put that output under either the runner.output directory, or (for increased prominence) the runner.output+'/..' directory (which is typically testRootDir unless an absolute --outdir path was provided) . A prefix of double underscore __pysys is recommended to distinguish dynamically created directories (ignored by version control) from the testcase directories (checked into version control).

Writer authors may wish to make use of these helpers:

replaceIllegalXMLCharacters(unicodeString[, …])

Utility function that takes a unicode character string and replaces all characters that are not permitted to appear in an XML document.

pysys.utils.logutils.stripANSIEscapeCodes(text)

Remove any ANSI escape sequences (for example to set console color) present in the specified string.

pysys.utils.logutils.stdoutPrint(s)

Writes the specified bytes or (preferably) unicode character string to stdout, avoiding any redirection to loggers performed by PySys, and performing replacements if needed based on the characters supported by the stdout encoding, and with support for output coloring using escape sequences (if enabled in PySys).

pysys.utils.logutils.ColorLogFormatter(…)

Formatter supporting colored output to a console.

BaseResultsWriter

class pysys.writer.api.BaseResultsWriter(logfile=None, **kwargs)[source]

Bases: object

Base class for all writers that get notified as and when test results are available.

Writer can additionally subclass ArtifactPublisher to be notified of artifacts produced by other writers that they wish to publish, or TestOutputVisitor to be notified of each file in the test output directory. If you are implementing a writer that needs a textual summary of the test outcomes, you can add TestOutcomeSummaryGenerator as a superclass to get this functionality.

Parameters

  • logfile (str) – Optional configuration property specifying a file to store output in. Does not apply to all writers, can be ignored if not needed.

  • kwargs – Additional keyword arguments may be added in a future release.

isEnabled(record=False, **kwargs)[source]

Determines whether this writer can be used in the current environment.

If set to False then after construction none of the other methods (including setup)) will be called.

Parameters

record – True if the user ran PySys with the --record flag, indicating that test results should be recorded.

Returns

For record writers, the default to enable only if record==True, but individual writers can use different criteria if desired, e.g. writers for logging output to a CI system may enable themselves based on environment variables indicating that system is present, even if record is not specified explicitly.

setup(numTests=0, cycles=1, xargs=None, threads=0, testoutdir='', runner=None, **kwargs)[source]

Called before any tests begin.

Before this method is called, for each property “PROP” specified for this writer in the project configuration file, the configured value will be assigned to self.PROP.

Parameters

  • numTests – The total number of tests (cycles*testids) to be executed

  • cycles – The number of cycles.

  • xargs – The runner’s xargs

  • threads – The number of threads used for running tests.

  • testoutdir – The output directory used for this test run (equal to runner.outsubdir), an identifying string which often contains the platform, or when there are multiple test runs on the same machine may be used to distinguish between them. This is usually a relative path but may be an absolute path.

  • runner – The runner instance that owns this writer. The default implementation of this methods sets the self.runner attribute to this value.

  • kwargs – Additional keyword arguments may be added in a future release.

cleanup(**kwargs)[source]

Called after all tests have finished executing (or been cancelled).

This is where file headers can be written, and open handles should be closed.

Parameters

kwargs – Additional keyword arguments may be added in a future release.

processResult(testObj, cycle=0, testTime=0, testStart=0, runLogOutput='', **kwargs)[source]

Called when each test has completed.

This method is always invoked under a lock that prevents multiple concurrent invocations so additional locking is not usually necessary.

Parameters

  • testObj (pysys.basetest.BaseTest) – Reference to an instance of a pysys.basetest.BaseTest class. The writer can extract data from this object but should not store a reference to it. The testObj.descriptor.id indicates the test that ran.

  • cycle (int) – The cycle number. These start from 0, so please add 1 to this value before using.

  • testTime (float) – Duration of the test in seconds as a floating point number.

  • testStart (float) – The time when the test started.

  • runLogOutput (str) – The logging output written to the console/run.log, as a unicode character string. This string will include ANSI escape codes if colored output is enabled; if desired these can be removed using pysys.utils.logutils.stripANSIEscapeCodes().

  • kwargs – Additional keyword arguments may be added in future releases.

processTestStarting(testObj, cycle=- 1, **kwargs)[source]

Called when a test is just about to begin executing.

Note on thread-safety: unlike the other methods on this interface, this is usually executed on a worker thread, so any data structures accessed in this method and others on this class must be synchronized if performing non-atomic operations.

Parameters

  • testObj – Reference to an instance of a pysys.basetest.BaseTest class. The writer can extract data from this object but should not store a reference to it. The testObj.descriptor.id indicates the test that ran.

  • cycle – The cycle number. These start from 0, so please add 1 to this value before using.

  • kwargs – Additional keyword arguments may be added in a future release.

BaseRecordResultsWriter

class pysys.writer.api.BaseRecordResultsWriter(logfile=None, **kwargs)[source]

Bases: pysys.writer.api.BaseResultsWriter

Base class for writers that record the results of tests, and are enabled only when the --record flag is specified.

For compatibility reasons writers that do not subclass BaseSummaryResultsWriter or BaseProgressResultsWriter are treated as “record” writers even if they do not inherit from this class.

BaseSummaryResultsWriter

class pysys.writer.api.BaseSummaryResultsWriter(logfile=None, **kwargs)[source]

Bases: pysys.writer.api.BaseResultsWriter

Base class for writers that display a summary of test results.

Summary writers are always enabled (regardless of whether --progress or --record are specified). If no “summary” writers are configured, a default ConsoleSummaryResultsWriter instance will be added automatically.

Summary writers are invoked after all other writers, ensuring that their output will be displayed after output from any other writer types.

BaseProgressResultsWriter

class pysys.writer.api.BaseProgressResultsWriter(logfile=None, **kwargs)[source]

Bases: pysys.writer.api.BaseResultsWriter

Base class for writers that display progress information while tests are running.

Progress writers are only enabled if the --progress flag is specified.

ArtifactPublisher

class pysys.writer.api.ArtifactPublisher[source]

Bases: object

Interface implemented by writers that implement publishing of file/directory artifacts.

For example, a writer for a CI provider that supports artifact uploading can subclass this interface to be notified when another writer (or performance reporter) produces an artifact. If implementing this interface, remember that the order each writer’s cleanup() is called is the same as the order the writers appear in the project configuration file, so if your writer relies on output published from another’s cleanup you may need to document this, or code your writer such that it doesn’t care what order the cleanup() methods are called.

To publish an artifact to all registered writers, call pysys.baserunner.BaseRunner.publishArtifact().

It is possible to restrict artifact publishing to just the categories you care about by setting the project property publishArtifactCategoryIncludeRegex which (if specified) must match the category name in order for writers to be notified.

New in version 1.6.0.

publishArtifact(path, category, **kwargs)[source]

Called when a file or directory artifact has been written and is ready to be published (e.g. by another writer).

Parameters

  • path (str) – Absolute path of the file or directory, using forward slashes as the path separator.

  • category (str) – A string identifying what kind of artifact this is, e.g. “TestOutputArchive” and “TestOutputArchiveDir” (from pysys.writer.testoutput.TestOutputArchiveWriter) or “CSVPerformanceReport” (from pysys.perf.reporters.CSVPerformanceReporter). If you create your own category, be sure to add an org/company name prefix to avoid clashes.

TestOutputVisitor

class pysys.writer.api.TestOutputVisitor[source]

Bases: object

Interface implemented by writers that wish to be notified of each file in the test output directory.

Implementing this interface is a lot more efficient than explicitly walking the directory tree. Note that in the interests of performance empty (zero byte) files are ignored.

New in version 1.6.0.

visitTestOutputFile(testObj, path, **kwargs)[source]

Called after execution of each test (and before purging of files) for each file found in the output directory.

Parameters

  • testObj (pysys.basetest.BaseTest) – The test object, which can be used to find the outcome etc.

  • str – The absolute, normalize path to the output file (will be a \?long path safe path on windows).

Return bool

Return True if this visitor has handled this file fully (e.g. by deleting it) and it should not be passed on to any other registered visitors.

TestOutcomeSummaryGenerator

class pysys.writer.api.TestOutcomeSummaryGenerator(logfile=None, **kwargs)[source]

Bases: pysys.writer.api.BaseResultsWriter

Mix-in helper class that can be inherited by any writer to allow (configurable) generation of a textual summary of the test outcomes.

If subclasses provide their own implementation of setup and processResult they must ensure this class’s methods of those names are also called. Then the summary can be obtained from logSummary or getSummaryText, typically in the writer’s cleanup method.

showOutcomeReason = True

Configures whether the summary includes the reason for each failure.

showOutputDir = True

Configures whether the summary includes the path to the output directory for each failure. This is relative to the current working directory unless project property pysysLogAbsolutePaths is True.

showTestDir = True

Configures whether the summary includes the path to the test directory for each failure, unless the output dir is displayed and the test dir is a parent of it. This is useful if you run tests with an absolute –outdir. This path is logged relative to the current working directory unless project property pysysLogAbsolutePaths is True.

showTestTitle = False

Configures whether the summary includes the test title for each failure.

showOutcomeStats = True

Configures whether the summary includes a count of the number of each outcomes.

showDuration = False

Configures whether the summary includes the total duration of all tests.

showRunDetails = False

Configures whether the summary includes the runDetails from the pysys.baserunner.BaseRunner, such as outDirName and hostname.

showInspectSummary = True

Configures whether the summary includes a summary of INSPECT outcomes (if any).

showNotVerifiedSummary = True

Configures whether the summary includes a summary of NOTVERIFIED outcomes (if any).

showTestIdList = False

Configures whether the summary includes a short list of the failing test ids in a form that’s easy to paste onto the command line to re-run the failed tests.

getSummaryText(**kwargs)[source]

Get the textual summary as a single string (with no coloring).

To customize what is included in the summary (rather than letting it be user-configurable), use the keyword arguments as for logSummary.

Return str

The summary as a string.

logSummary(log, showDuration=None, showOutcomeStats=None, showTestIdList=None, showFailureSummary=True, showRunDetails=None, **kwargs)[source]

Writes a textual summary using the specified log function, with colored output if enabled.

The keyword arguments can be used to disable sections of the output (overriding the settings) if needed by the caller.

Parameters

log (Callable[format,args,kwargs=]) – The function to call for each line of the summary (e.g. log.critical). The message is obtained with format % args, and color information is available from the extra= keyword argument.

replaceIllegalXMLCharacters

pysys.writer.api.replaceIllegalXMLCharacters(unicodeString, replaceWith='?')[source]

Utility function that takes a unicode character string and replaces all characters that are not permitted to appear in an XML document.

The XML specification says Char ::= #x9 | #xA | #xD | [#x20-#xD7FF] | [#xE000-#xFFFD] | [#x10000-#x10FFFF]

Currently Python’s XML libraries do not perform checking or removal of such characters, so if this function is not used then they may generate XML documents that no XML API (including Python’s) can read.

See https://bugs.python.org/issue5166

Parameters

  • unicodeString – a unicode character string (not a byte string). Since most XML documents are encoded in utf-8, typical usage would be to decode the UTF-8 bytes into characters before calling this function and then re-encode as UTF-8 again afterwards.

  • replaceWith – the unicode character string to replace each illegal character with.