pysys.process.monitor

Contains the process monitoring API used by pysys.basetest.BaseTest.startProcessMonitor.

The main components are the pysys.process.monitor.BaseProcessMonitor class, ProcessMonitorKey constants for identifying columns and the default ProcessMonitorTextFileHandler class for writing monitoring information to a file.

ProcessMonitorKey

class pysys.process.monitor.ProcessMonitorKey

Bases: object

Contains constants for supported process monitor keys.

Some of these keys are not currently returned on all platforms.

These constants provide the display names used for column headings etc.

Usually CPU_CORE_UTILIZATION is the best key for measuring CPU utilization and MEMORY_RESIDENT_KB is the most useful way to monitor memory usage.

CPU_CORE_UTILIZATION = 'CPU core %'

CPU utilization % scaled by the number of cores so that 100% indicates full use of one core, 200% indicates full use of two cores, etc.

The maximum value is 100 multiplied by the number of CPU cores (as given by multiprocessing.cpu_count()).

The type is int.

CPU_TOTAL_UTILIZATION = 'CPU total %'

Total utilization % of all available CPU cores, with a maximum value of 100%.

If you have 2 cores and one of them is 50% utilized, the value would be 25%.

The type is int.

MEMORY_RESIDENT_KB = 'Resident memory kB'

Resident / working set memory usage.

This is the non-swapped physical memory, and is usually a good statistic to use for checking for memory leaks and excessive memory usage.

On Unix it is equivalent to rss/RES and on Windows to the working set memory usage.

The type is int.

MEMORY_VIRTUAL_KB = 'Virtual memory kB'

Virtual memory of the process including memory that is swapped out.

On Unix it is equivalent to vsz/VIRT, and on Windows to the current page file usage.

The type is int.

DATE_TIME = 'Time'

String representation of the date and local time for this sample in yyyy-mm-dd HH:MM:SS format.

DATE_TIME_LEGACY = 'Time (legacy)'

String representation of the date and local time for this sample in a format compatible with v1.3.0 and earlier versions of PySys.

This is %d/%m/%y %H:%M:%S on Windows and %m/%d/%y %H:%M:%S on Unix.

Deprecated

Use DATE_TIME if possible.

SAMPLE = 'Sample'

A counter starting from 1 and incrementing with each new set of sample data.

The type is int.

BaseProcessMonitorHandler

class pysys.process.monitor.BaseProcessMonitorHandler

Bases: object

Interface to be implemented to provide a custom handler that records or processes data from a BaseProcessMonitor.

handleData(data, **kwargs)

Called on a background thread each time a new sample of monitoring data is available.

Parameters

• data – a dictionary whose keys are from ProcessMonitorKey (or any new keys added by custom process monitor implementations). The dictionary values are of type int, float or string.

• kwargs – Reserved for future use.

cleanup()

Called on a background thread to perform cleanup for this handler, for example closing file handles.

ProcessMonitorTextFileHandler

class pysys.process.monitor.ProcessMonitorTextFileHandler(file, columns=None, delimiter=None, writeHeaderLine=None)

Bases: pysys.process.monitor.BaseProcessMonitorHandler

Handles process monitor data by writing the data as delimited values to a file, usually with tab separated values (.tsv).

A new line is written every time the process monitor polls for a new sample of data. By default a header line starting with # is included at the start of the file to indicate the column headings.

If any value cannot be retrieved or is unavailable on this operating system, a -1 value will be written instead.

Uses default values set on the instance unless keyword overrides are provided; see setDefaults.

Parameters

• file (str) – An absolute path string or open file handle to which process monitor data lines will be written.

• columns (list[str]) – An ordered list of the columns from ProcessMonitorKey that should be included in the file. If not specified, the columns specified by DEFAULT_COLUMNS will be used.

• delimiter (str) – The delimiter string used between each column. If not specified, the string specified by DEFAULT_DELIMITER will be used.

• writeHeaderLine (bool) – Determines whether a header line prefixed by # will be written at the start of the file. If not overridden, the default is taken from DEFAULT_WRITE_HEADER_LINE.

DEFAULT_COLUMNS = ['Time', 'CPU core %', 'Resident memory kB', 'Virtual memory kB']

The default columns to write to the file.

If using a process monitor that comes with PySys, the values are from ProcessMonitorKey.

Additional columns may be added to the end of this list in future releases.

See setDefaults if you wish to change the defaults.

DEFAULT_WRITE_HEADER_LINE = True

Determines whether a header line prefixed by # will be written at the start of the file.

See setDefaults.

DEFAULT_DELIMITER = '\t'

The delimiter used between each column. Usually a tab character.

See setDefaults.

static setDefaults(columns, delimiter=None, writeHeaderLine=None)

Static helper method for setting the default columns or writeHeaderLine setting for all tests that use the ProcessMonitorTextFileHandler.

This method could be called from a custom runner’s pysys.baserunner.BaseRunner.setup method in order to take effect for all tests.

Do not call this from within an individual testcase since that could cause unwanted interference between different testcases.

Parameters

• columns – A list of the colums to be included, using values from ProcessMonitorKey. Since additional columns may be added to the end of DEFAULT_COLUMNS in future releases, when calling this method you should specify all the columns you want explicitly including the current defaults rather than writing DEFAULT_COLUMNS+[...].

• delimiter – The delimiter string used between each column.

• writeHeaderLine – Specifies whether a header line beginning with # should be written at the start of the file.

BaseProcessMonitor

class pysys.process.monitor.BaseProcessMonitor(owner, process, interval, handlers, **pmargs)

Bases: object

Process monitor for gathering statistics such as CPU and memory usage from a running process using a background thread.

For detail on the statistic keys available from the standard process monitor classes see ProcessMonitorKey.

The most convenient way to start the default process monitor for this operating system is to use pysys.basetest.BaseTest.startProcessMonitor.

You can create a custom subclass if you need to add support for a new OS or return additional monitoring statistics. For example, you could create a custom process monitor for Java processes that returned heap size and thread count, or you could create a simple process monitor wrapper around a Python library that provides a wider set of monitoring statistics, such as psutil. The fields and methods starting with a single underscore _ can be used or overridden by custom subclasses. To start the process monitor from a custom class, simply ensure you have passed the owner basetest to its constructor and then call the start method.

Monitors are automatically terminated during cleanup at the end of a test, or can be manually stopped before that using the stop method.

Parameters

• owner – The BaseTest owning this monitor.

• process – The process wrapper object. A numeric pid can be specified instead but with reduced functionality, so use a process object if you have one.

• interval – The interval in seconds between polling for each data sample.

• pmargs – Keyword arguments to allow parameterization of the returned data. An exception will be raised for any arguments not expected by this class.

process = None

The process object. Can be None if a pid was passed directly.

pid = None

The pid to be monitored.

owner = None

The test object that owns this monitor.

handlers = None

The list of handlers that will be notified each time the process is polled for new data.

thread = None

The background thread that responsible for monitoring the process.

start()

Called on the main test thread to start monitoring in the background.

Performs any required initialization of data structures then starts the background thread.

Returns

This instance.

Return type

BaseProcessMonitor

_preprocessData(data)

Called in the background thread with the data dictionary from each poll of the process, to allow OS-agnostic pre-processing and addition of derived data keys such as the date and time.

Parameters

data – The dictionary of process monitoring data. This method may add or modify the contents of this dictionary.

running()

Return the running status of the process monitor.

Returns

True if the process monitor background thread is still running.

Return type

bool

stop(timeout=None)

Request the process monitor thread to terminate.

Waits for the termination to complete if possible, but does not throw an exception if it fails.

Parameters

timeout (float) – Overrides the maximum time we will wait for the monitor to stop. By default this is ['WaitForProcessStop'] from constants.TIMEOUTS.

_getData(sample)

Implement gathering of latest monitoring data.

Called on the background monitoring thread regularly.

Parameters

sample – An integer starting at 1 and incrementing each time this method is called.

Returns

A dictionary of (typically numeric) values, keyed by ProcessMonitorKey.

Return type

dict

_cleanup()

Perform implementation-specific cleanup.

Called on the background monitoring thread when the monitor is stopping.