temci.report package

Submodules

temci.report.report module

class temci.report.report.AbstractReporter(misc_settings=None, stats_helper: Optional[RunDataStatsHelper] = None, excluded_properties: Optional[List[str]] = None)

Bases: object

Produces a meaningful report out of measured data.

Creates an instance.

Parameters

• misc_settings – configuration

• stats_helper – used stats helper instance

• excluded_properties – measured properties that are excluded from the reports

misc

Configuration

report()

Create a report and output or store it as configured.

stats

This object is used to simplify the work with the data and the statistics

stats_helper

Used starts helper

to_long_prop_dict

Maps a property name to a long property name

class temci.report.report.CSVReporter(misc_settings=None, stats_helper: Optional[RunDataStatsHelper] = None, excluded_properties: Optional[List[str]] = None)

Bases: AbstractReporter

Simple reporter that outputs just a configurable csv table with rows for each run block

Configuration format:

# List of valid column specs, format is a comma separated list of 'PROPERTY\[mod\]' or 'ATTRIBUTE' mod
# is one of: mean, stddev, property, min, max and stddev per mean, optionally a formatting option can
# be given viaPROPERTY\[mod|OPT1OPT2…\], where the OPTs are one of the following: % (format as
# percentage), p (wrap insignificant digits in parentheses (+- 2 std dev)), s (use scientific
# notation, configured in report/number) and o (wrap digits in the order of magnitude of 2 std devs in
# parentheses). PROPERTY can be either the description or the short version of the property. Configure
# the number formatting further via the number settings in the settings file
columns: [description]

# Output file name or standard out (-)
out: '-'

This reporter can be configured under the settings key report/csv_misc.

To use this reporter set the currently used reporter (at key report/reporter) to its name (csv). Other usable reporter are console and html2. The default reporter is console.

Creates an instance.

Parameters

• misc_settings – configuration

• stats_helper – used stats helper instance

• excluded_properties – measured properties that are excluded from the reports

report() → Optional[str]

Create an report and output it as configured.

Returns

the report string if to_string == True

class temci.report.report.Codespeed2Reporter(misc_settings=None, stats_helper: Optional[RunDataStatsHelper] = None, excluded_properties: Optional[List[str]] = None)

Bases: AbstractReporter

Reporter that outputs JSON as specified by the codespeed runner spec.

Creates an instance.

Parameters

• misc_settings – configuration

• stats_helper – used stats helper instance

• excluded_properties – measured properties that are excluded from the reports

report()

Create a report and output it as configured.

class temci.report.report.CodespeedReporter(misc_settings=None, stats_helper: Optional[RunDataStatsHelper] = None, excluded_properties: Optional[List[str]] = None)

Bases: AbstractReporter

Reporter that outputs JSON as expected by codespeed. Branch name and commit ID are taken from the current directory. Use it like this:

temci report --reporter codespeed ... | curl --data-urlencode json@- http://localhost:8000/result/add/json/

Configuration format:

# Branch name reported to codespeed. Defaults to current branch or else 'master'.
branch: ''

# Commit ID reported to codespeed. Defaults to current commit.
commit_id: ''

# Environment name reported to codespeed. Defaults to current host name.
environment: ''

# Executable name reported to codespeed. Defaults to the project name.
executable: ''

# Project name reported to codespeed.
project: ''

This reporter can be configured under the settings key report/codespeed_misc.

To use this reporter set the currently used reporter (at key report/reporter) to its name (codespeed). Other usable reporter are console, html2 and csv. The default reporter is console.

Creates an instance.

Parameters

• misc_settings – configuration

• stats_helper – used stats helper instance

• excluded_properties – measured properties that are excluded from the reports

report()

Create a report and output it as configured.

class temci.report.report.ConsoleReporter(misc_settings=None, stats_helper: Optional[RunDataStatsHelper] = None, excluded_properties: Optional[List[str]] = None)

Bases: AbstractReporter

Simple reporter that outputs just text.

Configuration format:

# Matches the baseline block
baseline: ''

# Position of the baseline comparison: each: after each block, after: after each cluster, both: after
# each and after cluster, instead: instead of the non baselined
baseline_position: each

# 'auto': report clusters (runs with the same description) and singles (clusters with a single entry,
# combined) separately, 'single': report all clusters together as one, 'cluster': report all clusters
# separately, 'both': append the output of 'cluster' to the output of 'single'
mode: auto

# Output file name or `-` (stdout)
out: '-'

# Report on the failing blocks
report_errors: true

# Print statistical tests for every property for every two programs
with_tester_results: true

This reporter can be configured under the settings key report/console_misc.

To use this reporter set the currently used reporter (at key report/reporter) to its name (console).

The default reporter is console.

Creates an instance.

Parameters

• misc_settings – configuration

• stats_helper – used stats helper instance

• excluded_properties – measured properties that are excluded from the reports

report(with_tester_results: bool = True, to_string: bool = False) → Optional[str]

Create an report and output it as configured.

Parameters

• with_tester_results – include the hypothesis tester results

• to_string – return the report as a string and don’t output it?

Returns

the report string if to_string == True

class temci.report.report.HTMLReporter(misc_settings=None, stats_helper: Optional[RunDataStatsHelper] = None, excluded_properties: Optional[List[str]] = None)

Bases: AbstractReporter

Deprecated reporter that just lives as a hull. It might be useful to revive it as a basic reporter without JavaScript.

Creates an instance.

Parameters

• misc_settings – configuration

• stats_helper – used stats helper instance

• excluded_properties – measured properties that are excluded from the reports

report()

Create a report and output or store it as configured.

class temci.report.report.HTMLReporter2(misc_settings=None, stats_helper: Optional[RunDataStatsHelper] = None, excluded_properties: Optional[List[str]] = None)

Bases: AbstractReporter

Reporter that produces a HTML based report with lot’s of graphics. A rewrite of the original HTMLReporter

Configuration format:

# Alpha value for confidence intervals
alpha: 0.05

# Height per run block for the big comparison box plots
boxplot_height: 2.0

# Width of all big plotted figures
fig_width_big: 25.0

# Width of all small plotted figures
fig_width_small: 15.0

# Format string used to format floats
float_format: '{:5.2e}'

# Override the contents of the output directory if it already exists?
force_override: false

# Generate pdf versions of the plotted figures?
gen_pdf: false

# Generate simple latex versions of the plotted figures?
gen_tex: true

# Generate excel files for all tables
gen_xls: false

# Hide warnings and errors related to statistical properties
hide_stat_warnings: false

# Name of the HTML file
html_filename: report.html

# Use local versions of all third party resources
local: false

# Show the mean related values in the big comparison table
mean_in_comparison_tables: true

# Show the mininmum related values in the big comparison table
min_in_comparison_tables: false

# Output directory
out: report

# Format string used to format floats as percentages
percent_format: '{:5.2%}'

# Show zoomed out (x min = 0) figures in the extended summaries?
show_zoomed_out: true

This reporter can be configured under the settings key report/html2_misc.

To use this reporter set the currently used reporter (at key report/reporter) to its name (html2). Another usable reporter is console. The default reporter is console.

Creates an instance.

Parameters

• misc_settings – configuration

• stats_helper – used stats helper instance

• excluded_properties – measured properties that are excluded from the reports

get_random_filename() → str

classmethod html_escape_property(property: str) → str

report()

Create a report and output or store it as configured.

class temci.report.report.ReporterRegistry

Bases: AbstractRegistry

Registry for reporters.

The used reporter can be configured by editing the settings key report/reporter. Possible reporter are ‘console’, ‘html2’, ‘csv’, ‘codespeed’, ‘codespeed2’ and ‘velcom’

default = 'console'

Name(s) of the class(es) used by default. Type depends on the use_list property.

plugin_synonym = ('reporter', 'reporter')

Singular and plural version of the word that is used in the documentation for the registered entities

registry = {'codespeed': <class 'temci.report.report.CodespeedReporter'>, 'codespeed2': <class 'temci.report.report.Codespeed2Reporter'>, 'console': <class 'temci.report.report.ConsoleReporter'>, 'csv': <class 'temci.report.report.CSVReporter'>, 'html2': <class 'temci.report.report.HTMLReporter2'>, 'velcom': <class 'temci.report.report.VelcomReporter'>}

Registered classes (indexed by their name)

settings_key_path = 'report'

Used settings key path

use_key = 'reporter'

Used key that sets which registered class is currently used

use_list = False

Allow more than one class to used at a specific moment in time

class temci.report.report.VelcomReporter(misc_settings=None, stats_helper: Optional[RunDataStatsHelper] = None, excluded_properties: Optional[List[str]] = None)

Bases: AbstractReporter

Reporter that outputs JSON as specified by the velcom runner spec.

Creates an instance.

Parameters

• misc_settings – configuration

• stats_helper – used stats helper instance

• excluded_properties – measured properties that are excluded from the reports

report()

Create a report and output it as configured.

temci.report.report.html_escape_property(property: str) → str

Escape the name of a measured property.

Parameters

property – name of a measured property

Returns

escaped property name

temci.report.report_processor module

class temci.report.report_processor.ReportProcessor(stats_helper: Optional[RunDataStatsHelper] = None)

Bases: object

Simplifies the work with reporters.

Creates an instance.

Parameters

stats_helper – used data wrapper or None if an empty one should be used

report()

Create a report with the used reporter

reporter

Used reporter

temci.report.rundata module

Contains the RunData object for benchmarking data of specific program block and the RunDataStatsHelper that provides helper methods for working with these objects.

class temci.report.rundata.ExcludedInvalidData

Bases: object

Info object that contains information about the excluded invalid data.

excluded_properties_per_run_data

Run data descriptions mapped to the excluded properties per run data. Only includes not fully excluded run datas.

excluded_run_datas

Descriptions of the fully excluded run datas

temci.report.rundata.Number

Numeric value

alias of Union[int, float]

class temci.report.rundata.RecordedError(message: str)

Bases: object

class temci.report.rundata.RecordedInternalError(message: str)

Bases: RecordedError

classmethod for_exception(ex: BaseException) → RecordedInternalError

class temci.report.rundata.RecordedProgramError(message: str, out: str, err: str, ret_code: int)

Bases: RecordedError

class temci.report.rundata.RunData(data: Optional[Dict[str, List[Union[int, float]]]] = None, attributes: Optional[Dict[str, str]] = None, recorded_error: Optional[RecordedError] = None, external: bool = False)

Bases: object

A set of benchmarking data for a specific program block.

An entry in the run output list

error:         Either(Dict({'message': Str(), 'return_code': Int(), 'output': Str(), 'error_output': Str()}, False, keys=Any, values=Any)|non existent)

internal_error:         Either(Dict({'message': Str()}, False, keys=Any, values=Any)|non existent)

# Optional attributes that describe the block
attributes:     
    description:         Optional(Str())

    # Tags of this block
    tags:         ListOrTuple(Str())

data:         Dict(, keys=Str(), values=List(Either(Int()|T(<class 'float'>))), default = {})
# with constrain: Either 'error' or 'internal_error' can be present

An entry in the run output list that specifies the long names for the properties

property_descriptions:         Either(non existent|Dict(, keys=Str(), values=Str(), default = {}))

Initializes a new run data object.

Parameters

• data – optional dictionary mapping each property to a list of actual values

• attributes – dictionary of optional attributes that describe its program block

• recorded_error – either program error or internal error

• external – does the data come from a prior benchmarking?

add_data_block(data_block: Dict[str, List[Union[int, float]]])

Adds a block of data.

Parameters

data_block – maps each of the run datas properties to list of actual values (from each benchmarking run).

attributes

Dictionary of optional attributes that describe its program block

benchmarks() → int

Returns the maximum number of measured values for the associated program block over all properties.

block_type_scheme = Dict('data': Dict(, keys=Str(), values=List(Either(Int()|T(<class 'float'>))), default = {}), 'attributes': Dict('tags': ListOrTuple(Str()), 'description': Optional(Str()), keys=Str(), values=Any, default = {'tags': [], 'description': ''}), 'error': Either(Dict({'message': Str(), 'return_code': Int(), 'output': Str(), 'error_output': Str()}, False, keys=Any, values=Any)|non existent), 'internal_error': Either(Dict({'message': Str()}, False, keys=Any, values=Any)|non existent), keys=Any, values=Any):Either 'error' or 'internal_error' can be present

clone(data: Optional[Dict[str, List[Union[int, float]]]] = None, attributes: Optional[Dict[str, str]] = None, recorded_error: Optional[RecordedError] = None, external: Optional[bool] = None) → RunData

Clone this instance and replaces thereby some instance properties.

Parameters

• data – optional dictionary mapping each property to a list of actual values

• attributes – dictionary of optional attributes that describe its program block

• external – does the data come from a prior benchmarking?

Returns

new instance

data

Raw benchmarking data, mapping properties to their corresponding values

description() → str

Description of this instance based on the attributes

env_info_scheme = env_info:         Either(non existent|List(Tuple(Str(), List(Tuple(Str(), Str())))))

exclude_invalid() → Tuple[Optional[RunData], List[str]]

Exclude properties that only have NaNs as measurements.

Returns

(new run data instance or None if all properties are excluded or the current if nothing changed, excluded properties)

exclude_properties(properties: List[str]) → RunData

Creates a new run data instance without the passed properties.

Parameters

properties – excluded properties

Returns

new run data instance

external

Does the data come from a prior benchmarking?

get_single_properties() → Dict[str, SingleProperty]

Returns single property stat objects per property that support statistical analysis (e.g. mean, standard deviation, …)

Returns

stat object per property

has_error() → bool

include_properties(properties: List[str]) → RunData

Creates a new run data instance with only the passed properties.

Parameters

properties – included properties, can be regular expressions

Returns

new run data instance

long_properties(long_versions: Dict[str, str]) → RunData

Replace the short properties names with their long version from the passed dictionary.

Parameters

long_versions – long versions of some properties

Returns

new run data instance (or current instance if nothing changed)

min_values() → int

Returns the minimum number of measured values for the associated program block over all properties.

properties

List of measured properties. They might not all be measured the same number of times.

property_descriptions_scheme = property_descriptions:         Either(non existent|Dict(, keys=Str(), values=Str(), default = {}))

to_dict() → Dict[str, Union[Dict[str, str], Dict[str, List[Union[int, float]]]]]

Returns a dictionary that represents this run data object.

class temci.report.rundata.RunDataStatsHelper(runs: List[RunData], tester: Optional[Tester] = None, external_count: int = 0, property_descriptions: Optional[Dict[str, str]] = None, errorneous_runs: Optional[List[RunData]] = None, included_blocks: Optional[str] = None, env_info: Optional[List[Tuple[str, List[Tuple[str, str]]]]] = None)

Bases: object

This class helps to simplify the work with a set of run data observations.

Don’t use the constructor use init_from_dicts if possible.

Parameters

• runs – list of run data objects

• tester – used tester or tester that is set in the settings

• external_count – Number of external program blocks (blocks for which the data was obtained in a

different benchmarking session) :param property_descriptions: mapping of some properties to their descriptions or longer versions :param errorneous_runs: runs that resulted in errors :param included_blocks: include query :param env_info: formatted environment info

add_data_block(program_id: int, data_block: Dict[str, List[Union[int, float]]])

Add block of data for the program block with the given id.

Parameters

• program_id – id of the program.

• data_block – list of data from several benchmarking runs of the program block

Raises

ValueError – if the program block with the given id doesn’t exist

add_error(program_id: id, error: RecordedError)

Set the error for a program

add_property_descriptions(property_descriptions: Dict[str, str])

Adds the given property descriptions.

Parameters

property_descriptions – mapping of some properties to their descriptions or longer versions

clone(runs: Optional[List[RunData]] = None, tester: Optional[Tester] = None, external_count: Optional[int] = None, property_descriptions: Optional[Dict[str, str]] = None) → RunDataStatsHelper

Clones this instance and replaces the given instance properties.

Parameters

• runs – list of run data objects

• tester – used tester or tester that is set in the settings

• external_count – Number of external program blocks (blocks for which the data was obtained in a

different benchmarking session) :param property_descriptions: mapping of some properties to their descriptions or longer versions :return: cloned instance

discard_run_data(id: int)

Disable that run data object with the given id.

estimate_time(run_bin_size: int, min_runs: int, max_runs: int) → float

Roughly erstimates the time needed to finish benchmarking all program blocks. It doesn’t take any parallelism into account. Therefore divide the number by the used parallel processes.

Warning

Doesn’t work well.

Parameters

• run_bin_size – times a program block is benchmarked in a single block of time

• min_runs – minimum number of allowed runs

• max_runs – maximum number of allowed runs

Returns

estimated time in seconds or float(“inf”) if no proper estimation could be made

estimate_time_for_next_round(run_bin_size: int, all: bool) → float

Roughly estimates the time needed for the next benchmarking round.

Parameters

• run_bin_size – times a program block is benchmarked in a single block of time and the size of a round

• all – expect all program block to be benchmarked

Returns

estimated time in seconds

exclude_invalid() → Tuple[RunDataStatsHelper, ExcludedInvalidData]

Exclude all properties of run datas that only have zeros or NaNs as measurements.

Returns

(new instance, info about the excluded data)

exclude_properties(properties: List[str]) → RunDataStatsHelper

Create a new instance without the passed properties.

Parameters

properties – excluded properties

Returns

new instance

external_count

Number of external program blocks (blocks for which the data was obtained in a different benchmarking session)

get_description_clusters() → Dict[str, List[RunData]]

Set of runs per description, call RunDataStatsHelper.make_descriptions_distinct first

Returns

set of runs per description

get_description_clusters_and_single() → Tuple[List[RunData], Dict[str, List[RunData]]]

Set of runs per description, call RunDataStatsHelper.make_descriptions_distinct first

Returns

set of runs per description

get_evaluation(with_equal: bool, with_unequal: bool, with_uncertain: bool, blocks: Optional[List[RunData]] = None) → dict

Structure of the returned list items:

- data: # set of two run data objects
  properties: # information for each property that is equal, ...
      -prop:
          - equal: True/False
            uncertain: True/False
            p_val: probability of the null hypothesis
            speed_up: speed up from the first to the second
            description: description of the property

Parameters

• with_equal – with tuple with at least one “equal” property

• with_unequal – … unequal property

• with_uncertain – include also uncertain properties

• blocks – blocks to compare

Returns

list of tuples for which at least one property matches the criteria

get_program_ids_to_bench() → List[int]

Returns the ids (the first gets id 0, …) of the program block / run data object that should be benchmarked again.

has_error(program_id: int) → bool

Is there an error recorded for the program with the given id?

include_properties(properties: List[str]) → RunDataStatsHelper

Create a new instance with only the passed properties

Parameters

properties – included properties, can be regular expressions

Returns

new instance

classmethod init_from_dicts(runs: Optional[List[Union[Dict[str, str], Dict[str, List[Union[int, float]]]]]] = None, external: bool = False, included_blocks: Optional[str] = None) → RunDataStatsHelper

Expected structure of the stats settings and the runs parameter:

"stats": {
    "tester": ...,
    "properties": ["prop1", ...],
    # or
    "properties": ["prop1", ...],
    "uncertainty_range": (0.1, 0.3)
}

"runs": [
    {"attributes": {"attr1": ..., ..., ["description": …], ["tags": …]},
     "data": {"__ov-time": [...], ...},
     "error": {"return_code": …, "output": "…", "error_output": "…"},
     "internal_error": {"message": "…"} (either "error" or "internal_error" might be present)
     ["property_descriptions": {"__ov-time": "Overall time", …}]
     ["env_info": … ]
     },
     ...
]

Parameters

• runs – list of dictionaries representing the benchmarking runs for each program block

• external – are the passed runs not from this benchmarking session but from another?

• included_blocks – include query

Raises

ValueError – if runs parameter has an incorrect structure

Returns

created stats helper

classmethod init_from_file(filename: str) → RunDataStatsHelper

Load the runs from the passed file. This file has to encode the runs using YAML in the format stated in RunDataStatsHelper.init_from_dicts()

Parameters

filename – name of the file

Raises

ValueError – if the file has an incorrect structure

Returns

created stats helper

is_equal(p_val: float) → bool

Is the passed value above the uncertainty range for null hypothesis probabilities?

is_uncertain(p_val: float) → bool

Does the passed probability of the null hypothesis for two samples lie in the uncertainty range? :param p_val: passed probability of the null hypothesis

is_unequal(p_val: float) → bool

Is the passed value above the uncertainty range for null hypothesis probabilities?

long_properties(property_format: str = '[{}]') → Tuple[RunDataStatsHelper, Dict[str, str]]

Replace the short properties names with their descriptions if possible.

Parameters

property_format – format string that gets a property description and produces a longer property name

Returns

new instance, a dict that returns a long property name for a given short property name

make_descriptions_distinct()

Append numbers to descriptions if needed to make them unique

properties() → List[str]

Returns a sorted list of all properties that exist in all run data blocks.

runs

Data of serveral runs from several measured program blocks

serialize() → List[Union[Dict[str, str], Dict[str, List[Union[int, float]]], List[Tuple[str, List[Tuple[str, str]]]]]]

Serialize this instance into a data structure that is accepted by the init_from_dicts method.

tester

Used statistical tester

update_env_info()

Obtain the environment information for the current system and store it

valid_runs() → List[RunData]

Number of valid (with measured data) runs

temci.report.rundata.get_for_tag(per_tag_settings_key: str, main_key: str, tag: Optional[str])

temci.report.rundata.get_for_tags(per_tag_settings_key: str, main_key: str, tags: List[str], combinator: Callable[[Any, Any], Any])

temci.report.stats module

Statistical helper classes for tested pairs and single blocks.

class temci.report.stats.BaseStatObject

Bases: object

Class that gives helper methods for the extending stat object classes.

description() → str

Returns the description of this stat object.

Should be implemented by sub classes of BaseStatObject.

eq_except_property(other) → bool

Is this stat object equal to the passed stat object (without regarding the actual property)?

Implemented by sub classes of BaseStatObject.

errors() → List[StatMessage]

Returns a list of all (merged) statistical errors for this stat object.

get_data_frame(**kwargs)

Get the data frame that is associated with this stat object.

Implemented by sub classes of BaseStatObject. :return pd.DataFrame instance

get_stat_messages() → List[StatMessage]

Returns a list of all (merged) statistical warnings and errors for this stat object.

has_errors() → bool

Does this stat object has any statistical errors?

has_warnings() → bool

Does this stat object has any statistical errors?

histogram(fig_width: Union[float, int], fig_height: Optional[Union[int, float]] = None, x_ticks: Optional[List[Union[int, float]]] = None, y_ticks: Optional[List[Union[int, float]]] = None, show_legend: Optional[bool] = None, type: Optional[str] = None, align: str = 'mid', x_label: Optional[str] = None, y_label: Optional[str] = None, zoom_in: bool = True, other_objs: Optional[List[BaseStatObject]] = None, other_obj_names: Optional[List[str]] = None, own_name: Optional[str] = None, **kwargs)

Plots a histogram as the current figure. Don’t forget to close it via fig.close()

Parameters

• fig_width – width of the figure in cm

• fig_height – height of the figure in cm

• x_ticks – None: use default ticks, list: use the given ticks

• y_ticks – None: use default ticks, list: use the given ticks

• show_legend – show a legend in the plot? If None only show one if there are more than one sub histograms

• type – histogram type (either ‘bar’, ‘barstacked’, ‘step’, ‘stepfilled’ or None for auto)

• align – controls where each bar centered (‘left’, ‘mid’ or ‘right’)

• x_label – if not None, shows the given x label

• y_label – if not None: shows the given y label

• zoom_in – does the x axis start at the minimum x value?

• kwargs – optional arguments passed to the get_data_frame method

• other_objs – addional objects to plot on the same histogram (only SingleProperty objects allowed)

• other_obj_names – names of the additional objects

• own_name – used with other_objs option

img_filename_ending = '.svg'

File ending for image file (and therefore their format)

is_single_valued() → bool

Does the data consist only of one unique value?

reset_plt()

Reset the current matplotlib plot style.

store_figure(filename: str, fig_width: float, fig_height: Optional[float] = None, pdf: bool = True, tex: bool = True, tex_standalone: bool = True, img: bool = True, zoom_in: bool = False) → Dict[str, str]

warnings() → List[StatMessage]

Returns a list of all (merged) statistical warnings for this stat object.

class temci.report.stats.EffectToSmallError(parent: BaseStatObject, properties: Union[List[str], str], values: Union[List[Union[int, float]], float, int])

Bases: EffectToSmallWarning

Error message regarding an only insignificant mean difference regarding the standard deviation.

Create a new instance.

Parameters

• parent – parent stat object

• properties – properties of the stat object for which this message is relevant

• values – a value of the related statistical property for each relevant property

border_value = 2

type = 10

Type of this StatMessage, warning or error

class temci.report.stats.EffectToSmallWarning(parent: BaseStatObject, properties: Union[List[str], str], values: Union[List[Union[int, float]], float, int])

Bases: StatWarning

Warning regarding a not really significant mean difference regarding the standard deviation.

Create a new instance.

Parameters

• parent – parent stat object

• properties – properties of the stat object for which this message is relevant

• values – a value of the related statistical property for each relevant property

border_value = 1

classmethod check_value(value) → bool

Is this type of message inappropriate for the passed value of the related statistical property?

hint = 'Try to reduce the standard deviation if you think that the measured difference is significant: If you use the exec run driver, consider using a preset. Also consider increasing the number of measured runs.'

Text that should give a hint about what to do to prevent the possible mistake

message = 'The mean difference per standard deviation of {props} is less than {b_val}.'

Format string for the message, {b_val} is replaced by the border value and {props} by the list of properties and program block names that have this problem.

value_format = '{:5.3f}'

Format string used to format the border and the actual value

class temci.report.stats.NotEnoughObservationsError(parent: BaseStatObject, properties: Union[List[str], str], values: Union[List[Union[int, float]], float, int])

Bases: NotEnoughObservationsWarning

Error message regarding too few observations or measurements.

Create a new instance.

Parameters

• parent – parent stat object

• properties – properties of the stat object for which this message is relevant

• values – a value of the related statistical property for each relevant property

border_value = 15

type = 10

Type of this StatMessage, warning or error

class temci.report.stats.NotEnoughObservationsWarning(parent: BaseStatObject, properties: Union[List[str], str], values: Union[List[Union[int, float]], float, int])

Bases: StatWarning

Warning regarding too few observations or measurements.

Create a new instance.

Parameters

• parent – parent stat object

• properties – properties of the stat object for which this message is relevant

• values – a value of the related statistical property for each relevant property

border_value = 30

classmethod check_value(value) → bool

Is this type of message inappropriate for the passed value of the related statistical property?

hint = 'Increase the number of measured runs.'

Text that should give a hint about what to do to prevent the possible mistake

message = 'The number of observations of {props} is less than {b_val}.'

Format string for the message, {b_val} is replaced by the border value and {props} by the list of properties and program block names that have this problem.

value_format = '{}'

Format string used to format the border and the actual value

temci.report.stats.Number

Numeric value type

alias of Union[float, int]

class temci.report.stats.SignificanceTooLowError(parent: BaseStatObject, properties: Union[List[str], str], values: Union[List[Union[int, float]], float, int])

Bases: SignificanceTooLowWarning

Error message regarding an only insignificant difference regarding a statistical test. The probability of the null hypothesis is larger than the larger end of the configured uncertainty range.

Create a new instance.

Parameters

• parent – parent stat object

• properties – properties of the stat object for which this message is relevant

• values – a value of the related statistical property for each relevant property

classmethod check_value(value) → bool

Is this type of message inappropriate for the passed value of the related statistical property?

type = 10

Type of this StatMessage, warning or error

class temci.report.stats.SignificanceTooLowWarning(parent: BaseStatObject, properties: Union[List[str], str], values: Union[List[Union[int, float]], float, int])

Bases: StatWarning

Warning regarding an only insignificant difference regarding a statistical test. The probability of the null hypothesis lies in the configured uncertainty range.

Create a new instance.

Parameters

• parent – parent stat object

• properties – properties of the stat object for which this message is relevant

• values – a value of the related statistical property for each relevant property

classmethod check_value(value) → bool

Is this type of message inappropriate for the passed value of the related statistical property?

hint = 'Increase the number of benchmarking runs.'

Text that should give a hint about what to do to prevent the possible mistake

message = 'The used statistical significance test showed that the significance of the\n    difference with {props} is too low.'

Format string for the message, {b_val} is replaced by the border value and {props} by the list of properties and program block names that have this problem.

class temci.report.stats.Single(data: Union[RunData, Single])

Bases: BaseStatObject

A statistical wrapper around a single run data object.

Create an instance.

Parameters

data – run data wrapped by this instance or another instance of which the run data is used

attributes

Attributes for this instance

description() → str

Returns the description of this stat object.

Should be implemented by sub classes of BaseStatObject.

eq_except_property(other) → bool

Is this stat object equal to the passed stat object (without regarding the actual property)?

Implemented by sub classes of BaseStatObject.

get_data_frame()

Get the data frame that is associated with this stat object.

Implemented by sub classes of BaseStatObject. :return pd.DataFrame instance

properties

SingleProperty objects for each property

rundata

Run data wrapped by this instance

class temci.report.stats.SingleProperty(parent: Single, data: Union[RunData, SingleProperty], property: str)

Bases: BaseStatObject

A statistical wrapper around a single run data block for a specific measured property.

Creates an instance.

Parameters

• parent – parent single object that contains this instance

• data – measured data for all properties or another single property instance of which the data is used

• property – actually measured property

array

NumPy array version of the measured data for the specific property. Using this in NumPy contexts might speed up the calculations.

data

Measured data for the specific property

description() → str

Returns the description of this stat object.

Should be implemented by sub classes of BaseStatObject.

eq_except_property(other) → bool

Is this stat object equal to the passed stat object (without regarding the actual property)?

Implemented by sub classes of BaseStatObject.

get_data_frame()

Get the data frame that is associated with this stat object.

Implemented by sub classes of BaseStatObject. :return pd.DataFrame instance

iqr() → float

Calculates the interquartile range.

is_single_valued() → bool

Does the data consist only of one unique value?

max() → float

Maximum value of the measurements

mean() → float

Mean value of the measurements

mean_ci(alpha: float) → Tuple[float, float]

Calculates the confidence interval in which the population mean lies with the given probability. Assumes normal distribution.

Adopted from http://stackoverflow.com/a/15034143

Parameters

alpha – given probability

Returns

lower, upper bound

median() → float

Median of the measurements

min() → float

Minimum value of the measurements

normality() → float

Calculates the probability of the data being normal distributed.

Warning

return NaN if the number of observations is less than 8

observations() → int

Number of measurements or observations

outliers(whis: float = 1.5) → List[float]

Returns the values that don’t lie in the in the range fenced by the whiskers.

parent

Parent single object that contains this instance

percentile(q: int) → float

Calculates the q th percentile. q must be between 0 and 100 inclusive.

property

Actually measured property

quartiles() → Tuple[float, float, float]

Calculates the 3 quartiles (1, 2 and 3)

rundata

RunData object that contains the measurements for all properties

sem() → float

Standard error of the mean (standard deviation / sqrt(observations))

skewedness() → float

Calculates the skewedness of the data.

Warning

return NaN if the number of observations is less than 8

std() → float

Standard deviation of the measurements

std_dev() → float

Standard deviation of the measurements

std_dev_ci(alpha: float) → Tuple[float, float]

Calculates the confidence interval in which the standard deviation lies with the given probability. Assumes normal distribution.

Adopted from http://www.stat.purdue.edu/~tlzhang/stat511/chapter7_4.pdf

Parameters

alpha – given probability

Returns

lower, upper bound

std_dev_per_mean() → float

Standard deviation per mean of the measurements (also known as variation coefficient)

std_devs() → Tuple[float, float]

Calculates the standard deviation of elements <= mean and of the elements > mean.

Returns

(lower, upper)

std_error_mean() → float

Standard error of the mean (standard deviation / sqrt(observations))

variance() → float

Variance of the measurements

whiskers(whis: float = 1.5) → Tuple[float, float]

Calculates the upper and the lower whisker for a boxplot. I.e. the minimum and the maximum value of the data set the lie in the range (Q1 - whis * IQR, Q3 + whis * IQR). IQR being the interquartil distance, Q1 the lower and Q2 the upper quartile.

Adapted from http://stackoverflow.com/a/20096945

class temci.report.stats.SinglesProperty(singles: List[Union[Single, SingleProperty]], property: str)

Bases: BaseStatObject

Creates an instance.

Parameters

• singles – compared single property objects or single objects that are turned into one

• property – regarded measured property

boxplot(fig_width: Union[float, int], fig_height: Optional[Union[int, float]] = None, zoom_in: bool = False)

Creates a (horizontal) box plot comparing all single object for a given property.

Parameters

• fig_width – width of the figure in cm

• fig_height – height of the figure in cm, if None it is calculated from the figure width using the aesthetic ratio

get_data_frame(**kwargs)

Get the data frame that is associated with this stat object.

Implemented by sub classes of BaseStatObject. :return pd.DataFrame instance

max() → float

Calculates the maximum value of all compared single property objects.

property

Regarded measured property

singles

Compared single property objects

class temci.report.stats.StatError(parent: BaseStatObject, properties: Union[List[str], str], values: Union[List[Union[int, float]], float, int])

Bases: StatWarning, StatMessage

A message that signals a possible warning that is probably severe

Create a new instance.

Parameters

• parent – parent stat object

• properties – properties of the stat object for which this message is relevant

• values – a value of the related statistical property for each relevant property

type = 10

Type of this StatMessage, warning or error

class temci.report.stats.StatMessage(parent: BaseStatObject, properties: Union[List[str], str], values: Union[List[Union[int, float]], float, int])

Bases: object

A message that warns of possible mistakes and gives hints. It’s related to a statistical property.

Create a new instance.

Parameters

• parent – parent stat object

• properties – properties of the stat object for which this message is relevant

• values – a value of the related statistical property for each relevant property

border_value = 0

classmethod check_value(value: Union[float, int]) → bool

Is this type of message inappropriate for the passed value of the related statistical property?

static combine(*messages: Tuple[Optional[StatMessage]]) → List[StatMessage]

Combines all message of the same type and with the same parent in the passed list. Ignores None entries.

Parameters

messages – passed list of messages

Returns

new reduced list

classmethod create_if_valid(parent: BaseStatObject, property: str, value: Union[float, int], **kwargs) → Optional[StatMessage]

Returns a message object if this type of message is appropriate for the passed value of the related statistical property or None otherwise.

Parameters

• parent – parent stat object

• property – property of the stat object for which this message is might by relevant and the value is given

• value – a value of the related statistical property

• kwargs – additional arguments for the constructor of this message type class

generate_msg_text(show_parent: bool) → str

Generates the text of this message object.

Parameters

show_parent – Is the parent shown in after the properties? E.g. “blub of bla parent: …”

Returns

message text

hint = ''

Text that should give a hint about what to do to prevent the possible mistake

message = '{props}: {b_val}'

Format string for the message, {b_val} is replaced by the border value and {props} by the list of properties and program block names that have this problem.

classmethod overridden(value: Union[int, float]) → bool

parent

Parent stat object

properties

Properties of the stat object for which this message is relevant

type = None

Type of this StatMessage, warning or error

value_format = '{:5.3f}'

Format string used to format the border and the actual value

values

a value of the related statistical property for each relevant property

class temci.report.stats.StatMessageType(value)

Bases: Enum

Types of StatMessages.

ERROR = 10

Error type that signals a possible error or mistake, but is called ‘severe warning’

WARNING = 5

Warning type that signals a possible but not severe mistake

class temci.report.stats.StatMessageValueFormat

Bases: object

Format strings for ints, floats and percentage values used in the StatMessages.

FLOAT = '{:5.3f}'

Format string for float values that aren’t shown as percentages

INT = '{}'

Format string for int values

PERCENT = '{:5.3%}'

Format string for float values that are shown as percentages

class temci.report.stats.StatWarning(parent: BaseStatObject, properties: Union[List[str], str], values: Union[List[Union[int, float]], float, int])

Bases: StatMessage

A message that signals a possible warning that isn’t severe

Create a new instance.

Parameters

• parent – parent stat object

• properties – properties of the stat object for which this message is relevant

• values – a value of the related statistical property for each relevant property

classmethod overridden(value: Union[int, float]) → bool

type = 5

Type of this StatMessage, warning or error

class temci.report.stats.StdDeviationToHighError(parent: BaseStatObject, properties: Union[List[str], str], values: Union[List[Union[int, float]], float, int])

Bases: StdDeviationToHighWarning

Error message regarding a far too high standard deviation.

Create a new instance.

Parameters

• parent – parent stat object

• properties – properties of the stat object for which this message is relevant

• values – a value of the related statistical property for each relevant property

border_value = 0.05

type = 10

Type of this StatMessage, warning or error

class temci.report.stats.StdDeviationToHighWarning(parent: BaseStatObject, properties: Union[List[str], str], values: Union[List[Union[int, float]], float, int])

Bases: StatWarning

A warning about a too high standard deviation.

Create a new instance.

Parameters

• parent – parent stat object

• properties – properties of the stat object for which this message is relevant

• values – a value of the related statistical property for each relevant property

border_value = 0.01

classmethod check_value(value) → bool

Is this type of message inappropriate for the passed value of the related statistical property?

hint = 'With the exec run driver you can probably use a preset. Also consider to increase the number of measured runs.'

Text that should give a hint about what to do to prevent the possible mistake

message = 'The standard deviation per mean of {props} is too high. It should be ≤ {b_val}.'

Format string for the message, {b_val} is replaced by the border value and {props} by the list of properties and program block names that have this problem.

value_format = '{:5.3%}'

Format string used to format the border and the actual value

class temci.report.stats.TestedPair(first: Union[RunData, Single], second: Union[RunData, Single], tester: Optional[Tester] = None)

Bases: BaseStatObject

A statistical wrapper around two run data objects that are compared via a tester.

Creates an instance.

Parameters

• first – first of the two compared run data or single objects

• second – second of the two compared run data or single objects

• tester – used statistical tester, if None the default tester is used (configured in the settings)

description() → str

Returns the description of this stat object.

Should be implemented by sub classes of BaseStatObject.

eq_except_property(other) → bool

Is this stat object equal to the passed stat object (without regarding the actual property)?

Implemented by sub classes of BaseStatObject.

first

First of the two compared single objects

first_rel_to_second() → float

Calculates the geometric mean of the first means relative to the second means. Ignores NaNs in calculating the geometric mean.

See http://www.cse.unsw.edu.au/~cs9242/15/papers/Fleming_Wallace_86.pdf

first_rel_to_second_std() → float

Calculates the geometric standard deviation for the first_rel_to_second method.

Ignores NaNs in calculating the geometric std.

properties

TestedPairProperty objects for each shared property of the inherited Single objects

second

Second of the two compared single objects

swap() → TestedPair

Creates a new pair with the elements swapped.

Returns

new pair object

tester

Used statistical tester for the comparisons

class temci.report.stats.TestedPairProperty(parent: TestedPair, first: Single, second: Single, property: str, tester: Optional[Tester] = None)

Bases: BaseStatObject

Statistical helper for comparing a pair of run data blocks for a specific measured property.

Creates an instance.

Parameters

• parent – parent tested pair object

• first – first of the two compared single objects

• second – second of the two compared single objects

• property – regarded property

• tester – used statistical tester, if None the default tester is used (configured in the settings)

Returns

description() → str

Returns the description of this stat object.

Should be implemented by sub classes of BaseStatObject.

eq_except_property(other) → bool

Is this stat object equal to the passed stat object (without regarding the actual property)?

Implemented by sub classes of BaseStatObject.

equal_prob() → float

Probability of the null hypothesis being not not correct (tertiary logic).

Returns

p value between 0 and 1

first

First of the two compared single property objects

first_rel_to_second() → float

Calculates the mean of the first relative to the mean of the second (mean(first) / mean(second)).

Returns 1 if both means are equal, regardless of their values.

get_data_frame(show_property=True)

Get the data frame that is associated with this stat object.

Implemented by sub classes of BaseStatObject. :return pd.DataFrame instance

is_equal() → Optional[bool]

Checks the null hypothesis.

Returns

True or False if the p val isn’t in the uncertainty range of the tester, None else

is_single_valued() → bool

Does the data consist only of one unique value?

max_rel_std_dev() → float

max_std_dev() → float

Calculates the maximum of the standard deviations of the first and the second.

mean_diff() → float

Calculates the difference of the means between the first and the second single property object (mean(first) - mean(second)).

mean_diff_ci(alpha: float) → Tuple[float, float]

Calculates the confidence interval in which the mean difference lies with the given probability. Assumes normal distribution.

Adopted from http://www.kean.edu/~fosborne/bstat/06b2means.html

Parameters

alpha – given probability

Returns

lower, upper bound

mean_diff_per_dev() → float

Calculates the mean difference per standard deviation (maximum of the first’s and the second’s deviation).

mean_diff_per_mean() → float

Calculates the mean difference relative to the mean of the first ((mean(first) - mean(second)) / mean(first)).

mean_std_dev() → float

Calculates the mean of the standard deviations of the first and the second.

min_observations() → int

Returns the minimum number of observations of the first and the second

parent

Parent tested pair object

property

Regarded specific property

second

Second of the two compared single property objects

swap() → TestedPairProperty

Swap the first and the second single object.

Returns

new instance

tester

Used statistical tester for the comparisons

class temci.report.stats.TestedPairsAndSingles(singles: List[Union[RunData, Single]], pairs: Optional[List[TestedPair]] = None)

Bases: BaseStatObject

A wrapper around a list of tested pairs and singles.

Creates an instance.

Parameters

• singles – compared single objects or run data objects that are turned into single object

• pairs – compared pairs of single objects, if None they created out of the passed single objects

get_pair(first_id: int, second_id: int) → TestedPair

Get the tested pair consisting of the two single objects with the passed ids. The id of a single objects is its index (starting at zero) in the internal single list.

Parameters

• first_id – id of the first single object

• second_id – id of the second single object

Returns

created tested pair object comparing the two single objects

get_stat_messages() → List[StatMessage]

Combines the messages for all inherited TestedPair and Single objects,

Returns

simplified list of all messages

number_of_singles() → int

Number of compared single objects

property pairs: List[TestedPair]

properties() → List[str]

Returns the properties that are shared among all single run data objects.

singles

Compared single objects

singles_properties

Singles property object for every measured property

temci.report.testers module

Contains the tester base class and several simple implementations that simplify the work with statistical hypothesis tests.

class temci.report.testers.AndersonTester(*args, **kwargs)

Bases: Tester

Tester that uses the Anderson statistic on 2 samples.

Creates a new instance. :param misc_settings: Additional settings :param uncertainty_range: (start, end) probability tuple that gives range in which the tester doesn’t give

a definitive result on the nullhypothesis check

name = 'anderson'

Name of the implemented statistical test

scipy_stat_method = 'anderson_ksamp'

Used method of the scipy.stats module if the _test_impl isn’t reimplemented

class temci.report.testers.KSTester(*args, **kwargs)

Bases: Tester

Tester that uses the Kolmogorov-Smirnov statistic on 2 samples.

Creates a new instance. :param misc_settings: Additional settings :param uncertainty_range: (start, end) probability tuple that gives range in which the tester doesn’t give

a definitive result on the nullhypothesis check

name = 'kolmogorov smirnov'

Name of the implemented statistical test

scipy_stat_method = 'ks_2samp'

Used method of the scipy.stats module if the _test_impl isn’t reimplemented

class temci.report.testers.TTester(*args, **kwargs)

Bases: Tester

Tester that uses the student’s t test.

Creates a new instance. :param misc_settings: Additional settings :param uncertainty_range: (start, end) probability tuple that gives range in which the tester doesn’t give

a definitive result on the nullhypothesis check

name = 't'

Name of the implemented statistical test

scipy_stat_method = 'ttest_ind'

Used method of the scipy.stats module if the _test_impl isn’t reimplemented

class temci.report.testers.Tester(*args, **kwargs)

Bases: object

A tester tests the probability of the nullh ypothesis of two same length list of observations.

This is a base class that shouldn’t be instantiated.

Creates a new instance. :param misc_settings: Additional settings :param uncertainty_range: (start, end) probability tuple that gives range in which the tester doesn’t give

a definitive result on the nullhypothesis check

estimate_needed_runs(data1: list, data2: list, run_bin_size: int, min_runs: int, max_runs: int) → int

Calculate a approximation of the needed length of both observations that is needed for the p value to lie outside the uncertainty range.

It uses the simple observation that the graph of the p value plotted against the size of the sets has a exponential, logarithmic or root shape.

Warning

Doesn’t work well.

Parameters

• data1 – list of observations

• data2 – list of observations

• run_bin_size – granularity of the observation (> 0)

• min_runs – minimum number of allowed runs

• max_runs – maximum number of allowed runs

Returns

approximation of needed runs or float(“inf”)

is_equal(data1: List[Union[int, float]], data2: List[Union[int, float]]) → bool

Are the two samples not significantly unequal regarding the probability of the null hypothesis?

is_uncertain(data1: List[Union[int, float]], data2: List[Union[int, float]]) → bool

Does the probability of the null hypothesis for two samples lie in the uncertainty range?

is_unequal(data1: List[Union[int, float]], data2: List[Union[int, float]]) → bool

Are the two samples significantly unequal regarding the probability of the null hypothesis?

misc_settings

Additional settings

name = ''

Name of the implemented statistical test

scipy_stat_method = None

Used method of the scipy.stats module if the _test_impl isn’t reimplemented

test(data1: List[Union[int, float]], data2: List[Union[int, float]]) → float

Calculates the probability of the null hypotheses for two samples.

uncertainty_range

(start, end) probability tuple that gives range in which the tester doesn’t give

a definitive result on the nullhypothesis check

class temci.report.testers.TesterRegistry

Bases: AbstractRegistry

The used tester can be configured by editing the settings key stats/tester. Possible testers are ‘t’, ‘ks’ and ‘anderson’

default = 't'

Name(s) of the class(es) used by default. Type depends on the use_list property.

plugin_synonym = ('tester', 'testers')

Singular and plural version of the word that is used in the documentation for the registered entities

registry = {'anderson': <class 'temci.report.testers.AndersonTester'>, 'ks': <class 'temci.report.testers.KSTester'>, 't': <class 'temci.report.testers.TTester'>}

Registered classes (indexed by their name)

settings_key_path = 'stats'

Used settings key path

use_key = 'tester'

Used key that sets which registered class is currently used

use_list = False

Allow more than one class to used at a specific moment in time

Module contents

This module is about generating meaningful reports an working with the resulting measurements of serveral benchmarks.