temci.misc package

Submodules

temci.misc.game module

Benchmarks game inspired comparison of different implementations for a given language.

It doesn’t really belong directly to the temci tool, but uses big parts of it. It’s currently in a pre alpha state as it’s a part of the evaluation for my bachelor thesis that I’m currently doing,

temci.misc.game.AV_GHC_VERSIONS = ['7.0.1', '7.2.1', '7.4.1', '7.6.1', '7.8.1', '7.10.1', '8.0.1']

These are (currently) the versions installable via the ppa on https://launchpad.net/~hvr/+archive/ubuntu/ghc Older versions can’t be installed due to version conflicts and missing libraries

class temci.misc.game.BOTableColumn(title: str, format_str: str, property: Callable[[SingleProperty, List[float], List[SingleProperty], int], float], reduce: Callable[[List[float]], Any])

Bases: object

Column for BaseObject table_html_for_vals_per_impl

class temci.misc.game.BaseObject(name: str, children: Optional[Union[Dict[str, BaseObject], InsertionTimeOrderedDict]] = None)

Bases: object

A base class for all other classes that provides helper methods.

boxplot_html(base_file_name: str, singles: List[SingleProperty], zoom_in: bool = False) → str

boxplot_html_for_data(name: str, base_file_name: str, data: Dict[str, List[float]], zoom_in: bool = False)

build(base_dir: str) → List[dict]

classmethod from_config_dict(*args) → BaseObject

get_geom_over_rel_means() → Dict[str, float]

get_geom_over_rel_stds() → Dict[str, float]

get_geom_std_over_rel_means() → Dict[str, float]

get_gsd_for_x_per_impl(property: Callable[[SingleProperty, List[float], List[SingleProperty], int], float]) → Dict[str, float]

Calculates the geometric standard deviation for the property for each implementation.

get_reduced_x_per_impl(property: Callable[[SingleProperty, List[float], List[SingleProperty], int], float], reduce: Callable[[List[float]], Any], x_per_impl_func: Optional[Callable[[Callable[[SingleProperty, List[float], List[SingleProperty], int], float]], Dict[str, List[float]]]] = None) → Dict[str, float]

Returns the reduced [property] for each implementation. To reduce the list of [property] it uses the passed reduce function. The returned implementations doesn’t depend on one of the parameters.

get_x_per_impl(property: Callable[[SingleProperty, List[float], List[SingleProperty], int], float]) → Dict[str, List[float]]

Returns a list of [property] for each implementation.

Parameters

property – property function that gets a SingleProperty object and a list of all means and returns a float

table_html_for_vals_per_impl(columns: List[Union[BOTableColumn, Callable[[], BOTableColumn]]], base_file_name: str, x_per_impl_func: Optional[Callable[[Callable[[SingleProperty, List[float], List[SingleProperty], int], float]], Dict[str, List[float]]]] = None) → str

Returns the html for a table that has a row for each implementation and several columns (the first is the implementation column).

class temci.misc.game.Implementation(parent: ProgramWithInput, name: str, run_cmd: str, build_cmd: Optional[str] = None, run_data: Optional[List[Union[int, float]]] = None)

Bases: BaseObject

Represents an implementation of a program.

build(base_dir: str) → List[dict]

classmethod from_config_dict(parent: ProgramWithInput, config: dict) → Implementation

get_single_property() → SingleProperty

get_x_per_impl(property: Callable[[SingleProperty, List[float], List[SingleProperty], int], float])

Returns a list of [property] for each implementation.

Parameters

property – property function that gets a SingleProperty object and a list of all means and returns a float

mean() → float

class temci.misc.game.Input(prefix: Optional[str] = None, number: Optional[Union[int, float]] = None, appendix: Optional[str] = None)

Bases: object

Input with a variable numeric part.

classmethod from_config_dict(config: dict) → Input

classmethod list_from_numbers(*numbers: List[Union[int, float]]) → List[Input]

replace(search: str, replacement: str) → Input

Returns an input object in which the search string is replaced in the prefix and the appendix.

to_dict() → dict

class temci.misc.game.Language(name: str, categories: List[ProgramCategory])

Bases: BaseObject

apply_program_filter(filter: ~typing.Callable[[int, ~typing.List[~temci.misc.game.Program]], bool] = <function id_program_filter>)

build(base_dir: str, multiprocess: bool = True) → List[dict]

create_temci_run_file(base_build_dir: str, file: str)

classmethod from_config_dict(config: dict) → Language

get_box_plot_html(base_file_name: str) → str

get_box_plot_per_input_per_impl_html(base_file_name: str, input_num: int) → str

A box plot for each input that shows the mean scores (over all programs) for each implementation.

get_full_html(base_dir: str, html_func: Optional[Callable[[str, int, bool], str]] = None) → str

get_html(base_file_name: str, h_level: int, with_header: bool = True, multiprocess: bool = False) → str

get_html2(base_file_name: str, h_level: int, with_header: bool = True, multiprocess: bool = False, show_entropy_distinction: bool = True)

get_impl_mean_scores() → Dict[str, float]

get_max_input_num() → int

get_scores_per_impl() → Dict[str, List[float]]

get_statistical_property_scores(func: ~typing.Callable[[~temci.report.stats.SingleProperty], float], reduce: ~typing.Callable[[~typing.List[float]], ~temci.utils.typecheck.Any] = <function gmean>) → Dict[str, float]

get_statistical_property_scores_per_impl(func: Callable[[SingleProperty], float]) → Dict[str, List[float]]

get_statistical_property_scores_per_input_per_impl(func: ~typing.Callable[[~temci.report.stats.SingleProperty], float], input_num: int, reduce: ~typing.Callable[[~typing.List[float]], ~temci.utils.typecheck.Any] = <function gmean>) → Dict[str, List[float]]

Assumptions:

• Most programs have the same number of input (known as max input number)

• The input number n takes roughly the same amount of time for every program category

get_x_per_impl_and_input(property: Callable[[SingleProperty, List[float], List[SingleProperty], int], float], input_num: int) → Dict[str, List[float]]

classmethod merge_different_versions_of_the_same(configs: List[dict], config_impl_apps: List[str], group_by_app: bool)

process_result_file(file: str, property: str = 'task-clock')

set_difference_from_two_result_dicts(run_datas: Tuple[List[Dict[str, Any]]], app: str, property: str = 'task-clock')

First - Second for each measured value

set_merged_run_data_from_result_dict(run_datas: List[List[Dict[str, Any]]], impl_apps: List[str], property: str = 'task-clock')

set_run_data_from_result_dict(run_datas: List[Dict[str, Any]], property: str = 'task-clock')

store_html(base_dir: str, clear_dir: bool = True, html_func: Optional[Callable[[str, int, bool], str]] = None)

class temci.misc.game.Mode(value)

Bases: Enum

An enumeration.

geom_mean_rel_to_best = 1

calculate all mean scores as “mean / best mean” and use the geometric mean for summaries

mean_rel_to_first = 2

calculate all mean scores as “mean / mean of first” and use the arithmetic mean for summaries

mean_rel_to_one = 3

calculate all mean scores as “mean / 1” and use the arithmetic mean for summaries

class temci.misc.game.Program(parent: ProgramCategory, name: str, file: str, prog_inputs: Optional[List[ProgramWithInput]] = None, copied_files: Optional[List[str]] = None)

Bases: BaseObject

A program with several different inputs.

build(base_dir: str) → List[dict]

entropy

Entropy of the implementation

classmethod from_config_dict(parent: ProgramCategory, config: dict) → Implementation

get_box_plot_html(base_file_name: str) → str

get_box_plot_per_input_per_impl_html(base_file_name: str, input: str) → str

A box plot for each input that shows the execution times for each implementation.

get_html(base_file_name: str, h_level: int) → str

get_html2(base_file_name: str, h_level: int)

get_impl_mean_scores() → Dict[str, List[float]]

Geometric mean over the means relative to best per implementation (per input).

get_statistical_property_scores(func: Callable[[SingleProperty], float]) → Dict[str, List[float]]

get_statistical_property_scores_per_input_per_impl(func: Callable[[SingleProperty], float], input: str) → Dict[str, float]

class temci.misc.game.ProgramCategory(parent: Language, name: str, programs: List[Program])

Bases: BaseObject

Represents a specific abstract program that gives the specification for several implementations (aka “program”s).

apply_program_filter(filter: ~typing.Callable[[int, ~typing.List[~temci.misc.game.Program]], bool] = <function id_program_filter>)

Filters the programs that make up self.children and self.programs. It stores the original set of programs else where.

Parameters

filter – the used filter, the id filter resets the original state

build(base_dir: str) → List[dict]

classmethod from_config_dict(parent: Language, config: dict) → ProgramCategory

get_box_plot_html(base_file_name: str) → str

get_box_plot_per_input_per_impl_html(base_file_name: str, input: str) → str

A box plot for each input that shows the mean scores (over all programs) for each implementation.

get_html(base_file_name: str, h_level: int) → str

get_html2(base_file_name: str, h_level: int)

get_impl_mean_scores() → Dict[str, float]

get_input_strs() → List[str]

get_scores_per_impl() → Dict[str, List[float]]

get_statistical_property_scores(func: ~typing.Callable[[~temci.report.stats.SingleProperty], float], reduce: ~typing.Callable[[~typing.List[float]], ~temci.utils.typecheck.Any] = <function gmean>) → Dict[str, float]

get_statistical_property_scores_per_impl(func: ~typing.Callable[[~temci.report.stats.SingleProperty], float], reduce: ~typing.Callable[[~typing.List[float]], ~temci.utils.typecheck.Any] = <function gmean>) → Dict[str, float]

get_statistical_property_scores_per_input_per_impl(func: Callable[[SingleProperty], float], input: str) → Dict[str, List[float]]

get_x_per_impl_and_input(property: Callable[[SingleProperty, List[float], List[SingleProperty], int], float], input: str) → Dict[str, List[float]]

temci.misc.game.ProgramFilterFunc

A function that get’s the current program index in the also passed list of all other programs and returns True if the program is okay and False otherwise.

alias of Callable[[int, List[Program]], bool]

class temci.misc.game.ProgramWithInput(parent: Program, input: Input, impls: List[Implementation], id: int)

Bases: BaseObject

This represents the program with a specific input. It has several program implementations.

build(base_dir: str) → List[dict]

get_box_plot_html(base_file_name: str) → str

get_html(base_file_name: str, h_level: int) → str

get_html2(base_file_name: str, h_level: int) → str

get_means_rel_to_best() → Dict[str, float]

get_single()

get_single_properties() → List[Tuple[str, SingleProperty]]

get_statistical_properties_for_each(func: Callable[[SingleProperty], float]) → Dict[str, float]

get_x_per_impl(property: Callable[[SingleProperty, List[float], List[SingleProperty], int], float]) → Dict[str, List[float]]

Returns a list of [property] for each implementation.

Parameters

property – property function that gets a SingleProperty object and a list of all means and returns a float

temci.misc.game.ReduceFunc

Gets passed a list of values and returns a single value, e.g. stats.gmean

alias of Callable[[List[float]], Any]

temci.misc.game.StatProperty

Gets passed a SingleProperty object, the list of means (containing the object’s mean), the list of all SingleProperty objects and the index of the first in it and returns a float.

alias of Callable[[SingleProperty, List[float], List[SingleProperty], int], float]

temci.misc.game.StatisticalPropertyFunc

Get’s passed the SingleProperty object to process and min mean

alias of Callable[[SingleProperty], float]

temci.misc.game.amean_std(values: List[float]) → float

Calculates the arithmetic mean.

temci.misc.game.bench_categories(ending: str, inputs: Dict[str, List[Input]]) → List[dict]

temci.misc.game.bench_category(category: str, ending: str, inputs: List[Input], numbers: Optional[List[int]] = None) → dict

temci.misc.game.bench_file(category: str, ending: str, number: int = 1) → str

temci.misc.game.bench_program(category: str, ending: str, inputs: List[Input], number: int = 1) → dict

temci.misc.game.c_config(inputs_per_category: Dict[str, List[Input]], optimisation: str = '-O2', clang_version='3.7') → Dict[str, Union[str, dict]]

Generates a game config that compares gcc and clang.

temci.misc.game.cparser_config(inputs_per_category: Dict[str, List[Input]], optimisation: str = '-O2', clang_version='3.7') → Dict[str, Union[str, dict]]

Generates a game config that compares gcc, clang and cparser.

temci.misc.game.divide_inputs(inputs_per_category: Dict[str, List[Input]], divisor: Union[int, float]) → Dict[str, List[Input]]

temci.misc.game.empty_inputs(inputs_per_category: Dict[str, List[Input]]) → Dict[str, List[Input]]

temci.misc.game.file_entropy(file: str) → int

Calculates the entropy of given file by taking the length of its gzip compressed content

temci.misc.game.file_lines(file: str) → int

Number of non empty lines in the file

temci.misc.game.first(values: List[float]) → float

temci.misc.game.first_inputs(inputs_per_category: Dict[str, List[Input]]) → Dict[str, List[Input]]

temci.misc.game.haskel_config(inputs_per_category: Dict[str, List[Input]], optimisation: str, ghc_versions: Optional[List[str]] = None, used_c_compilers: Optional[List[str]] = None) → Dict[str, Union[str, dict]]

Generate a game config comparing all available ghc versions

Parameters

• inputs_per_category –

• optimisation – optimisation flags, e.g. ‘-Odph’ or ‘-O’

• ghc_versions – compared ghc versions, if None, AV_GHC_VERSIONS is used

temci.misc.game.id_program_filter(_1, _2)

temci.misc.game.last_inputs(inputs_per_category: Dict[str, List[Input]]) → Dict[str, List[Input]]

temci.misc.game.mean_rel_std()

temci.misc.game.mean_score_column()

temci.misc.game.mean_score_std_column()

temci.misc.game.prefix_inputs(prefix: str, inputs: List[Input]) → List[Input]

temci.misc.game.process(config: ~typing.Dict[str, ~typing.Union[str, dict]], name: ~typing.Optional[str] = None, build_dir: ~typing.Optional[str] = None, build: bool = True, benchmark: bool = True, report: bool = True, temci_runs: int = 15, temci_options: str = '--discarded_blocks 1', temci_stop_start: bool = True, report_dir: ~typing.Optional[str] = None, property: str = 'task-clock', report_modes: ~typing.List[~temci.misc.game.Mode] = [<Mode.mean_rel_to_first: 2>, <Mode.geom_mean_rel_to_best: 1>])

Process a config dict. Simplifies the build, benchmarking and report generating.

Parameters

• config – processed config dict

• name – the name of the whole configuration (used to generate the file names), default “{config[‘language]}”

• build_dir – build dir that is used to build the programs, default is “/tmp/{name}”

• build – make a new build of all programs? (results in a “{name}.exec.yaml” file for temci)

• benchmark – benchmark the “{name}.exec.yaml” file (from a built)? (results in a “{name}.yaml” result file)

• report – generate a game report? (results in a report placed into the report_dir)

• temci_runs – number of benchmarking runs (if benchmark=True)

• temci_options – used options for temci

• temci_stop_start – does temci use the StopStart plugin for decreasing the variance while benchmarking?

• report_dir – the directory to place the report in, default is “{name}_report”

• property – measured property for which the report is generated, default is “task-clock”

temci.misc.game.produce_ttest_comparison_table(datas: List[List[Dict[str, Union[Dict[str, List[float]], Any]]]], impls: List[str], data_descrs: List[str], filename: str, property: str = 'task-clock', alpha: float = 0.05, tester_name: str = 't', ratio_format: str = '{:3.0%}')

temci.misc.game.property_filter_half(cur_index: int, all: List[Program], property_func: Callable[[Program], float], remove_upper_half: bool) → bool

Note: if the number of programs is uneven, then one program will belong to the upper and the lower half.

temci.misc.game.ref(name: str, value=None, _store={'fasta': ['250000', '2500000', '25000000']})

A simple YAML like reference utility. It to easily store a value under a given key and return it.

Parameters

• name – name of the reference

• value – new value of the reference (if value isn’t None)

• _store – dict to store everything in

Returns

the value of the reference

temci.misc.game.rel_mean_func(x, min)

temci.misc.game.rel_mean_property(single: SingleProperty, means: List[float], *args) → float

A property function that returns the relative mean (the mean of the single / minimum of means)

temci.misc.game.rel_std_dev_func(x: SingleProperty, min: float) → float

temci.misc.game.rel_std_dev_to_min_func(x: SingleProperty, min: float) → float

temci.misc.game.rel_std_property(single: SingleProperty, means: List[float], *args) → float

A property function that returns the relative standard deviation (relative to single’s mean)

temci.misc.game.replace_run_with_build_cmd(config_dict: Dict[str, Union[str, dict]]) → Dict[str, Union[str, dict]]

temci.misc.game.rust_config(inputs_per_category: Dict[str, List[Input]], optimisation: int = 3) → Dict[str, Union[str, dict]]

Generates a game config that compares the different rust versions.

temci.misc.game.ttest_rel_to_first_property(single: SingleProperty, _, all_singles: List[SingleProperty], index: int) → float

temci.misc.game.ttest_summarize(values: List[float]) → float

temci.misc.game.ttest_to_first()

temci.misc.game.used_rel_mean_property(single: SingleProperty, means: List[float], *args) → float

temci.misc.game.used_std_property(single: SingleProperty, means: List[float], *args) → float

temci.misc.game.used_summarize_mean(values: List[float]) → float

temci.misc.game.used_summarize_mean_std(values: List[float]) → float

Module contents

The stuff in this folder doesn’t really belong to the temci tool but builds on top of it some cool applications, like a benchmarksgame inspired comparison of different implementations of several languages. The tools may depend on other code or packages than the temci tool itself.