API documentation¶

A test set is a Python script defining a subclass of TestSet, whose class name should match the name of the test-set script in PascalCase. Running the qpbenchmark command on this script solves its problems with all available solvers and settings, stores results and writes a report.

Test sets¶

TestSet ¶

TestSet()

Bases: ABC

Abstract base class for a test set.

A test set is a collection of problems with solver settings.

Attributes:

Name	Type	Description
`solver_settings`	`Dict[str, SolverSettings]`	Dictionary of solver parameters for each settings.
`tolerances`	`Dict[str, Tolerance]`	Validation tolerances.

Initialize test set.

description `abstractmethod` `property` ¶

description: str

Test set description.

sparse_only `abstractmethod` `property` ¶

sparse_only: bool

If True, restrict test set to solvers with a sparse matrix API.

title `abstractmethod` `property` ¶

title: str

Report title.

iter `abstractmethod` ¶

__iter__() -> Iterator[Problem]

Yield test-set problems one by one.

define_tolerances ¶

define_tolerances(runtime: float = 10.0) -> None

Define validation tolerances.

This function can be overridden by child test-set classes.

Parameters:

Name	Type	Description	Default
`runtime`	`float`	Maximum QP solver runtime in seconds.	`10.0`

define_solver_settings ¶

define_solver_settings() -> None

Define solver settings.

This function can be overridden by child test-set classes.

count_problems ¶

count_problems() -> int

Count the number of problems in the set.

Returns:

Type	Description
`int`	Number of problems in the test set.

get_problem ¶

get_problem(name: str) -> Optional[Problem]

Get a specific test set problem.

Parameters:

Name	Type	Description	Default
`name`	`str`	Problem name.	required

Returns:

Type	Description
`Optional[Problem]`	Problem if found, None otherwise.

skip_solver_issue ¶

skip_solver_issue(problem: Problem, solver: str) -> bool

Skip known solver issue.

Parameters:

Name	Type	Description	Default
`problem`	`Problem`	Problem to solve.	required
`solver`	`str`	QP solver.	required

Returns:

Type	Description
`bool`	True if `solver` is known to fail on `problem`.

skip_solver_timeout ¶

skip_solver_timeout(
    time_limit: float,
    problem: Problem,
    solver: str,
    settings: str,
) -> bool

Skip known solver timeouts.

Parameters:

Name	Type	Description	Default
`time_limit`	`float`	Time limit in seconds.	required
`problem`	`Problem`	Problem to solve.	required
`solver`	`str`	QP solver.	required
`settings`	`str`	QP solver settings.	required

Note

This function only checks for timeouts that the solvers are not able to handle by themselves, e.g. for those who do not provide a time limit parameter.

Returns:

Type	Description
`bool`	True if `solver` is known to take more than `time_limit` seconds on
`bool`	`problem` when using the solver parameters defined in `settings`.

ParquetTestSet ¶

ParquetTestSet(path: Union[Path, str])

Bases: TestSet

Test set read from a Parquet file.

Initialize test set.

Parameters:

Name	Type	Description	Default
`path`	`Union[Path, str]`	Path to Parquet file to read problems from.	required

iter ¶

__iter__() -> Iterator[Problem]

Yield test-set problems one by one.

SolverSettings ¶

SolverSettings()

Settings for multiple solvers.

Initialize settings.

solvers `property` ¶

solvers: Iterator[str]

List solvers configured in these settings.

is_implemented `classmethod` ¶

is_implemented(solver: str)

Check whether a solver is implemented by this class.

getitem ¶

__getitem__(solver: str) -> Dict[str, Any]

Get settings dictionary of a given solver.

Parameters:

Name	Type	Description	Default
`solver`	`str`	Name of the QP solver.	required

Returns:

Type	Description
`Dict[str, Any]`	Dictionary of custom solver settings.

set_eps_abs ¶

set_eps_abs(eps_abs: float) -> None

Set absolute tolerances for solvers that support it.

Parameters:

Name	Type	Description	Default
`eps_abs`	`float`	Absolute primal, dual and duality-gap tolerance.	required

Notes

When we set an absolute tolerance :math:\epsilon_{abs} on residuals, we ask the solver to find an approximation of the optimum such that the primal residual, dual residual and duality gap are below :math:\epsilon_{abs}, that is:

.. math::

\begin{align}
r_p := \max(\| A x - b \|_\infty, [G x - h]^+, [lb - x]^+,
[x - ub]^+) & \leq \epsilon_{abs} \\
r_d := \| P x + q + A^T y + G^T z + z_{box} \|_\infty &
\leq \epsilon_{abs} \\
r_g := | x^T P x + q^T x + b^T y + h^T z + lb^T z_{box}^- +
ub^T z_{box}^+ | & \leq \epsilon_{abs}
\end{align}

were :math:v^- = \min(v, 0) and :math:v^+ = \max(v, 0). The tolerance on the primal residual is called "feasibility tolerance" by some solvers, for instance CVXOPT and ECOS. See this note <https://scaron.info/blog/optimality-conditions-and-numerical-tolerances-in-qp-solvers.html>__ for more details.

set_eps_rel ¶

set_eps_rel(eps_rel: float) -> None

Set relative tolerances for solvers that support it.

Parameters:

Name	Type	Description	Default
`eps_rel`	`float`	Relative primal, dual and duality-gap tolerance.	required

set_time_limit ¶

set_time_limit(time_limit: float) -> None

Apply time limits to all solvers that support it.

Parameters:

Name	Type	Description	Default
`time_limit`	`float`	Time limit in seconds.	required

set_verbosity ¶

set_verbosity(verbose: bool) -> None

Apply verbosity settings to all solvers.

Parameters:

Name	Type	Description	Default
`verbose`	`bool`	Verbosity boolean.	required

get_param ¶

get_param(solver: str, param: str, default: str) -> Any

Get solver parameter in these settings.

Parameters:

Name	Type	Description	Default
`solver`	`str`	QP solver.	required
`param`	`str`	Parameter name.	required
`default`	`str`	Value to return if the parameter is not configured.	required

Returns:

Type	Description
`Any`	Corresponding solver parameter, or default value.

set_param ¶

set_param(solver: str, param: str, value: Any) -> None

Set solver parameter.

Parameters:

Name	Type	Description	Default
`solver`	`str`	QP solver.	required
`param`	`str`	Parameter name.	required
`value`	`Any`	Parameter value.	required

Tolerance `dataclass` ¶

Tolerance(
    primal: float, dual: float, gap: float, runtime: float
)

Tolerances on solver solution validation.

Attributes:

Name	Type	Description
`primal`	`float`	Tolerance on primal residuals.
`dual`	`float`	Tolerance on dual residuals.
`gap`	`float`	Tolerance on duality gaps.
`runtime`	`float`	Time limit in seconds.

from_metric ¶

from_metric(metric: str) -> float

Get tolerance corresponding to a given metric.

Parameters:

Name	Type	Description	Default
`metric`	`str`	Metric to get the tolerance of.	required

Returns:

Type	Description
`float`	Corresponding tolerance.

Problems¶

Problem ¶

Problem(
    P: Union[ndarray, csc_matrix],
    q: ndarray,
    G: Optional[Union[ndarray, csc_matrix]],
    h: Optional[ndarray],
    A: Optional[Union[ndarray, csc_matrix]],
    b: Optional[ndarray],
    lb: Optional[ndarray],
    ub: Optional[ndarray],
    name: str,
)

Bases: Problem

Quadratic program.

Attributes:

Name	Type	Description
`name`	`str`	Name of the problem, for reporting.

Quadratic program in qpsolvers format.

from_qpsolvers `staticmethod` ¶

from_qpsolvers(qp: Problem, name: str) -> Problem

Stick a name to a generic problem from qpsolvers.

Parameters:

Name	Type	Description	Default
`qp`	`Problem`	Quadratic program.	required
`name`	`str`	Name of the problem.	required

Returns:

Type	Description
`Problem`	Benchmark version of the problem.

to_dense ¶

to_dense() -> Problem

Return dense version.

Returns:

Type	Description
`Problem`	Dense version of the present problem.

to_sparse ¶

to_sparse() -> Problem

Return sparse version.

Returns:

Type	Description
`Problem`	Sparse version of the present problem.

load `staticmethod` ¶

load(file: str)

Load problem from file.

Parameters:

Name	Type	Description	Default
`file`	`str`	Path to the file to read.	required

ProblemList ¶

ProblemList()

List of problems saved to and read from Parquet files.

Initialize to an empty list.

append ¶

append(problem: Problem) -> None

Append a problem to the list.

Parameters:

Name	Type	Description	Default
`problem`	`Problem`	Problem to append.	required

extend ¶

extend(
    problem_list: Union[ProblemList, List[Problem]],
) -> None

Extend problem list with another.

Parameters:

Name	Type	Description	Default
`problem_list`	`Union[ProblemList, List[Problem]]`	Other problem list.	required

to_parquet ¶

to_parquet(path: str) -> None

Save sequence of problems to a Parquet file.

Parameters:

Name	Type	Description	Default
`path`	`str`	Path to the Parquet file to save problems to.	required

Running a test set¶

run ¶

run(
    test_set: TestSet,
    results: Results,
    only_problem: Optional[str] = None,
    only_settings: Optional[str] = None,
    only_solver: Optional[str] = None,
    rerun: bool = False,
    rerun_timeouts: bool = False,
    verbose: bool = False,
) -> None

Run a given test set and store results.

Parameters:

Name	Type	Description	Default
`test_set`	`TestSet`	Test set to run.	required
`results`	`Results`	Results instance to write to.	required
`only_problem`	`Optional[str]`	If set, only run that specific problem in the set.	`None`
`only_settings`	`Optional[str]`	If set, only run with these solver settings.	`None`
`only_solver`	`Optional[str]`	If set, only run that specific solver.	`None`
`rerun`	`bool`	If set, rerun instances that already have a result.	`False`
`rerun_timeouts`	`bool`	If set, also rerun known timeouts.	`False`
`verbose`	`bool`	If set, log info messages for each QP solver call.	`False`

main ¶

main(
    test_set_path: Optional[Union[Path, str]] = None,
    results_path: Optional[Union[Path, str]] = None,
)

Main function of the script.

Parameters:

Name	Type	Description	Default
`test_set_path`	`Optional[Union[Path, str]]`	If set, load test set from this Python file.	`None`
`results_path`	`Optional[Union[Path, str]]`	Path to the results CSV file.	`None`

Results¶

Results ¶

Results(
    file_path: Optional[Union[str, Path]], test_set: TestSet
)

Test set results.

Attributes:

Name	Type	Description
`df`	`DataFrame`	Data frame storing the results.
`file_path`	`Optional[Path]`	Path to the results CSV file.
`test_set`	`TestSet`	Test set from which results were produced.

Initialize results.

Parameters:

Name	Type	Description	Default
`file_path`	`Optional[Union[str, Path]]`	Path to the results file (format: CSV or Parquet), or `None` if there is no file associated with these results.	required
`test_set`	`TestSet`	Test set from which results were produced.	required

nb_rows `property` ¶

nb_rows: int

Number of rows in the dataframe.

check_df `staticmethod` ¶

check_df(df) -> None

Check consistency of a full results dataframe.

Raises:

Type	Description
`ResultsError`	if the dataframe is inconsitent.

read_from_file `staticmethod` ¶

read_from_file(
    path: Union[str, Path],
) -> Optional[pandas.DataFrame]

Load a pandas dataframe from a CSV or Parquet file.

Parameters:

Name	Type	Description	Default
`path`	`Union[str, Path]`	Path to the file to load.	required

Returns:

Type	Description
`Optional[DataFrame]`	Loaded dataframe, or None if the file does not exist.

write ¶

write(path: Optional[Union[str, Path]] = None) -> None

Write results to their CSV file for persistence.

Parameters:

Name	Type	Description	Default
`path`	`Optional[Union[str, Path]]`	Optional path to a separate file to write to.	`None`

has ¶

has(problem: Problem, solver: str, settings: str) -> bool

Check if results contain a given run of a solver on a problem.

Parameters:

Name	Type	Description	Default
`problem`	`Problem`	Test set problem.	required
`solver`	`str`	Name of the QP solver.	required
`settings`	`str`	Name of the corresponding solver settings.	required

Returns:

Type	Description
`bool`	True if a result for this instance is present.

is_timeout ¶

is_timeout(
    problem: Problem,
    solver: str,
    settings: str,
    time_limit: float,
) -> bool

Check whether a particular result was a timeout.

update ¶

update(
    problem: Problem,
    solver: str,
    settings: str,
    solution: Solution,
    runtime: float,
) -> None

Update entry for a given (problem, solver) pair.

Parameters:

Name	Type	Description	Default
`problem`	`Problem`	Problem solved.	required
`solver`	`str`	Solver name.	required
`settings`	`str`	Solver settings.	required
`solution`	`Solution`	Solution found by the solver.	required
`runtime`	`float`	Duration the solver took, in seconds.	required

build_success_rate_df ¶

build_success_rate_df(
    primal_tolerances: Dict[str, float],
    dual_tolerances: Dict[str, float],
    gap_tolerances: Dict[str, float],
) -> Tuple[pandas.DataFrame, pandas.DataFrame]

Build the success-rate data frame.

Parameters:

Name	Type	Description	Default
`primal_tolerances`	`Dict[str, float]`	Primal-residual tolerance for each settings.	required
`dual_tolerances`	`Dict[str, float]`	Dual-residual tolerance for each settings.	required
`gap_tolerances`	`Dict[str, float]`	Duality-gap tolerance for each settings.	required

Returns:

Type	Description
`Tuple[DataFrame, DataFrame]`	Success-rate data frames.

build_correct_rate_df ¶

build_correct_rate_df(
    primal_tolerances: Dict[str, float],
    dual_tolerances: Dict[str, float],
    gap_tolerances: Dict[str, float],
) -> Tuple[pandas.DataFrame, pandas.DataFrame]

Build the correctness-rate data frame.

Parameters:

Name	Type	Description	Default
`primal_tolerances`	`Dict[str, float]`	Primal-residual tolerance for each settings.	required
`dual_tolerances`	`Dict[str, float]`	Dual-residual tolerance for each settings.	required
`gap_tolerances`	`Dict[str, float]`	Duality-gap tolerance for each settings.	required

Returns:

Type	Description
`Tuple[DataFrame, DataFrame]`	Correctness-rate data frames.

get_shgeom_for_metric_and_settings ¶

get_shgeom_for_metric_and_settings(
    metric: str,
    settings: str,
    shift: float,
    not_found_value: float,
    floor: Optional[float] = None,
) -> Dict[str, float]

Get shifted geometric means for a given metric with given settings.

Parameters:

Name	Type	Description	Default
`metric`	`str`	Name of the metric column to average.	required
`settings`	`str`	Name of the settings column to filter on.	required
`shift`	`float`	Shift of the shifted geometric mean.	required
`not_found_value`	`float`	Value to apply when a solver has not found a solution.	required
`floor`	`Optional[float]`	If set, clip metric values from below to this value before averaging. This is used to floor residuals at the requested tolerance, so that over-accuracy (e.g. 1e-15 versus a 1e-9 residual) is neither penalized nor rewarded.	`None`

Returns:

Type	Description
`Dict[str, float]`	Dictionary with the shifted geometric mean of each solver.

build_shgeom_df ¶

build_shgeom_df(
    metric: str,
    shift: float,
    not_found_values: Dict[str, float],
    floors: Optional[Dict[str, float]] = None,
) -> pandas.DataFrame

Compute the shifted geometric mean for a given metric.

Parameters:

Name	Type	Description	Default
`metric`	`str`	Name of the metric column to average.	required
`shift`	`float`	Shift of the shifted geometric mean.	required
`not_found_values`	`Dict[str, float]`	Values to apply when a solver has not found a solution (one per settings). For instance, time limits are used for the runtime of a solver that fails to solve a problem.	required
`floors`	`Optional[Dict[str, float]]`	Per-settings lower bounds applied to metric values before averaging (see :func:`get_shgeom_for_metric_and_settings`). Used to clip residuals at their corresponding tolerances.	`None`

Returns:

Type	Description
`DataFrame`	Shifted geometric mean of the prescribed column.

Report ¶

Report(author: str, results: Results)

Report generated from benchmark results.

Attributes:

Name	Type	Description
`author`	`str`	GitHub username of the person who generated the report.
`results`	`Results`	Results from which the report should be generated.
`solver_settings`	`Dict[str, SolverSettings]`	Dictionary of solver parameters for each settings.
`test_set`	`TestSet`	Test set from which results were generated.

Initialize report.

Parameters:

Name	Type	Description	Default
`author`	`str`	GitHub username of the person who generated the report.	required
`results`	`Results`	Results from which the report should be generated.	required

get_tolerances_table ¶

get_tolerances_table() -> str

Get tolerances Markdown table.

Returns:

Type	Description
`str`	Tolerances Markdown table.

get_solver_settings_table ¶

get_solver_settings_table() -> str

Get Markdown table for solver settings.

Returns:

Type	Description
`str`	Solver settings Markdown table.

get_solver_versions_table ¶

get_solver_versions_table() -> str

Get Markdown table for solver versions.

Returns:

Type	Description
`str`	Solver versions Markdown table.

write ¶

write(path: str) -> None

Write report to a given path.

Parameters:

Name	Type	Description	Default
`path`	`str`	Path to the Markdown file to write the report to.	required

Note

We make sure each table in the report is preceded by a single line title, for instance "Precentage of problems each solver is able to solve:". This comes in handy for @@ anchorage when doing the diff of two reports.

Exceptions¶

BenchmarkError ¶

Bases: Exception

Base class for benchmark exceptions.

ProblemNotFound ¶

Bases: BenchmarkError

Exception raised when a requested problem is not part of a test set.

ResultsError ¶

Bases: BenchmarkError

Exception raised when results formatting is wrong.

API documentation¶

Test sets¶

TestSet ¶

description abstractmethod property ¶

sparse_only abstractmethod property ¶

title abstractmethod property ¶

__iter__ abstractmethod ¶

define_tolerances ¶

define_solver_settings ¶

count_problems ¶

get_problem ¶

skip_solver_issue ¶

skip_solver_timeout ¶

ParquetTestSet ¶

__iter__ ¶

SolverSettings ¶

solvers property ¶

is_implemented classmethod ¶

__getitem__ ¶

set_eps_abs ¶

set_eps_rel ¶

set_time_limit ¶

set_verbosity ¶

get_param ¶

set_param ¶

Tolerance dataclass ¶

from_metric ¶

Problems¶

Problem ¶

from_qpsolvers staticmethod ¶

to_dense ¶

to_sparse ¶

load staticmethod ¶

ProblemList ¶

append ¶

extend ¶

to_parquet ¶

Running a test set¶

run ¶

main ¶

Results¶

Results ¶

nb_rows property ¶

check_df staticmethod ¶

read_from_file staticmethod ¶

write ¶

has ¶

is_timeout ¶

update ¶

build_success_rate_df ¶

build_correct_rate_df ¶

get_shgeom_for_metric_and_settings ¶

build_shgeom_df ¶

Report ¶

get_tolerances_table ¶

get_solver_settings_table ¶

get_solver_versions_table ¶

write ¶

Exceptions¶

BenchmarkError ¶

ProblemNotFound ¶

ResultsError ¶

description `abstractmethod` `property` ¶

sparse_only `abstractmethod` `property` ¶

title `abstractmethod` `property` ¶

iter `abstractmethod` ¶

iter ¶

solvers `property` ¶

is_implemented `classmethod` ¶

getitem ¶

Tolerance `dataclass` ¶

from_qpsolvers `staticmethod` ¶

load `staticmethod` ¶

nb_rows `property` ¶

check_df `staticmethod` ¶

read_from_file `staticmethod` ¶