👷 Contributing¶

This project's goal is to facilitate the comparison of quadratic programming solvers for the community of solver users. All contributions are welcome, for example here are some ways to help:

Add a solver to the benchmark
Submit a new problem to the free-for-all test set, e.g. one that reflects an application you are working on.
Create a new test set, distribute it in its own repository, and open a PR to reference it here
Share your results from running the benchmark on your machine

Add a solver to the benchmark¶

Add a corresponding entry (in the Added section of the upcoming version) to the changelog
Add the solver name to the IMPLEMENTED_SOLVERS list in solver_settings.py
Add the solver to the solvers pixi feature in pyproject.toml (update the version of qpsolvers if applicable)
Add the solver to the Solvers table in the readme
Set any other relevant solver settings in the define_solver_settings function in test_set.py
Set the solver's absolute tolerance in the set_eps_abs function in solver_settings.py
Set the solver's relative tolerance (if applicable) in the set_eps_rel function in solver_settings.py
Set the solver's time limit (if applicable) in the set_time_limit function in solver_settings.py

Creating a new test set¶

The benchmark is organized into test sets. A test set is a Git repository that contains QP problems and results from running the benchmark (all solvers, all settings) on them. To create a new test set:

Create a new repository for your test set
- You can name your repository X_qpbenchmark to indicate the topic X and that it is a qpbenchmark test set (examples: mpc_qpbenchmark, ik_qpbenchmark, ...)
- Store your QP problem files in a data/ sub-directory
- Create a results/ sub-directory to store the solver performance database and test-set report
Add a main Python file named after the test set, say foo_bar.py
Implement a FooBar class in this file deriving from qpbenchmark.TestSet
- The class name should match the file name in PascalCase

For instance, here is a minimal foo_bar.py for problems saved in data/ with qpbenchmark.Problem.save:

import os

import qpbenchmark
from qpbenchmark.benchmark import main

DATA_DIR = os.path.join(os.path.dirname(os.path.abspath(__file__)), "data")


class FooBar(qpbenchmark.TestSet):
    """Test set of problems stored in the data/ directory."""

    @property
    def description(self) -> str:
        return "Problems arising from foo bar applications."

    @property
    def title(self) -> str:
        return "Foo bar test set"

    @property
    def sparse_only(self) -> bool:
        return False

    def __iter__(self):
        for file_name in sorted(os.listdir(DATA_DIR)):
            if file_name.endswith(".npz"):
                yield qpbenchmark.Problem.load(os.path.join(DATA_DIR, file_name))


if __name__ == "__main__":
    test_set_path = os.path.abspath(__file__)
    test_set_dir = os.path.dirname(test_set_path)
    main(
        test_set_path=test_set_path,
        results_path=f"{test_set_dir}/results/foo_bar.parquet",
    )

You can then check your problems with python foo_bar.py check and run the benchmark over the test set with python foo_bar.py run. You can check out the existing test sets for more complete examples. Once your test set is ready and has generated its results, feel free to add it as a git submodule in the test_sets/ directory.

Solver performance depends on the machine, so we welcome results from a variety of machines. To share yours:

Run a test set as described in the quick start, with the --rerun flag so that all results are computed on your machine.
Check out the report written to the results/ directory of the test set: it records your CPU, solver versions and solver settings.
Post the report in the Results category of the test set's discussions forum, for instance: free-for-all, Maros-Meszaros or model predictive control.

👷 Contributing¶

Add a solver to the benchmark¶

Creating a new test set¶

Share your results¶