Free-for-all test set¶
| Number of problems | 28 |
|---|---|
| Benchmark version | 2.7.1 |
| Date | 2026-07-23 16:20:46.720828+00:00 |
| CPU | AMD Ryzen 7 8845HS w/ Radeon 780M Graphics |
| Run by | @stephane-caron |
| Results file | free_for_all.parquet |
| Results checksum | bd181b93bf088a257d7ce5111d0b9cf002739e125a6c7b5bf9b990699b9911f8 |
Benchmark reports are copious as we aim to document comparison factors as much as possible. You can also jump to results directly.
Contents¶
Description¶
Community-built test set to benchmark QP solvers.
Solvers¶
| solver | version |
|---|---|
| clarabel | 0.11.1 |
| cvxopt | 1.3.3 |
| gurobi | 13.0.2 (size-limited) |
| highs | 1.15.1 |
| kvxopt | 1.3.3.1 |
| osqp | 1.1.3 |
| piqp | 0.6.3 |
| proxqp | 0.7.3 |
| qpalm | 1.2.6 |
| scs | 3.2.11 |
| sip | 0.0.2 |
All solvers were called via qpsolvers v4.13.0.
Results by settings¶
Default settings¶
Solvers are compared over the whole test set by shifted geometric mean (shm). Lower is better, 1.0 is the best.
| Success rate (%) | Runtime (shm) | Primal residual (shm) | Dual residual (shm) | Duality gap (shm) | |
|---|---|---|---|---|---|
| clarabel | 92.9 | 2.4 | 1.0 | 1.0 | 1.0 |
| cvxopt | 67.9 | 6.6 | 1.0 | 1.0 | 1.0 |
| gurobi | 60.7 | 5.7 | 1.0 | 1.0 | 1.0 |
| highs | 67.9 | 3.5 | 1.0 | 20.3 | 1.0 |
| kvxopt | 75.0 | 12.4 | 1.0 | 1.0 | 1.0 |
| osqp | 78.6 | 1.9 | 1.0 | 3.1 | 1.0 |
| piqp | 100.0 | 1.0 | 1.0 | 1.0 | 1.0 |
| proxqp | 75.0 | 3.0 | 1.0 | 1.0 | 1.0 |
| qpalm | 92.9 | 1.2 | 1.0 | 1.0 | 1.0 |
| scs | 78.6 | 3.3 | 1.0 | 1.0 | 1.0 |
| sip | 89.3 | 1.6 | 1.0 | 1.0 | 1.0 |
High accuracy settings¶
Solvers are compared over the whole test set by shifted geometric mean (shm). Lower is better, 1.0 is the best.
| Success rate (%) | Runtime (shm) | Primal residual (shm) | Dual residual (shm) | Duality gap (shm) | |
|---|---|---|---|---|---|
| clarabel | 92.9 | 1.2 | 1.0 | 1.0 | 1.0 |
| cvxopt | 17.9 | 3.3 | 1.0 | 1.0 | 99068.8 |
| gurobi | 25.0 | 2.8 | 1.0 | 2038629.1 | 408.7 |
| highs | 0.0 | 1.8 | 1.0 | 17777465770.7 | 17263445.2 |
| kvxopt | 17.9 | 6.2 | 1.0 | 1.0 | 99094.4 |
| osqp | 57.1 | 3.2 | 1.0 | 1.0 | 1.0 |
| piqp | 92.9 | 1.0 | 1.0 | 1.0 | 1.0 |
| proxqp | 67.9 | 2.3 | 1.0 | 1.0 | 1.0 |
| qpalm | 64.3 | 1.4 | 1.0 | 1.0 | 1.9 |
| scs | 71.4 | 1.7 | 1.1 | 1.0 | 1.0 |
| sip | 82.1 | 1.3 | 1.0 | 1.0 | 1.0 |
Low accuracy settings¶
Solvers are compared over the whole test set by shifted geometric mean (shm). Lower is better, 1.0 is the best.
| Success rate (%) | Runtime (shm) | Primal residual (shm) | Dual residual (shm) | Duality gap (shm) | |
|---|---|---|---|---|---|
| clarabel | 92.9 | 2.8 | 1.0 | 1.0 | 1.0 |
| cvxopt | 64.3 | 8.6 | 1.0 | 1.0 | 1.0 |
| gurobi | 57.1 | 7.4 | 1.0 | 3.0 | 1.0 |
| highs | 53.6 | 3.2 | 1.0 | 17780.0 | 89.6 |
| kvxopt | 71.4 | 16.2 | 1.0 | 1.0 | 1.0 |
| osqp | 78.6 | 3.8 | 1.0 | 1.0 | 1.0 |
| piqp | 100.0 | 1.0 | 1.0 | 1.0 | 1.0 |
| proxqp | 78.6 | 3.8 | 1.0 | 1.0 | 1.0 |
| qpalm | 60.7 | 2.3 | 1.0 | 1.0 | 4.9 |
| scs | 78.6 | 4.1 | 1.0 | 1.0 | 1.0 |
| sip | 85.7 | 2.6 | 1.0 | 1.0 | 1.0 |
Mid accuracy settings¶
Solvers are compared over the whole test set by shifted geometric mean (shm). Lower is better, 1.0 is the best.
| Success rate (%) | Runtime (shm) | Primal residual (shm) | Dual residual (shm) | Duality gap (shm) | |
|---|---|---|---|---|---|
| clarabel | 92.9 | 1.7 | 1.0 | 1.0 | 1.0 |
| cvxopt | 46.4 | 4.9 | 1.0 | 1.0 | 99.8 |
| gurobi | 53.6 | 4.2 | 1.0 | 2039.6 | 1.3 |
| highs | 14.3 | 2.6 | 1.0 | 17777500.3 | 17264.2 |
| kvxopt | 53.6 | 9.2 | 1.0 | 1.0 | 99.8 |
| osqp | 64.3 | 3.8 | 1.0 | 1.0 | 1.0 |
| piqp | 96.4 | 1.0 | 1.0 | 1.0 | 1.0 |
| proxqp | 75.0 | 2.6 | 1.0 | 1.0 | 1.0 |
| qpalm | 71.4 | 1.6 | 1.0 | 1.0 | 2.5 |
| scs | 75.0 | 2.6 | 1.0 | 1.0 | 1.0 |
| sip | 85.7 | 1.5 | 1.0 | 1.0 | 10.4 |
Results by metric¶
Success rate¶
Precentage of problems each solver is able to solve:
| default | high_accuracy | low_accuracy | mid_accuracy | |
|---|---|---|---|---|
| clarabel | 93 | 93 | 93 | 93 |
| cvxopt | 68 | 18 | 64 | 46 |
| gurobi | 61 | 25 | 57 | 54 |
| highs | 68 | 0 | 54 | 14 |
| kvxopt | 75 | 18 | 71 | 54 |
| osqp | 79 | 57 | 79 | 64 |
| piqp | 100 | 93 | 100 | 96 |
| proxqp | 75 | 68 | 79 | 75 |
| qpalm | 93 | 64 | 61 | 71 |
| scs | 79 | 71 | 79 | 75 |
| sip | 89 | 82 | 86 | 86 |
Rows are solvers and columns are settings. We consider that a solver successfully solved a problem when (1) it returned with a success status and (2) its solution satisfies optimality conditions within tolerance. The second table below summarizes the frequency at which solvers return success (1) and the corresponding solution did indeed pass tolerance checks.
Percentage of problems where "solved" return codes are correct:
| default | high_accuracy | low_accuracy | mid_accuracy | |
|---|---|---|---|---|
| clarabel | 100 | 100 | 100 | 100 |
| cvxopt | 100 | 50 | 96 | 79 |
| gurobi | 100 | 64 | 96 | 93 |
| highs | 93 | 25 | 71 | 39 |
| kvxopt | 100 | 43 | 96 | 79 |
| osqp | 93 | 100 | 100 | 100 |
| piqp | 100 | 100 | 100 | 100 |
| proxqp | 96 | 100 | 100 | 100 |
| qpalm | 100 | 82 | 71 | 86 |
| scs | 100 | 96 | 100 | 96 |
| sip | 100 | 100 | 96 | 96 |
Computation time¶
We compare solver computation times over the whole test set using the shifted geometric mean. Intuitively, a solver with a shifted-geometric-mean runtime of Y is Y times slower than the best solver over the test set. See Metrics for details.
Shifted geometric mean of solver computation times (1.0 is the best):
| default | high_accuracy | low_accuracy | mid_accuracy | |
|---|---|---|---|---|
| clarabel | 2.4 | 1.2 | 2.8 | 1.7 |
| cvxopt | 6.6 | 3.3 | 8.6 | 4.9 |
| gurobi | 5.7 | 2.8 | 7.4 | 4.2 |
| highs | 3.5 | 1.8 | 3.2 | 2.6 |
| kvxopt | 12.4 | 6.2 | 16.2 | 9.2 |
| osqp | 1.9 | 3.2 | 3.8 | 3.8 |
| piqp | 1.0 | 1.0 | 1.0 | 1.0 |
| proxqp | 3.0 | 2.3 | 3.8 | 2.6 |
| qpalm | 1.2 | 1.4 | 2.3 | 1.6 |
| scs | 3.3 | 1.7 | 4.1 | 2.6 |
| sip | 1.6 | 1.3 | 2.6 | 1.5 |
Rows are solvers and columns are solver settings. The shift is sh = 10. As in the OSQP and ProxQP benchmarks, we assume a solver's run time is at the time limit when it fails to solve a problem.
Optimality conditions¶
Primal residual¶
The primal residual measures the maximum (equality and inequality) constraint violation in the solution returned by a solver. We use the shifted geometric mean to compare solver primal residuals over the whole test set. Intuitively, a solver with a shifted-geometric-mean primal residual of Y is Y times less precise on constraints than the best solver over the test set. See Metrics for details.
Shifted geometric means of primal residuals (1.0 is the best):
| default | high_accuracy | low_accuracy | mid_accuracy | |
|---|---|---|---|---|
| clarabel | 1.0 | 1.0 | 1.0 | 1.0 |
| cvxopt | 1.0 | 1.0 | 1.0 | 1.0 |
| gurobi | 1.0 | 1.0 | 1.0 | 1.0 |
| highs | 1.0 | 1.0 | 1.0 | 1.0 |
| kvxopt | 1.0 | 1.0 | 1.0 | 1.0 |
| osqp | 1.0 | 1.0 | 1.0 | 1.0 |
| piqp | 1.0 | 1.0 | 1.0 | 1.0 |
| proxqp | 1.0 | 1.0 | 1.0 | 1.0 |
| qpalm | 1.0 | 1.0 | 1.0 | 1.0 |
| scs | 1.0 | 1.1 | 1.0 | 1.0 |
| sip | 1.0 | 1.0 | 1.0 | 1.0 |
Rows are solvers and columns are solver settings. The shift is sh = 10. A solver that fails to find a solution receives a primal residual equal to the full primal tolerance.
Dual residual¶
The dual residual measures the maximum violation of the dual feasibility condition in the solution returned by a solver. We use the shifted geometric mean to compare solver dual residuals over the whole test set. Intuitively, a solver with a shifted-geometric-mean dual residual of Y is Y times less precise on the dual feasibility condition than the best solver over the test set. See Metrics for details.
Shifted geometric means of dual residuals (1.0 is the best):
| default | high_accuracy | low_accuracy | mid_accuracy | |
|---|---|---|---|---|
| clarabel | 1.0 | 1.0 | 1.0 | 1.0 |
| cvxopt | 1.0 | 1.0 | 1.0 | 1.0 |
| gurobi | 1.0 | 2038629.1 | 3.0 | 2039.6 |
| highs | 20.3 | 17777465770.7 | 17780.0 | 17777500.3 |
| kvxopt | 1.0 | 1.0 | 1.0 | 1.0 |
| osqp | 3.1 | 1.0 | 1.0 | 1.0 |
| piqp | 1.0 | 1.0 | 1.0 | 1.0 |
| proxqp | 1.0 | 1.0 | 1.0 | 1.0 |
| qpalm | 1.0 | 1.0 | 1.0 | 1.0 |
| scs | 1.0 | 1.0 | 1.0 | 1.0 |
| sip | 1.0 | 1.0 | 1.0 | 1.0 |
Rows are solvers and columns are solver settings. The shift is sh = 10. A solver that fails to find a solution receives a dual residual equal to the full dual tolerance.
Duality gap¶
The duality gap measures the consistency of the primal and dual solutions returned by a solver. A duality gap close to zero ensures that the complementarity slackness optimality condition is satisfied. We use the shifted geometric mean to compare solver duality gaps over the whole test set. Intuitively, a solver with a shifted-geometric-mean duality gap of Y is Y times less precise on the complementarity slackness condition than the best solver over the test set. See Metrics for details.
Shifted geometric means of duality gaps (1.0 is the best):
| default | high_accuracy | low_accuracy | mid_accuracy | |
|---|---|---|---|---|
| clarabel | 1.0 | 1.0 | 1.0 | 1.0 |
| cvxopt | 1.0 | 99068.8 | 1.0 | 99.8 |
| gurobi | 1.0 | 408.7 | 1.0 | 1.3 |
| highs | 1.0 | 17263445.2 | 89.6 | 17264.2 |
| kvxopt | 1.0 | 99094.4 | 1.0 | 99.8 |
| osqp | 1.0 | 1.0 | 1.0 | 1.0 |
| piqp | 1.0 | 1.0 | 1.0 | 1.0 |
| proxqp | 1.0 | 1.0 | 1.0 | 1.0 |
| qpalm | 1.0 | 1.9 | 4.9 | 2.5 |
| scs | 1.0 | 1.0 | 1.0 | 1.0 |
| sip | 1.0 | 1.0 | 1.0 | 10.4 |
Rows are solvers and columns are solver settings. The shift is sh = 10. A solver that fails to find a solution receives a duality gap equal to the full gap tolerance.
Settings¶
There are 4 settings: default, high_accuracy, low_accuracy and mid_accuracy. They validate solutions using the following tolerances:
| tolerance | default | high_accuracy | low_accuracy | mid_accuracy |
|---|---|---|---|---|
dual |
1 | 1e-09 | 0.001 | 1e-06 |
gap |
1 | 1e-09 | 0.001 | 1e-06 |
primal |
1 | 1e-09 | 0.001 | 1e-06 |
runtime |
10 | 10 | 10 | 10 |
Solvers for each settings are configured as follows:
| solver | parameter | default | high_accuracy | low_accuracy | mid_accuracy |
|---|---|---|---|---|---|
| clarabel | tol_feas |
- | 1e-09 | 0.001 | 1e-06 |
| clarabel | tol_gap_abs |
- | 1e-09 | 0.001 | 1e-06 |
| clarabel | tol_gap_rel |
- | 0 | 0 | 0 |
| cvxopt | feastol |
- | 1e-09 | 0.001 | 1e-06 |
| gurobi | FeasibilityTol |
- | 1e-09 | 0.001 | 1e-06 |
| gurobi | OptimalityTol |
- | 1e-09 | 0.001 | 1e-06 |
| gurobi | TimeLimit |
10.0 | 10 | 10 | 10 |
| highs | dual_feasibility_tolerance |
- | 1e-09 | 0.001 | 1e-06 |
| highs | primal_feasibility_tolerance |
- | 1e-09 | 0.001 | 1e-06 |
| highs | time_limit |
10.0 | 10 | 10 | 10 |
| kvxopt | feastol |
- | 1e-09 | 0.001 | 1e-06 |
| osqp | eps_abs |
- | 1e-09 | 0.001 | 1e-06 |
| osqp | eps_rel |
- | 0 | 0 | 0 |
| osqp | time_limit |
10.0 | 10 | 10 | 10 |
| piqp | check_duality_gap |
- | 1 | 1 | 1 |
| piqp | eps_abs |
- | 1e-09 | 0.001 | 1e-06 |
| piqp | eps_duality_gap_abs |
- | 1e-09 | 0.001 | 1e-06 |
| piqp | eps_duality_gap_rel |
- | 0 | 0 | 0 |
| piqp | eps_rel |
- | 0 | 0 | 0 |
| proxqp | check_duality_gap |
- | 1 | 1 | 1 |
| proxqp | eps_abs |
- | 1e-09 | 0.001 | 1e-06 |
| proxqp | eps_duality_gap_abs |
- | 1e-09 | 0.001 | 1e-06 |
| proxqp | eps_duality_gap_rel |
- | 0 | 0 | 0 |
| proxqp | eps_rel |
- | 0 | 0 | 0 |
| qpalm | eps_abs |
- | 1e-09 | 0.001 | 1e-06 |
| qpalm | eps_rel |
- | 0 | 0 | 0 |
| qpalm | time_limit |
10.0 | 10 | 10 | 10 |
| scs | eps_abs |
- | 1e-09 | 0.001 | 1e-06 |
| scs | eps_rel |
- | 0 | 0 | 0 |
| scs | time_limit_secs |
10.0 | 10 | 10 | 10 |
| sip | eps_abs |
- | 1e-09 | 0.001 | 1e-06 |
| sip | eps_rel |
- | 0 | 0 | 0 |
| sip | time_limit |
10.0 | 10 | 10 | 10 |
Known limitations¶
The following issues have been identified as impacting the fairness of this benchmark. Keep them in mind when drawing conclusions from the results.
CPU info¶
| Property | Value |
|---|---|
arch |
X86_64 |
arch_string_raw |
x86_64 |
bits |
64 |
brand_raw |
AMD Ryzen 7 8845HS w/ Radeon 780M Graphics |
count |
16 |
family |
25 |
flags |
3dnowext, 3dnowprefetch, abm, adx, aes, amd_lbr_v2, aperfmperf, apic, arat, avx, avx2, avx512_bf16, avx512_bitalg, avx512_vbmi2, avx512_vnni, avx512_vpopcntdq, avx512bitalg, avx512bw, avx512cd, avx512dq, avx512f, avx512ifma, avx512vbmi, avx512vbmi2, avx512vl, avx512vnni, avx512vpopcntdq, bmi1, bmi2, bpext, cat_l3, cdp_l3, clflush, clflushopt, clwb, clzero, cmov, cmp_legacy, constant_tsc, cpb, cppc, cpuid, cpuid_fault, cqm, cqm_llc, cqm_mbm_local, cqm_mbm_total, cqm_occup_llc, cr8_legacy, cx16, cx8, dbx, de, decodeassists, erms, extapic, extd_apicid, f16c, flush_l1d, flushbyasid, fma, fpu, fsgsbase, fsrm, fxsr, fxsr_opt, gfni, ht, hw_pstate, ibpb, ibrs, ibrs_enhanced, ibs, invpcid, irperf, lahf_lm, lbrv, lm, mba, mca, mce, misalignsse, mmx, mmxext, monitor, movbe, msr, mtrr, mwaitx, nonstop_tsc, nopl, npt, nrip_save, nx, ospke, osvw, osxsave, overflow_recov, pae, pat, pausefilter, pci_l2i, pclmulqdq, pdpe1gb, perfctr_core, perfctr_llc, perfctr_nb, perfmon_v2, pfthreshold, pge, pku, pni, popcnt, pqe, pqm, pse, pse36, rapl, rdpid, rdpru, rdrand, rdrnd, rdseed, rdt_a, rdtscp, rep_good, sep, sha, sha_ni, skinit, smap, smca, smep, ssbd, sse, sse2, sse4_1, sse4_2, sse4a, ssse3, stibp, succor, svm, svm_lock, syscall, tce, topoext, tsc, tsc_scale, umip, user_shstk, v_spec_ctrl, vaes, vgif, vmcb_clean, vme, vmmcall, vnmi, vpclmulqdq, wbnoinvd, wdt, x2apic, x2avic, xgetbv1, xsave, xsavec, xsaveerptr, xsaveopt, xsaves, xtopology |
l1_data_cache_size |
262144 |
l1_instruction_cache_size |
262144 |
l2_cache_associativity |
6 |
l2_cache_line_size |
1024 |
l2_cache_size |
8388608 |
l3_cache_size |
1048576 |
model |
117 |
python_version |
3.14.6.final.0 (64 bit) |
stepping |
2 |
vendor_id_raw |
AuthenticAMD |