As in the summarized results (Comparing Minimizers), the best possible results with Mantid minimizers are given a top score “1” (lowest error, or lowest run time).
The following color codes are used to distinguish different levels of performance, note the irregular (increasing) difference between level boundaries):
The integer numbers 1 and 2 next to the problem names denote the starting point (all NIST problems are usually tested for two different starting points). By clicking on the names of the test problems (leftmost column) you can see the original definition of the problem.
For details on the ranking approach and the color codes see above and the summarized results. When interpreting these results note that the median shown in the summarized results is calculated excluding undefined values. Undefined accuracy values are obtained when the minimizers fail to produce any result and are shown in the detailed results as “nan”. For the more difficult groups of problems the proportion of undefined values is higher for some minimizers. This can bias the statistics shown in the summary table in favor of the minimizers that fail more often, as some of the the hard problems are excluded from the median instead of contributing to a higher median (lower ranking).
Alternatively, see the summary and detailed results when using weighted least squares as cost function.
| BFGS | Conjugate gradient (Fletcher-Reeves imp.) | Conjugate gradient (Polak-Ribiere imp.) | Damping | Levenberg-Marquardt | Levenberg-MarquardtMD | Simplex | SteepestDescent | |
|---|---|---|---|---|---|---|---|---|
| Misra1a 1 | 156.5 | 156.7 | 156.7 | 1 | 1 | 1 | 1 | 156.7 |
| Misra1a 2 | 1.829 | 2.253 | 2.253 | 1 | 1 | 1 | 1 | 2.253 |
| Chwirut2 1 | 1.067 | 1.013 | 1 | 1 | 1 | 1 | 1.068 | 1.066 |
| Chwirut2 2 | 1.049 | 1.002 | 1 | 1 | 1 | 1 | 1.197 | 1.004 |
| Chwirut1 1 | 1.097 | 1.001 | 1 | 1 | 1 | 1 | 1.112 | 1.097 |
| Chwirut1 2 | 1.017 | 1.007 | 1 | 1 | 1 | 1 | 1.109 | 1.017 |
| Lanczos3 1 | 7.683e+06 | 5.433e+04 | 4.963e+04 | 1 | 1 | 1 | 973.7 | 7.797e+04 |
| Lanczos3 2 | 2.652e+05 | 6.206e+04 | 6.264e+04 | 1 | 1 | 1 | 1.807e+04 | 1.285e+05 |
| Gauss1 1 | 5.6 | 1.745 | 1.044 | 1 | 1 | 1 | 1 | 5.6 |
| Gauss1 2 | 1.655 | 9.182 | 9.182 | 1 | 1 | 1 | 1 | 9.168 |
| Gauss2 1 | 7.272 | 7.341 | 7.341 | 1 | 1 | 1 | 1.022 | 7.272 |
| Gauss2 2 | 2.081 | 3.754 | 3.754 | 1 | 1 | 1 | 1 | 3.393 |
| DanWood 1 | 72.41 | 1 | 1 | 1 | 1 | 1 | 1.011 | 55.52 |
| DanWood 2 | 1.244 | 1 | 1 | 1 | 1 | 1 | 1.031 | 1.97 |
| Misra1b 1 | 96.76 | 96.97 | 96.97 | 1 | 1 | 1 | 1 | 96.97 |
| Misra1b 2 | 8.521 | 16.43 | 16.43 | 1 | 1 | 1 | 1 | 16.43 |
| BFGS | Conjugate gradient (Fletcher-Reeves imp.) | Conjugate gradient (Polak-Ribiere imp.) | Damping | Levenberg-Marquardt | Levenberg-MarquardtMD | Simplex | SteepestDescent | |
|---|---|---|---|---|---|---|---|---|
| Kirby2 1 | 423 | 25.07 | 25.37 | 1 | 1 | 1 | 956.3 | 510.9 |
| Kirby2 2 | 2.554 | 1.717 | 1.68 | 1 | 1 | 1 | 1.396 | 121 |
| Hahn1 1 | 4.508e+04 | 45.51 | 217 | 69.31 | 1 | 1 | 200.6 | 4962 |
| Hahn1 2 | 4000 | 35.48 | 211.6 | 1 | 1 | 1 | 157.2 | 4896 |
| MGH17 1 | 1.419e+04 | 1.42e+04 | 1.42e+04 | nan | 1 | 754.5 | 1.421e+04 | 1.413e+04 |
| MGH17 2 | 2473 | 1.414 | 1.422 | 1 | 1 | 1 | 4179 | 1762 |
| Lanczos1 1 | 1.561e+18 | 1.107e+16 | 1.01e+16 | 151.7 | 1 | 21.29 | 2.182e+15 | 1.583e+16 |
| Lanczos1 2 | 1.559e+17 | 3.652e+16 | 3.687e+16 | 1 | 1 | 28.59 | 1.064e+16 | 7.518e+16 |
| Lanczos2 1 | 5.553e+09 | 3.939e+07 | 3.591e+07 | 1 | 1 | 1 | 7.765e+06 | 5.631e+07 |
| Lanczos2 2 | 1.917e+08 | 4.491e+07 | 4.533e+07 | 1 | 1 | 1 | 1.308e+07 | 9.22e+07 |
| Gauss3 1 | 15.19 | 1.815 | 1.568 | 1 | 1 | 1 | 1.036 | 15.19 |
| Gauss3 2 | 3.147 | 2.44 | 2.763 | 7.912 | 1 | 1 | 1.25 | 8.944 |
| Misra1c 1 | 115 | 115.8 | 115.8 | 1 | 1 | 1 | 1 | 115.8 |
| Misra1c 2 | 6.595 | 6.702 | 6.702 | 1 | 1 | 1 | 1 | 6.702 |
| Misra1d 1 | 12.67 | 19.62 | 19.62 | 1 | 1 | 1 | 1 | 19.62 |
| Misra1d 2 | 1.03 | 1.939 | 1.939 | 1 | 1 | 1 | 1 | 1.939 |
| ENSO 1 | 1.259 | 1.025 | 1.024 | 1.524 | 1 | 1.524 | 1.014 | 1.202 |
| ENSO 2 | 1.033 | 1.018 | 1.029 | 1.087 | 1 | 1.087 | 1.028 | 1.052 |
| BFGS | Conjugate gradient (Fletcher-Reeves imp.) | Conjugate gradient (Polak-Ribiere imp.) | Damping | Levenberg-Marquardt | Levenberg-MarquardtMD | Simplex | SteepestDescent | |
|---|---|---|---|---|---|---|---|---|
| MGH09 1 | 7.705 | 25.02 | 25.02 | 1.817e+04 | 3.342 | 1 | 20.62 | 24.19 |
| MGH09 2 | 1.62 | 1.621 | 1.62 | 256.6 | 1 | 1 | 1.025 | 1.604 |
| Thurber 1 | 228.3 | 175.3 | 173.2 | 2.693 | 1 | 1 | 176.5 | 177.2 |
| Thurber 2 | 422.3 | 20.02 | 40.38 | 1 | 1 | 1 | 4.43 | 368.9 |
| BoxBOD 1 | 8.366 | 8.366 | 8.366 | nan | 8.366 | 1 | 8.366 | 1.515 |
| BoxBOD 2 | 33.46 | 8.366 | 8.366 | 1 | 1 | 1 | 8.366 | 6.177 |
| Rat42 1 | 13.33 | 13.27 | 13.27 | 4.759e+06 | 1 | 1 | 16.48 | 193.6 |
| Rat42 2 | 1.331 | 1.213 | 1.302 | 1 | 1 | 1 | 1.429 | 1.449 |
| MGH10 1 | 6.367e+05 | 6.362e+05 | 6.344e+05 | 2.984e+47 | 1 | 5.306e+05 | 6.422e+05 | 6.37e+05 |
| MGH10 2 | 4.08e+04 | 664.2 | 664 | 1.394e+51 | 1 | 41.54 | 4.368e+04 | 4.132e+04 |
| Eckerle4 1 | 478.1 | 478.1 | 478.1 | 366 | 1 | 1 | 478.1 | 478.1 |
| Eckerle4 2 | 31.01 | 1.02 | 1.026 | 1 | 1 | 1 | 1 | 1.003 |
| Rat43 1 | 39.21 | 1.02 | 1.02 | nan | 1.136 | 1 | 51.2 | 21.8 |
| Rat43 2 | 1.07 | 1.001 | 1 | 1 | 1 | 1 | 1 | 1.001 |
| Bennett5 1 | 1294 | 648.7 | 1.028 | 1 | 1 | 1 | 1646 | 1307 |
| Bennett5 2 | 690.3 | 3.347 | 1.165 | 1 | 1 | 1.065 | 1572 | 698.8 |
For details on the ranking approach and the color codes see above and the summarized results. Note that this comparison is approximate and was performed on a particular platform (Ubuntu) and machine.
Alternatively, see the summary and detailed results when using weighted least squares as cost function.
| BFGS | Conjugate gradient (Fletcher-Reeves imp.) | Conjugate gradient (Polak-Ribiere imp.) | Damping | Levenberg-Marquardt | Levenberg-MarquardtMD | Simplex | SteepestDescent | |
|---|---|---|---|---|---|---|---|---|
| Misra1a 1 | 1.042 | 1.03 | 1.046 | 1 | 1.116 | 1.011 | 1.768 | 1.95 |
| Misra1a 2 | 1.008 | 1.036 | 1.107 | 1.005 | 1.011 | 1 | 1.421 | 2.223 |
| Chwirut2 1 | 1.165 | 19.75 | 19.91 | 1 | 1.066 | 1.075 | 1.287 | 10.02 |
| Chwirut2 2 | 1.213 | 22.14 | 22.13 | 1.014 | 1.049 | 1 | 1.359 | 10.83 |
| Chwirut1 1 | 1.691 | 50.26 | 13.62 | 1 | 1.073 | 1.187 | 1.354 | 21.98 |
| Chwirut1 2 | 1.763 | 8.55 | 26.32 | 1 | 1.145 | 1.038 | 1.624 | 24.16 |
| Lanczos3 1 | 1.304 | 3.878 | 4.103 | 1 | 2.617 | 5.288 | 5.068 | 12.83 |
| Lanczos3 2 | 1.379 | 4.23 | 2.776 | 1 | 1.053 | 1.028 | 1.728 | 12.84 |
| Gauss1 1 | 1.427 | 7.891 | 45.86 | 1 | 1.181 | 1.047 | 7.806 | 56.56 |
| Gauss1 2 | 2.962 | 1 | 1.004 | 1.308 | 1.493 | 1.372 | 9.91 | 73.93 |
| Gauss2 1 | 1.991 | 1 | 1.005 | 1.306 | 1.487 | 1.376 | 12.36 | 73.53 |
| Gauss2 2 | 2.526 | 1 | 1.003 | 1.312 | 1.496 | 1.39 | 11.63 | 74.15 |
| DanWood 1 | 1.079 | 1.601 | 1.527 | 1 | 1.006 | 1.034 | 1.158 | 5.121 |
| DanWood 2 | 1.029 | 1.223 | 1.208 | 1 | 1.036 | 1.03 | 1.102 | 5.405 |
| Misra1b 1 | 1.047 | 1.096 | 1.091 | 1.022 | 1.163 | 1 | 1.62 | 2.519 |
| Misra1b 2 | 1.142 | 1.108 | 1.255 | 1.073 | 1 | 1.024 | 1.495 | 2.948 |
| BFGS | Conjugate gradient (Fletcher-Reeves imp.) | Conjugate gradient (Polak-Ribiere imp.) | Damping | Levenberg-Marquardt | Levenberg-MarquardtMD | Simplex | SteepestDescent | |
|---|---|---|---|---|---|---|---|---|
| Kirby2 1 | 1.463 | 37.51 | 37.56 | 1 | 1.058 | 1.027 | 2.042 | 21.9 |
| Kirby2 2 | 1.355 | 38.81 | 39.63 | 1.025 | 1.104 | 1 | 2.745 | 23.81 |
| Hahn1 1 | 1.884 | 41.9 | 21.86 | 2.089 | 1.006 | 1 | 1.53 | 23 |
| Hahn1 2 | 1.298 | 52.93 | 22.61 | 1 | 1.128 | 1.08 | 5.577 | 28.7 |
| MGH17 1 | 1 | 1.184 | 1.169 | 1.448 | 11.24 | 5.252 | 1.553 | 9.74 |
| MGH17 2 | 1.185 | 14.82 | 15.27 | 1 | 1.283 | 1.454 | 1.421 | 12.09 |
| Lanczos1 1 | 1.268 | 3.7 | 3.995 | 1 | 2.5 | 5.376 | 4.295 | 12.5 |
| Lanczos1 2 | 1.392 | 4.178 | 2.727 | 1 | 1.064 | 1.018 | 1.742 | 12.84 |
| Lanczos2 1 | 1.317 | 3.918 | 4.009 | 1 | 2.495 | 5.654 | 4.272 | 12.6 |
| Lanczos2 2 | 1.401 | 4.34 | 2.706 | 1 | 1.056 | 1.043 | 1.759 | 13.1 |
| Gauss3 1 | 1.336 | 55.55 | 49.51 | 1 | 1.136 | 1.056 | 8.739 | 51.18 |
| Gauss3 2 | 1.661 | 45.89 | 11.86 | 45.69 | 1.116 | 1 | 7.838 | 46.25 |
| Misra1c 1 | 1.121 | 1.09 | 1.024 | 1.059 | 1 | 1.008 | 1.535 | 3.129 |
| Misra1c 2 | 1.095 | 1.098 | 1.085 | 1 | 1.019 | 1.019 | 1.476 | 2.974 |
| Misra1d 1 | 1.062 | 1.066 | 1.082 | 1 | 1.008 | 1.018 | 1.399 | 2.411 |
| Misra1d 2 | 1.075 | 1.096 | 1.084 | 1 | 1.008 | 1.094 | 1.374 | 2.042 |
| ENSO 1 | 2.804 | 185.3 | 173.5 | 1 | 137.4 | 1.046 | 20.45 | 129.8 |
| ENSO 2 | 5.554 | 183.9 | 112.1 | 1.022 | 138 | 1 | 16.54 | 118.1 |
| BFGS | Conjugate gradient (Fletcher-Reeves imp.) | Conjugate gradient (Polak-Ribiere imp.) | Damping | Levenberg-Marquardt | Levenberg-MarquardtMD | Simplex | SteepestDescent | |
|---|---|---|---|---|---|---|---|---|
| MGH09 1 | 1.149 | 1 | 1.038 | 1.658 | 1.486 | 2.59 | 1.214 | 1.537 |
| MGH09 2 | 1 | 1.058 | 1.041 | 1.81 | 1.057 | 1.212 | 1.728 | 3.476 |
| Thurber 1 | 1 | 6.162 | 24.11 | 1.518 | 1.145 | 1.183 | 1.255 | 11.6 |
| Thurber 2 | 1.098 | 15.26 | 25.03 | 1.517 | 1 | 1.065 | 4.53 | 13.11 |
| BoxBOD 1 | 1.041 | 1.097 | 1.114 | 1.5 | 1 | 1.333 | 1.297 | 4.698 |
| BoxBOD 2 | 1.026 | 1.076 | 1.095 | 1.002 | 1 | 1.062 | 3.725 | 4.483 |
| Rat42 1 | 1.18 | 1.509 | 1.407 | 1 | 1.039 | 1.169 | 1.167 | 5.347 |
| Rat42 2 | 1 | 7.957 | 1.876 | 1.005 | 1.018 | 1.012 | 1.161 | 5.295 |
| MGH10 1 | 1 | 2.99 | 2.939 | 1.231 | 6.398 | 5.026 | 1.048 | 3.473 |
| MGH10 2 | 1 | 2.035 | 2.433 | 1.925 | 5.985 | 5.84 | 1.122 | 6.092 |
| Eckerle4 1 | 1 | 1.088 | 1.088 | 1.258 | 1.171 | 5.581 | 1.127 | 1.39 |
| Eckerle4 2 | 1.014 | 1.644 | 1.699 | 1 | 1.049 | 1.033 | 1.197 | 3.224 |
| Rat43 1 | 1.115 | 15.44 | 3.682 | 1.431 | 1 | 1.14 | 1.314 | 7.362 |
| Rat43 2 | 1.002 | 1.548 | 4.422 | 1.019 | 1 | 1.051 | 1.183 | 8.667 |
| Bennett5 1 | 1.173 | 40.41 | 4.457 | 1.017 | 1 | 15.08 | 1.181 | 13.89 |
| Bennett5 2 | 1.233 | 43.28 | 4.892 | 1 | 17.72 | 14.75 | 1.246 | 14.83 |