Comparison of minimizers for individual test problems

As in the summarized results (Comparing Minimizers), the best possible results with Mantid minimizers are given a top score “1” (lowest error, or lowest run time).

The following color codes are used to distinguish different levels of performance, note the irregular (increasing) difference between level boundaries):

  • best or within 10% (ranking < 1.1)
  • within 33% over the best (1.1 < ranking < 1.33)
  • within 75% over the best (1.33 < ranking < 1.75)
  • within 200% over the best (1.75 < ranking < 3)
  • over 200% of best (ranking > 3)

The integer numbers 1 and 2 next to the problem names denote the starting point (all NIST problems are usually tested for two different starting points). By clicking on the names of the test problems (leftmost column) you can see the original definition of the problem.

Comparison in terms of accuracy

For details on the ranking approach and the color codes see above and the summarized results. When interpreting these results note that the median shown in the summarized results is calculated excluding undefined values. Undefined accuracy values are obtained when the minimizers fail to produce any result and are shown in the detailed results as “nan”. For the more difficult groups of problems the proportion of undefined values is higher for some minimizers. This can bias the statistics shown in the summary table in favor of the minimizers that fail more often, as some of the the hard problems are excluded from the median instead of contributing to a higher median (lower ranking).

Alternatively, see the summary and detailed results when using weighted least squares as cost function.

Accuracy for individual NIST problems, “lower” difficulty

  BFGS Conjugate gradient (Fletcher-Reeves imp.) Conjugate gradient (Polak-Ribiere imp.) Damping Levenberg-Marquardt Levenberg-MarquardtMD Simplex SteepestDescent Trust Region
Misra1a 1 156.5 156.7 156.7 1 1 1 1 156.7 1
Misra1a 2 1.829 2.253 2.253 1 1 1 1 2.253 1
Chwirut2 1 1.067 1.013 1 1 1 1 1.068 1.066 1
Chwirut2 2 1.049 1.002 1 1 1 1 1.197 1.004 1
Chwirut1 1 1.097 1.001 1 1 1 1 1.112 1.097 1
Chwirut1 2 1.017 1.007 1 1 1 1 1.109 1.017 1
Lanczos3 1 7.683e+06 5.433e+04 4.963e+04 1 1 1 973.7 7.797e+04 1
Lanczos3 2 2.652e+05 6.206e+04 6.264e+04 1 1 1 1.807e+04 1.285e+05 1
Gauss1 1 5.6 1.745 1.044 1 1 1 1 5.6 1
Gauss1 2 1.655 9.182 9.182 1 1 1 1 9.168 1
Gauss2 1 7.272 7.341 7.341 1 1 1 1.022 7.272 1
Gauss2 2 2.081 3.754 3.754 1 1 1 1 3.393 1
DanWood 1 72.41 1 1 1 1 1 1.011 55.52 1
DanWood 2 1.244 1 1 1 1 1 1.031 1.97 1
Misra1b 1 96.76 96.97 96.97 1 1 1 1 96.97 1
Misra1b 2 8.521 16.43 16.43 1 1 1 1 16.43 1

Accuracy for individual NIST problems, “average” difficulty

  BFGS Conjugate gradient (Fletcher-Reeves imp.) Conjugate gradient (Polak-Ribiere imp.) Damping Levenberg-Marquardt Levenberg-MarquardtMD Simplex SteepestDescent Trust Region
Kirby2 1 423 25.07 25.37 1 1 1 956.3 510.9 1
Kirby2 2 2.554 1.717 1.68 1 1 1 1.396 121 1
Hahn1 1 4.508e+04 45.51 217 69.31 1 1 200.6 4962 17.97
Hahn1 2 4000 35.48 211.6 1 1 1 157.2 4896 226.2
MGH17 1 1.419e+04 1.42e+04 1.42e+04 nan 1 754.5 1.421e+04 1.413e+04 1.421e+04
MGH17 2 2473 1.414 1.422 1 1 1 4179 1762 1
Lanczos1 1 1.561e+18 1.107e+16 1.01e+16 151.7 1 21.29 2.182e+15 1.583e+16 9.922e+08
Lanczos1 2 1.559e+17 3.652e+16 3.687e+16 1 1 28.59 1.064e+16 7.518e+16 2.033e+06
Lanczos2 1 5.553e+09 3.939e+07 3.591e+07 1 1 1 7.765e+06 5.631e+07 3.949
Lanczos2 2 1.917e+08 4.491e+07 4.533e+07 1 1 1 1.308e+07 9.22e+07 1.002
Gauss3 1 15.19 1.815 1.568 1 1 1 1.036 15.19 1
Gauss3 2 3.147 2.44 2.763 7.912 1 1 1.25 8.944 1
Misra1c 1 115 115.8 115.8 1 1 1 1 115.8 1
Misra1c 2 6.595 6.702 6.702 1 1 1 1 6.702 1
Misra1d 1 12.67 19.62 19.62 1 1 1 1 19.62 1
Misra1d 2 1.03 1.939 1.939 1 1 1 1 1.939 1
ENSO 1 1.264 1.029 1.028 1.529 1.004 1.529 1.018 1.207 1
ENSO 2 1.033 1.018 1.029 1.087 1 1.087 1.029 1.052 1

Accuracy for individual NIST problems, “higher” difficulty

  BFGS Conjugate gradient (Fletcher-Reeves imp.) Conjugate gradient (Polak-Ribiere imp.) Damping Levenberg-Marquardt Levenberg-MarquardtMD Simplex SteepestDescent Trust Region
MGH09 1 7.705 25.02 25.02 1.817e+04 3.342 1 20.62 24.19 3.347
MGH09 2 1.62 1.621 1.62 256.6 1 1 1.025 1.604 1
Thurber 1 228.3 175.3 173.2 2.693 1 1 176.5 177.2 1
Thurber 2 422.3 20.02 40.38 1 1 1 4.43 368.9 1
BoxBOD 1 8.366 8.366 8.366 nan 8.366 1 8.366 1.515 8.366
BoxBOD 2 33.46 8.366 8.366 1 1 1 8.366 6.177 1
Rat42 1 13.33 13.27 13.27 4.759e+06 1 1 16.48 193.6 1
Rat42 2 1.331 1.213 1.302 1 1 1 1.429 1.449 1
MGH10 1 6.367e+05 6.362e+05 6.344e+05 2.984e+47 1 5.306e+05 6.422e+05 6.37e+05 6.271e+05
MGH10 2 7.834e+04 1275 1275 2.677e+51 1.92 79.76 8.385e+04 7.934e+04 1
Eckerle4 1 478.1 478.1 478.1 366 1 1 478.1 478.1 340.6
Eckerle4 2 31.01 1.02 1.026 1 1 1 1 1.003 1
Rat43 1 93.81 2.441 2.44 nan 2.717 2.393 122.5 52.15 1
Rat43 2 1.07 1.001 1 1 1 1 1 1.001 1
Bennett5 1 1294 648.7 1.028 1 1 1 1646 1307 1
Bennett5 2 690.3 3.347 1.165 1 1 1.065 1572 698.8 1

Accuracy for individual “CUTEst” problems

  BFGS Conjugate gradient (Fletcher-Reeves imp.) Conjugate gradient (Polak-Ribiere imp.) Damping Levenberg-Marquardt Levenberg-MarquardtMD Simplex SteepestDescent Trust Region
PALMER6C 372.9 156 179.2 1.367e+06 1 1.367e+06 373 491.7 nan
PALMER7C 67.84 14.16 16.92 4.478e+05 2415 4.478e+05 74.4 115.4 1
PALMER8C 224.5 20.27 20.86 3.601e+05 338 3.601e+05 118.8 345.2 1
YFITU 4.981e+14 1.495e+14 2.976e+13 1 1 1 4.094e+04 3.382e+14 1
VESUVIOLS 1.613 1.611 1.611 1.508 1 1.423 4.669 1.611 1
DMN15102LS 8.623 4.69 4.412 556.8 4.822 1.336 6.504 4.911 1

Accuracy for individual Neutron Data problems

  BFGS Conjugate gradient (Fletcher-Reeves imp.) Conjugate gradient (Polak-Ribiere imp.) Damping Levenberg-Marquardt Levenberg-MarquardtMD Simplex SteepestDescent Trust Region
ENGINX 193749 calibration, spectrum 651, peak 19 1.047 1.047 1.047 2.532e+09 1 1.024 1.045 1.399 1.001
ENGINX 193749 calibration, spectrum 651, peak 20 1.478 1.459 1.469 nan 1 1.48 1.457 6.097 1
ENGINX 193749 calibration, spectrum 651, peak 23 1.079 1.096 1.064 nan 1.146 1.089 1.055 11.32 1
ENGINX 193749 calibration, spectrum 651, peak 5 1.286 1.225 1.224 2.34 1 1.003 1.204 1.454 1
ENGINX 193749 calibration, spectrum 651, peak 6 1.047 1.004 1.005 nan 1.094 1.222 1 1.481 1.093
ENGINX 236516 vanadium, bank 1, 10 brk 1 1 1 1.138 1 1.138 1.023 1 1
ENGINX 236516 vanadium, bank 1, 30 brk 1 1 1 1.138 1 1.138 1.106 1 1
EVS14188-90_processed Gaussian peaks 1 1.634 1.63 1.63 2.261 1.723 1 2.396 2.4 1
EVS14188-90_processed Gaussian peaks 1 1.538 1.538 1.538 1 1.063 1 2.337 2.344 1
GEM 13 IC peaks 38.01 5.84 3.141 nan 1.06 1 14.9 37.32 1
WISH17701 tube 3 calibration, peak 1 1.005 5.036 5.081 1 13.44 1 5.281 7.485 1
WISH17701 tube 3 calibration, peak 2 1 1.267 1.189 1 1 1 1.511 3.546 1
WISH17701 tube 3 calibration, peak 3 2.604 10.96 9.291 1 1 1 11.54 21.39 1
WISH17701 tube 3 calibration, peak 4 16.21 16.31 17.19 1 1 1 18.63 36.03 1
WISH17701 tube 3 calibration, peak 5 1.667 1.871 1.643 1 1 1 2.334 3.402 1
WISH17701 tube 3 calibration, peak 6 3.922 3.822 3.923 1 1 1 5.876 16.78 1
WISH17701 tube 3 calibration, peak 7 4.736 1.344 4.82 2.948e+06 1 1 5.245 6.1 1
WISH17701 tube 3 calibration, peak 8 2.154 2.198 1.992 1 1 1 2.799 3.845 1
WISH17701 tube 3 calibration, peak 9 1.133 1.132 1.28 1 1 1 1.283 1.594 1

Comparison in terms of run time

For details on the ranking approach and the color codes see above and the summarized results. Note that this comparison is approximate and was performed on a particular platform (Ubuntu) and machine.

Alternatively, see the summary and detailed results when using weighted least squares as cost function.

Run time for individual NIST problems, “lower” difficulty

  BFGS Conjugate gradient (Fletcher-Reeves imp.) Conjugate gradient (Polak-Ribiere imp.) Damping Levenberg-Marquardt Levenberg-MarquardtMD Simplex SteepestDescent Trust Region
Misra1a 1 3.309 1 1.123 1.037 1.157 1.06 1.552 1.605 1.254
Misra1a 2 1.037 1.116 1.187 1.01 1.079 1.011 1.182 1.692 1
Chwirut2 1 2.904 12.18 12.32 1.004 1 1.017 1.226 6.411 10.04
Chwirut2 2 1.144 14.81 14.24 1.001 1.077 1 1.132 6.965 10.61
Chwirut1 1 2.805 35.57 9.038 1 1.042 1.169 1.185 15.05 20.69
Chwirut1 2 1.511 6.314 17.72 1.059 1.206 1 1.413 17.15 25.93
Lanczos3 1 1.997 2.014 2.13 1 1.56 2.577 2.396 5.87 1.522
Lanczos3 2 1.358 3.221 2.311 1.013 1 1.04 1.593 9.186 1.126
Gauss1 1 1.854 6.151 33.63 1.106 1 1.025 5.618 41.71 2.911
Gauss1 2 2.692 1.021 1 1.275 1.471 1.302 7.81 58.05 1.899
Gauss2 1 2.597 1 1.347 1.336 1.381 1.415 9.748 60.03 72.46
Gauss2 2 2.251 1 1.263 1.296 1.399 1.366 9.315 60.99 71.38
DanWood 1 5.911 1.425 1.422 1 1.088 1.064 1.145 3.716 1.053
DanWood 2 1.056 1.44 1.367 1 1.007 1.016 1.126 3.488 6.863
Misra1b 1 3.567 1.107 1.116 1.004 1.243 1 1.27 1.79 1.221
Misra1b 2 1.04 1.023 1.08 1.024 1.475 1 1.242 2.032 7.078

Run time for individual NIST problems, “average” difficulty

  BFGS Conjugate gradient (Fletcher-Reeves imp.) Conjugate gradient (Polak-Ribiere imp.) Damping Levenberg-Marquardt Levenberg-MarquardtMD Simplex SteepestDescent Trust Region
Kirby2 1 1.904 26.47 26.83 1.01 1.089 1 1.725 15.63 1.106
Kirby2 2 1.646 27.81 28.82 1 1.116 1.003 2.125 16.92 1.59
Hahn1 1 2.561 39.16 20.65 2.061 1 1.242 1.556 21.35 24.92
Hahn1 2 1.481 45.04 18.88 1 1.15 1.197 4.776 24.01 27.06
MGH17 1 2.589 1.054 1 1.195 7.081 3.388 1.175 6.162 1.996
MGH17 2 1.179 9.803 10.13 1 1.175 1.301 1.276 8.077 1.2
Lanczos1 1 2.886 2.828 2.975 1 2.054 3.892 3.054 8.937 1.962
Lanczos1 2 1.275 3.053 2.171 1 1.045 1.004 1.472 8.765 1.008
Lanczos2 1 4.376 2.982 2.967 1 2.098 4.038 3.033 8.663 2.11
Lanczos2 2 1.289 3.112 2.005 1.035 1.006 1 1.923 8.702 1.057
Gauss3 1 1.817 44.7 39.07 1 1.096 1.007 6.95 40.6 1.479
Gauss3 2 1.622 38.18 10.42 38.51 1.199 1 6.354 38.78 45.31
Misra1c 1 3.003 1.054 1 1.023 1.024 1.029 1.358 2.201 1.15
Misra1c 2 1.066 1.531 1.053 1 1.065 1.095 1.244 2.17 1.03
Misra1d 1 3.291 1.162 1.215 1 1.114 1.154 1.308 2.02 7.472
Misra1d 2 1.067 1.035 1.006 1.01 1 1.501 1.253 1.581 6.866
ENSO 1 3.148 122.2 114.2 1 86.69 1.139 12.49 80.31 111.8
ENSO 2 3.918 123 75.31 1.013 95.06 1 10.85 79.66 117.9

Run time for individual NIST problems, “higher” difficulty

  BFGS Conjugate gradient (Fletcher-Reeves imp.) Conjugate gradient (Polak-Ribiere imp.) Damping Levenberg-Marquardt Levenberg-MarquardtMD Simplex SteepestDescent Trust Region
MGH09 1 3.377 1.047 1 1.462 1.306 1.93 1.15 1.363 1.553
MGH09 2 1 1.083 1.019 1.553 1.035 1.228 1.885 2.875 1.239
Thurber 1 2.309 4.403 16.13 1.172 1.004 1 1.045 7.911 1.906
Thurber 2 1.048 10.52 17.23 1.324 1 1.058 3.317 9.513 1.764
BoxBOD 1 2.701 1.186 1 1.226 1.046 1.199 1.066 2.931 1.4
BoxBOD 2 1.227 1.158 1.143 1.095 1 1.1 2.75 3.453 5.641
Rat42 1 3.106 1.28 1.269 1 1.113 1.119 1.173 3.64 5.789
Rat42 2 1.009 5.551 1.721 1 1.104 1.006 1.101 3.639 6.701
MGH10 1 2.822 2.502 2.186 1.083 4.295 3.219 1 2.871 5.13
MGH10 2 1 1.364 1.736 1.567 3.9 3.662 1.03 4.135 1.835
Eckerle4 1 2.676 1 1.021 1.083 1.114 3.506 1.007 1.157 1.431
Eckerle4 2 1.087 1.571 1.54 1.15 1.226 1 1.327 2.771 1.178
Rat43 1 2.515 11.4 3.306 1.477 1 1.127 1.245 5.629 1.269
Rat43 2 1.292 1.691 3.626 1.392 1 1.007 1.013 5.831 8.317
Bennett5 1 2.454 29.53 3.52 1 1.004 10.75 1.08 10.47 1.227
Bennett5 2 1.057 26.43 3.493 1 11.13 9.95 1.284 9.243 1.652

Run time for individual “CUTEst” problems

  BFGS Conjugate gradient (Fletcher-Reeves imp.) Conjugate gradient (Polak-Ribiere imp.) Damping Levenberg-Marquardt Levenberg-MarquardtMD Simplex SteepestDescent Trust Region
PALMER6C 5.659 39.11 38.28 1 10.81 1.001 1.971 17.68 2.897
PALMER7C 3.822 22.58 20.15 1.192 2.506 1 1.899 20.01 30.98
PALMER8C 3.192 32.92 32.48 1 2.433 1.023 2.76 17.17 28.1
YFITU 2.88 8.55 7.987 1 1.032 1.22 1.979 4.444 1.08
VESUVIOLS 1.396 8.362 5.688 38.52 1.602 1.046 1 37.74 45.01
DMN15102LS 4.42 19.22 15.29 1 7.917 5.026 3.786 22.67 53.85

Run time for individual Neutron Data problems

  BFGS Conjugate gradient (Fletcher-Reeves imp.) Conjugate gradient (Polak-Ribiere imp.) Damping Levenberg-Marquardt Levenberg-MarquardtMD Simplex SteepestDescent Trust Region
ENGINX 193749 calibration, spectrum 651, peak 19 1.407 3.497 4.965 1 1.916 1.651 3.105 18.34 22.31
ENGINX 193749 calibration, spectrum 651, peak 20 1.26 5.343 5.152 1 1.958 1.267 1.457 15.92 21.66
ENGINX 193749 calibration, spectrum 651, peak 23 1.093 2.561 10.11 1 1.254 1.469 3.082 16.36 21.79
ENGINX 193749 calibration, spectrum 651, peak 5 2.439 16.67 7.301 1 2.653 2.675 3.366 25.02 49.29
ENGINX 193749 calibration, spectrum 651, peak 6 1.228 26.73 22.33 1.018 7.748 1 1.635 15.47 1.644
ENGINX 236516 vanadium, bank 1, 10 brk 14.89 21.38 21.23 1 7.994 1.017 14.5 180.5 4.503
ENGINX 236516 vanadium, bank 1, 30 brk 21.13 53.58 50.38 1.039 9.629 1 7.574 180.1 4.352
EVS14188-90_processed Gaussian peaks 1 31.98 40.88 38.89 1 1.83 7.479 1.271 24.95 4.362
EVS14188-90_processed Gaussian peaks 1 18.35 33.2 33.08 1.613 1 2.204 1.04 20.61 2.048
GEM 13 IC peaks 1.189 21.56 101.7 1 9.353 7.766 15.62 26.93 100.2
WISH17701 tube 3 calibration, peak 1 8.515 5.339 5.301 1.107 1 1.055 1.648 3.757 1.127
WISH17701 tube 3 calibration, peak 2 10.4 5.375 5.607 1 1.017 1.224 1.576 4.426 1.268
WISH17701 tube 3 calibration, peak 3 7.95 5.367 5.636 1.189 1 1.007 1.502 3.782 1.052
WISH17701 tube 3 calibration, peak 4 3.437 6.265 6.307 1 1.094 1.144 1.697 4.308 1.195
WISH17701 tube 3 calibration, peak 5 9.633 5.586 5.461 1.043 1.021 1 1.462 4.114 1.18
WISH17701 tube 3 calibration, peak 6 3.342 5.376 5.292 1 1.337 1.192 1.449 3.869 1.11
WISH17701 tube 3 calibration, peak 7 4.351 6.014 5.729 1 1.094 1.113 1.566 4.127 1.669
WISH17701 tube 3 calibration, peak 8 4.847 6.11 5.976 1 1.047 1.087 1.411 4.306 1.231
WISH17701 tube 3 calibration, peak 9 3.221 5.451 5.535 1.151 1.022 1 1.335 3.888 1.114