Comparison of minimizers, with weighted least squares

Here we provide a summary and detailed results for a comparison of minimizers in Mantid when using weights in the cost function. For an explanation of the comparison approach and the results obtained when not using weights see the General comparison of minimizers. In the results presented here the cost function used is least squares and weigths are defined from input observational error estimates. See the general concept page on fitting for an explanation on how the error estimates are used in the least squares cost function. This cost function is named in the list of cost functions available in Mantid as “Least squares”. It is the default cost function in Mantid and the most commonly used for neutron data.

For the Neutron data problems true observational errors are used as weights, as commonly done in Mantid. As the NIST and CUTEst problems do not include measurement errors, assuming that these datasets would represent data from a typical Mantid workspace we introduce observational error estimates calculated as the square root of the observations.

Comparison in terms of accuracy

Summary, median ranking

  BFGS Conjugate gradient (Fletcher-Reeves imp.) Conjugate gradient (Polak-Ribiere imp.) Damping Levenberg-Marquardt Levenberg-MarquardtMD Simplex SteepestDescent Trust Region
NIST, “lower” difficulty 4.051 3.009 3.009 1 1 1 1.001 6.453 1
NIST, “average” difficulty 215.2 27.14 25.43 1.001 1 1 15.12 203.2 1
NIST, “higher” difficulty 26.92 11.91 5.804 1.005 1 1.006 6.393 40.18 1.001
CUTEst 36.53 10.1 9.345 1.579e+05 2.464 5315 14.31 58.03 1
Neutron data 1.667 1.566 1.643 1 1 1 2.334 3.546 1

Accuracy for individual NIST problems, “lower” difficulty

  BFGS Conjugate gradient (Fletcher-Reeves imp.) Conjugate gradient (Polak-Ribiere imp.) Damping Levenberg-Marquardt Levenberg-MarquardtMD Simplex SteepestDescent Trust Region
Misra1a 1 152.6 151.7 151.7 1 1 1 1.001 151.7 1
Misra1a 2 2.233 2.266 2.266 1 1 1 1.001 2.265 1
Chwirut2 1 1.08 1.043 1.001 1 1 1 1.049 1.081 1
Chwirut2 2 1.004 1 1 1 1 1 1.247 1.005 1
Chwirut1 1 1.134 1.11 1.006 1.285 1.012 1.012 1 1.132 1.012
Chwirut1 2 1.023 1 1.011 1.012 1.012 1.012 1.087 1.024 1.012
Lanczos3 1 1.683e+06 8.746e+04 8.828e+04 1 1 1 1.833e+05 1.191e+06 1
Lanczos3 2 1.982e+05 5.844e+04 5.685e+04 1.241 1.241 1 3540 1.866e+05 1.241
Gauss1 1 5.637 5.596 5.596 1 1 1 1 5.637 1
Gauss1 2 9.281 1.077 1.027 1 1 1 1.032 9.278 1
Gauss2 1 2.465 7.338 7.338 1 1 1 1 7.269 1
Gauss2 2 2.151 3.752 3.752 1 1 1 1 3.404 1
DanWood 1 73.76 1 1.002 1 1 1 1.012 53.75 1
DanWood 2 1.231 1.005 1.005 1.005 1.005 1.005 1 2.04 1.005
Misra1b 1 101.2 97.62 97.62 1 1 1 1 97.58 1
Misra1b 2 15.72 15.44 15.44 4.476e+05 1 1 1 15.44 1

Accuracy for individual NIST problems, “average” difficulty

  BFGS Conjugate gradient (Fletcher-Reeves imp.) Conjugate gradient (Polak-Ribiere imp.) Damping Levenberg-Marquardt Levenberg-MarquardtMD Simplex SteepestDescent Trust Region
Kirby2 1 309.1 13.83 13.49 1 1 1 210.8 285.4 1
Kirby2 2 1.042 1 1.106 1.042 1.042 1.042 4.37 70.06 1.042
Hahn1 1 2.788e+04 2.666e+04 74.31 472.2 15.15 19.09 280.6 9883 1
Hahn1 2 6474 37.17 345.4 1 1 1 25.87 1.001e+04 1
MGH17 1 1.271e+07 1.271e+07 1.271e+07 nan 1 5.207e+07 1.761e+07 1.271e+07 6.607e+15
MGH17 2 604 33.64 30.24 3.75e+13 1.016 1 1523 657.6 1.017
Lanczos1 1 1.049e+18 5.459e+16 5.507e+16 285.4 1 5.136 1.082e+17 9.626e+17 2.374e+09
Lanczos1 2 1.818e+17 5.367e+16 5.221e+16 1 1 213.2 5.065e+15 1.625e+17 2.606e+06
Lanczos2 1 1.364e+09 7.103e+07 7.166e+07 1 1 1 1.446e+08 1.081e+09 1
Lanczos2 2 1.295e+08 3.823e+07 3.719e+07 1 1 1 3.506e+06 1.155e+08 1
Gauss3 1 15.22 15.19 15.19 1 1 1 1 15.22 1
Gauss3 2 3.172 2.023 2.573 8.2 1 1 1.001 9.047 1
Misra1c 1 121.4 121.1 121.1 nan 1 1 1 121.1 1
Misra1c 2 6.516 6.517 6.517 1 1 1 1 6.518 1
Misra1d 1 20.25 20.63 20.63 1.001 1.001 1.001 1 20.63 1.001
Misra1d 2 2.008 2.002 2.002 1 1 1 1.001 2.001 1
ENSO 1 1.315 1.026 1.067 1.526 1.003 1.526 1 1.081 1.001
ENSO 2 1.06 1.025 1.025 1.084 1.002 1.084 1.065 1.061 1

Accuracy for individual NIST problems, “higher” difficulty

  BFGS Conjugate gradient (Fletcher-Reeves imp.) Conjugate gradient (Polak-Ribiere imp.) Damping Levenberg-Marquardt Levenberg-MarquardtMD Simplex SteepestDescent Trust Region
MGH09 1 18.15 52.96 52.96 338.1 3.947 1 7.13 46.53 3.948
MGH09 2 1.807 1.181 1 4.196 1.063 1.063 1.182 1.718 1.063
Thurber 1 1.359 1.49 1 18.14 6.775 6.787 5.655 1.946 6.77
Thurber 2 2.689 1.419 1.446 1 1.001 1.003 2.293 2.572 1.001
BoxBOD 1 2.968 2.968 2.968 nan 2.968 2.968 2.968 1 2.968
BoxBOD 2 22.91 8.641 8.641 1 1 1 8.641 8.382 1
Rat42 1 152.7 15.17 15.24 1 1 1 14.83 207.2 1
Rat42 2 1.301 1.289 1.044 1 1 1 1.448 1.422 1
MGH10 1 6.234e+05 6.282e+05 6.215e+05 9.056e+49 1 5.269e+05 6.462e+05 6.282e+05 6.097e+05
MGH10 2 8.218e+04 1338 1335 nan 3.105 147.2 8.409e+04 8.091e+04 1
Eckerle4 1 478 490.5 490.5 478 1 1 478 490.5 340.7
Eckerle4 2 30.92 33.83 33.83 1 1 1 1.001 33.83 1
Rat43 1 251.4 178.7 83.69 nan 1 2.938 1 242.9 1
Rat43 2 1.071 1.006 1.006 1.01 1.01 1.01 1.01 1 1.01
Bennett5 1 1334 566.4 1.028 nan 1 1 1819 1318 1
Bennett5 2 713.7 6.524 2.247 1 1 1.062 1571 705 1

Accuracy for individual “CUTEst” problems

  BFGS Conjugate gradient (Fletcher-Reeves imp.) Conjugate gradient (Polak-Ribiere imp.) Damping Levenberg-Marquardt Levenberg-MarquardtMD Simplex SteepestDescent Trust Region
PALMER6C 2.874 1 2.322 1.063e+04 9.331 1.063e+04 2.904 3.739 nan
PALMER7C 66.85 14.78 14.85 4.319e+05 2.624 4.319e+05 74.53 111.7 1
PALMER8C 196.8 18.62 23.16 3.051e+05 2.304 3.051e+05 23.38 268.5 1
YFITU 4.743e+14 1.18e+14 6.325e+13 1.032 1.032 1.032 2.848e+14 3.346e+14 1
VESUVIOLS 1.614 1.608 1.608 5.156e+33 1 1.403 5.241 1.608 1
DMN15102LS 6.216 5.426 3.84 8.583 8.351 1 3.745 4.341 5.634

Accuracy for individual Neutron Data problems

  BFGS Conjugate gradient (Fletcher-Reeves imp.) Conjugate gradient (Polak-Ribiere imp.) Damping Levenberg-Marquardt Levenberg-MarquardtMD Simplex SteepestDescent Trust Region
ENGINX 193749 calibration, spectrum 651, peak 19 1 1.015 1.058 nan 1.143 1.064 1.168 1.381 1.139
ENGINX 193749 calibration, spectrum 651, peak 20 1.813 1.735 1.725 nan 1 1.808 1.216 6.084 1.001
ENGINX 193749 calibration, spectrum 651, peak 23 3.102 2.989 2.923 nan 1 3.267 2.864 21.55 3.245
ENGINX 193749 calibration, spectrum 651, peak 5 1.057 1.077 1.077 nan 1.137 1.222 1.353 1 1.145
ENGINX 193749 calibration, spectrum 651, peak 6 1.136 1.021 1.141 nan 1.016 1.051 1 1.132 1.016
ENGINX 236516 vanadium, bank 1, 10 brk 1 1 1 1.138 1 1.138 1.038 1.02 1
ENGINX 236516 vanadium, bank 1, 30 brk 1 1 1 1.138 1 1.138 1.113 1.03 1
EVS14188-90_processed Gaussian peaks 1 1.914 1.566 1.277 2.228 1.649 1.003 2.301 2.305 1
EVS14188-90_processed Gaussian peaks 1 1.538 1.538 1.538 1 1.053 1 2.337 2.344 1
GEM 13 IC peaks 17.09 3.571 14.59 nan 1 1.018 8.178 26.04 1.003
WISH17701 tube 3 calibration, peak 1 1.005 5.036 5.081 1 13.44 1 5.281 7.485 1
WISH17701 tube 3 calibration, peak 2 1 1.267 1.189 1 1 1 1.511 3.546 1
WISH17701 tube 3 calibration, peak 3 2.604 10.96 9.291 1 1 1 11.54 21.39 1
WISH17701 tube 3 calibration, peak 4 16.21 16.31 17.19 1 1 1 18.63 36.03 1
WISH17701 tube 3 calibration, peak 5 1.667 1.871 1.643 1 1 1 2.334 3.402 1
WISH17701 tube 3 calibration, peak 6 3.922 3.822 3.923 1 1 1 5.876 16.78 1
WISH17701 tube 3 calibration, peak 7 4.736 1.344 4.82 2.948e+06 1 1 5.245 6.1 1
WISH17701 tube 3 calibration, peak 8 2.154 2.198 1.992 1 1 1 2.799 3.845 1
WISH17701 tube 3 calibration, peak 9 1.133 1.132 1.28 1 1 1 1.283 1.594 1

Comparison in terms of run time

Summary, median ranking

  BFGS Conjugate gradient (Fletcher-Reeves imp.) Conjugate gradient (Polak-Ribiere imp.) Damping Levenberg-Marquardt Levenberg-MarquardtMD Simplex SteepestDescent Trust Region
NIST, “lower” difficulty 1.917 1.763 1.31 1.035 1.065 1.047 1.466 8.084 1.214
NIST, “average” difficulty 2.376 2.513 3.014 1.018 1.256 1.075 2.117 9.286 1.718
NIST, “higher” difficulty 1.193 1.453 2.154 1.093 1.148 1.765 1.16 3.619 1.323
CUTEst 3.255 18.66 13.39 1.318 1.829 1 1.695 16.48 12.33
Neutron data 3.545 5.796 5.746 1 1.099 1.077 1.715 4.205 2.495

Run time for individual NIST problems, “lower” difficulty

  BFGS Conjugate gradient (Fletcher-Reeves imp.) Conjugate gradient (Polak-Ribiere imp.) Damping Levenberg-Marquardt Levenberg-MarquardtMD Simplex SteepestDescent Trust Region
Misra1a 1 3.628 1.143 1 1.005 1.146 1.018 1.428 1.924 1.15
Misra1a 2 1 1.525 1.051 1.054 1.049 1.058 1.288 2.419 1.189
Chwirut2 1 3.623 13.04 12.65 1 2.051 1.09 1.272 7.138 2.322
Chwirut2 2 1.247 4.151 14.19 1 1.067 1.011 1.405 7.675 1.166
Chwirut1 1 2.651 30.15 28.61 1 1.322 1.045 1.189 13.2 18.16
Chwirut1 2 1.413 41.07 36.49 1.029 1 1.018 1.453 17.24 1.236
Lanczos3 1 3.605 3.162 2.54 1.048 2.752 1 2.125 8.493 2.576
Lanczos3 2 1.338 2.545 3.203 1 1.061 5.59 3.318 9.571 1.18
Gauss1 1 1.521 1 1.15 1.283 1.518 1.517 6.998 61.43 70.51
Gauss1 2 1.263 19.64 56.44 1.161 1.022 1 5.12 45.89 52.21
Gauss2 1 3.714 1 1.018 1.294 1.323 1.548 6.346 57.34 66.04
Gauss2 2 2.313 1 1.256 1.225 1.404 1.4 9.05 60.21 69.86
DanWood 1 3.447 2 1.365 1.041 1 1.049 1.113 3.466 1.079
DanWood 2 1 1.18 1.22 1.013 1.015 1.058 1.547 3.745 1.072
Misra1b 1 3.045 1.101 1.081 1 1.064 1.005 1.48 1.622 1.193
Misra1b 2 1.038 1.061 1.14 1.616 1.031 1 1.394 2.065 1.119

Run time for individual NIST problems, “average” difficulty

  BFGS Conjugate gradient (Fletcher-Reeves imp.) Conjugate gradient (Polak-Ribiere imp.) Damping Levenberg-Marquardt Levenberg-MarquardtMD Simplex SteepestDescent Trust Region
Kirby2 1 2.466 27.66 28.18 1.089 1.087 1 2.104 15.74 2.708
Kirby2 2 1.496 28.78 28.33 1 1.18 1.092 2.112 18.05 2.242
Hahn1 1 2.338 1.79 37.52 17.6 23.66 1.1 1.575 20.91 1
Hahn1 2 1.074 46.11 25.85 1.015 1.106 1 2.759 24.71 1.352
MGH17 1 2.492 1.035 1 1.31 8.089 2.115 1.125 2.459 1.491
MGH17 2 1.427 1.768 2.957 1 7.974 5.232 1.128 7.163 9.131
Lanczos1 1 2.841 3.231 2.486 1.006 2.584 1 2.122 8.269 1.944
Lanczos1 2 1.234 2.577 3.071 1 1.024 4.233 2.784 9.353 1.008
Lanczos2 1 3.627 3.309 2.695 1 2.763 1.05 2.346 8.758 2.182
Lanczos2 2 1.265 2.449 3.131 1.021 1.089 4.293 3.322 9.219 1
Gauss3 1 2.414 1 1.077 1.695 1.785 1.786 10.87 62.85 72.52
Gauss3 2 1.487 36.57 15.59 4.071 1.331 1 6.78 39.31 45.25
Misra1c 1 3.008 1.068 1.009 1.279 1 1.058 1.282 2.089 1.168
Misra1c 2 1.069 1.157 1.043 1.451 1.036 1 1.257 2.124 1.378
Misra1d 1 3.784 1.249 1.146 1 1.139 1.099 1.761 2.127 1.329
Misra1d 2 1 1.067 1.088 1.039 1.046 1.017 1.301 1.549 1.091
ENSO 1 3.544 115.8 74.75 1 108 1.386 16.06 87.49 107.7
ENSO 2 3.77 98.17 147.5 1 104.3 1.03 12.13 82.23 106.9

Run time for individual NIST problems, “higher” difficulty

  BFGS Conjugate gradient (Fletcher-Reeves imp.) Conjugate gradient (Polak-Ribiere imp.) Damping Levenberg-Marquardt Levenberg-MarquardtMD Simplex SteepestDescent Trust Region
MGH09 1 5.288 1 1.078 1.698 1.34 2.401 2.913 1.775 2.08
MGH09 2 1.03 1.334 1.512 1.47 1.09 1.07 1 3.695 1.1
Thurber 1 1.006 7.124 7.251 10.08 3.972 2.645 1 3.85 4.471
Thurber 2 1 6.238 15.17 25.36 9.846 6.471 1.664 9.338 11.1
BoxBOD 1 2.342 1.056 1.052 1.218 1.008 1 1.125 3.096 1.161
BoxBOD 2 1.045 1.073 1.098 1.063 1.618 1.073 1.056 3.153 1
Rat42 1 3.149 1.34 1.245 1.022 1.052 1 1.265 3.512 1.057
Rat42 2 1.116 4.528 5.022 1 1.056 1.001 1.166 3.542 6.594
MGH10 1 5.74 1 3.761 1.141 4.286 3.409 1.155 2.312 5.005
MGH10 2 1 3.895 5.263 1.362 4.603 4.039 1.021 3.807 4.765
Eckerle4 1 3.128 1.022 1.18 1 1.156 3.707 1.089 1.467 1.193
Eckerle4 2 1.051 1.085 1.08 1 1.206 1.085 1.197 1.078 1.17
Rat43 1 2.505 1.566 1.581 1.123 1.046 1 2.815 4.697 1.453
Rat43 2 1 7.314 5.865 1.037 1.022 1.129 1.712 5.968 1.046
Bennett5 1 2.741 29.52 2.727 1.004 1 10.99 1.072 9.937 1.599
Bennett5 2 1.27 24.75 2.849 1 1.139 11.25 1.218 9.038 1.177

Run time for individual “CUTEst” problems

  BFGS Conjugate gradient (Fletcher-Reeves imp.) Conjugate gradient (Polak-Ribiere imp.) Damping Levenberg-Marquardt Levenberg-MarquardtMD Simplex SteepestDescent Trust Region
PALMER6C 3.203 37.49 14.98 1.107 2.015 1 1.682 16.59 3.542
PALMER7C 3.71 28.69 34.96 1.047 19.52 1 1.708 17.97 27.65
PALMER8C 3.308 31.74 34.65 1.529 17.39 1 4.452 16.36 25.99
YFITU 3.193 8.638 9.006 1.582 1.049 1 1.077 4.637 1.186
VESUVIOLS 1.209 7.944 11.8 1.612 1.603 1.14 1 41.34 1.609
DMN15102LS 5.209 5.145 5.854 1 1.644 3.548 2.691 8.067 21.12

Run time for individual Neutron Data problems

  BFGS Conjugate gradient (Fletcher-Reeves imp.) Conjugate gradient (Polak-Ribiere imp.) Damping Levenberg-Marquardt Levenberg-MarquardtMD Simplex SteepestDescent Trust Region
ENGINX 193749 calibration, spectrum 651, peak 19 1.245 3.983 2.818 1 2.72 1.285 3.901 14.78 17.93
ENGINX 193749 calibration, spectrum 651, peak 20 1.674 8.281 4.713 1 4.067 1.361 3.286 17.37 21.73
ENGINX 193749 calibration, spectrum 651, peak 23 1.359 2.566 2.889 1.032 1.542 1 1.297 14.64 19.83
ENGINX 193749 calibration, spectrum 651, peak 5 1.118 4.324 1.408 1 1.099 1.175 4.255 6.036 2.991
ENGINX 193749 calibration, spectrum 651, peak 6 1 2.992 1.967 1.108 2.317 1.125 3.347 5.617 30.43
ENGINX 236516 vanadium, bank 1, 10 brk 19.23 24.26 25.23 1.069 5.64 1 17.53 190.6 2.93
ENGINX 236516 vanadium, bank 1, 30 brk 32.84 101 135.2 1 6.909 1.004 7.944 175 2.661
EVS14188-90_processed Gaussian peaks 1 83.93 39.91 40.09 1 2.115 3.198 1.162 25.14 3.585
EVS14188-90_processed Gaussian peaks 1 17.25 35.46 36.52 2.102 1 1.913 1.018 21.93 2.495
GEM 13 IC peaks 1.354 41.38 44.17 1 4.599 3.344 10.27 3.188 10.59
WISH17701 tube 3 calibration, peak 1 7.068 5.796 5.604 1.103 1 1.076 1.895 4.189 1.268
WISH17701 tube 3 calibration, peak 2 5.019 5.489 5.62 1.011 1 1.276 1.767 4.071 1.276
WISH17701 tube 3 calibration, peak 3 9.898 6.215 6.334 1.818 1.096 1 1.715 4.205 1.191
WISH17701 tube 3 calibration, peak 4 3.493 5.53 5.487 1 1.072 1.01 1.426 3.866 1.154
WISH17701 tube 3 calibration, peak 5 9.501 5.589 6.026 1.063 1 1.052 1.446 4.002 1.185
WISH17701 tube 3 calibration, peak 6 3.091 5.472 5.569 1 1.137 1.077 1.391 3.901 1.106
WISH17701 tube 3 calibration, peak 7 3.545 6.025 5.746 1 1.037 1.092 1.5 4.034 1.274
WISH17701 tube 3 calibration, peak 8 4.326 5.938 5.948 1 1.014 1.068 1.43 4.077 1.313
WISH17701 tube 3 calibration, peak 9 3.089 5.646 5.913 1.083 1.075 1 1.384 4.042 1.253