Comparison of minimizers, with weighted least squares

Here we provide a summary and detailed results for a comparison of minimizers in Mantid when using weights in the cost function. For an explanation of the comparison approach and the results obtained when not using weights see the General comparison of minimizers. In the alternative results presented here the cost function used is least squares and weigths are defined from input observational error estimates. See the general concept page on fitting for an explanation on how the error estimates are used in the least squares cost function. This cost function is named in the list of cost functions available in Mantid as “Least squares”. It is the default cost function in Mantid and the most commonly used for neutron data.

As the NIST problems do not include measurement errors, assuming that these datasets would represent data from a typical Mantid workspace we introduce observational error estimates calculated as the square root of the observations.

Comparison in terms of accuracy

Summary, median ranking

  BFGS Conjugate gradient (Fletcher-Reeves imp.) Conjugate gradient (Polak-Ribiere imp.) Damping Levenberg-Marquardt Levenberg-MarquardtMD Simplex SteepestDescent
NIST, “lower” difficulty 4.051 3.009 3.009 1 1 1 1.001 6.453
NIST, “average” difficulty 215.2 27.14 17.91 1.001 1 1 11.44 203.2
NIST, “higher” difficulty 26.92 11.91 5.804 1.005 1 1.006 6.393 40.18

Accuracy for individual NIST problems, “lower” difficulty

  BFGS Conjugate gradient (Fletcher-Reeves imp.) Conjugate gradient (Polak-Ribiere imp.) Damping Levenberg-Marquardt Levenberg-MarquardtMD Simplex SteepestDescent
Misra1a 1 152.6 151.7 151.7 1 1 1 1.001 151.7
Misra1a 2 2.233 2.266 2.266 1 1 1 1.001 2.265
Chwirut2 1 1.08 1.043 1.001 1 1 1 1.049 1.081
Chwirut2 2 1.004 1 1 1 1 1 1.247 1.005
Chwirut1 1 1.134 1.11 1.006 1.285 1.012 1.012 1 1.132
Chwirut1 2 1.023 1 1.011 1.012 1.012 1.012 1.087 1.024
Lanczos3 1 1.683e+06 8.746e+04 8.828e+04 1 1 1 1.833e+05 1.191e+06
Lanczos3 2 1.982e+05 5.844e+04 5.685e+04 1.241 1.241 1 3540 1.866e+05
Gauss1 1 5.637 5.596 5.596 1 1 1 1 5.637
Gauss1 2 9.281 1.077 1.027 1 1 1 1.032 9.278
Gauss2 1 2.465 7.338 7.338 1 1 1 1 7.269
Gauss2 2 2.151 3.752 3.752 1 1 1 1 3.404
DanWood 1 73.76 1 1.002 1 1 1 1.012 53.75
DanWood 2 1.231 1.005 1.005 1.005 1.005 1.005 1 2.04
Misra1b 1 101.2 97.62 97.62 1 1 1 1 97.58
Misra1b 2 15.72 15.44 15.44 4.476e+05 1 1 1 15.44

Accuracy for individual NIST problems, “average” difficulty

  BFGS Conjugate gradient (Fletcher-Reeves imp.) Conjugate gradient (Polak-Ribiere imp.) Damping Levenberg-Marquardt Levenberg-MarquardtMD Simplex SteepestDescent
Kirby2 1 309.1 13.83 13.49 1 1 1 210.8 285.4
Kirby2 2 1.042 1 1.106 1.042 1.042 1.042 4.37 70.06
Hahn1 1 1840 1759 4.905 31.16 1 1.26 18.52 652.2
Hahn1 2 6474 37.17 345.4 1 1 1 25.87 1.001e+04
MGH17 1 1.271e+07 1.271e+07 1.271e+07 nan 1 5.207e+07 1.761e+07 1.271e+07
MGH17 2 604 33.64 30.24 3.75e+13 1.016 1 1523 657.6
Lanczos1 1 1.049e+18 5.459e+16 5.507e+16 285.4 1 5.136 1.082e+17 9.626e+17
Lanczos1 2 1.818e+17 5.367e+16 5.221e+16 1 1 213.2 5.065e+15 1.625e+17
Lanczos2 1 1.364e+09 7.103e+07 7.166e+07 1 1 1 1.446e+08 1.081e+09
Lanczos2 2 1.295e+08 3.823e+07 3.719e+07 1 1 1 3.506e+06 1.155e+08
Gauss3 1 15.22 15.19 15.19 1 1 1 1 15.22
Gauss3 2 3.172 2.023 2.573 8.2 1 1 1.001 9.047
Misra1c 1 121.4 121.1 121.1 nan 1 1 1 121.1
Misra1c 2 6.516 6.517 6.517 1 1 1 1 6.518
Misra1d 1 20.25 20.63 20.63 1.001 1.001 1.001 1 20.63
Misra1d 2 2.008 2.002 2.002 1 1 1 1.001 2.001
ENSO 1 1.315 1.026 1.067 1.526 1.003 1.526 1 1.081
ENSO 2 1.058 1.023 1.023 1.082 1 1.082 1.063 1.059

Accuracy for individual NIST problems, “higher” difficulty

  BFGS Conjugate gradient (Fletcher-Reeves imp.) Conjugate gradient (Polak-Ribiere imp.) Damping Levenberg-Marquardt Levenberg-MarquardtMD Simplex SteepestDescent
MGH09 1 18.15 52.96 52.96 338.1 3.947 1 7.13 46.53
MGH09 2 1.807 1.181 1 4.196 1.063 1.063 1.182 1.718
Thurber 1 1.359 1.49 1 18.14 6.775 6.787 5.655 1.946
Thurber 2 2.689 1.419 1.446 1 1.001 1.003 2.293 2.572
BoxBOD 1 2.968 2.968 2.968 nan 2.968 2.968 2.968 1
BoxBOD 2 22.91 8.641 8.641 1 1 1 8.641 8.382
Rat42 1 152.7 15.17 15.24 1 1 1 14.83 207.2
Rat42 2 1.301 1.289 1.044 1 1 1 1.448 1.422
MGH10 1 6.234e+05 6.282e+05 6.215e+05 9.056e+49 1 5.269e+05 6.462e+05 6.282e+05
MGH10 2 2.647e+04 431.1 430.1 nan 1 47.4 2.708e+04 2.606e+04
Eckerle4 1 478 490.5 490.5 478 1 1 478 490.5
Eckerle4 2 30.92 33.83 33.83 1 1 1 1.001 33.83
Rat43 1 251.4 178.7 83.69 nan 1 2.938 1 242.9
Rat43 2 1.071 1.006 1.006 1.01 1.01 1.01 1.01 1
Bennett5 1 1334 566.4 1.028 nan 1 1 1819 1318
Bennett5 2 713.7 6.524 2.247 1 1 1.062 1571 705

Comparison in terms of run time

Summary, median ranking

  BFGS Conjugate gradient (Fletcher-Reeves imp.) Conjugate gradient (Polak-Ribiere imp.) Damping Levenberg-Marquardt Levenberg-MarquardtMD Simplex SteepestDescent
NIST, “lower” difficulty 1.33 1.358 1.363 1 1.06 1.044 1.654 11.53
NIST, “average” difficulty 1.385 3.221 4.185 1.036 1.161 1.044 2.914 13.18
NIST, “higher” difficulty 1.031 1.668 2.677 1.157 1.098 2.097 1.267 5.309

Run time for individual NIST problems, “lower” difficulty

  BFGS Conjugate gradient (Fletcher-Reeves imp.) Conjugate gradient (Polak-Ribiere imp.) Damping Levenberg-Marquardt Levenberg-MarquardtMD Simplex SteepestDescent
Misra1a 1 1.015 1.01 1.027 1 1.119 1.044 1.649 2.208
Misra1a 2 1.035 1.084 1.165 1.003 1 1.004 1.413 2.322
Chwirut2 1 1.304 18.74 18.73 1 1.03 1.101 1.336 10.21
Chwirut2 2 1.289 5.725 20.42 1 1.046 1.016 1.393 10.85
Chwirut1 1 1.388 43.42 41.59 1.027 1 1.084 1.325 19.69
Chwirut1 2 1.524 55.4 48.98 1 1.069 1.022 1.658 24.45
Lanczos3 1 1.407 4.416 3.944 1.199 4.139 1 2.694 12.21
Lanczos3 2 1.376 3.33 4.185 1 1.05 7.605 4.11 12.56
Gauss1 1 1.358 1.005 1 1.314 1.565 1.518 8.433 74.81
Gauss1 2 1.356 24.14 71.14 1 1.116 1.045 6.557 56.12
Gauss2 1 2.8 1.001 1 1.299 1.476 1.359 8.172 73.63
Gauss2 2 2.472 1 1.006 1.283 1.473 1.366 11.13 73.01
DanWood 1 1.052 1.493 1.478 1 1.02 1.007 1.134 5.23
DanWood 2 1.04 1.224 1.249 1 1 1.034 1.142 5.29
Misra1b 1 1 1.098 1.061 1.013 1.139 1.012 1.773 2.254
Misra1b 2 1.041 1.159 1.158 1 1.023 1.047 1.62 2.593

Run time for individual NIST problems, “average” difficulty

  BFGS Conjugate gradient (Fletcher-Reeves imp.) Conjugate gradient (Polak-Ribiere imp.) Damping Levenberg-Marquardt Levenberg-MarquardtMD Simplex SteepestDescent
Kirby2 1 1.391 38.34 39.24 1 1.115 1.036 2.543 22.24
Kirby2 2 1.697 41.62 40.93 1 1.139 1.052 2.785 24.72
Hahn1 1 1.671 1.559 38.24 17.66 24.05 1 1.468 21.44
Hahn1 2 1.144 57.27 31.42 1 1.183 1.065 3.302 30.39
MGH17 1 1 1.204 1.206 1.741 13.58 3.088 1.529 3.751
MGH17 2 1 2.267 4.228 1.148 12.18 7.853 1.279 10.89
Lanczos1 1 1.548 4.513 3.444 1 3.626 1.093 3.091 12.83
Lanczos1 2 1.381 3.273 4.143 1 1.066 6.14 4.026 13.21
Lanczos2 1 2.101 4.443 3.319 1 3.805 1.08 3.044 13.11
Lanczos2 2 1.374 3.169 4.287 1 1.053 6.051 3.957 13.14
Gauss3 1 1.389 1.003 1 1.623 1.809 1.693 12.52 74.15
Gauss3 2 1.512 40.91 17.35 4.645 1.137 1 7.576 45.67
Misra1c 1 1.023 1.04 1.004 1.223 1 1.005 1.561 2.725
Misra1c 2 1.034 1.158 1.152 1.009 1.042 1 1.604 3.056
Misra1d 1 1.128 1.124 1.13 1.049 1 1.012 1.435 2.505
Misra1d 2 1 1.05 1.131 1.045 1.019 1 1.395 1.952
ENSO 1 3.305 169.9 108.5 1.052 154.9 1 23.31 125.4
ENSO 2 4.739 138.8 210.1 1.027 149.5 1 16.94 115.7

Run time for individual NIST problems, “higher” difficulty

  BFGS Conjugate gradient (Fletcher-Reeves imp.) Conjugate gradient (Polak-Ribiere imp.) Damping Levenberg-Marquardt Levenberg-MarquardtMD Simplex SteepestDescent
MGH09 1 1.162 1.015 1 1.901 1.439 3.099 3.89 2.023
MGH09 2 1 1.311 1.448 1.727 1.055 1.062 1.241 5.183
Thurber 1 1 23.67 23.88 33.75 13.13 8.502 3.13 11.92
Thurber 2 1 8.796 21.86 36.27 14.06 9.137 1.919 12.62
BoxBOD 1 1.025 1.081 1.081 1.342 1 1.024 1.274 4.704
BoxBOD 2 1.049 1.099 1.116 1.016 1 1.062 1.271 4.538
Rat42 1 1 1.44 1.338 1.018 1.002 1.031 1.227 4.983
Rat42 2 1.098 6.961 7.725 1.013 1.031 1 1.188 5.436
MGH10 1 1.01 1 5.243 1.199 6.158 5.036 1.084 2.986
MGH10 2 1 5.66 7.736 1.545 6.873 5.875 1.167 5.715
Eckerle4 1 1.063 1.014 1.017 1.115 1.327 5.807 1.289 1
Eckerle4 2 1.114 1.037 1.02 1.082 1.103 1.092 1.257 1
Rat43 1 1 1.897 1.994 1.286 1.112 1.095 3.447 7.278
Rat43 2 1.038 10.86 8.574 1.003 1 1.062 2.13 8.624
Bennett5 1 1.127 39.59 3.36 1.044 1 15.01 1.133 13.25
Bennett5 2 1.202 33.01 3.592 1 1.093 14.61 1.263 12.26