Comparison of minimizers, with weighted least squares
Here we provide a summary and detailed results for a comparison of
minimizers in Mantid when using weights in the cost function. For an
explanation of the comparison approach and the results obtained when
not using weights see the General comparison of minimizers. In the alternative results presented here the
cost function used is least squares and weigths are defined from input
observational error estimates. See the general concept page on
fitting for an explanation on how the error estimates are
used in the least squares cost function. This cost function is named
in the list of cost functions available in Mantid as “Least
squares”. It is the default cost function in Mantid and the most
commonly used for neutron data.
As the NIST problems do not include measurement errors, assuming that
these datasets would represent data from a typical Mantid workspace we
introduce observational error estimates calculated as the square root
of the observations.
Comparison in terms of accuracy
Accuracy for individual NIST problems, “lower” difficulty
|
BFGS |
Conjugate gradient (Fletcher-Reeves imp.) |
Conjugate gradient (Polak-Ribiere imp.) |
Damping |
Levenberg-Marquardt |
Levenberg-MarquardtMD |
Simplex |
SteepestDescent |
Misra1a 1 |
152.6 |
151.7 |
151.7 |
1 |
1 |
1 |
1.001 |
151.7 |
Misra1a 2 |
2.233 |
2.266 |
2.266 |
1 |
1 |
1 |
1.001 |
2.265 |
Chwirut2 1 |
1.08 |
1.043 |
1.001 |
1 |
1 |
1 |
1.049 |
1.081 |
Chwirut2 2 |
1.004 |
1 |
1 |
1 |
1 |
1 |
1.247 |
1.005 |
Chwirut1 1 |
1.134 |
1.11 |
1.006 |
1.285 |
1.012 |
1.012 |
1 |
1.132 |
Chwirut1 2 |
1.023 |
1 |
1.011 |
1.012 |
1.012 |
1.012 |
1.087 |
1.024 |
Lanczos3 1 |
1.683e+06 |
8.746e+04 |
8.828e+04 |
1 |
1 |
1 |
1.833e+05 |
1.191e+06 |
Lanczos3 2 |
1.982e+05 |
5.844e+04 |
5.685e+04 |
1.241 |
1.241 |
1 |
3540 |
1.866e+05 |
Gauss1 1 |
5.637 |
5.596 |
5.596 |
1 |
1 |
1 |
1 |
5.637 |
Gauss1 2 |
9.281 |
1.077 |
1.027 |
1 |
1 |
1 |
1.032 |
9.278 |
Gauss2 1 |
2.465 |
7.338 |
7.338 |
1 |
1 |
1 |
1 |
7.269 |
Gauss2 2 |
2.151 |
3.752 |
3.752 |
1 |
1 |
1 |
1 |
3.404 |
DanWood 1 |
73.76 |
1 |
1.002 |
1 |
1 |
1 |
1.012 |
53.75 |
DanWood 2 |
1.231 |
1.005 |
1.005 |
1.005 |
1.005 |
1.005 |
1 |
2.04 |
Misra1b 1 |
101.2 |
97.62 |
97.62 |
1 |
1 |
1 |
1 |
97.58 |
Misra1b 2 |
15.72 |
15.44 |
15.44 |
4.476e+05 |
1 |
1 |
1 |
15.44 |
Accuracy for individual NIST problems, “average” difficulty
|
BFGS |
Conjugate gradient (Fletcher-Reeves imp.) |
Conjugate gradient (Polak-Ribiere imp.) |
Damping |
Levenberg-Marquardt |
Levenberg-MarquardtMD |
Simplex |
SteepestDescent |
Kirby2 1 |
309.1 |
13.83 |
13.49 |
1 |
1 |
1 |
210.8 |
285.4 |
Kirby2 2 |
1.042 |
1 |
1.106 |
1.042 |
1.042 |
1.042 |
4.37 |
70.06 |
Hahn1 1 |
1840 |
1759 |
4.905 |
31.16 |
1 |
1.26 |
18.52 |
652.2 |
Hahn1 2 |
6474 |
37.17 |
345.4 |
1 |
1 |
1 |
25.87 |
1.001e+04 |
MGH17 1 |
1.271e+07 |
1.271e+07 |
1.271e+07 |
nan |
1 |
5.207e+07 |
1.761e+07 |
1.271e+07 |
MGH17 2 |
604 |
33.64 |
30.24 |
3.75e+13 |
1.016 |
1 |
1523 |
657.6 |
Lanczos1 1 |
1.049e+18 |
5.459e+16 |
5.507e+16 |
285.4 |
1 |
5.136 |
1.082e+17 |
9.626e+17 |
Lanczos1 2 |
1.818e+17 |
5.367e+16 |
5.221e+16 |
1 |
1 |
213.2 |
5.065e+15 |
1.625e+17 |
Lanczos2 1 |
1.364e+09 |
7.103e+07 |
7.166e+07 |
1 |
1 |
1 |
1.446e+08 |
1.081e+09 |
Lanczos2 2 |
1.295e+08 |
3.823e+07 |
3.719e+07 |
1 |
1 |
1 |
3.506e+06 |
1.155e+08 |
Gauss3 1 |
15.22 |
15.19 |
15.19 |
1 |
1 |
1 |
1 |
15.22 |
Gauss3 2 |
3.172 |
2.023 |
2.573 |
8.2 |
1 |
1 |
1.001 |
9.047 |
Misra1c 1 |
121.4 |
121.1 |
121.1 |
nan |
1 |
1 |
1 |
121.1 |
Misra1c 2 |
6.516 |
6.517 |
6.517 |
1 |
1 |
1 |
1 |
6.518 |
Misra1d 1 |
20.25 |
20.63 |
20.63 |
1.001 |
1.001 |
1.001 |
1 |
20.63 |
Misra1d 2 |
2.008 |
2.002 |
2.002 |
1 |
1 |
1 |
1.001 |
2.001 |
ENSO 1 |
1.315 |
1.026 |
1.067 |
1.526 |
1.003 |
1.526 |
1 |
1.081 |
ENSO 2 |
1.058 |
1.023 |
1.023 |
1.082 |
1 |
1.082 |
1.063 |
1.059 |
Accuracy for individual NIST problems, “higher” difficulty
|
BFGS |
Conjugate gradient (Fletcher-Reeves imp.) |
Conjugate gradient (Polak-Ribiere imp.) |
Damping |
Levenberg-Marquardt |
Levenberg-MarquardtMD |
Simplex |
SteepestDescent |
MGH09 1 |
18.15 |
52.96 |
52.96 |
338.1 |
3.947 |
1 |
7.13 |
46.53 |
MGH09 2 |
1.807 |
1.181 |
1 |
4.196 |
1.063 |
1.063 |
1.182 |
1.718 |
Thurber 1 |
1.359 |
1.49 |
1 |
18.14 |
6.775 |
6.787 |
5.655 |
1.946 |
Thurber 2 |
2.689 |
1.419 |
1.446 |
1 |
1.001 |
1.003 |
2.293 |
2.572 |
BoxBOD 1 |
2.968 |
2.968 |
2.968 |
nan |
2.968 |
2.968 |
2.968 |
1 |
BoxBOD 2 |
22.91 |
8.641 |
8.641 |
1 |
1 |
1 |
8.641 |
8.382 |
Rat42 1 |
152.7 |
15.17 |
15.24 |
1 |
1 |
1 |
14.83 |
207.2 |
Rat42 2 |
1.301 |
1.289 |
1.044 |
1 |
1 |
1 |
1.448 |
1.422 |
MGH10 1 |
6.234e+05 |
6.282e+05 |
6.215e+05 |
9.056e+49 |
1 |
5.269e+05 |
6.462e+05 |
6.282e+05 |
MGH10 2 |
2.647e+04 |
431.1 |
430.1 |
nan |
1 |
47.4 |
2.708e+04 |
2.606e+04 |
Eckerle4 1 |
478 |
490.5 |
490.5 |
478 |
1 |
1 |
478 |
490.5 |
Eckerle4 2 |
30.92 |
33.83 |
33.83 |
1 |
1 |
1 |
1.001 |
33.83 |
Rat43 1 |
251.4 |
178.7 |
83.69 |
nan |
1 |
2.938 |
1 |
242.9 |
Rat43 2 |
1.071 |
1.006 |
1.006 |
1.01 |
1.01 |
1.01 |
1.01 |
1 |
Bennett5 1 |
1334 |
566.4 |
1.028 |
nan |
1 |
1 |
1819 |
1318 |
Bennett5 2 |
713.7 |
6.524 |
2.247 |
1 |
1 |
1.062 |
1571 |
705 |
Comparison in terms of run time
Run time for individual NIST problems, “lower” difficulty
|
BFGS |
Conjugate gradient (Fletcher-Reeves imp.) |
Conjugate gradient (Polak-Ribiere imp.) |
Damping |
Levenberg-Marquardt |
Levenberg-MarquardtMD |
Simplex |
SteepestDescent |
Misra1a 1 |
1.015 |
1.01 |
1.027 |
1 |
1.119 |
1.044 |
1.649 |
2.208 |
Misra1a 2 |
1.035 |
1.084 |
1.165 |
1.003 |
1 |
1.004 |
1.413 |
2.322 |
Chwirut2 1 |
1.304 |
18.74 |
18.73 |
1 |
1.03 |
1.101 |
1.336 |
10.21 |
Chwirut2 2 |
1.289 |
5.725 |
20.42 |
1 |
1.046 |
1.016 |
1.393 |
10.85 |
Chwirut1 1 |
1.388 |
43.42 |
41.59 |
1.027 |
1 |
1.084 |
1.325 |
19.69 |
Chwirut1 2 |
1.524 |
55.4 |
48.98 |
1 |
1.069 |
1.022 |
1.658 |
24.45 |
Lanczos3 1 |
1.407 |
4.416 |
3.944 |
1.199 |
4.139 |
1 |
2.694 |
12.21 |
Lanczos3 2 |
1.376 |
3.33 |
4.185 |
1 |
1.05 |
7.605 |
4.11 |
12.56 |
Gauss1 1 |
1.358 |
1.005 |
1 |
1.314 |
1.565 |
1.518 |
8.433 |
74.81 |
Gauss1 2 |
1.356 |
24.14 |
71.14 |
1 |
1.116 |
1.045 |
6.557 |
56.12 |
Gauss2 1 |
2.8 |
1.001 |
1 |
1.299 |
1.476 |
1.359 |
8.172 |
73.63 |
Gauss2 2 |
2.472 |
1 |
1.006 |
1.283 |
1.473 |
1.366 |
11.13 |
73.01 |
DanWood 1 |
1.052 |
1.493 |
1.478 |
1 |
1.02 |
1.007 |
1.134 |
5.23 |
DanWood 2 |
1.04 |
1.224 |
1.249 |
1 |
1 |
1.034 |
1.142 |
5.29 |
Misra1b 1 |
1 |
1.098 |
1.061 |
1.013 |
1.139 |
1.012 |
1.773 |
2.254 |
Misra1b 2 |
1.041 |
1.159 |
1.158 |
1 |
1.023 |
1.047 |
1.62 |
2.593 |
Run time for individual NIST problems, “average” difficulty
|
BFGS |
Conjugate gradient (Fletcher-Reeves imp.) |
Conjugate gradient (Polak-Ribiere imp.) |
Damping |
Levenberg-Marquardt |
Levenberg-MarquardtMD |
Simplex |
SteepestDescent |
Kirby2 1 |
1.391 |
38.34 |
39.24 |
1 |
1.115 |
1.036 |
2.543 |
22.24 |
Kirby2 2 |
1.697 |
41.62 |
40.93 |
1 |
1.139 |
1.052 |
2.785 |
24.72 |
Hahn1 1 |
1.671 |
1.559 |
38.24 |
17.66 |
24.05 |
1 |
1.468 |
21.44 |
Hahn1 2 |
1.144 |
57.27 |
31.42 |
1 |
1.183 |
1.065 |
3.302 |
30.39 |
MGH17 1 |
1 |
1.204 |
1.206 |
1.741 |
13.58 |
3.088 |
1.529 |
3.751 |
MGH17 2 |
1 |
2.267 |
4.228 |
1.148 |
12.18 |
7.853 |
1.279 |
10.89 |
Lanczos1 1 |
1.548 |
4.513 |
3.444 |
1 |
3.626 |
1.093 |
3.091 |
12.83 |
Lanczos1 2 |
1.381 |
3.273 |
4.143 |
1 |
1.066 |
6.14 |
4.026 |
13.21 |
Lanczos2 1 |
2.101 |
4.443 |
3.319 |
1 |
3.805 |
1.08 |
3.044 |
13.11 |
Lanczos2 2 |
1.374 |
3.169 |
4.287 |
1 |
1.053 |
6.051 |
3.957 |
13.14 |
Gauss3 1 |
1.389 |
1.003 |
1 |
1.623 |
1.809 |
1.693 |
12.52 |
74.15 |
Gauss3 2 |
1.512 |
40.91 |
17.35 |
4.645 |
1.137 |
1 |
7.576 |
45.67 |
Misra1c 1 |
1.023 |
1.04 |
1.004 |
1.223 |
1 |
1.005 |
1.561 |
2.725 |
Misra1c 2 |
1.034 |
1.158 |
1.152 |
1.009 |
1.042 |
1 |
1.604 |
3.056 |
Misra1d 1 |
1.128 |
1.124 |
1.13 |
1.049 |
1 |
1.012 |
1.435 |
2.505 |
Misra1d 2 |
1 |
1.05 |
1.131 |
1.045 |
1.019 |
1 |
1.395 |
1.952 |
ENSO 1 |
3.305 |
169.9 |
108.5 |
1.052 |
154.9 |
1 |
23.31 |
125.4 |
ENSO 2 |
4.739 |
138.8 |
210.1 |
1.027 |
149.5 |
1 |
16.94 |
115.7 |
Run time for individual NIST problems, “higher” difficulty
|
BFGS |
Conjugate gradient (Fletcher-Reeves imp.) |
Conjugate gradient (Polak-Ribiere imp.) |
Damping |
Levenberg-Marquardt |
Levenberg-MarquardtMD |
Simplex |
SteepestDescent |
MGH09 1 |
1.162 |
1.015 |
1 |
1.901 |
1.439 |
3.099 |
3.89 |
2.023 |
MGH09 2 |
1 |
1.311 |
1.448 |
1.727 |
1.055 |
1.062 |
1.241 |
5.183 |
Thurber 1 |
1 |
23.67 |
23.88 |
33.75 |
13.13 |
8.502 |
3.13 |
11.92 |
Thurber 2 |
1 |
8.796 |
21.86 |
36.27 |
14.06 |
9.137 |
1.919 |
12.62 |
BoxBOD 1 |
1.025 |
1.081 |
1.081 |
1.342 |
1 |
1.024 |
1.274 |
4.704 |
BoxBOD 2 |
1.049 |
1.099 |
1.116 |
1.016 |
1 |
1.062 |
1.271 |
4.538 |
Rat42 1 |
1 |
1.44 |
1.338 |
1.018 |
1.002 |
1.031 |
1.227 |
4.983 |
Rat42 2 |
1.098 |
6.961 |
7.725 |
1.013 |
1.031 |
1 |
1.188 |
5.436 |
MGH10 1 |
1.01 |
1 |
5.243 |
1.199 |
6.158 |
5.036 |
1.084 |
2.986 |
MGH10 2 |
1 |
5.66 |
7.736 |
1.545 |
6.873 |
5.875 |
1.167 |
5.715 |
Eckerle4 1 |
1.063 |
1.014 |
1.017 |
1.115 |
1.327 |
5.807 |
1.289 |
1 |
Eckerle4 2 |
1.114 |
1.037 |
1.02 |
1.082 |
1.103 |
1.092 |
1.257 |
1 |
Rat43 1 |
1 |
1.897 |
1.994 |
1.286 |
1.112 |
1.095 |
3.447 |
7.278 |
Rat43 2 |
1.038 |
10.86 |
8.574 |
1.003 |
1 |
1.062 |
2.13 |
8.624 |
Bennett5 1 |
1.127 |
39.59 |
3.36 |
1.044 |
1 |
15.01 |
1.133 |
13.25 |
Bennett5 2 |
1.202 |
33.01 |
3.592 |
1 |
1.093 |
14.61 |
1.263 |
12.26 |