\(\renewcommand\AA{\unicode{x212B}}\)

Comparison of minimizers, with weighted least squares

Here we provide a summary and detailed results for a comparison of minimizers in Mantid when using weights in the cost function. For an explanation of the comparison approach and the results obtained when not using weights see the General comparison of minimizers. In the results presented here the cost function used is least squares and weigths are defined from input observational error estimates. See the general concept page on fitting for an explanation on how the error estimates are used in the least squares cost function. This cost function is named in the list of cost functions available in Mantid as “Least squares”. It is the default cost function in Mantid and the most commonly used for neutron data.

For the Neutron data problems true observational errors are used as weights, as commonly done in Mantid. As the NIST and CUTEst problems do not include measurement errors, assuming that these datasets would represent data from a typical Mantid workspace we introduce observational error estimates calculated as the square root of the observations.

Comparison in terms of accuracy

Summary, median ranking

BFGS

Conjugate gradient (Fletcher-Reeves imp.)

Conjugate gradient (Polak-Ribiere imp.)

Damping

Levenberg-Marquardt

Levenberg-MarquardtMD

Simplex

SteepestDescent

Trust Region

NIST, “lower” difficulty

4.051

3.009

3.009

1

1

1

1.001

6.453

1

NIST, “average” difficulty

215.2

27.14

25.43

1.001

1

1

15.12

203.2

1

NIST, “higher” difficulty

26.92

11.91

5.804

1.005

1

1.006

6.393

40.18

1.001

CUTEst

36.53

10.1

9.345

1.579e+05

2.464

5315

14.31

58.03

1

Neutron data

1.667

1.566

1.643

1

1

1

2.334

3.546

1

Accuracy for individual NIST problems, “lower” difficulty

BFGS

Conjugate gradient (Fletcher-Reeves imp.)

Conjugate gradient (Polak-Ribiere imp.)

Damping

Levenberg-Marquardt

Levenberg-MarquardtMD

Simplex

SteepestDescent

Trust Region

Misra1a 1

152.6

151.7

151.7

1

1

1

1.001

151.7

1

Misra1a 2

2.233

2.266

2.266

1

1

1

1.001

2.265

1

Chwirut2 1

1.08

1.043

1.001

1

1

1

1.049

1.081

1

Chwirut2 2

1.004

1

1

1

1

1

1.247

1.005

1

Chwirut1 1

1.134

1.11

1.006

1.285

1.012

1.012

1

1.132

1.012

Chwirut1 2

1.023

1

1.011

1.012

1.012

1.012

1.087

1.024

1.012

Lanczos3 1

1.683e+06

8.746e+04

8.828e+04

1

1

1

1.833e+05

1.191e+06

1

Lanczos3 2

1.982e+05

5.844e+04

5.685e+04

1.241

1.241

1

3540

1.866e+05

1.241

Gauss1 1

5.637

5.596

5.596

1

1

1

1

5.637

1

Gauss1 2

9.281

1.077

1.027

1

1

1

1.032

9.278

1

Gauss2 1

2.465

7.338

7.338

1

1

1

1

7.269

1

Gauss2 2

2.151

3.752

3.752

1

1

1

1

3.404

1

DanWood 1

73.76

1

1.002

1

1

1

1.012

53.75

1

DanWood 2

1.231

1.005

1.005

1.005

1.005

1.005

1

2.04

1.005

Misra1b 1

101.2

97.62

97.62

1

1

1

1

97.58

1

Misra1b 2

15.72

15.44

15.44

4.476e+05

1

1

1

15.44

1

Accuracy for individual NIST problems, “average” difficulty

BFGS

Conjugate gradient (Fletcher-Reeves imp.)

Conjugate gradient (Polak-Ribiere imp.)

Damping

Levenberg-Marquardt

Levenberg-MarquardtMD

Simplex

SteepestDescent

Trust Region

Kirby2 1

309.1

13.83

13.49

1

1

1

210.8

285.4

1

Kirby2 2

1.042

1

1.106

1.042

1.042

1.042

4.37

70.06

1.042

Hahn1 1

2.788e+04

2.666e+04

74.31

472.2

15.15

19.09

280.6

9883

1

Hahn1 2

6474

37.17

345.4

1

1

1

25.87

1.001e+04

1

MGH17 1

1.271e+07

1.271e+07

1.271e+07

nan

1

5.207e+07

1.761e+07

1.271e+07

6.607e+15

MGH17 2

604

33.64

30.24

3.75e+13

1.016

1

1523

657.6

1.017

Lanczos1 1

1.049e+18

5.459e+16

5.507e+16

285.4

1

5.136

1.082e+17

9.626e+17

2.374e+09

Lanczos1 2

1.818e+17

5.367e+16

5.221e+16

1

1

213.2

5.065e+15

1.625e+17

2.606e+06

Lanczos2 1

1.364e+09

7.103e+07

7.166e+07

1

1

1

1.446e+08

1.081e+09

1

Lanczos2 2

1.295e+08

3.823e+07

3.719e+07

1

1

1

3.506e+06

1.155e+08

1

Gauss3 1

15.22

15.19

15.19

1

1

1

1

15.22

1

Gauss3 2

3.172

2.023

2.573

8.2

1

1

1.001

9.047

1

Misra1c 1

121.4

121.1

121.1

nan

1

1

1

121.1

1

Misra1c 2

6.516

6.517

6.517

1

1

1

1

6.518

1

Misra1d 1

20.25

20.63

20.63

1.001

1.001

1.001

1

20.63

1.001

Misra1d 2

2.008

2.002

2.002

1

1

1

1.001

2.001

1

ENSO 1

1.315

1.026

1.067

1.526

1.003

1.526

1

1.081

1.001

ENSO 2

1.06

1.025

1.025

1.084

1.002

1.084

1.065

1.061

1

Accuracy for individual NIST problems, “higher” difficulty

BFGS

Conjugate gradient (Fletcher-Reeves imp.)

Conjugate gradient (Polak-Ribiere imp.)

Damping

Levenberg-Marquardt

Levenberg-MarquardtMD

Simplex

SteepestDescent

Trust Region

MGH09 1

18.15

52.96

52.96

338.1

3.947

1

7.13

46.53

3.948

MGH09 2

1.807

1.181

1

4.196

1.063

1.063

1.182

1.718

1.063

Thurber 1

1.359

1.49

1

18.14

6.775

6.787

5.655

1.946

6.77

Thurber 2

2.689

1.419

1.446

1

1.001

1.003

2.293

2.572

1.001

BoxBOD 1

2.968

2.968

2.968

nan

2.968

2.968

2.968

1

2.968

BoxBOD 2

22.91

8.641

8.641

1

1

1

8.641

8.382

1

Rat42 1

152.7

15.17

15.24

1

1

1

14.83

207.2

1

Rat42 2

1.301

1.289

1.044

1

1

1

1.448

1.422

1

MGH10 1

6.234e+05

6.282e+05

6.215e+05

9.056e+49

1

5.269e+05

6.462e+05

6.282e+05

6.097e+05

MGH10 2

8.218e+04

1338

1335

nan

3.105

147.2

8.409e+04

8.091e+04

1

Eckerle4 1

478

490.5

490.5

478

1

1

478

490.5

340.7

Eckerle4 2

30.92

33.83

33.83

1

1

1

1.001

33.83

1

Rat43 1

251.4

178.7

83.69

nan

1

2.938

1

242.9

1

Rat43 2

1.071

1.006

1.006

1.01

1.01

1.01

1.01

1

1.01

Bennett5 1

1334

566.4

1.028

nan

1

1

1819

1318

1

Bennett5 2

713.7

6.524

2.247

1

1

1.062

1571

705

1

Accuracy for individual “CUTEst” problems

BFGS

Conjugate gradient (Fletcher-Reeves imp.)

Conjugate gradient (Polak-Ribiere imp.)

Damping

Levenberg-Marquardt

Levenberg-MarquardtMD

Simplex

SteepestDescent

Trust Region

PALMER6C

2.874

1

2.322

1.063e+04

9.331

1.063e+04

2.904

3.739

nan

PALMER7C

66.85

14.78

14.85

4.319e+05

2.624

4.319e+05

74.53

111.7

1

PALMER8C

196.8

18.62

23.16

3.051e+05

2.304

3.051e+05

23.38

268.5

1

YFITU

4.743e+14

1.18e+14

6.325e+13

1.032

1.032

1.032

2.848e+14

3.346e+14

1

VESUVIOLS

1.614

1.608

1.608

5.156e+33

1

1.403

5.241

1.608

1

DMN15102LS

6.216

5.426

3.84

8.583

8.351

1

3.745

4.341

5.634

Accuracy for individual Neutron Data problems

BFGS

Conjugate gradient (Fletcher-Reeves imp.)

Conjugate gradient (Polak-Ribiere imp.)

Damping

Levenberg-Marquardt

Levenberg-MarquardtMD

Simplex

SteepestDescent

Trust Region

ENGINX 193749 calibration, spectrum 651, peak 19

1

1.015

1.058

nan

1.143

1.064

1.168

1.381

1.139

ENGINX 193749 calibration, spectrum 651, peak 20

1.813

1.735

1.725

nan

1

1.808

1.216

6.084

1.001

ENGINX 193749 calibration, spectrum 651, peak 23

3.102

2.989

2.923

nan

1

3.267

2.864

21.55

3.245

ENGINX 193749 calibration, spectrum 651, peak 5

1.057

1.077

1.077

nan

1.137

1.222

1.353

1

1.145

ENGINX 193749 calibration, spectrum 651, peak 6

1.136

1.021

1.141

nan

1.016

1.051

1

1.132

1.016

ENGINX 236516 vanadium, bank 1, 10 brk

1

1

1

1.138

1

1.138

1.038

1.02

1

ENGINX 236516 vanadium, bank 1, 30 brk

1

1

1

1.138

1

1.138

1.113

1.03

1

EVS14188-90_processed Gaussian peaks 1

1.914

1.566

1.277

2.228

1.649

1.003

2.301

2.305

1

EVS14188-90_processed Gaussian peaks 1

1.538

1.538

1.538

1

1.053

1

2.337

2.344

1

GEM 13 IC peaks

17.09

3.571

14.59

nan

1

1.018

8.178

26.04

1.003

WISH17701 tube 3 calibration, peak 1

1.005

5.036

5.081

1

13.44

1

5.281

7.485

1

WISH17701 tube 3 calibration, peak 2

1

1.267

1.189

1

1

1

1.511

3.546

1

WISH17701 tube 3 calibration, peak 3

2.604

10.96

9.291

1

1

1

11.54

21.39

1

WISH17701 tube 3 calibration, peak 4

16.21

16.31

17.19

1

1

1

18.63

36.03

1

WISH17701 tube 3 calibration, peak 5

1.667

1.871

1.643

1

1

1

2.334

3.402

1

WISH17701 tube 3 calibration, peak 6

3.922

3.822

3.923

1

1

1

5.876

16.78

1

WISH17701 tube 3 calibration, peak 7

4.736

1.344

4.82

2.948e+06

1

1

5.245

6.1

1

WISH17701 tube 3 calibration, peak 8

2.154

2.198

1.992

1

1

1

2.799

3.845

1

WISH17701 tube 3 calibration, peak 9

1.133

1.132

1.28

1

1

1

1.283

1.594

1

Comparison in terms of run time

Summary, median ranking

BFGS

Conjugate gradient (Fletcher-Reeves imp.)

Conjugate gradient (Polak-Ribiere imp.)

Damping

Levenberg-Marquardt

Levenberg-MarquardtMD

Simplex

SteepestDescent

Trust Region

NIST, “lower” difficulty

1.917

1.763

1.31

1.035

1.065

1.047

1.466

8.084

1.214

NIST, “average” difficulty

2.376

2.513

3.014

1.018

1.256

1.075

2.117

9.286

1.718

NIST, “higher” difficulty

1.193

1.453

2.154

1.093

1.148

1.765

1.16

3.619

1.323

CUTEst

3.255

18.66

13.39

1.318

1.829

1

1.695

16.48

12.33

Neutron data

3.545

5.796

5.746

1

1.099

1.077

1.715

4.205

2.495

Run time for individual NIST problems, “lower” difficulty

BFGS

Conjugate gradient (Fletcher-Reeves imp.)

Conjugate gradient (Polak-Ribiere imp.)

Damping

Levenberg-Marquardt

Levenberg-MarquardtMD

Simplex

SteepestDescent

Trust Region

Misra1a 1

3.628

1.143

1

1.005

1.146

1.018

1.428

1.924

1.15

Misra1a 2

1

1.525

1.051

1.054

1.049

1.058

1.288

2.419

1.189

Chwirut2 1

3.623

13.04

12.65

1

2.051

1.09

1.272

7.138

2.322

Chwirut2 2

1.247

4.151

14.19

1

1.067

1.011

1.405

7.675

1.166

Chwirut1 1

2.651

30.15

28.61

1

1.322

1.045

1.189

13.2

18.16

Chwirut1 2

1.413

41.07

36.49

1.029

1

1.018

1.453

17.24

1.236

Lanczos3 1

3.605

3.162

2.54

1.048

2.752

1

2.125

8.493

2.576

Lanczos3 2

1.338

2.545

3.203

1

1.061

5.59

3.318

9.571

1.18

Gauss1 1

1.521

1

1.15

1.283

1.518

1.517

6.998

61.43

70.51

Gauss1 2

1.263

19.64

56.44

1.161

1.022

1

5.12

45.89

52.21

Gauss2 1

3.714

1

1.018

1.294

1.323

1.548

6.346

57.34

66.04

Gauss2 2

2.313

1

1.256

1.225

1.404

1.4

9.05

60.21

69.86

DanWood 1

3.447

2

1.365

1.041

1

1.049

1.113

3.466

1.079

DanWood 2

1

1.18

1.22

1.013

1.015

1.058

1.547

3.745

1.072

Misra1b 1

3.045

1.101

1.081

1

1.064

1.005

1.48

1.622

1.193

Misra1b 2

1.038

1.061

1.14

1.616

1.031

1

1.394

2.065

1.119

Run time for individual NIST problems, “average” difficulty

BFGS

Conjugate gradient (Fletcher-Reeves imp.)

Conjugate gradient (Polak-Ribiere imp.)

Damping

Levenberg-Marquardt

Levenberg-MarquardtMD

Simplex

SteepestDescent

Trust Region

Kirby2 1

2.466

27.66

28.18

1.089

1.087

1

2.104

15.74

2.708

Kirby2 2

1.496

28.78

28.33

1

1.18

1.092

2.112

18.05

2.242

Hahn1 1

2.338

1.79

37.52

17.6

23.66

1.1

1.575

20.91

1

Hahn1 2

1.074

46.11

25.85

1.015

1.106

1

2.759

24.71

1.352

MGH17 1

2.492

1.035

1

1.31

8.089

2.115

1.125

2.459

1.491

MGH17 2

1.427

1.768

2.957

1

7.974

5.232

1.128

7.163

9.131

Lanczos1 1

2.841

3.231

2.486

1.006

2.584

1

2.122

8.269

1.944

Lanczos1 2

1.234

2.577

3.071

1

1.024

4.233

2.784

9.353

1.008

Lanczos2 1

3.627

3.309

2.695

1

2.763

1.05

2.346

8.758

2.182

Lanczos2 2

1.265

2.449

3.131

1.021

1.089

4.293

3.322

9.219

1

Gauss3 1

2.414

1

1.077

1.695

1.785

1.786

10.87

62.85

72.52

Gauss3 2

1.487

36.57

15.59

4.071

1.331

1

6.78

39.31

45.25

Misra1c 1

3.008

1.068

1.009

1.279

1

1.058

1.282

2.089

1.168

Misra1c 2

1.069

1.157

1.043

1.451

1.036

1

1.257

2.124

1.378

Misra1d 1

3.784

1.249

1.146

1

1.139

1.099

1.761

2.127

1.329

Misra1d 2

1

1.067

1.088

1.039

1.046

1.017

1.301

1.549

1.091

ENSO 1

3.544

115.8

74.75

1

108

1.386

16.06

87.49

107.7

ENSO 2

3.77

98.17

147.5

1

104.3

1.03

12.13

82.23

106.9

Run time for individual NIST problems, “higher” difficulty

BFGS

Conjugate gradient (Fletcher-Reeves imp.)

Conjugate gradient (Polak-Ribiere imp.)

Damping

Levenberg-Marquardt

Levenberg-MarquardtMD

Simplex

SteepestDescent

Trust Region

MGH09 1

5.288

1

1.078

1.698

1.34

2.401

2.913

1.775

2.08

MGH09 2

1.03

1.334

1.512

1.47

1.09

1.07

1

3.695

1.1

Thurber 1

1.006

7.124

7.251

10.08

3.972

2.645

1

3.85

4.471

Thurber 2

1

6.238

15.17

25.36

9.846

6.471

1.664

9.338

11.1

BoxBOD 1

2.342

1.056

1.052

1.218

1.008

1

1.125

3.096

1.161

BoxBOD 2

1.045

1.073

1.098

1.063

1.618

1.073

1.056

3.153

1

Rat42 1

3.149

1.34

1.245

1.022

1.052

1

1.265

3.512

1.057

Rat42 2

1.116

4.528

5.022

1

1.056

1.001

1.166

3.542

6.594

MGH10 1

5.74

1

3.761

1.141

4.286

3.409

1.155

2.312

5.005

MGH10 2

1

3.895

5.263

1.362

4.603

4.039

1.021

3.807

4.765

Eckerle4 1

3.128

1.022

1.18

1

1.156

3.707

1.089

1.467

1.193

Eckerle4 2

1.051

1.085

1.08

1

1.206

1.085

1.197

1.078

1.17

Rat43 1

2.505

1.566

1.581

1.123

1.046

1

2.815

4.697

1.453

Rat43 2

1

7.314

5.865

1.037

1.022

1.129

1.712

5.968

1.046

Bennett5 1

2.741

29.52

2.727

1.004

1

10.99

1.072

9.937

1.599

Bennett5 2

1.27

24.75

2.849

1

1.139

11.25

1.218

9.038

1.177

Run time for individual “CUTEst” problems

BFGS

Conjugate gradient (Fletcher-Reeves imp.)

Conjugate gradient (Polak-Ribiere imp.)

Damping

Levenberg-Marquardt

Levenberg-MarquardtMD

Simplex

SteepestDescent

Trust Region

PALMER6C

3.203

37.49

14.98

1.107

2.015

1

1.682

16.59

3.542

PALMER7C

3.71

28.69

34.96

1.047

19.52

1

1.708

17.97

27.65

PALMER8C

3.308

31.74

34.65

1.529

17.39

1

4.452

16.36

25.99

YFITU

3.193

8.638

9.006

1.582

1.049

1

1.077

4.637

1.186

VESUVIOLS

1.209

7.944

11.8

1.612

1.603

1.14

1

41.34

1.609

DMN15102LS

5.209

5.145

5.854

1

1.644

3.548

2.691

8.067

21.12

Run time for individual Neutron Data problems

BFGS

Conjugate gradient (Fletcher-Reeves imp.)

Conjugate gradient (Polak-Ribiere imp.)

Damping

Levenberg-Marquardt

Levenberg-MarquardtMD

Simplex

SteepestDescent

Trust Region

ENGINX 193749 calibration, spectrum 651, peak 19

1.245

3.983

2.818

1

2.72

1.285

3.901

14.78

17.93

ENGINX 193749 calibration, spectrum 651, peak 20

1.674

8.281

4.713

1

4.067

1.361

3.286

17.37

21.73

ENGINX 193749 calibration, spectrum 651, peak 23

1.359

2.566

2.889

1.032

1.542

1

1.297

14.64

19.83

ENGINX 193749 calibration, spectrum 651, peak 5

1.118

4.324

1.408

1

1.099

1.175

4.255

6.036

2.991

ENGINX 193749 calibration, spectrum 651, peak 6

1

2.992

1.967

1.108

2.317

1.125

3.347

5.617

30.43

ENGINX 236516 vanadium, bank 1, 10 brk

19.23

24.26

25.23

1.069

5.64

1

17.53

190.6

2.93

ENGINX 236516 vanadium, bank 1, 30 brk

32.84

101

135.2

1

6.909

1.004

7.944

175

2.661

EVS14188-90_processed Gaussian peaks 1

83.93

39.91

40.09

1

2.115

3.198

1.162

25.14

3.585

EVS14188-90_processed Gaussian peaks 1

17.25

35.46

36.52

2.102

1

1.913

1.018

21.93

2.495

GEM 13 IC peaks

1.354

41.38

44.17

1

4.599

3.344

10.27

3.188

10.59

WISH17701 tube 3 calibration, peak 1

7.068

5.796

5.604

1.103

1

1.076

1.895

4.189

1.268

WISH17701 tube 3 calibration, peak 2

5.019

5.489

5.62

1.011

1

1.276

1.767

4.071

1.276

WISH17701 tube 3 calibration, peak 3

9.898

6.215

6.334

1.818

1.096

1

1.715

4.205

1.191

WISH17701 tube 3 calibration, peak 4

3.493

5.53

5.487

1

1.072

1.01

1.426

3.866

1.154

WISH17701 tube 3 calibration, peak 5

9.501

5.589

6.026

1.063

1

1.052

1.446

4.002

1.185

WISH17701 tube 3 calibration, peak 6

3.091

5.472

5.569

1

1.137

1.077

1.391

3.901

1.106

WISH17701 tube 3 calibration, peak 7

3.545

6.025

5.746

1

1.037

1.092

1.5

4.034

1.274

WISH17701 tube 3 calibration, peak 8

4.326

5.938

5.948

1

1.014

1.068

1.43

4.077

1.313

WISH17701 tube 3 calibration, peak 9

3.089

5.646

5.913

1.083

1.075

1

1.384

4.042

1.253

Category: Concepts