\(\renewcommand\AA{\unicode{x212B}}\)

Comparison of minimizers for individual test problems

As in the summarized results (Which minimizers to use with Mantid), the best possible results with Mantid minimizers are given a top score “1” (lowest error, or lowest run time).

The following color codes are used to distinguish different levels of performance, note the irregular (increasing) difference between level boundaries):

  • best or within 10% (ranking < 1.1)

  • within 33% over the best (1.1 < ranking < 1.33)

  • within 75% over the best (1.33 < ranking < 1.75)

  • within 200% over the best (1.75 < ranking < 3)

  • over 200% of best (ranking > 3)

The integer numbers 1 and 2 next to the problem names denote the starting point (all NIST problems are usually tested for two different starting points). By clicking on the names of the test problems (leftmost column) you can see the original definition of the problem.

Comparison in terms of accuracy

For details on the ranking approach and the color codes see above and the summarized results. When interpreting these results note that the median shown in the summarized results is calculated excluding undefined values. Undefined accuracy values are obtained when the minimizers fail to produce any result and are shown in the detailed results as “nan”. For the more difficult groups of problems the proportion of undefined values is higher for some minimizers. This can bias the statistics shown in the summary table in favor of the minimizers that fail more often, as some of the hard problems are excluded from the median instead of contributing to a higher median (lower ranking).

Alternatively, see the summary and detailed results when using weighted least squares as cost function.

Accuracy for individual NIST problems, “lower” difficulty

BFGS

Conjugate gradient (Fletcher-Reeves imp.)

Conjugate gradient (Polak-Ribiere imp.)

Damping

Levenberg-Marquardt

Levenberg-MarquardtMD

Simplex

SteepestDescent

Trust Region

Misra1a 1

156.5

156.7

156.7

1

1

1

1

156.7

1

Misra1a 2

1.829

2.253

2.253

1

1

1

1

2.253

1

Chwirut2 1

1.067

1.013

1

1

1

1

1.068

1.066

1

Chwirut2 2

1.049

1.002

1

1

1

1

1.197

1.004

1

Chwirut1 1

1.097

1.001

1

1

1

1

1.112

1.097

1

Chwirut1 2

1.017

1.007

1

1

1

1

1.109

1.017

1

Lanczos3 1

7.683e+06

5.433e+04

4.963e+04

1

1

1

973.7

7.797e+04

1

Lanczos3 2

2.652e+05

6.206e+04

6.264e+04

1

1

1

1.807e+04

1.285e+05

1

Gauss1 1

5.6

1.745

1.044

1

1

1

1

5.6

1

Gauss1 2

1.655

9.182

9.182

1

1

1

1

9.168

1

Gauss2 1

7.272

7.341

7.341

1

1

1

1.022

7.272

1

Gauss2 2

2.081

3.754

3.754

1

1

1

1

3.393

1

DanWood 1

72.41

1

1

1

1

1

1.011

55.52

1

DanWood 2

1.244

1

1

1

1

1

1.031

1.97

1

Misra1b 1

96.76

96.97

96.97

1

1

1

1

96.97

1

Misra1b 2

8.521

16.43

16.43

1

1

1

1

16.43

1

Accuracy for individual NIST problems, “average” difficulty

BFGS

Conjugate gradient (Fletcher-Reeves imp.)

Conjugate gradient (Polak-Ribiere imp.)

Damping

Levenberg-Marquardt

Levenberg-MarquardtMD

Simplex

SteepestDescent

Trust Region

Kirby2 1

423

25.07

25.37

1

1

1

956.3

510.9

1

Kirby2 2

2.554

1.717

1.68

1

1

1

1.396

121

1

Hahn1 1

4.508e+04

45.51

217

69.31

1

1

200.6

4962

17.97

Hahn1 2

4000

35.48

211.6

1

1

1

157.2

4896

226.2

MGH17 1

1.419e+04

1.42e+04

1.42e+04

nan

1

754.5

1.421e+04

1.413e+04

1.421e+04

MGH17 2

2473

1.414

1.422

1

1

1

4179

1762

1

Lanczos1 1

1.561e+18

1.107e+16

1.01e+16

151.7

1

21.29

2.182e+15

1.583e+16

9.922e+08

Lanczos1 2

1.559e+17

3.652e+16

3.687e+16

1

1

28.59

1.064e+16

7.518e+16

2.033e+06

Lanczos2 1

5.553e+09

3.939e+07

3.591e+07

1

1

1

7.765e+06

5.631e+07

3.949

Lanczos2 2

1.917e+08

4.491e+07

4.533e+07

1

1

1

1.308e+07

9.22e+07

1.002

Gauss3 1

15.19

1.815

1.568

1

1

1

1.036

15.19

1

Gauss3 2

3.147

2.44

2.763

7.912

1

1

1.25

8.944

1

Misra1c 1

115

115.8

115.8

1

1

1

1

115.8

1

Misra1c 2

6.595

6.702

6.702

1

1

1

1

6.702

1

Misra1d 1

12.67

19.62

19.62

1

1

1

1

19.62

1

Misra1d 2

1.03

1.939

1.939

1

1

1

1

1.939

1

ENSO 1

1.264

1.029

1.028

1.529

1.004

1.529

1.018

1.207

1

ENSO 2

1.033

1.018

1.029

1.087

1

1.087

1.029

1.052

1

Accuracy for individual NIST problems, “higher” difficulty

BFGS

Conjugate gradient (Fletcher-Reeves imp.)

Conjugate gradient (Polak-Ribiere imp.)

Damping

Levenberg-Marquardt

Levenberg-MarquardtMD

Simplex

SteepestDescent

Trust Region

MGH09 1

7.705

25.02

25.02

1.817e+04

3.342

1

20.62

24.19

3.347

MGH09 2

1.62

1.621

1.62

256.6

1

1

1.025

1.604

1

Thurber 1

228.3

175.3

173.2

2.693

1

1

176.5

177.2

1

Thurber 2

422.3

20.02

40.38

1

1

1

4.43

368.9

1

BoxBOD 1

8.366

8.366

8.366

nan

8.366

1

8.366

1.515

8.366

BoxBOD 2

33.46

8.366

8.366

1

1

1

8.366

6.177

1

Rat42 1

13.33

13.27

13.27

4.759e+06

1

1

16.48

193.6

1

Rat42 2

1.331

1.213

1.302

1

1

1

1.429

1.449

1

MGH10 1

6.367e+05

6.362e+05

6.344e+05

2.984e+47

1

5.306e+05

6.422e+05

6.37e+05

6.271e+05

MGH10 2

7.834e+04

1275

1275

2.677e+51

1.92

79.76

8.385e+04

7.934e+04

1

Eckerle4 1

478.1

478.1

478.1

366

1

1

478.1

478.1

340.6

Eckerle4 2

31.01

1.02

1.026

1

1

1

1

1.003

1

Rat43 1

93.81

2.441

2.44

nan

2.717

2.393

122.5

52.15

1

Rat43 2

1.07

1.001

1

1

1

1

1

1.001

1

Bennett5 1

1294

648.7

1.028

1

1

1

1646

1307

1

Bennett5 2

690.3

3.347

1.165

1

1

1.065

1572

698.8

1

Accuracy for individual “CUTEst” problems

BFGS

Conjugate gradient (Fletcher-Reeves imp.)

Conjugate gradient (Polak-Ribiere imp.)

Damping

Levenberg-Marquardt

Levenberg-MarquardtMD

Simplex

SteepestDescent

Trust Region

PALMER6C

372.9

156

179.2

1.367e+06

1

1.367e+06

373

491.7

nan

PALMER7C

67.84

14.16

16.92

4.478e+05

2415

4.478e+05

74.4

115.4

1

PALMER8C

224.5

20.27

20.86

3.601e+05

338

3.601e+05

118.8

345.2

1

YFITU

4.981e+14

1.495e+14

2.976e+13

1

1

1

4.094e+04

3.382e+14

1

VESUVIOLS

1.613

1.611

1.611

1.508

1

1.423

4.669

1.611

1

DMN15102LS

8.623

4.69

4.412

556.8

4.822

1.336

6.504

4.911

1

Accuracy for individual Neutron Data problems

BFGS

Conjugate gradient (Fletcher-Reeves imp.)

Conjugate gradient (Polak-Ribiere imp.)

Damping

Levenberg-Marquardt

Levenberg-MarquardtMD

Simplex

SteepestDescent

Trust Region

ENGINX 193749 calibration, spectrum 651, peak 19

1.047

1.047

1.047

2.532e+09

1

1.024

1.045

1.399

1.001

ENGINX 193749 calibration, spectrum 651, peak 20

1.478

1.459

1.469

nan

1

1.48

1.457

6.097

1

ENGINX 193749 calibration, spectrum 651, peak 23

1.079

1.096

1.064

nan

1.146

1.089

1.055

11.32

1

ENGINX 193749 calibration, spectrum 651, peak 5

1.286

1.225

1.224

2.34

1

1.003

1.204

1.454

1

ENGINX 193749 calibration, spectrum 651, peak 6

1.047

1.004

1.005

nan

1.094

1.222

1

1.481

1.093

ENGINX 236516 vanadium, bank 1, 10 brk

1

1

1

1.138

1

1.138

1.023

1

1

ENGINX 236516 vanadium, bank 1, 30 brk

1

1

1

1.138

1

1.138

1.106

1

1

EVS14188-90_processed Gaussian peaks 1

1.634

1.63

1.63

2.261

1.723

1

2.396

2.4

1

EVS14188-90_processed Gaussian peaks 1

1.538

1.538

1.538

1

1.063

1

2.337

2.344

1

GEM 13 IC peaks

38.01

5.84

3.141

nan

1.06

1

14.9

37.32

1

WISH17701 tube 3 calibration, peak 1

1.005

5.036

5.081

1

13.44

1

5.281

7.485

1

WISH17701 tube 3 calibration, peak 2

1

1.267

1.189

1

1

1

1.511

3.546

1

WISH17701 tube 3 calibration, peak 3

2.604

10.96

9.291

1

1

1

11.54

21.39

1

WISH17701 tube 3 calibration, peak 4

16.21

16.31

17.19

1

1

1

18.63

36.03

1

WISH17701 tube 3 calibration, peak 5

1.667

1.871

1.643

1

1

1

2.334

3.402

1

WISH17701 tube 3 calibration, peak 6

3.922

3.822

3.923

1

1

1

5.876

16.78

1

WISH17701 tube 3 calibration, peak 7

4.736

1.344

4.82

2.948e+06

1

1

5.245

6.1

1

WISH17701 tube 3 calibration, peak 8

2.154

2.198

1.992

1

1

1

2.799

3.845

1

WISH17701 tube 3 calibration, peak 9

1.133

1.132

1.28

1

1

1

1.283

1.594

1

Comparison in terms of run time

For details on the ranking approach and the color codes see above and the summarized results. Note that this comparison is approximate and was performed on a particular platform (Ubuntu) and machine.

Alternatively, see the summary and detailed results when using weighted least squares as cost function.

Run time for individual NIST problems, “lower” difficulty

BFGS

Conjugate gradient (Fletcher-Reeves imp.)

Conjugate gradient (Polak-Ribiere imp.)

Damping

Levenberg-Marquardt

Levenberg-MarquardtMD

Simplex

SteepestDescent

Trust Region

Misra1a 1

3.309

1

1.123

1.037

1.157

1.06

1.552

1.605

1.254

Misra1a 2

1.037

1.116

1.187

1.01

1.079

1.011

1.182

1.692

1

Chwirut2 1

2.904

12.18

12.32

1.004

1

1.017

1.226

6.411

10.04

Chwirut2 2

1.144

14.81

14.24

1.001

1.077

1

1.132

6.965

10.61

Chwirut1 1

2.805

35.57

9.038

1

1.042

1.169

1.185

15.05

20.69

Chwirut1 2

1.511

6.314

17.72

1.059

1.206

1

1.413

17.15

25.93

Lanczos3 1

1.997

2.014

2.13

1

1.56

2.577

2.396

5.87

1.522

Lanczos3 2

1.358

3.221

2.311

1.013

1

1.04

1.593

9.186

1.126

Gauss1 1

1.854

6.151

33.63

1.106

1

1.025

5.618

41.71

2.911

Gauss1 2

2.692

1.021

1

1.275

1.471

1.302

7.81

58.05

1.899

Gauss2 1

2.597

1

1.347

1.336

1.381

1.415

9.748

60.03

72.46

Gauss2 2

2.251

1

1.263

1.296

1.399

1.366

9.315

60.99

71.38

DanWood 1

5.911

1.425

1.422

1

1.088

1.064

1.145

3.716

1.053

DanWood 2

1.056

1.44

1.367

1

1.007

1.016

1.126

3.488

6.863

Misra1b 1

3.567

1.107

1.116

1.004

1.243

1

1.27

1.79

1.221

Misra1b 2

1.04

1.023

1.08

1.024

1.475

1

1.242

2.032

7.078

Run time for individual NIST problems, “average” difficulty

BFGS

Conjugate gradient (Fletcher-Reeves imp.)

Conjugate gradient (Polak-Ribiere imp.)

Damping

Levenberg-Marquardt

Levenberg-MarquardtMD

Simplex

SteepestDescent

Trust Region

Kirby2 1

1.904

26.47

26.83

1.01

1.089

1

1.725

15.63

1.106

Kirby2 2

1.646

27.81

28.82

1

1.116

1.003

2.125

16.92

1.59

Hahn1 1

2.561

39.16

20.65

2.061

1

1.242

1.556

21.35

24.92

Hahn1 2

1.481

45.04

18.88

1

1.15

1.197

4.776

24.01

27.06

MGH17 1

2.589

1.054

1

1.195

7.081

3.388

1.175

6.162

1.996

MGH17 2

1.179

9.803

10.13

1

1.175

1.301

1.276

8.077

1.2

Lanczos1 1

2.886

2.828

2.975

1

2.054

3.892

3.054

8.937

1.962

Lanczos1 2

1.275

3.053

2.171

1

1.045

1.004

1.472

8.765

1.008

Lanczos2 1

4.376

2.982

2.967

1

2.098

4.038

3.033

8.663

2.11

Lanczos2 2

1.289

3.112

2.005

1.035

1.006

1

1.923

8.702

1.057

Gauss3 1

1.817

44.7

39.07

1

1.096

1.007

6.95

40.6

1.479

Gauss3 2

1.622

38.18

10.42

38.51

1.199

1

6.354

38.78

45.31

Misra1c 1

3.003

1.054

1

1.023

1.024

1.029

1.358

2.201

1.15

Misra1c 2

1.066

1.531

1.053

1

1.065

1.095

1.244

2.17

1.03

Misra1d 1

3.291

1.162

1.215

1

1.114

1.154

1.308

2.02

7.472

Misra1d 2

1.067

1.035

1.006

1.01

1

1.501

1.253

1.581

6.866

ENSO 1

3.148

122.2

114.2

1

86.69

1.139

12.49

80.31

111.8

ENSO 2

3.918

123

75.31

1.013

95.06

1

10.85

79.66

117.9

Run time for individual NIST problems, “higher” difficulty

BFGS

Conjugate gradient (Fletcher-Reeves imp.)

Conjugate gradient (Polak-Ribiere imp.)

Damping

Levenberg-Marquardt

Levenberg-MarquardtMD

Simplex

SteepestDescent

Trust Region

MGH09 1

3.377

1.047

1

1.462

1.306

1.93

1.15

1.363

1.553

MGH09 2

1

1.083

1.019

1.553

1.035

1.228

1.885

2.875

1.239

Thurber 1

2.309

4.403

16.13

1.172

1.004

1

1.045

7.911

1.906

Thurber 2

1.048

10.52

17.23

1.324

1

1.058

3.317

9.513

1.764

BoxBOD 1

2.701

1.186

1

1.226

1.046

1.199

1.066

2.931

1.4

BoxBOD 2

1.227

1.158

1.143

1.095

1

1.1

2.75

3.453

5.641

Rat42 1

3.106

1.28

1.269

1

1.113

1.119

1.173

3.64

5.789

Rat42 2

1.009

5.551

1.721

1

1.104

1.006

1.101

3.639

6.701

MGH10 1

2.822

2.502

2.186

1.083

4.295

3.219

1

2.871

5.13

MGH10 2

1

1.364

1.736

1.567

3.9

3.662

1.03

4.135

1.835

Eckerle4 1

2.676

1

1.021

1.083

1.114

3.506

1.007

1.157

1.431

Eckerle4 2

1.087

1.571

1.54

1.15

1.226

1

1.327

2.771

1.178

Rat43 1

2.515

11.4

3.306

1.477

1

1.127

1.245

5.629

1.269

Rat43 2

1.292

1.691

3.626

1.392

1

1.007

1.013

5.831

8.317

Bennett5 1

2.454

29.53

3.52

1

1.004

10.75

1.08

10.47

1.227

Bennett5 2

1.057

26.43

3.493

1

11.13

9.95

1.284

9.243

1.652

Run time for individual “CUTEst” problems

BFGS

Conjugate gradient (Fletcher-Reeves imp.)

Conjugate gradient (Polak-Ribiere imp.)

Damping

Levenberg-Marquardt

Levenberg-MarquardtMD

Simplex

SteepestDescent

Trust Region

PALMER6C

5.659

39.11

38.28

1

10.81

1.001

1.971

17.68

2.897

PALMER7C

3.822

22.58

20.15

1.192

2.506

1

1.899

20.01

30.98

PALMER8C

3.192

32.92

32.48

1

2.433

1.023

2.76

17.17

28.1

YFITU

2.88

8.55

7.987

1

1.032

1.22

1.979

4.444

1.08

VESUVIOLS

1.396

8.362

5.688

38.52

1.602

1.046

1

37.74

45.01

DMN15102LS

4.42

19.22

15.29

1

7.917

5.026

3.786

22.67

53.85

Run time for individual Neutron Data problems

BFGS

Conjugate gradient (Fletcher-Reeves imp.)

Conjugate gradient (Polak-Ribiere imp.)

Damping

Levenberg-Marquardt

Levenberg-MarquardtMD

Simplex

SteepestDescent

Trust Region

ENGINX 193749 calibration, spectrum 651, peak 19

1.407

3.497

4.965

1

1.916

1.651

3.105

18.34

22.31

ENGINX 193749 calibration, spectrum 651, peak 20

1.26

5.343

5.152

1

1.958

1.267

1.457

15.92

21.66

ENGINX 193749 calibration, spectrum 651, peak 23

1.093

2.561

10.11

1

1.254

1.469

3.082

16.36

21.79

ENGINX 193749 calibration, spectrum 651, peak 5

2.439

16.67

7.301

1

2.653

2.675

3.366

25.02

49.29

ENGINX 193749 calibration, spectrum 651, peak 6

1.228

26.73

22.33

1.018

7.748

1

1.635

15.47

1.644

ENGINX 236516 vanadium, bank 1, 10 brk

14.89

21.38

21.23

1

7.994

1.017

14.5

180.5

4.503

ENGINX 236516 vanadium, bank 1, 30 brk

21.13

53.58

50.38

1.039

9.629

1

7.574

180.1

4.352

EVS14188-90_processed Gaussian peaks 1

31.98

40.88

38.89

1

1.83

7.479

1.271

24.95

4.362

EVS14188-90_processed Gaussian peaks 1

18.35

33.2

33.08

1.613

1

2.204

1.04

20.61

2.048

GEM 13 IC peaks

1.189

21.56

101.7

1

9.353

7.766

15.62

26.93

100.2

WISH17701 tube 3 calibration, peak 1

8.515

5.339

5.301

1.107

1

1.055

1.648

3.757

1.127

WISH17701 tube 3 calibration, peak 2

10.4

5.375

5.607

1

1.017

1.224

1.576

4.426

1.268

WISH17701 tube 3 calibration, peak 3

7.95

5.367

5.636

1.189

1

1.007

1.502

3.782

1.052

WISH17701 tube 3 calibration, peak 4

3.437

6.265

6.307

1

1.094

1.144

1.697

4.308

1.195

WISH17701 tube 3 calibration, peak 5

9.633

5.586

5.461

1.043

1.021

1

1.462

4.114

1.18

WISH17701 tube 3 calibration, peak 6

3.342

5.376

5.292

1

1.337

1.192

1.449

3.869

1.11

WISH17701 tube 3 calibration, peak 7

4.351

6.014

5.729

1

1.094

1.113

1.566

4.127

1.669

WISH17701 tube 3 calibration, peak 8

4.847

6.11

5.976

1

1.047

1.087

1.411

4.306

1.231

WISH17701 tube 3 calibration, peak 9

3.221

5.451

5.535

1.151

1.022

1

1.335

3.888

1.114