ISIS SANS Reduction Back-end¶

Introduction and Motivation¶

The ISIS SANS v2 reduction back-end is a more modern and updated version of the original ISIS SANS reduction back-end which has been in use for almost 10 years.

Users who sets up a SANS reduction work-flow have control over a vast number of settings (>50) in addition to settings which are extracted from the provided workspaces and instrument specific settings. The total number of settings which define a SANS data reduction can be close to 100.

The previous implementation of the SANS data reduction stored the settings non-centrally and allowed the overall state to be mutable. This made it extremely hard to reason about the overall state of a data reduction and lead to unnecessary data reloads, degrading the overall performance. Also, the direct coupling of the state to the algorithms does not allow for extending them to other facilities.

The new implementation of the SANS data reduction aims to avoid these pitfalls and focusses on robustness, maintainability and performance. The main way to achieve this is to use a simple state object which stores the reduction-relevant information centrally.

Overview¶

The reduction back-end consists of three components:

• the SANSState approach to centrally store the state of the reduction
• a set of work-flow algorithms which perform the individual reduction steps
• algorithms which orchestrate the work-flow algorithms.

Motivation¶

As mentioned above, the amount of parameters that can be set by the user makes the SANS reduction one of the more complex ones in the Mantid ecosystem. Previous implementations stored the settings non-centrally which led to many difficult-to-find bugs and a lot of uncertainty about the current settings of the reduction as they were changed during the reduction.

This has been the main bottleneck of the previous reduction framework. To overcome this, the new implementation of the SANS data reduction uses a simple state object which stores the reduction-relevant information centrally. This SANSState approach is the corner stone of the new design.

The SANSState is:

• self-validating

• immutable (currently this is not enforced on the object itself, but should be added in the future.

  The reduction code is written however such that it does not make sense to mutate the state while a reduction is running.)

• typed

• serializable

• easy to reason about

• modular (sub-states for units of work)

This approach allows us to identify issues with the settings before a lengthy data reduction has been started.

Components¶

This section describes the essential components of the state mechanism. These include the states themselves, the parameters in a state and the state construction.

State Generation¶

User input can come in the form of user files, the Python interface or the GUI. In some of these cases the order in which the parameters are set is not always in the same order and sometimes a parameter can be set multiple times (e.g. via the user file). These settings are captured in an input dictionary and then processed by builder classes which are coordinated by a state director. These components are described below.

{FitId.general: fit_general(start=1.0,
                            stop=3.0,
                            fit_type=FitType.Polynomial,
                            data_type=DataType.Can,
                            polynomial_order=2)

class StateScaleBuilder(object):
    @automatic_setters(StateScale, exclusions=[])
    def __init__(self, data_info):
      ...

SANSCalculateTransmission¶

The SANSCalculateTransmission algorithm is one of the more complex algorithms in the reduction chain with many sub-steps and a wide variety of parameters which can be set by the users.

The algorithm performs the following steps:

1. Select the incident monitor. If this is not explicitly set then the default value is taken.
2. Select the transmission detector ids. The detector ids are chosen via the following preference:
   1. If available, get detector ids from region-of-interest selection on detector
   2. Else if available get detector ids from transmission monitor setting
   3. Else get default transmission monitor
3. Get the corrected transmission workspace. The sub-steps are:
   1. Load the transmission workspace
   2. Extract the transmission detector ids with ExtractSpectra
   3. Perform prompt peak correction
   4. Perform flat background correction to monitors (if applicable) using CalculateFlatBackground
   5. Perform flat background correction to other detectors (if applicable) using CalculateFlatBackground
   6. Convert to wavelength and rebin using SANSConvertToWavelengthAndRebin
4. Get the corrected direct workspace. The sub-steps are:
   1. Load the direct workspace
   2. Extract the transmission detector ids with ExtractSpectra
   3. Perform prompt peak correction (if applicable) using RemoveBins
   4. Perform flat background correction to monitors (if applicable) using CalculateFlatBackground
   5. Perform flat background correction to other detectors (if applicable) using CalculateFlatBackground
   6. Convert to wavelength and rebin using SANSConvertToWavelengthAndRebin
5. Perform fitting for the transmission calculation. The sub-steps are:
   1. Use incident monitor, wavelength settings, transmission detector ids, fit settings as well as the corrected transmission (step 3) and direct (step4) workspaces to initialize CalculateTransmission
   2. Execute CalculateTransmission
   3. Get the fitted and unfitted output workspaces
6. Set the fitted and unfitted workspaces on the output of the algorithm.

SANSConvertToQ¶

The SANSConvertToQ algorithm is the most essential algorithm in the reduction chain. It coordinates the final conversion from wavelength units to momentum transfer units.

If a 1D reduction has been selected then the algorithm will perform the follow sub-steps:

1. Calculate the momentum transfer resolution workspace using TOFSANSResolutionByPixel (if applicable)
2. Set data workspace, adjustment workspaces, momentum transfer resolution workspace (if applicable), radius and wavelength cut-offs, momentum transfer limits and the gravity correction on Q1D
3. Execute Q1D
4. Get reduced workspace, the sum-of-counts workspace and the sum-of-norm workspaces and set on the output of the algorithm

If a 2D reduction has been selected then the algorithm will perform the following sub-steps:

1. Set data workspace, adjustment workspaces, momentum transfer resolution workspace (if applicable), radius and wavelength cut-offs, momentum transfer limits and the gravity correction on Qxy
2. Execute Qxy
3. Get reduced workspace, the sum-of-counts workspace and the sum-of-norm workspaces and set on the output of the algorithm

SANSConvertToWavelength¶

The SANSConvertToWavelength algorithm acts as a wrapper around SANSConvertToWavelengthAndRebin. Unlike SANSConvertToWavelengthAndRebin it takes a SANSState object as its input. This algorithm is used for the wavelength conversion of the scatter workspace.

SANSConvertToWavelengthAndRebin¶

The SANSConvertToWavelengthAndRebin algorithm is one of the few which does not take a SANSState object as an input.

The algorithm performs the following steps:

1. Unit conversion from time-of-flight units to wavelength units using ConvertUnits
2. Performs a rebin operation using either Rebin or InterpolatingRebin

SANSCreateWavelengthAndPixelAdjustment¶

The SANSCreateWavelengthAndPixelAdjustment algorithm combines the output of the SANSCalculateTransmission algorithm, the output of the SANSNormalizeToMonitor algorithm and flood and direct files to produce the correction workspaces which are required for SANSConvertToQ.

The sub-steps of the algorithm are:

1. Create the wavelength-adjustment workspace. The sub-steps are:
   1. Get the calculate-transmission workspace from the input
   2. Get the normalization-to-monitor workspace from the input
   3. Load the wavelength-adjustment file using LoadRKH
   4. Provide all of the above workspaces with the same binning using Rebin and multiply them using Multiply
2. Create the pixel-adjustment workspace. The sub-states are:
   1. Load the pixel-adjustment file using LoadRKH
   2. Crop the pixel-adjustment workspace to the desired detector using SANSCrop
3. Set the pixel-adjustment and wavelength-adjustment workspaces on the output of the algorithm

SANSCrop¶

The SANSCrop algorithm crops the input workspace to a specified component using CropToComponent.

SANSLoad¶

The SANSLoad algorithm is responsible for loading data and applying the calibration where required. This algorithm loads SANS data sets. The loading can handle nexus and raw files which can be plain or multi-period data. In addition the algorithm has to be able to handle added files. The SANS data sets which can be loaded with this algorithm are:

• sample scatter data which is the actual data under investigation. The algorithm loads the corresponding monitor workspace separately
• sample transmission data
• sample direct data
• can scatter data. The algorithm also loads the corresponding monitor workspace separately
• can transmission data
• can direct data

In addition a calibration file which is applied after the data has been loaded can be specified. The calibration performs micro-adjustments to the detectors.

The algorithm sub-steps are:

1. Based on the input data a loading strategy is selected.
2. For each workspace in the StateData state object we load the data (with the loading strategy from step 1). The sub-states are:
   1. If optimizations are enabled check if the desired workspace already exists on the ADS. If so, fetch it and return it. We are done with loading this data set.
   2. Else get the correct loader strategy (e.g. for event-mode files) and load the data. This will load either all periods or just the specified period where applicable. If scatter data is loaded, then the monitor data is loaded into a separate workspace with the suffix “_monitors”.
3. Apply calibration if required. Note that the algorithm loads the calibration workspace from the ADS if it exists there when optimizations are enabled. Else it loads it from file and places it on the ADS.
4. Set the loaded workspaces on the output of the algorithm.
5. For LOQ apply transmission corrections if applicable. This will apply a different instrument definition for transmission runs.

SANSMaskWorkspace¶

The SANSMaskWorkspace algorithm is responsible for masking detectors and time bins on the scatter workspaces. There are several types of masking which are currently supported:

• Time/Bin masking.
• Radius masking.
• Mask files.
• Spectrum masking which includes individual spectra, spectra ranges, spectra blocks and spectra cross blocks. These masks are partially specified on a detector level (see below).
• Angle masking.
• Beam stop masking.

Note that only those of the following steps are executed where the user has specified a particular masking instruction. The algorithm sub-steps are:

1. Select the correct masking strategy (currently only ISIS)
2. Apply time bin masking. The sub-steps are:
   1. Apply general time bin masks using MaskBins
   2. Apply detector specific time bin masks using MaskBins
3. Apply cylinder masking. This generates a hollow cylinder which masks the beam stop (defined by an inner radius) and anything outside of an area of interest (defined by an outer radius). The sub-steps are:
   1. Set up the inner and the outer radius of the cylinder mask.
   2. Mask everything outside of the hollow cylinder using MaskDetectorsInShape
4. Apply a list of mask files. For each mask file the sub-steps are:
   1. Load the mask file into a workspace using LoadMask
   2. Apply the mask workspace to the scatter workspace using MaskDetectors
5. Apply spectrum masks. The sub-steps are:
   1. Get the spectra masks for single spectra, spectrum ranges, single horizontal spectrum strips, single vertical spectrum strips, horizontal spectrum range (several strips next to each other), vertical spectrum range (several strips next to each other), block masks and block cross masks
   2. Mask the selected spectra using MaskDetectors
6. Apply angle masking. This is used for pizza-slice masking and uses MaskDetectorsInShape
7. Apply beam stop masking. The beam stop consists of a disc where the beam is located and a connection arm (rod) which holds the disc. The disc has at this point already been masked by a cylinder mask. This step masks the connection arm using MaskDetectorsInShape

SANSMove¶

The SANSMove algorithm moves the instrument component of the SANS workspace according to the settings in the state object. Additionally the user can specify the beam centre. Note that if the beam centre is also specified in the state object, then the manual selection takes precedence. The way we perform a move is highly-instrument and in fact data-dependent. Currently the move mechanism is implemented for SANS2D, LOQ, LARMOR and ZOOM.

The main purpose is to shift a freshly loaded data set into its default position. Note that each instrument has its own way of displacing the instrument. In general this is achieved by a combination of translations and rotations using MoveInstrumentComponent and RotateInstrumentComponent.

SANSNormalizeToMonitor¶

The SANSNormalizeToMonitor algorithm provides a monitor normalization workspace for subsequent wavelength correction in Q1D v2 or Qxy v1. The settings of the algorithm are provided by the state object. The user can provide a ScaleFactor which is normally obtained during event slicing.

The sub-steps of this algorithm are:

1. Get the incident monitor spectrum number and the scale factor
2. Extract the monitor spectrum using ExtractSingleSpectrum into a monitor workspace
3. Apply the scale factor to the monitor workspace using Scale
4. Perform a prompt peak correction (if applicable) using RemoveBins
5. Perform a flat background correction (if applicable) using CalculateFlatBackground
6. Convert to wavelength units and rebin using SANSConvertToWavelengthAndRebin

SANSSave¶

The SANSSave algorithm performs two steps:

1. Create a cloned workspace where the zero-error values are inflated (if this is requested)
2. Save the workspace into each specified file format.

Zero-error inflation is useful for data points where the error is 0. When performing any form of regression of this the zero-valued error will fix the model at this point. If we inflate the error at this point then it does not contribute to the regression.

SANSScale¶

The SANSScale algorithm scales a SANS workspace according to the settings in the state object. The scaling includes division by the volume of the sample and multiplication by an absolute scale.

The sub-steps of this algorithm are:

1. Multiply by the absolute scale. The sub-steps are:
   1. If a scale is specified, multiply the scale by 100, else set the scale to 100
   2. If the instrument is LOQ divide by
   3. Multiply the scatter workspace by the scale using Multiply (and CreateSingleValuedWorkspace)
2. Divide by the sample volume. The sub-steps are:
   1. Calculate the sample volume based either on the user settings or the sample information from the file.
   2. Divide by the scatter workspace by the sample volume using Divide (and CreateSingleValuedWorkspace)

SANSSliceEvent¶

The SANSSliceEvent algorithm creates a sliced workspaces from an event-based SANS input workspace according to the settings in the state object. The algorithm will extract a slice based on a start and end time which are set in the state object. In addition, the data type, i.e. if the slice is to be taken from a sample or a can workspace, can be specified. Note that the monitor workspace is not being sliced but scaled by the ratio of the proton charge of the sliced workspace to the proton charnge of the full workspace.

The sub-states of this algorithm are:

1. Get the start time and the end time of the time slice
2. If the data set is from a Can measurement, then don’t perform a slice
3. Slice the scatter workspace using the start and end time and FilterByTime
4. Get the partial charge for the sliced data and calculate the slice factor which is (partial charge) / (total charge)
5. Multiply the monitor workspace with the scale factor
6. Set the sliced scatter data, the scaled monitor data and the slice factor on the output of this algorithm

SANSBatchReduction¶

This class is the entry point for any reduction and is not an algorithm but rather a script. It takes three important inputs:

• A list of SANS state objects. Each state object defines a reduction. In fact if the state object contains period data with periods and time slices it will in fact define reductions.
• A use_optimizations boolean flag. If true, the data loading mechanism will check the ADS first if the required data is available from there and only load the data if it is not present. It will place newly loaded data into the ADS. The ADS is also checked for can reductions.
• An output_mode enum, which can be:
  • PublishToADS means that the reduced data is added to the ADS
  • SaveToFile means that the reduced data is saved only to file
  • Both means that the reduced data is added to the ADS and saved to file

SANSBatchReduction reduces the list of states sequentially. The for-loop in the execute method lends itself to parallelization via MPI, hence this could be a potential future optimization if this should be required. The sub-steps to handle each state object are:

1. Load the data which is relevant for the particular reduction (make use of optimizations if applicable)
2. If the state object contains multi-period data with periods and/or time slices then generate state objects
3. For each state object run the SANSSingleReduction algorithm
4. Group the output workspaces if required, e.g. for reduced multi-period data
5. Provide workspaces to the selected output channel, i.e. ADS, files or both.

Note that SANSBatchReduction also sets the name of the reduced data.

The users can interact with the new SANS reduction back-end either via the GUI or the Python interface. Both of these methods utilize the SANSBatchReduction to perform the reduction.

SANSSingleReduction¶

The SANSSingleReduction algorithm defines a single complete reduction of a data set, i.e. it will run the reduction for the Sample and Can and perform the subtraction of these results if the reduction has been set up to do this. In particular this algorithm stitches the reduced workspaces of the different detectors using SANSStitch, again only if the reduction has been set up to do this.

SANSReductionCore¶

This work-flow algorithm actually defines the order of the reduction steps and is the inner core of the orchestration mechanism. The inputs to this algorithm are

• A SANS state object
• Several input workspaces
• A detector type selection, i.e. LAB or HAB
• A data type selection, i.e. Sample or Can

The sub-steps of this algorithm are:

1. Get the cropped input ScatterWorkspace. The cropping is defined by the selected detector type. The underlying algorithm is SANSCrop.
2. Create an event slice of the input workspace using SANSSliceEvent. Note that event slicing is only applied to event-mode workspaces and only when it has been specified by the user. During this step the scatter workspace is sliced and the associated monitor workspace is scaled. The scaling factor is the ratio of the charge of the sliced data set and the charge of the entire data set.
3. If we are dealing with an even-mode workspace and the compatibility mode has been chosen then either a custom binning or the monitor binning is applied using Rebin or RebinToWorkspace, respectively.
4. Both the data and the monitor workspace perform an initial move operation using SANSMove. The algorithm is applied twice. The first time using the SetToZero mode in case the algorithm had been loaded and moved already previously. This resets the instrument position of the workspace to the positions of the base instrument. The second time the move algorithm is operated in InitialMove mode.
5. The data workspace is masked using SANSMaskWorkspace. Note that only the general masks and the masks for the selected component are applied. Note that all steps up until now were performed in the time-of-flight domain.
6. Convert the data from the time-of-flight to the wavelength domain using SANSConvertToWavelength.
7. Scale the data set using SANSScale. This will multiply the data set with the absolute scale and divide by the sample volume.
8. This step creates the adjustment workspaces using SANSCreateAdjustmentWorkspaces. This uses the input TransmissionWorkspace and DirectWorkspace workspaces. Note that the instruments referenced in the workspace are moved using SANSMove before they are used by the adjustment algorithm. The outputs are a wavelength-adjustment workspace, a pixel-adjustment workspace and a wavelength-and-pixel adjustment workspace. Note that their creation is optional.
9. Convert the data workspace into histogram-mode using RebinToWorkspace. This is only relevant for event-mode workspaces where the compatibility mode has not been used. Up until now the event-mode workspace could be used as an event workspace, but the momentum transfer conversion (the next step) requires a histogram-mode workspace.
10. The final step, the conversion to momentum transfer units, either uses Q1D or Qxy depending on the setting of the reduction dimensionality. This step uses the data workspace as well as all of the adjustment workspaces which have been provided earlier on. The resulting OutputWorkspace and the SumOfCounts as well as SumOfNormFactors counts are provided as outputs.

Category: Development