Grouping-schema map (as GroupingMap object):

Overview:

Each detector on the instrument is comprised of multiple pixels, which are the smallest physical elements used to measure particle flux. These pixels are identified by their indices (global to the instrument), which are colloquially referred to as detector-ids.

During the first stages of processing data from a specified run, data from groups of pixels are combined and processed together. A grouping schema is used to assign pixels to groups, thereby associating each detector-id with a group number. Such a schema is most commonly represented in Mantid as a grouping workspace. The specific representation of the grouping workspace is as a map between detector-id and group number, implemented as a vector of consecutive spectra, where each spectrum number is the detector-id, and each single value is the corresponding group number.

Depending on the requirements of data processing, several possible grouping schema may be used. Examples of available grouping schemas are, “all”: a single group including all pixels in the instrument; “column”: each group includes pixels belonging to a column of detectors; “bank”: each group includes pixels belonging to a detector bank.

Implementation:

The available grouping schema have been identified by the CIS, and are collected together in a default grouping-schema map represented on-disk as defaultGroupingMap.json, located at <instrument.calibration.powder.grouping.home>. The grouping-schema map provides a mapping between the common name for a grouping schema, and an XML or HDF5 format file which represents the specifics of that schema, which may then be loaded into a grouping workspace. The grouping-schema map is represented on-disk in JSON format, and each schema entry consists of the common-name for the schema, e.g. “column”, followed by either a relative or absolute path to the filename for that schema. In this filename specification any relative path will be relative to <instrument.calibration.powder.grouping.home>, although an absolute path can be used to refer to a grouping schema that isn’t located in that directory. In the JSON file, these lists of available schema are additionally divided into those applicable to native mode, and those applicable to lite mode.

For a specific instrument state, it is possible that only a subset of the default grouping schemas is applicable, or perhaps there may be additional grouping schemas which are required. For this reason, during state initialization a state-registered copy of the JSON-file for the default grouping-schema map groupingMap.json is created at the state root directory. This file may then be edited as required, to either add or exclude applicable grouping schemas. In this state-specific JSON-file, the stateID field is used to register this grouping-schema map to the specific state.

To facilitate editing by hand, the JSON representation of the grouping-schema map consists of two lists of FocusGroup representations. One list includes FocusGroup applicable to the instrument in native mode, and the other list those applicable in lite mode.

Usage:

At any point after a state has been initialized, the state’s grouping-schema map may be accessed using:

  1. Import the necessary classes:

from from snapred.backend.dao.state import GroupingMap
from snapred.backend.data.DataFactoryService import DataFactoryService

  1. Access the state’s grouping map:

runNumber = "52280"
dataService = DataFactoryService()
groupingMap = dataService.getGroupingMap(runNumber)

  1. Obtain the filename for the “column” (lite mode) grouping schema:

groupingScheme = "column"
useLiteMode = True

# A relative path here will refer to <instrument.calibration.powder.grouping.home>
columnFilename = groupingMap.getMap(useLiteMode)[groupingScheme].definition

Initialization specifics:

The template file defaultGroupingMap.json must be filled-in and placed “by hand” at <instrument.calibration.powder.grouping.home> by the CIS.

The creation of the groupingMap.json file located at the state-root directory is triggered at two distinct code locations, whenever a state-root directory doesn’t exist. These trigger locations are at: 1) LocalDataService.readStateConfig, and (2) LocalDataService.initializeState. Once the groupingMap.json file has been created, as it may contain user-specified information, it will never be overwritten.

The GroupingMap data object is an attribute of the StateConfig data object, however it is represented on disk as a separate JSON file. In many cases, the GroupingMap is required when the complete StateConfig is not. For this reason, it is generally accessed using LocalDataService.readGroupingMap (see the “defects” section below), rather than LocalDataService.readStateConfig. This is why two separate initialization paths have been implemented. Generally, the initializeState path is used by the application code and unit-test code, whereas the readStateConfig path at present only seems to be used by the unit-test code.

Validation specifics:

Pydantic validators are used to validate the GroupingMap instance at point of loading. Any grouping-schema files which do not exist (or have an incorrect format) are not loaded to the resident GroupingMap – in these cases, warnings will be logged but the loading will be allowed to continue. When the GroupingMap is loaded using the readStateConfig path, the validity of the GroupingMap.stateId is checked to ensure that it corresponds to that of the state itself.

Potential defects:

  • Although this is a fine point, the fact that we now have a public method LocalDataService.readGroupingMap which falls back automatically to load the defaultGroupingMap.json when the state has not been initialized is a potential defect. During the initial design phase it was intended that the GroupingMap would always be accessed only via its parent StateConfig object. Since accessing the GroupingMap by itself is definitely an acceptable goal, perhaps a better implementation would provide the current _readGroupingMap as the public method, and would produce an error if called on an uninitialized state. On this topic, it will also be noted that this access requirement possibly indicates that GroupingMap should not really be part of Stateconfig at all, or that some aspect of StateConfig needs to be redesigned so that its loading is no longer an issue in these cases (e.g. Optional[Calibration] is a possibility here).

  • If the path provided to a grouping schema (e.g. in defaultGroupingMap.json) is a relative path, it is relative to <instrument.calibration.powder.grouping.home>. This relative-path aspect provides additional information to the user (e.g. that this is a “standard” grouping schema, in some sense). For this reason, it is a defect (to be fixed shortly) that when the groupingMap.json is written to disk, the paths have all been converted to an absolute form.