Reference Files¶
The flat_field step uses four different types of reference files, depending on the
type of data being processed. Most cases just use the FLAT reference file, while NIRSpec
spectroscopic exposures use the three reference files FFLAT (fore optics),
SFLAT (spectrograph optics), and DFLAT (detector).
FLAT Reference File¶
- REFTYPE:
FLAT
- Data model:
The FLAT reference file contains pixel-by-pixel detector response values. It is used for all instrument modes except the NIRSpec spectroscopic modes.
Reference Selection Keywords for FLAT¶
CRDS selects appropriate FLAT references based on the following keywords. FLAT is not applicable for instruments not in the table. Non-standard keywords used for file selection are required.
Instrument |
Keywords |
|---|---|
FGS |
INSTRUME, DETECTOR, EXP_TYPE, DATE-OBS, TIME-OBS |
MIRI |
INSTRUME, DETECTOR, FILTER, BAND, READPATT, SUBARRAY, DATE-OBS, TIME-OBS |
NIRCam |
INSTRUME, DETECTOR, FILTER, PUPIL, DATE-OBS, TIME-OBS |
NIRISS |
INSTRUME, DETECTOR, FILTER, PUPIL, DATE-OBS, TIME-OBS |
NIRSpec |
INSTRUME, DETECTOR, FILTER, GRATING, EXP_TYPE, DATE-OBS, TIME-OBS |
Standard Keywords¶
The following table lists the keywords that are required to be present in all reference files. The first column gives the FITS keyword names. The second column gives the jwst data model name for each keyword, which is useful when using data models in creating and populating a new reference file. The third column gives the equivalent meta tag in ASDF reference file headers, which is the same as the name within the data model meta tree (second column).
FITS Keyword |
Data Model Name |
ASDF meta tag |
|---|---|---|
AUTHOR |
model.meta.author |
author |
DATAMODL |
model.meta.model_type |
model_type |
DATE |
model.meta.date |
date |
DESCRIP |
model.meta.description |
description |
FILENAME |
model.meta.filename |
N/A |
INSTRUME |
model.meta.instrument.name |
instrument: {name} |
PEDIGREE |
model.meta.pedigree |
pedigree |
REFTYPE |
model.meta.reftype |
reftype |
TELESCOP |
model.meta.telescope |
telescope |
USEAFTER |
model.meta.useafter |
useafter |
NOTE: More information on standard required keywords can be found here: Standard Required Keywords
Type Specific Keywords for FLAT¶
In addition to the standard reference file keywords listed above, the following keywords are required in FLAT reference files, because they are used as CRDS selectors (see Reference Selection Keywords for FLAT):
Keyword |
Data Model Name |
Instruments |
|---|---|---|
DETECTOR |
model.meta.instrument.detector |
All |
EXP_TYPE |
model.meta.exposure.type |
FGS, NIRSpec |
FILTER |
model.meta.instrument.filter |
MIRI, NIRCam, NIRISS, NIRSpec |
PUPIL |
model.meta.instrument.pupil |
NIRCam, NIRISS |
BAND |
model.meta.instrument.band |
MIRI |
READPATT |
model.meta.exposure.readpatt |
MIRI |
SUBARRAY |
model.meta.subarray.name |
MIRI |
GRATING |
model.meta.instrument.grating |
NIRSpec |
Reference File Format¶
FLAT reference files are FITS format, with 3 IMAGE extensions and 1 BINTABLE extension. The FITS primary HDU does not contain a data array. The format and content of the file is as follows:
EXTNAME |
XTENSION |
NAXIS |
Dimensions |
Data type |
|---|---|---|---|---|
SCI |
IMAGE |
2 |
ncols x nrows |
float |
ERR |
IMAGE |
2 |
ncols x nrows |
float |
DQ |
IMAGE |
2 |
ncols x nrows |
integer |
DQ_DEF |
BINTABLE |
2 |
TFIELDS = 4 |
N/A |
The DQ_DEF table extension lists the bit assignments for the flag conditions
used in the DQ array.
The DQ_DEF extension contains the bit assignments used in the DQ array. It contains the following 4 columns:
TTYPE |
TFORM |
Description |
|---|---|---|
BIT |
integer |
The bit number, starting at zero |
VALUE |
integer |
The equivalent base-10 value of BIT |
NAME |
string |
The mnemonic name of the data quality condition |
DESCRIPTION |
string |
A description of the data quality condition |
NOTE: For more information on standard bit definitions see: Data Quality Flags.
For application to imaging data, the FITS file contains a single set of SCI, ERR, DQ, and DQ_DEF extensions. Image dimensions should be 2048x2048 for the NIR detectors and 1032x1024 for MIRI (i.e. they include reference pixels), unless data were taken in subarray mode.
Reference Files for NIRSpec Spectroscopy¶
For NIRSpec spectroscopic data, the flat-field reference files allow for variations in the flat field with wavelength, as well as from pixel to pixel. There is a separate flat-field reference file for each of three sections of the instrument: the fore optics (FFLAT), the spectrograph (SFLAT), and the detector (DFLAT). The contents of the reference files differ from one mode to another (see below), but in general they may contain a flat-field image and a 1-D array. The image provides pixel-to-pixel values for the flat field that may vary slowly (or not at all) with wavelength, while the 1-D array is for a pixel-independent fast variation with wavelength. Details of the file formats are given in the following sections.
If there is no significant slow variation with wavelength, the image will be a 2-D array; otherwise, the image will be a 3-D array, with each plane corresponding to a different wavelength. In the latter case, the wavelength for each plane will be given in a table extension called WAVELENGTH in the flat-field reference file. The fast variation is given in a table extension called FAST_VARIATION, with column names “slit_name”, “nelem”, “wavelength”, and “data” (an array of wavelength-dependent flat-field values). Each row of the table contains a slit name (for fixed-slit data, otherwise “ANY”), an array of flat-field values, an array of the corresponding wavelengths, and the number of elements (“nelem”) of “data” and “wavelength” that are populated, because the allocated array size can be larger than needed. For some reference files there will not be any image array, in which case all the flat field information will be taken from the FAST_VARIATION table.
The SCI extension of the reference files may contain NaNs. If so, the
flat_field step will replace these values with 1 and will flag the
corresponding pixel in the DQ extension with NO_FLAT_FIELD. The WAVELENGTH
extension is not expected to contain NaNs.
For the detector section, there is only one flat-field reference file for each detector. For the fore optics and the spectrograph sections, however, there are different flat fields for fixed-slit data, IFU data, and for multi-object spectroscopic data. Here is a summary of the contents of these files.
For the fore optics (FFLAT), the flat field for fixed-slit data contains just a FAST_VARIATION table (i.e. there is no image). This table has five rows, one for each of the fixed slits. The FFLAT for IFU data also contains just a FAST_VARIATION table, but it has only one row with the value “ANY” in the “slit_name” column. For multi-object spectroscopic data, the FFLAT contains four sets of images (one for each MSA quadrant), WAVELENGTH tables, and FAST_VARIATION tables. The images are unique to the FFLATs, however. The image “pixels” correspond to micro-shutter array slits, rather than to detector pixels. The array size is 365 columns by 171 rows, and there are multiple planes to handle the slow variation of flat field with wavelength.
For the spectrograph optics (SFLAT), the flat-field files have nearly the same format for fixed-slit data, IFU, and multi-object data. The difference is that for fixed-slit and IFU data, the image is just a single plane, i.e. the only variation with wavelength is in the FAST_VARIATION table, while there are multiple planes in the image for multi-object spectroscopic data (and therefore there is also a corresponding WAVELENGTH table, with one row for each plane of the image).
For the detector section, the DFLAT file contains a 3-D image (i.e. the flat field at multiple wavelengths), a corresponding WAVELENGTH table, and a FAST_VARIATION table with one row.
As just described, there are 3 types of reference files for NIRSpec (FFLAT, SFLAT, and DFLAT), and within each of these types, there are several formats, which are now described.
FFLAT Reference File¶
- REFTYPE:
FFLAT
There are 3 forms of NIRSpec FFLAT reference files: fixed slit, MSA spec, and IFU. For each type the primary HDU does not contain a data array.
Reference Selection Keywords for FFLAT¶
CRDS selects appropriate FFLAT references based on the following keywords. FFLAT is not applicable for instruments not in the table. Non-standard keywords used for file selection are required.
Instrument |
Keywords |
|---|---|
NIRSpec |
INSTRUME, FILTER, EXP_TYPE, DATE-OBS, TIME-OBS |
Fixed Slit¶
- Data model:
The fixed slit FFLAT files have EXP_TYPE=NRS_FIXEDSLIT, and have a single BINTABLE extension, labeled FAST_VARIATION.
The table contains four columns:
slit_name: string, name of slit
nelem: integer, maximum number of wavelengths
wavelength: float 1-D array, values of wavelength
data: float 1-D array, flat field values for each wavelength
The number of rows in the table is given by NAXIS2, and each row corresponds to a separate slit.
MSA Spec¶
- Data model:
The MSA Spec FFLAT files have EXP_TYPE=NRS_MSASPEC, and contain data pertaining to each of the 4 quadrants. For each quadrant, there is a set of 5 extensions - SCI, ERR, DQ, WAVELENGTH, and FAST_VARIATION. The file also contains a single DQ_DEF extension.
The extensions have the following characteristics:
EXTNAME |
XTENSION |
NAXIS |
Dimensions |
Data type |
|---|---|---|---|---|
SCI |
IMAGE |
3 |
ncols x nrows x nelem |
float |
ERR |
IMAGE |
3 |
ncols x nrows x nelem |
float |
DQ |
IMAGE |
3 |
ncols x nrows x nelem |
integer |
WAVELENGTH |
BINTABLE |
2 |
TFIELDS = 1 |
N/A |
FAST_VARIATION |
BINTABLE |
2 |
TFIELDS = 4 |
N/A |
DQ_DEF |
BINTABLE |
2 |
TFIELDS = 4 |
N/A |
The DQ_DEF extension contains the bit assignments used in the DQ array. It contains the following 4 columns:
TTYPE |
TFORM |
Description |
|---|---|---|
BIT |
integer |
The bit number, starting at zero |
VALUE |
integer |
The equivalent base-10 value of BIT |
NAME |
string |
The mnemonic name of the data quality condition |
DESCRIPTION |
string |
A description of the data quality condition |
NOTE: For more information on standard bit definitions see: Data Quality Flags.
For the 5 extensions that appear multiple times, the EXTVER keyword indicates the quadrant number, 1 to 4. Each plane of the SCI array gives the throughput value for every shutter in the MSA quadrant for the corresponding wavelength, which is specified in the WAVELENGTH table. These wavelength-dependent values are combined with the FAST_VARIATION array, and are then applied to the science spectrum based on the wavelength of each pixel.
The WAVELENGTH table contains a single column:
wavelength: float 1-D array, values of wavelength
Each of these wavelength values corresponds to a single plane of the IMAGE arrays.
The FAST_VARIATION table contains four columns:
slit_name: the string “ANY”
nelem: integer, maximum number of wavelengths
wavelength: float 1-D array, values of wavelength
data: float 1-D array, flat field values for each wavelength
The flat field values in this table are used to account for a wavelength-dependence on a much finer scale than given by the values in the SCI array. There is a single row in this table, which contains 1-D arrays of wavelength and flat-field values. The same wavelength-dependent value is applied to all pixels in a quadrant.
IFU¶
- Data model:
The IFU FFLAT files have EXP_TYPE=NRS_IFU. These have one extension, a BINTABLE extension labeled FAST_VARIATION.
The FAST_VARIATION table contains four columns:
slit_name: the string “ANY”
nelem: integer, maximum number of wavelengths
wavelength: float 1-D array, values of wavelength
data: float 1-D array, flat field values for each wavelength
For each pixel in the science data, the wavelength of the light that fell on that pixel will be determined from the WAVELENGTH array in the science exposure (in the absence of that array, it will be computed using the WCS interface). The flat-field value for that pixel will then be obtained by interpolating within the wavelength and data arrays from the FAST_VARIATION table.
SFLAT Reference File¶
- REFTYPE:
SFLAT
- Data model:
There are 3 types of NIRSpec SFLAT reference files: fixed slit, MSA spec, and IFU. For each type the primary HDU does not contain a data array.
Reference Selection Keywords for SFLAT¶
CRDS selects appropriate SFLAT references based on the following keywords. SFLAT is not applicable for instruments not in the table. Non-standard keywords used for file selection are required.
Instrument |
Keywords |
|---|---|
NIRSpec |
INSTRUME, DETECTOR, FILTER, GRATING, EXP_TYPE, LAMP, OPMODE, DATE-OBS, TIME-OBS |
Fixed Slit¶
The fixed slit references files have EXP_TYPE=NRS_FIXEDSLIT, and have a BINTABLE extension labeled FAST_VARIATION. The table contains four columns:
slit_name: string, name of slit
nelem: integer, maximum number of wavelengths
wavelength: float 1-D array, values of wavelength
data: float 1-D array, flat field values for each wavelength
The number of rows in the table is given by NAXIS2, and each row corresponds to a separate slit.
MSA Spec¶
The MSA Spec SFLAT files have EXP_TYPE=NRS_MSASPEC. They contain 6 extensions, with the following characteristics:
EXTNAME |
XTENSION |
NAXIS |
Dimensions |
Data type |
|---|---|---|---|---|
SCI |
IMAGE |
3 |
ncols x nrows x n_wl |
float |
ERR |
IMAGE |
3 |
ncols x nrows x n_wl |
float |
DQ |
IMAGE |
3 |
ncols x nrows x n_wl |
integer |
WAVELENGTH |
BINTABLE |
2 |
TFIELDS = 1 |
N/A |
FAST_VARIATION |
BINTABLE |
2 |
TFIELDS = 4 |
N/A |
DQ_DEF |
BINTABLE |
2 |
TFIELDS = 4 |
N/A |
The DQ_DEF extension contains the bit assignments used in the DQ array. It contains the following 4 columns:
TTYPE |
TFORM |
Description |
|---|---|---|
BIT |
integer |
The bit number, starting at zero |
VALUE |
integer |
The equivalent base-10 value of BIT |
NAME |
string |
The mnemonic name of the data quality condition |
DESCRIPTION |
string |
A description of the data quality condition |
NOTE: For more information on standard bit definitions see: Data Quality Flags.
The keyword NAXIS3 in the 3 IMAGE extensions specifies the number, n_wl, of monochromatic slices, each of which gives the flat_field value for every pixel for the corresponding wavelength, which is specified in the WAVELENGTH table.
The WAVELENGTH table contains a single column:
wavelength: float 1-D array, values of wavelength
Each of these wavelength values corresponds to a single plane of the IMAGE arrays.
The FAST_VARIATION table contains four columns:
slit_name: the string “ANY”
nelem: integer, maximum number of wavelengths
wavelength: float 1-D array, values of wavelength
data: float 1-D array, flat field values for each wavelength
The flat field values in this table are used to account for a wavelength-dependence on a much finer scale than given by the values in the SCI array. For each pixel in the science data, the wavelength of the light that fell on that pixel will be read from the WAVELENGTH array in the science exposure (if that array is absent, it will be computed using the WCS interface). The flat-field value for that pixel will then be obtained by interpolating within the wavelength and data arrays from the FAST_VARIATION table.
DFLAT Reference File¶
- REFTYPE:
DFLAT
- Data model:
Reference Selection Keywords for DFLAT¶
CRDS selects appropriate DFLAT references based on the following keywords. DFLAT is not applicable for instruments not in the table. Non-standard keywords used for file selection are required.
Instrument |
Keywords |
|---|---|
NIRSpec |
INSTRUME, DETECTOR, EXP_TYPE, DATE-OBS, TIME-OBS |
There is one type of DFLAT reference file, containing 6 extensions with the following characteristics:
EXTNAME |
XTENSION |
NAXIS |
Dimensions |
Data type |
|---|---|---|---|---|
SCI |
IMAGE |
3 |
ncols x nrows x n_wl |
float |
ERR |
IMAGE |
2 |
ncols x nrows |
float |
DQ |
IMAGE |
2 |
ncols x nrows |
integer |
WAVELENGTH |
BINTABLE |
2 |
TFIELDS = 1 |
N/A |
FAST_VARIATION |
BINTABLE |
2 |
TFIELDS = 4 |
N/A |
DQ_DEF |
BINTABLE |
2 |
TFIELDS = 4 |
N/A |
The DQ_DEF extension contains the bit assignments used in the DQ array. It contains the following 4 columns:
TTYPE |
TFORM |
Description |
|---|---|---|
BIT |
integer |
The bit number, starting at zero |
VALUE |
integer |
The equivalent base-10 value of BIT |
NAME |
string |
The mnemonic name of the data quality condition |
DESCRIPTION |
string |
A description of the data quality condition |
NOTE: For more information on standard bit definitions see: Data Quality Flags.
The keyword NAXIS3 in the SCI extension specifies the number, n_wl, of monochromatic slices, each of which gives the flat_field value for every pixel for the corresponding wavelength, which is specified in the WAVELENGTH table.
The WAVELENGTH table contains a single column:
wavelength: float 1-D array, values of wavelength
Each of these wavelength values corresponds to a single plane of the SCI IMAGE array.
The FAST_VARIATION table contains four columns:
slit_name: the string “ANY”
nelem: integer, maximum number of wavelengths
wavelength: float 1-D array, values of wavelength
data: float 1-D array, flat field values for each wavelength
The flat field values in this table are used to account for a wavelength-dependence on a much finer scale than given by the values in the SCI array. There is a single row in this table, which contains 1-D arrays of wavelength and flat-field values. The same wavelength-dependent value is applied to all pixels in a quadrant.