Reference Files

The flat_field step uses four different types of reference files, depending on the type of data being processed. Most cases just use the FLAT reference file, while NIRSpec spectroscopic exposures use the three reference files FFLAT (fore optics), SFLAT (spectrograph optics), and DFLAT (detector).

FLAT Reference File

REFTYPE

FLAT

Data model

FlatModel

The FLAT reference file contains pixel-by-pixel detector response values. It is used for all instrument modes except the NIRSpec spectroscopic modes.

Reference Selection Keywords for FLAT

CRDS selects appropriate FLAT references based on the following keywords. FLAT is not applicable for instruments not in the table. Non-standard keywords used for file selection are required.

Instrument

Keywords

FGS

INSTRUME, DETECTOR, EXP_TYPE, DATE-OBS, TIME-OBS

MIRI

INSTRUME, DETECTOR, FILTER, BAND, READPATT, SUBARRAY, DATE-OBS, TIME-OBS

NIRCam

INSTRUME, DETECTOR, FILTER, PUPIL, DATE-OBS, TIME-OBS

NIRISS

INSTRUME, DETECTOR, FILTER, PUPIL, DATE-OBS, TIME-OBS

NIRSpec

INSTRUME, DETECTOR, FILTER, GRATING, EXP_TYPE, DATE-OBS, TIME-OBS

Standard Keywords

The following table lists the keywords that are required to be present in all reference files. The first column gives the FITS keyword names. The second column gives the jwst data model name for each keyword, which is useful when using data models in creating and populating a new reference file. The third column gives the equivalent meta tag in ASDF reference file headers, which is the same as the name within the data model meta tree (second column).

FITS Keyword

Data Model Name

ASDF meta tag

AUTHOR

model.meta.author

author

DATAMODL

model.meta.model_type

model_type

DATE

model.meta.date

date

DESCRIP

model.meta.description

description

FILENAME

model.meta.filename

N/A

INSTRUME

model.meta.instrument.name

instrument: {name}

PEDIGREE

model.meta.pedigree

pedigree

REFTYPE

model.meta.reftype

reftype

TELESCOP

model.meta.telescope

telescope

USEAFTER

model.meta.useafter

useafter

NOTE: More information on standard required keywords can be found here: Standard Required Keywords

Type Specific Keywords for FLAT

In addition to the standard reference file keywords listed above, the following keywords are required in FLAT reference files, because they are used as CRDS selectors (see Reference Selection Keywords for FLAT):

Keyword

Data Model Name

Instruments

DETECTOR

model.meta.instrument.detector

All

EXP_TYPE

model.meta.exposure.type

FGS, NIRSpec

FILTER

model.meta.instrument.filter

MIRI, NIRCam, NIRISS, NIRSpec

PUPIL

model.meta.instrument.pupil

NIRCam, NIRISS

BAND

model.meta.instrument.band

MIRI

READPATT

model.meta.exposure.readpatt

MIRI

SUBARRAY

model.meta.subarray.name

MIRI

GRATING

model.meta.instrument.grating

NIRSpec

Reference File Format

FLAT reference files are FITS format, with 3 IMAGE extensions and 1 BINTABLE extension. The FITS primary HDU does not contain a data array. The format and content of the file is as follows:

EXTNAME

XTENSION

NAXIS

Dimensions

Data type

SCI

IMAGE

2

ncols x nrows

float

ERR

IMAGE

2

ncols x nrows

float

DQ

IMAGE

2

ncols x nrows

integer

DQ_DEF

BINTABLE

2

TFIELDS = 4

N/A

The DQ_DEF table extension lists the bit assignments for the flag conditions used in the DQ array.

The DQ_DEF extension contains the bit assignments used in the DQ array. It contains the following 4 columns:

TTYPE

TFORM

Description

BIT

integer

The bit number, starting at zero

VALUE

integer

The equivalent base-10 value of BIT

NAME

string

The mnemonic name of the data quality condition

DESCRIPTION

string

A description of the data quality condition

NOTE: For more information on standard bit definitions see: Data Quality Flags.

For application to imaging data, the FITS file contains a single set of SCI, ERR, DQ, and DQ_DEF extensions. Image dimensions should be 2048x2048 for the NIR detectors and 1032x1024 for MIRI (i.e. they include reference pixels), unless data were taken in subarray mode.

Reference Files for NIRSpec Spectroscopy

For NIRSpec spectroscopic data, the flat-field reference files allow for variations in the flat field with wavelength, as well as from pixel to pixel. There is a separate flat-field reference file for each of three sections of the instrument: the fore optics (FFLAT), the spectrograph (SFLAT), and the detector (DFLAT). The contents of the reference files differ from one mode to another (see below), but in general they may contain a flat-field image and a 1-D array. The image provides pixel-to-pixel values for the flat field that may vary slowly (or not at all) with wavelength, while the 1-D array is for a pixel-independent fast variation with wavelength. Details of the file formats are given in the following sections.

If there is no significant slow variation with wavelength, the image will be a 2-D array; otherwise, the image will be a 3-D array, with each plane corresponding to a different wavelength. In the latter case, the wavelength for each plane will be given in a table extension called WAVELENGTH in the flat-field reference file. The fast variation is given in a table extension called FAST_VARIATION, with column names “slit_name”, “nelem”, “wavelength”, and “data” (an array of wavelength-dependent flat-field values). Each row of the table contains a slit name (for fixed-slit data, otherwise “ANY”), an array of flat-field values, an array of the corresponding wavelengths, and the number of elements (“nelem”) of “data” and “wavelength” that are populated, because the allocated array size can be larger than needed. For some reference files there will not be any image array, in which case all the flat field information will be taken from the FAST_VARIATION table.

The SCI extension of the reference files may contain NaNs. If so, the flat_field step will replace these values with 1 and will flag the corresponding pixel in the DQ extension with NO_FLAT_FIELD. The WAVELENGTH extension is not expected to contain NaNs.

For the detector section, there is only one flat-field reference file for each detector. For the fore optics and the spectrograph sections, however, there are different flat fields for fixed-slit data, IFU data, and for multi-object spectroscopic data. Here is a summary of the contents of these files.

For the fore optics (FFLAT), the flat field for fixed-slit data contains just a FAST_VARIATION table (i.e. there is no image). This table has five rows, one for each of the fixed slits. The FFLAT for IFU data also contains just a FAST_VARIATION table, but it has only one row with the value “ANY” in the “slit_name” column. For multi-object spectroscopic data, the FFLAT contains four sets of images (one for each MSA quadrant), WAVELENGTH tables, and FAST_VARIATION tables. The images are unique to the FFLATs, however. The image “pixels” correspond to micro-shutter array slits, rather than to detector pixels. The array size is 365 columns by 171 rows, and there are multiple planes to handle the slow variation of flat field with wavelength.

For the spectrograph optics (SFLAT), the flat-field files have nearly the same format for fixed-slit data, IFU, and multi-object data. The difference is that for fixed-slit and IFU data, the image is just a single plane, i.e. the only variation with wavelength is in the FAST_VARIATION table, while there are multiple planes in the image for multi-object spectroscopic data (and therefore there is also a corresponding WAVELENGTH table, with one row for each plane of the image).

For the detector section, the DFLAT file contains a 3-D image (i.e. the flat field at multiple wavelengths), a corresponding WAVELENGTH table, and a FAST_VARIATION table with one row.

As just described, there are 3 types of reference files for NIRSpec (FFLAT, SFLAT, and DFLAT), and within each of these types, there are several formats, which are now described.

FFLAT Reference File

REFTYPE

FFLAT

There are 3 forms of NIRSpec FFLAT reference files: fixed slit, MSA spec, and IFU. For each type the primary HDU does not contain a data array.

Reference Selection Keywords for FFLAT

CRDS selects appropriate FFLAT references based on the following keywords. FFLAT is not applicable for instruments not in the table. Non-standard keywords used for file selection are required.

Instrument

Keywords

NIRSpec

INSTRUME, FILTER, EXP_TYPE, DATE-OBS, TIME-OBS

Fixed Slit

Data model

NirspecFlatModel

The fixed slit FFLAT files have EXP_TYPE=NRS_FIXEDSLIT, and have a single BINTABLE extension, labeled FAST_VARIATION.

The table contains four columns:

  • slit_name: string, name of slit

  • nelem: integer, maximum number of wavelengths

  • wavelength: float 1-D array, values of wavelength

  • data: float 1-D array, flat field values for each wavelength

The number of rows in the table is given by NAXIS2, and each row corresponds to a separate slit.

MSA Spec

Data model

NirspecQuadFlatModel

The MSA Spec FFLAT files have EXP_TYPE=NRS_MSASPEC, and contain data pertaining to each of the 4 quadrants. For each quadrant, there is a set of 5 extensions - SCI, ERR, DQ, WAVELENGTH, and FAST_VARIATION. The file also contains a single DQ_DEF extension.

The extensions have the following characteristics:

EXTNAME

XTENSION

NAXIS

Dimensions

Data type

SCI

IMAGE

3

ncols x nrows x nelem

float

ERR

IMAGE

3

ncols x nrows x nelem

float

DQ

IMAGE

3

ncols x nrows x nelem

integer

WAVELENGTH

BINTABLE

2

TFIELDS = 1

N/A

FAST_VARIATION

BINTABLE

2

TFIELDS = 4

N/A

DQ_DEF

BINTABLE

2

TFIELDS = 4

N/A

The DQ_DEF extension contains the bit assignments used in the DQ array. It contains the following 4 columns:

TTYPE

TFORM

Description

BIT

integer

The bit number, starting at zero

VALUE

integer

The equivalent base-10 value of BIT

NAME

string

The mnemonic name of the data quality condition

DESCRIPTION

string

A description of the data quality condition

NOTE: For more information on standard bit definitions see: Data Quality Flags.

For the 5 extensions that appear multiple times, the EXTVER keyword indicates the quadrant number, 1 to 4. Each plane of the SCI array gives the throughput value for every shutter in the MSA quadrant for the corresponding wavelength, which is specified in the WAVELENGTH table. These wavelength-dependent values are combined with the FAST_VARIATION array, and are then applied to the science spectrum based on the wavelength of each pixel.

The WAVELENGTH table contains a single column:

  • wavelength: float 1-D array, values of wavelength

Each of these wavelength values corresponds to a single plane of the IMAGE arrays.

The FAST_VARIATION table contains four columns:

  • slit_name: the string “ANY”

  • nelem: integer, maximum number of wavelengths

  • wavelength: float 1-D array, values of wavelength

  • data: float 1-D array, flat field values for each wavelength

The flat field values in this table are used to account for a wavelength-dependence on a much finer scale than given by the values in the SCI array. There is a single row in this table, which contains 1-D arrays of wavelength and flat-field values. The same wavelength-dependent value is applied to all pixels in a quadrant.

IFU

Data model

NirspecFlatModel

The IFU FFLAT files have EXP_TYPE=NRS_IFU. These have one extension, a BINTABLE extension labeled FAST_VARIATION.

The FAST_VARIATION table contains four columns:

  • slit_name: the string “ANY”

  • nelem: integer, maximum number of wavelengths

  • wavelength: float 1-D array, values of wavelength

  • data: float 1-D array, flat field values for each wavelength

For each pixel in the science data, the wavelength of the light that fell on that pixel will be determined from the WAVELENGTH array in the science exposure (in the absence of that array, it will be computed using the WCS interface). The flat-field value for that pixel will then be obtained by interpolating within the wavelength and data arrays from the FAST_VARIATION table.

SFLAT Reference File

REFTYPE

SFLAT

Data model

NirspecFlatModel

There are 3 types of NIRSpec SFLAT reference files: fixed slit, MSA spec, and IFU. For each type the primary HDU does not contain a data array.

Reference Selection Keywords for SFLAT

CRDS selects appropriate SFLAT references based on the following keywords. SFLAT is not applicable for instruments not in the table. Non-standard keywords used for file selection are required.

Instrument

Keywords

NIRSpec

INSTRUME, DETECTOR, FILTER, GRATING, EXP_TYPE, DATE-OBS, TIME-OBS

Fixed Slit

The fixed slit references files have EXP_TYPE=NRS_FIXEDSLIT, and have a BINTABLE extension labeled FAST_VARIATION. The table contains four columns:

  • slit_name: string, name of slit

  • nelem: integer, maximum number of wavelengths

  • wavelength: float 1-D array, values of wavelength

  • data: float 1-D array, flat field values for each wavelength

The number of rows in the table is given by NAXIS2, and each row corresponds to a separate slit.

MSA Spec

The MSA Spec SFLAT files have EXP_TYPE=NRS_MSASPEC. They contain 6 extensions, with the following characteristics:

EXTNAME

XTENSION

NAXIS

Dimensions

Data type

SCI

IMAGE

3

ncols x nrows x n_wl

float

ERR

IMAGE

3

ncols x nrows x n_wl

float

DQ

IMAGE

3

ncols x nrows x n_wl

integer

WAVELENGTH

BINTABLE

2

TFIELDS = 1

N/A

FAST_VARIATION

BINTABLE

2

TFIELDS = 4

N/A

DQ_DEF

BINTABLE

2

TFIELDS = 4

N/A

The DQ_DEF extension contains the bit assignments used in the DQ array. It contains the following 4 columns:

TTYPE

TFORM

Description

BIT

integer

The bit number, starting at zero

VALUE

integer

The equivalent base-10 value of BIT

NAME

string

The mnemonic name of the data quality condition

DESCRIPTION

string

A description of the data quality condition

NOTE: For more information on standard bit definitions see: Data Quality Flags.

The keyword NAXIS3 in the 3 IMAGE extensions specifies the number, n_wl, of monochromatic slices, each of which gives the flat_field value for every pixel for the corresponding wavelength, which is specified in the WAVELENGTH table.

The WAVELENGTH table contains a single column:

  • wavelength: float 1-D array, values of wavelength

Each of these wavelength values corresponds to a single plane of the IMAGE arrays.

The FAST_VARIATION table contains four columns:

  • slit_name: the string “ANY”

  • nelem: integer, maximum number of wavelengths

  • wavelength: float 1-D array, values of wavelength

  • data: float 1-D array, flat field values for each wavelength

The flat field values in this table are used to account for a wavelength-dependence on a much finer scale than given by the values in the SCI array. For each pixel in the science data, the wavelength of the light that fell on that pixel will be read from the WAVELENGTH array in the science exposure (if that array is absent, it will be computed using the WCS interface). The flat-field value for that pixel will then be obtained by interpolating within the wavelength and data arrays from the FAST_VARIATION table.

DFLAT Reference File

REFTYPE

DFLAT

Data model

NirspecFlatModel

Reference Selection Keywords for DFLAT

CRDS selects appropriate DFLAT references based on the following keywords. DFLAT is not applicable for instruments not in the table. Non-standard keywords used for file selection are required.

Instrument

Keywords

NIRSpec

INSTRUME, DETECTOR, EXP_TYPE, DATE-OBS, TIME-OBS

There is one type of DFLAT reference file, containing 6 extensions with the following characteristics:

EXTNAME

XTENSION

NAXIS

Dimensions

Data type

SCI

IMAGE

3

ncols x nrows x n_wl

float

ERR

IMAGE

2

ncols x nrows

float

DQ

IMAGE

2

ncols x nrows

integer

WAVELENGTH

BINTABLE

2

TFIELDS = 1

N/A

FAST_VARIATION

BINTABLE

2

TFIELDS = 4

N/A

DQ_DEF

BINTABLE

2

TFIELDS = 4

N/A

The DQ_DEF extension contains the bit assignments used in the DQ array. It contains the following 4 columns:

TTYPE

TFORM

Description

BIT

integer

The bit number, starting at zero

VALUE

integer

The equivalent base-10 value of BIT

NAME

string

The mnemonic name of the data quality condition

DESCRIPTION

string

A description of the data quality condition

NOTE: For more information on standard bit definitions see: Data Quality Flags.

The keyword NAXIS3 in the SCI extension specifies the number, n_wl, of monochromatic slices, each of which gives the flat_field value for every pixel for the corresponding wavelength, which is specified in the WAVELENGTH table.

The WAVELENGTH table contains a single column:

  • wavelength: float 1-D array, values of wavelength

Each of these wavelength values corresponds to a single plane of the SCI IMAGE array.

The FAST_VARIATION table contains four columns:

  • slit_name: the string “ANY”

  • nelem: integer, maximum number of wavelengths

  • wavelength: float 1-D array, values of wavelength

  • data: float 1-D array, flat field values for each wavelength

The flat field values in this table are used to account for a wavelength-dependence on a much finer scale than given by the values in the SCI array. There is a single row in this table, which contains 1-D arrays of wavelength and flat-field values. The same wavelength-dependent value is applied to all pixels in a quadrant.