Introduction

This document is intended to be a core reference guide to the formats, naming convention and data quality flags used by the reference files for pipeline steps requiring them, and is not intended to be a detailed description of each of those pipeline steps. It also does not give details on pipeline steps that do not use reference files. The present manual is referred to by several other documentation pages, such as the JWST pipeline and JDocs.

Reference File Naming Convention

Before reference files are ingested into CRDS, they are renamed following a convention used by the pipeline. As with any other changes undergone by the reference files, the previous names are kept in header keywords, so the Instrument Teams can easily track which delivered file is being used by the pipeline in each step.

The naming of reference files uses the following syntax:

jwst_<instrument>_<reftype>_<version>.<extension>

where

• instrument is one of “fgs”, “miri”, “nircam”, “niriss”, and “nirspec”

• reftype is one of the type names listed in the table below

• version is a 4-digit version number (e.g. 0042)

• extension gives the file format, such as “fits” or “asdf”

An example NIRCam GAIN reference file name would be “jwst_nircam_gain_0042.fits”.

The HISTORY header keyword of each reference file includes details on specific processing undergone by the files before being ingested in CRDS.

Reference File Types

Most reference files have a one-to-one relationship with calibration steps, e.g. there is one step that uses one type of reference file. Some steps, however, use several types of reference files and some reference file types are used by more than one step. The tables below show the correspondence between pipeline steps and reference file types. The first table is ordered by pipeline step, while the second is ordered by reference file type. Links to the reference file types provide detailed documentation on each reference file.

Pipeline Step	Reference File Type (REFTYPE)
align_refs	PSFMASK
ami_analyze	THROUGHPUT
assign_wcs	CAMERA
	COLLIMATOR
	DISPERSER
	DISTORTION
	FILTEROFFSET
	FORE
	FPA
	IFUFORE
	IFUPOST
	IFUSLICER
	MSA
	OTE
	SPECWCS
	REGIONS
	WAVELENGTHRANGE
background	WFSSBKG
	WAVELENGTHRANGE
barshadow	BARSHADOW
cube_build	CUBEPAR
dark_current	DARK
dq_init	MASK
emicorr	EMICORR
extract_1d	EXTRACT1D
	APCORR
	SPECKERNEL (NIRISS SOSS ATOCA only)
	SPECPROFILE (NIRISS SOSS ATOCA only)
	SPECTRACE (NIRISS SOSS ATOCA only)
	WAVEMAP (NIRISS SOSS ATOCA only)
extract_2d	WAVECORR
	WAVELENGTHRANGE
flatfield	FLAT
	DFLAT
	FFLAT
	SFLAT
fringe	FRINGE
gain_scale	GAIN
ipc	IPC
jump	GAIN
	READNOISE
linearity	LINEARITY
msaflagopen	MSAOPER
pathloss	PATHLOSS
persistence	PERSAT
	TRAPDENSITY
	TRAPPARS
photom	PHOTOM
	AREA
ramp_fitting	GAIN
	READNOISE
refpix	REFPIX
resample	DRIZPARS
reset	RESET
residual_fringe	FRINGEFREQ
	REGIONS
rscd	RSCD
saturation	SATURATION
source_catalog	APCORR
	ABVEGAOFFSET
straylight	MRSXARTCORR
spectral_leak	MRSPTCORR
superbias	SUPERBIAS
tso_photometry	TSOPHOT
wavecorr	WAVECORR

Reference File Type (REFTYPE)	Pipeline Step
ABVEGAOFFSET	source_catalog
APCORR	extract_1d
	source_catalog
AREA	photom
BARSHADOW	barshadow
CAMERA	assign_wcs
COLLIMATOR	assign_wcs
CUBEPAR	cube_build
DARK	dark_current
DFLAT	flatfield
DISPERSER	assign_wcs
DISTORTION	assign_wcs
DRIZPARS	resample
EMICORR	emicorr
EXTRACT1D	extract_1d
FFLAT	flatfield
FILTEROFFSET	assign_wcs
FLAT	flatfield
FORE	assign_wcs
FPA	assign_wcs
FRINGE	fringe
FRINGEFREQ	residual_fringe
GAIN	gain_scale
	jump
	ramp_fitting
IFUFORE	assign_wcs
IFUPOST	assign_wcs
IFUSLICER	assign_wcs
IPC	ipc
LINEARITY	linearity
MASK	dq_init
MRSPTCORR	spectral_leak
MRSXARTCORR	straylight
MSA	assign_wcs
MSAOPER	msaflagopen
OTE	assign_wcs
PATHLOSS	pathloss
PERSAT	persistence
PHOTOM	photom
PSFMASK	align_refs
READNOISE	jump
	ramp_fitting
REFPIX	refpix
REGIONS	assign_wcs
	residual_fringe
RESET	reset
RSCD	rscd
SATURATION	saturation
SFLAT	flatfield
SPECWCS	assign_wcs
SUPERBIAS	superbias
THROUGHPUT	ami_analyze
TRAPDENSITY	persistence
TRAPPARS	persistence
TSOPHOT	tso_photometry
WAVELENGTHRANGE	assign_wcs
	background
	extract_2d
WAVECORR	wavecorr
WFSSBKG	background

Step Parameters Reference Types

When each Step is instantiated, a CRDS look-up, based on the Step class name and input data, is made to retrieve a parameter file. The reftype for such parameter files is pars-<class name>. For example, for the step jwst.persistence.PersistenceStep, the reftype would be pars-persistencestep.

For more information, see Parameter Files.

Standard Required Keywords

At present, most JWST science and reference files are FITS files with image or table extensions. The FITS primary data unit is always empty. The primary header contains all keywords not specific to individual extensions. Keywords specific to a particular extension are contained in the header of that extension.

The required Keywords Documenting Contents of Reference Files are:

Keyword	Comment
REFTYPE	WFSSBKG    Required values are listed in the discussion of each pipeline step.
DESCRIP	Summary of file content and/or reason for delivery
AUTHOR	Fred Jones     Person(s) who created the file
USEAFTER	YYYY-MM-DDThh:mm:ss Date and time after the reference files will be used. The T is required. Time string may NOT be omitted; use T00:00:00 if no meaningful value is available.
PEDIGREE	Options are 'SIMULATION' 'GROUND' 'DUMMY' 'INFLIGHT YYYY-MM-DD YYYY-MM-DD'
HISTORY	Description of Reference File Creation
HISTORY	DOCUMENT: Name of document describing the strategy and algorithms used to create file.
HISTORY	SOFTWARE: Description, version number, location of software used to create file.
HISTORY	DATA USED: Data used to create file
HISTORY	DIFFERENCES: How is this version different from the one that it replaces?
HISTORY	If your text spills over to the next line, begin it with another HISTORY keyword, as in this example.
TELESCOP	JWST   Name of the telescope/project.
INSTRUME	FGS   Instrument name. Allowed values: FGS, NIRCAM, NIRISS, NIRSPEC, MIRI
SUBARRAY	FULL, GENERIC, SUBS200A1, ...   (XXX abstract technical description of SUBARRAY)
SUBSTRT1	1        Starting pixel index along axis 1 (1-indexed)
SUBSIZE1	2048     Size of subarray along axis 1
SUBSTRT2	1        Starting pixel index along axis 2 (1-indexed)
SUBSIZE2	2048     Size of subarray along axis 2
FASTAXIS	1        Fast readout direction relative to image axes for Amplifier #1 (1 = +x axis, 2 = +y axis, -1 = -x axis, -2 = -y axis) SEE NOTE BELOW.
SLOWAXIS	2        Slow readout direction relative to image axes for all amplifiers (1 = +x axis, 2 = +y axis, -1 = -x axis, -2 = -y axis)

Observing Mode Keywords

A pipeline module may require separate reference files for each instrument, detector, filter, observation date, etc. The values of these parameters must be included in the reference file header. The observing-mode keyword values are vital to the process of ingesting reference files into CRDS, as they are used to establish the mapping between observing modes and specific reference files. Some observing-mode keywords are also used in the pipeline processing steps. If an observing-mode keyword is irrelevant to a particular observing mode (such as GRATING for the MIRI imager mode or the NIRCam and NIRISS instruments), then it may be omitted from the file header.

The Keywords Documenting the Observing Mode are:

Keyword	Sample Value	Comment
PUPIL	NRM	Pupil wheel element. Required only for NIRCam and NIRISS. NIRCam allowed values: CLEAR, F162M, F164N, F323N, F405N, F466N, F470N, GRISMV2, GRISMV3 NIRISS allowed values: CLEARP, F090W, F115W, F140M, F150W, F158M, F200W, GR700XD, NRM
FILTER	F2100W	Filter wheel element. Allowed values: too many to list here
GRATING	G395M	Required only for NIRSpec. NIRSpec allowed values: G140M, G235M, G395M, G140H, G235H, G395H, PRISM, MIRROR
EXP_TYPE	MIR_MRS	Exposure type. FGS allowed values: FGS_IMAGE, FGS_FOCUS, FGS_SKYFLAT, FGS_INTFLAT, FGS_DARK MIRI allowed values: MIR_IMAGE, MIR_TACQ, MIR_LYOT, MIR_4QPM, MIR_LRS-FIXEDSLIT, MIR_LRS-SLITLESS, MIR_MRS, MIR_DARK, MIR_FLATIMAGE, MIR_FLATMRS, MIR_CORONCAL NIRCam allowed values: NRC_IMAGE, NRC_GRISM, NRC_TACQ, NRC_TACONFIRM, NRC_CORON, NRC_TSIMAGE, NRC_TSGRISM, NRC_FOCUS, NRC_DARK, NRC_FLAT, NRC_LED NIRISS allowed values: NIS_IMAGE, NIS_TACQ, NIS_TACONFIRM, NIS_WFSS, NIS_SOSS, NIS_AMI, NIS_FOCUS, NIS_DARK, NIS_LAMP NIRSpec allowed values: NRS_TASLIT, NRS_TACQ, NRS_TACONFIRM, NRS_CONFIRM, NRS_FIXEDSLIT, NRS_AUTOWAVE, NRS_IFU, NRS_MSASPEC, NRS_AUTOFLAT, NRS_IMAGE, NRS_FOCUS, NRS_DARK, NRS_LAMP, NRS_BOTA, NRS_BRIGHTOBJ, NRS_MIMF
DETECTOR	MIRIFULONG	Allowed values: GUIDER1, GUIDER2 NIS NRCA1, NRCA2, NRCA3, NRCA4, NRCB1, NRCB2, NRCB3, NRCB4, NRCALONG, NRCBLONG NRS1, NRS2 MIRIFULONG, MIRIFUSHORT, MIRIMAGE
CHANNEL	12	MIRI MRS (IFU) channel. Allowed values: 1, 2, 3, 4, 12, 34 SHORT NIRCam channel. Allowed values: SHORT, LONG
BAND	MEDIUM	IFU band. Required only for MIRI. Allowed values are SHORT, MEDIUM, LONG, and N/A, as well as any allowable combination of two values (SHORT-MEDIUM, LONG-SHORT, etc.). (Also used as a header keyword for selection of all MIRI Flat files, Imager included.)
READPATT	FAST	Name of the readout pattern used for the exposure. Each pattern represents a particular combination of parameters like nframes and groups. For MIRI, FAST and SLOW refer to the rate at which the detector is read. MIRI allowed values: SLOW, FAST, FASTGRPAVG, FASTINTAVG NIRCam allowed values: DEEP8, DEEP2, MEDIUM8, MEDIUM2, SHALLOW4, SHALLOW2, BRIGHT2, BRIGHT1, RAPID NIRSpec allowed values: NRSRAPID, NRS, NRSN16R4, NRSIRS2RAPID NIRISS allowed values: NIS, NISRAPID FGS allowed values: ID, ACQ1, ACQ2, TRACK, FINEGUIDE, FGS60, FGS840, FGS7850, FGSRAPID, FGS
NRS_NORM	16	Required only for NIRSpec.
NRS_REF	4	Required only for NIRSpec.
P_XXXXXX	P_READPA	pattern keywords used by CRDS for JWST to describe the intended uses of a reference file using or’ed combinations of values. Only a subset of P_pattern keywords are supported.

Note: For the NIR detectors, the fast readout direction changes sign from one amplifier to the next. It is +1, -1, +1, and -1, for amps 1, 2, 3, and 4, respectively. The keyword FASTAXIS refers specifically to amp 1. That way, it is entirely correct for single-amp readouts and correct at the origin for 4-amp readouts. For MIRI, FASTAXIS is always +1.

Tracking Pipeline Progress

As each pipeline step is applied to a science data product, it will record a status indicator in a header keyword of the science data product. The current list of step status keyword names is given in the following table. These status keywords may be included in the primary header of reference files, in order to maintain a history of the data that went into creating the reference file. Allowed values for the status keywords are ‘COMPLETE’ and ‘SKIPPED’. Absence of a particular keyword is understood to mean that step was not even attempted.

Table 1. Keywords Documenting Which Pipeline Steps Have Been Performed.

S_AMIANA	AMI fringe analysis
S_AMIAVG	AMI fringe averaging
S_AMINOR	AMI fringe normalization
S_BARSHA	Bar shadow correction
S_BKDSUB	Background subtraction
S_COMB1D	1-D spectral combination
S_DARK	Dark subtraction
S_DQINIT	DQ initialization
S_EXTR1D	1-D spectral extraction
S_EXTR2D	2-D spectral extraction
S_FLAT	Flat field correction
S_FRINGE	Fringe correction
S_FRSTFR	MIRI first frame correction
S_GANSCL	Gain scale correction
S_GRPSCL	Group scale correction
S_GUICDS	Guide mode CDS computation
S_IFUCUB	IFU cube creation
S_IMPRNT	NIRSpec MSA imprint subtraction
S_IPC	IPC correction
S_JUMP	Jump detection
S_KLIP	Coronagraphic PSF subtraction
S_LASTFR	MIRI last frame correction
S_LINEAR	Linearity correction
S_MIREMI	MIRI EMI correction
S_MRSMAT	MIRI MRS background matching
S_MSAFLG	NIRSpec MSA failed shutter flagging
S_OUTLIR	Outlier detection
S_PERSIS	Persistence correction
S_PHOTOM	Photometric (absolute flux) calibration
S_PSFALI	Coronagraphic PSF alignment
S_PSFSTK	Coronagraphic PSF stacking
S_PTHLOS	Pathloss correction
S_RAMP	Ramp fitting
S_REFPIX	Reference pixel correction
S_RESAMP	Resampling (drizzling)
S_RESET	MIRI reset correction
S_RSCD	MIRI RSCD correction
S_SATURA	Saturation check
S_SKYMAT	Sky matching
S_SRCCAT	Source catalog creation
S_SRCTYP	Source type determination
S_STRAY	Straylight correction
S_SUPERB	Superbias subtraction
S_TELEMI	Telescope emission correction
S_TSPHOT	TSO imaging photometry
S_TWKREG	Tweakreg image alignment
S_WCS	WCS assignment
S_WFSCOM	Wavefront sensing image combination
S_WHTLIT	TSO white-light curve generation

Orientation of Detector Image

All steps in the pipeline assume the data are in the DMS (science) orientation, not the native readout orientation. The pipeline does NOT check or correct for the orientation of the reference data. It assumes that all files ingested into CRDS have been put into the science orientation. All header keywords documenting the observing mode (Table 2) should likewise be transformed into the DMS orientation. For square data array dimensions it’s not possible to infer the actual orientation directly so reference file authors must manage orientation carefully.

Table 2. Correct values for FASTAXIS and SLOWAXIS for each detector.

DETECTOR	FASTAXIS	SLOWAXIS
MIRIMAGE	1	2
MIRIFULONG	1	2
MIRIFUSHORT	1	2
NRCA1	-1	2
NRCA2	1	-2
NRCA3	-1	2
NRCA4	1	-2
NRCALONG	-1	2
NRCB1	1	-2
NRCB2	-1	2
NRCB3	1	-2
NRCB4	-1	2
NRCBLONG	1	-2
NRS1	2	1
NRS2	-2	-1
NIS	-2	-1
GUIDER1	-2	-1
GUIDER2	2	-1

Differing values for these keywords will be taken as an indicator that neither the keyword value nor the array orientation are correct.

P_pattern keywords

P_ pattern keywords used by CRDS for JWST to describe the intended uses of a reference file using or’ed combinations

For example, if the same NIRISS SUPERBIAS should be used for

READPATT=NIS

or

READPATT=NISRAPID

the definition of READPATT in the calibration s/w datamodels schema does not allow it. READPATT can specify one or the other but not both.

To support expressing combinations of values, CRDS and the CAL s/w have added “pattern keywords” which nominally begin with P_ followed by the ordinary keyword, truncated as needed to 8 characters. In this case, P_READPA corresponds to READPATT.

Pattern keywords override the corresponding ordinary keyword for the purposes of automatically updating CRDS rmaps. Pattern keywords describe intended use.

In this example, the pattern keyword:

P_READPA = NIS | NISRAPID |

can be used to specify the intent “use for NIS or for NISRAPID”.

Only or-ed combinations of the values used in ordinary keywords are valid for pattern keywords.

Patterns appear in a slightly different form in rmaps than they do in P_ keywords. The value of a P_ keyword always ends with a trailing or-bar. In rmaps, no trailing or-bar is used so the equivalent of the above in an rmap is:

‘NIS|NISRAPID’

From a CRDS perspective, the P_ pattern keywords and their corresponding datamodels paths currently supported can be found in the JWST Pattern Keywords section of the CRDS documentation.

Currently all P_ keywords correspond to basic keywords found only in the primary headers of reference files and are typically only valid for FITS format..

The translation from these P_ pattern keywords are completely generic in CRDS and can apply to any reference file type so they should be assumed to be reserved whether a particular type uses them or not. Defining non-pattern keywords with the prefix P_ is strongly discouraged.

Data Quality Flags

Within science data files, the PIXELDQ flags are stored as 32-bit integers; the GROUPDQ flags are 8-bit integers. The meaning of each bit is specified in a separate binary table extension called DQ_DEF. The binary table has the format presented in Table 3, which represents the master list of DQ flags. Only the first eight entries in the table below are relevant to the GROUPDQ array. All calibrated data from a particular instrument and observing mode have the same set of DQ flags in the same (bit) order. For Build 7, this master list will be used to impose this uniformity. We may eventually use different master lists for different instruments or observing modes.

Within reference files for some steps, the Data Quality arrays for some steps are stored as 8-bit integers to conserve memory. Only the flags actually used by a reference file are included in its DQ array. The meaning of each bit in the DQ array is stored in the DQ_DEF extension, which is a binary table having the following fields: Bit, Value, Name, and Description.

Table 3. Flags for the DQ, PIXELDQ, and GROUPDQ Arrays (Format of DQ_DEF Extension).

Bit	Value	Name	Description
0	1	DO_NOT_USE	Bad pixel. Do not use.
1	2	SATURATED	Pixel saturated during exposure
2	4	JUMP_DET	Jump detected during exposure
3	8	DROPOUT	Data lost in transmission
4	16	OUTLIER	Flagged by outlier detection
5	32	PERSISTENCE	High persistence
6	64	AD_FLOOR	Below A/D floor
7	128	CHARGELOSS	Charge Migration
8	256	UNRELIABLE_ERROR	Uncertainty exceeds quoted error
9	512	NON_SCIENCE	Pixel not on science portion of detector
10	1024	DEAD	Dead pixel
11	2048	HOT	Hot pixel
12	4096	WARM	Warm pixel
13	8192	LOW_QE	Low quantum efficiency
14	16384	RC	RC pixel
15	32768	TELEGRAPH	Telegraph pixel
16	65536	NONLINEAR	Pixel highly nonlinear
17	131072	BAD_REF_PIXEL	Reference pixel cannot be used
18	262144	NO_FLAT_FIELD	Flat field cannot be measured
19	524288	NO_GAIN_VALUE	Gain cannot be measured
20	1048576	NO_LIN_CORR	Linearity correction not available
21	2097152	NO_SAT_CHECK	Saturation check not available
22	4194304	UNRELIABLE_BIAS	Bias variance large
23	8388608	UNRELIABLE_DARK	Dark variance large
24	16777216	UNRELIABLE_SLOPE	Slope variance large (i.e., noisy pixel)
25	33554432	UNRELIABLE_FLAT	Flat variance large
26	67108864	OPEN	Open pixel (counts move to adjacent pixels)
27	134217728	ADJ_OPEN	Adjacent to open pixel
28	268435456	FLUX_ESTIMATED	Pixel flux estimated due to missing/bad data
29	536870912	MSA_FAILED_OPEN	Pixel sees light from failed-open shutter
30	1073741824	OTHER_BAD_PIXEL	A catch-all flag
31	2147483648	REFERENCE_PIXEL	Pixel is a reference pixel

Note: Words like “highly” and “large” will be defined by each instrument team. They are likely to vary from one detector to another – or even from one observing mode to another.

Parameter Specification

There are a number of steps, such as OutlierDetectionStep or SkyMatchStep, that define what data quality flags a pixel is allowed to have to be considered in calculations. Such parameters can be set in a number of ways.

First, the flag can be defined as the integer sum of all the DQ bit values from the input images DQ arrays that should be considered “good”. For example, if pixels in the DQ array can have combinations of 1, 2, 4, and 8 and one wants to consider DQ flags 2 and 4 as being acceptable for computations, then the parameter value should be set to “6” (2+4). In this case a pixel having DQ values 2, 4, or 6 will be considered a good pixel, while a pixel with a DQ value, e.g., 1+2=3, 4+8=”12”, etc. will be flagged as a “bad” pixel.

Alternatively, one can enter a comma-separated or ‘+’ separated list of integer bit flags that should be summed to obtain the final “good” bits. For example, both “4,8” and “4+8” are equivalent to a setting of “12”.

Finally, instead of integers, the JWST mnemonics, as defined above, may be used. For example, all the following specifications are equivalent:

"12" == "4+8" == "4, 8" == "JUMP_DET, DROPOUT"

Note

The default value (0) will make all non-zero pixels in the DQ mask be considered “bad” pixels and the corresponding pixels will not be used in computations.

Setting to None will turn off the use of the DQ array for computations.

In order to reverse the meaning of the flags from indicating values of the “good” DQ flags to indicating the “bad” DQ flags, prepend ‘~’ to the string value. For example, in order to exclude pixels with DQ flags 4 and 8 for computations and to consider as “good” all other pixels (regardless of their DQ flag), use a value of ~4+8, or ~4,8. A string value of ~0 would be equivalent to a setting of None.