Introduction¶

This document is intended to be a core reference guide to the formats, naming convention and data quality flags used by the reference files for pipeline steps requiring them, and is not intended to be a detailed description of each of those pipeline steps. It also does not give details on pipeline steps that do not use reference files. The present manual is referred to by several other documentation pages, such as the JWST pipeline and JDocs.

Reference File Naming Convention¶

Before reference files are ingested into CRDS, they are renamed following a convention used by the pipeline. As with any other changes undergone by the reference files, the previous names are kept in header keywords, so the Instrument Teams can easily track which delivered file is being used by the pipeline in each step.

The naming of reference files uses the following syntax:

jwst_<instrument>_<reftype>_<version>.<extension>


where

• instrument is one of “fgs”, “miri”, “nircam”, “niriss”, and “nirspec”

• reftype is one of the type names listed in the table below

• version is a 4-digit version number (e.g. 0042)

• extension gives the file format, such as “fits” or “asdf”

An example NIRCam GAIN reference file name would be “jwst_nircam_gain_0042.fits”.

The HISTORY header keyword of each reference file includes details on specific processing undergone by the files before being ingested in CRDS.

Reference File Types¶

Most reference files have a one-to-one relationship with calibration steps, e.g. there is one step that uses one type of reference file. Some steps, however, use several types of reference files and some reference file types are used by more than one step. The tables below show the correspondence between pipeline steps and refernece file types. The first table is ordered by pipeline step, while the second is ordered by reference file type. Links to the reference file types provide detailed documentation on each reference file.

Pipeline Step

Reference File Type (REFTYPE)

align_refs

ami_analyze

THROUGHPUT

assign_wcs

CAMERA

COLLIMATOR

DISPERSER

DISTORTION

FILTEROFFSET

FORE

FPA

IFUFORE

IFUPOST

IFUSLICER

MSA

OTE

SPECWCS

REGIONS

WAVELENGTHRANGE

background

WFSSBKG

WAVELENGTHRANGE

cube_build

CUBEPAR

RESOL

dark_current

DARK

dq_init

extract_1d

EXTRACT1D

APCORR

extract_2d

WAVECORR

WAVELENGTHRANGE

flatfield

FLAT

DFLAT

FFLAT

SFLAT

fringe

FRINGE

gain_scale

GAIN

ipc

IPC

jump

GAIN

linearity

LINEARITY

msaflagopen

MSAOPER

pathloss

PATHLOSS

persistence

PERSAT

TRAPDENSITY

TRAPPARS

photom

PHOTOM

AREA

ramp_fitting

GAIN

refpix

REFPIX

reset

RESET

rscd

RSCD

saturation

SATURATION

source_catalog

APCORR

ABVEGAOFFSET

straylight

REGIONS

superbias

SUPERBIAS

tso_photometry

TSOPHOT

Step Parameters Reference Types¶

When each Step is instantiated, a CRDS look-up, based on the Step class name and input data, is made to retrieve a configuration file. The reftype for such configuration files is pars-<class name>. For example, for the step jwst.persistence.PersistenceStep, the reftype would be pars-persistencestep.

Standard Required Keywords¶

At present, most JWST science and reference files are FITS files with image or table extensions. The FITS primary data unit is always empty. The primary header contains all keywords not specific to individual extensions. Keywords specific to a particular extension are contained in the header of that extension.

The required Keywords Documenting Contents of Reference Files are:

Keyword

Comment

REFTYPE

WFSSBKG    Required values are listed in the discussion of each pipeline step.

DESCRIP

Summary of file content and/or reason for delivery

AUTHOR

Fred Jones     Person(s) who created the file

USEAFTER

YYYY-MM-DDThh:mm:ss Date and time after the reference files will be used. The T is required. Time string may NOT be omitted; use T00:00:00 if no meaningful value is available.

PEDIGREE

Options are 'SIMULATION' 'GROUND' 'DUMMY' 'INFLIGHT YYYY-MM-DD YYYY-MM-DD'

HISTORY

Description of Reference File Creation

HISTORY

DOCUMENT: Name of document describing the strategy and algorithms used to create file.

HISTORY

SOFTWARE: Description, version number, location of software used to create file.

HISTORY

DATA USED: Data used to create file

HISTORY

DIFFERENCES: How is this version different from the one that it replaces?

HISTORY

If your text spills over to the next line, begin it with another HISTORY keyword, as in this example.

TELESCOP

JWST   Name of the telescope/project.

INSTRUME

FGS   Instrument name. Allowed values: FGS, NIRCAM, NIRISS, NIRSPEC, MIRI

SUBARRAY

FULL, GENERIC, SUBS200A1, ...   (XXX abstract technical description of SUBARRAY)

SUBSTRT1

1        Starting pixel index along axis 1 (1-indexed)

SUBSIZE1

2048     Size of subarray along axis 1

SUBSTRT2

1        Starting pixel index along axis 2 (1-indexed)

SUBSIZE2

2048     Size of subarray along axis 2

FASTAXIS

1        Fast readout direction relative to image axes for Amplifier #1 (1 = +x axis, 2 = +y axis, -1 = -x axis, -2 = -y axis) SEE NOTE BELOW.

SLOWAXIS

2        Slow readout direction relative to image axes for all amplifiers (1 = +x axis, 2 = +y axis, -1 = -x axis, -2 = -y axis)

Observing Mode Keywords¶

A pipeline module may require separate reference files for each instrument, detector, filter, observation date, etc. The values of these parameters must be included in the reference file header. The observing-mode keyword values are vital to the process of ingesting reference files into CRDS, as they are used to establish the mapping between observing modes and specific reference files. Some observing-mode keywords are also used in the pipeline processing steps. If an observing-mode keyword is irrelevant to a particular observing mode (such as GRATING for the MIRI imager mode or the NIRCam and NIRISS instruments), then it may be omitted from the file header.

The Keywords Documenting the Observing Mode are:

Keyword

Sample Value

Comment

PUPIL

NRM

Pupil wheel element. Required only for NIRCam and NIRISS. NIRCam allowed values: CLEAR, F162M, F164N, F323N, F405N, F466N, F470N, GRISMV2, GRISMV3 NIRISS allowed values: CLEARP, F090W, F115W, F140M, F150W, F158M, F200W, GR700XD, NRM

FILTER

F2100W

Filter wheel element. Allowed values: too many to list here

GRATING

G395M

Required only for NIRSpec.

NIRSpec allowed values: G140M, G235M, G395M, G140H, G235H, G395H, PRISM, MIRROR

EXP_TYPE

MIR_MRS

Exposure type.

FGS allowed values: FGS_IMAGE, FGS_FOCUS, FGS_SKYFLAT, FGS_INTFLAT, FGS_DARK

MIRI allowed values: MIR_IMAGE, MIR_TACQ, MIR_LYOT, MIR_4QPM, MIR_LRS-FIXEDSLIT, MIR_LRS-SLITLESS, MIR_MRS, MIR_DARK, MIR_FLATIMAGE, MIR_FLATMRS, MIR_CORONCAL

NIRCam allowed values: NRC_IMAGE, NRC_GRISM, NRC_TACQ, NRC_TACONFIRM, NRC_CORON, NRC_TSIMAGE, NRC_TSGRISM, NRC_FOCUS, NRC_DARK, NRC_FLAT, NRC_LED

NIRISS allowed values: NIS_IMAGE, NIS_TACQ, NIS_TACONFIRM, NIS_WFSS, NIS_SOSS, NIS_AMI, NIS_FOCUS, NIS_DARK, NIS_LAMP

NIRSpec allowed values: NRS_TASLIT, NRS_TACQ, NRS_TACONFIRM, NRS_CONFIRM, NRS_FIXEDSLIT, NRS_AUTOWAVE, NRS_IFU, NRS_MSASPEC, NRS_AUTOFLAT, NRS_IMAGE, NRS_FOCUS, NRS_DARK, NRS_LAMP, NRS_BOTA, NRS_BRIGHTOBJ, NRS_MIMF

DETECTOR

MIRIFULONG

Allowed values: GUIDER1, GUIDER2

NIS

NRCA1, NRCA2, NRCA3, NRCA4, NRCB1, NRCB2, NRCB3, NRCB4, NRCALONG, NRCBLONG

NRS1, NRS2

MIRIFULONG, MIRIFUSHORT, MIRIMAGE

CHANNEL

12

MIRI MRS (IFU) channel. Allowed values: 1, 2, 3, 4, 12, 34 SHORT NIRCam channel. Allowed values: SHORT, LONG

BAND

MEDIUM

IFU band. Required only for MIRI. Allowed values are SHORT, MEDIUM, LONG, and N/A, as well as any allowable combination of two values (SHORT-MEDIUM, LONG-SHORT, etc.). (Also used as a header keyword for selection of all MIRI Flat files, Imager included.)

FAST

Name of the readout pattern used for the exposure. Each pattern represents a particular combination of parameters like nframes and groups. For MIRI, FAST and SLOW refer to the rate at which the detector is read.

MIRI allowed values: SLOW, FAST, FASTGRPAVG, FASTINTAVG

NIRCam allowed values: DEEP8, DEEP2, MEDIUM8, MEDIUM2, SHALLOW4, SHALLOW2, BRIGHT2, BRIGHT1, RAPID

NIRSpec allowed values: NRSRAPID, NRS, NRSN16R4, NRSIRS2RAPID

NIRISS allowed values: NIS, NISRAPID

FGS allowed values: ID, ACQ1, ACQ2, TRACK, FINEGUIDE, FGS60, FGS840, FGS7850, FGSRAPID, FGS

NRS_NORM

16

Required only for NIRSpec.

NRS_REF

4

Required only for NIRSpec.

P_XXXXXX

pattern keywords used by CRDS for JWST to describe the intended uses of a reference file using or’ed combinations of values. Only a subset of P_pattern keywords are supported.

Note: For the NIR detectors, the fast readout direction changes sign from one amplifier to the next. It is +1, -1, +1, and -1, for amps 1, 2, 3, and 4, respectively. The keyword FASTAXIS refers specifically to amp 1. That way, it is entirely correct for single-amp readouts and correct at the origin for 4-amp readouts. For MIRI, FASTAXIS is always +1.

Tracking Pipeline Progress¶

As each pipeline step is applied to a science data product, it will record a status indicator in a header keyword of the science data product. The current list of step status keyword names is given in the following table. These status keywords may be included in the primary header of reference files, in order to maintain a history of the data that went into creating the reference file. Allowed values for the status keywords are ‘COMPLETE’ and ‘SKIPPED’. Absence of a particular keyword is understood to mean that step was not even attempted.

Table 1. Keywords Documenting Which Pipeline Steps Have Been Performed.

 S_AMIANA AMI fringe analysis S_AMIAVG AMI fringe averaging S_AMINOR AMI fringe normalization S_BARSHA Bar shadow correction S_BKDSUB Background subtraction S_COMB1D 1-D spectral combination S_DARK Dark subtraction S_DQINIT DQ initialization S_ERRINI ERR initialization S_EXTR1D 1-D spectral extraction S_EXTR2D 2-D spectral extraction S_FLAT Flat field correction S_FRINGE Fringe correction S_FRSTFR MIRI first frame correction S_GANSCL Gain scale correction S_GRPSCL Group scale correction S_GUICDS Guide mode CDS computation S_IFUCUB IFU cube creation S_IMPRNT NIRSpec MSA imprint subtraction S_IPC IPC correction S_JUMP Jump detection S_KLIP Coronagraphic PSF subtraction S_LASTFR MIRI last frame correction S_LINEAR Linearity correction S_MRSMAT MIRI MRS background matching S_MSAFLG NIRSpec MSA failed shutter flagging S_OUTLIR Outlier detection S_PERSIS Persistence correction S_PHOTOM Photometric (absolute flux) calibration S_PSFALI Coronagraphic PSF alignment S_PSFSTK Coronagraphic PSF stacking S_PTHLOS Pathloss correction S_RAMP Ramp fitting S_REFPIX Reference pixel correction S_RESAMP Resampling (drizzling) S_RESET MIRI reset correction S_RSCD MIRI RSCD correction S_SATURA Saturation check S_SKYMAT Sky matching S_SRCCAT Source catalog creation S_SRCTYP Source type determination S_STRAY Straylight correction S_SUPERB Superbias subtraction S_TELEMI Telescope emission correction S_TSPHOT TSO imaging photometry S_TWKREG Tweakreg image alignment S_WCS WCS assignment S_WFSCOM Wavefront sensing image combination S_WHTLIT TSO white-light curve generation

Orientation of Detector Image¶

All steps in the pipeline assume the data are in the DMS (science) orientation, not the native readout orientation. The pipeline does NOT check or correct for the orientation of the reference data. It assumes that all files ingested into CRDS have been put into the science orientation. All header keywords documenting the observing mode (Table 2) should likewise be transformed into the DMS orientation. For square data array dimensions it’s not possible to infer the actual orientation directly so reference file authors must manage orientation carefully.

Table 2. Correct values for FASTAXIS and SLOWAXIS for each detector.

DETECTOR

FASTAXIS

SLOWAXIS

MIRIMAGE

1

2

MIRIFULONG

1

2

MIRIFUSHORT

1

2

NRCA1

-1

2

NRCA2

1

-2

NRCA3

-1

2

NRCA4

1

-2

NRCALONG

-1

2

NRCB1

1

-2

NRCB2

-1

2

NRCB3

1

-2

NRCB4

-1

2

NRCBLONG

1

-2

NRS1

2

1

NRS2

-2

-1

NIS

-2

-1

GUIDER1

-2

-1

GUIDER2

2

-1

Differing values for these keywords will be taken as an indicator that neither the keyword value nor the array orientation are correct.

P_pattern keywords¶

P_ pattern keywords used by CRDS for JWST to describe the intended uses of a reference file using or’ed combinations

For example, if the same NIRISS SUPERBIAS should be used for

or

the definition of READPATT in the calibration s/w datamodels schema does not allow it. READPATT can specify one or the other but not both.

To support expressing combinations of values, CRDS and the CAL s/w have added “pattern keywords” which nominally begin with P_ followed by the ordinary keyword, truncated as needed to 8 characters. In this case, P_READPA corresponds to READPATT.

Pattern keywords override the corresponding ordinary keyword for the purposes of automatically updating CRDS rmaps. Pattern keywords describe intended use.

In this example, the pattern keyword:

P_READPA = NIS | NISRAPID |

can be used to specify the intent “use for NIS or for NISRAPID”.

Only or-ed combinations of the values used in ordinary keywords are valid for pattern keywords.

Patterns appear in a slightly different form in rmaps than they do in P_ keywords. The value of a P_ keyword always ends with a trailing or-bar. In rmaps, no trailing or-bar is used so the equivalient of the above in an rmap is:

‘NIS|NISRAPID’

From a CRDS perspective, the P_ pattern keywords and their corresponding datamodels paths currently supported can be found in the JWST Pattern Keywords section of the CRDS documentation.

Currently all P_ keywords correspond to basic keywords found only in the primary headers of reference files and are typically only valid for FITS format..

The traslation from these P_ pattern keywords are completely generic in CRDS and can apply to any reference file type so they should be assumed to be reserved whether a particular type uses them or not. Defining non-pattern keywords with the prefix P_ is strongly discouraged.

Data Quality Flags¶

Within science data files, the PIXELDQ flags are stored as 32-bit integers; the GROUPDQ flags are 8-bit integers. The meaning of each bit is specified in a separate binary table extension called DQ_DEF. The binary table has the format presented in Table 3, which represents the master list of DQ flags. Only the first eight entries in the table below are relevant to the GROUPDQ array. All calibrated data from a particular instrument and observing mode have the same set of DQ flags in the same (bit) order. For Build 7, this master list will be used to impose this uniformity. We may eventually use different master lists for different instruments or observing modes.

Within reference files for some steps, the Data Quality arrays for some steps are stored as 8-bit integers to conserve memory. Only the flags actually used by a reference file are included in its DQ array. The meaning of each bit in the DQ array is stored in the DQ_DEF extension, which is a binary table having the following fields: Bit, Value, Name, and Description.

Table 3. Flags for the DQ, PIXELDQ, and GROUPDQ Arrays (Format of DQ_DEF Extension).

Bit

Value

Name

Description

0

1

DO_NOT_USE

1

2

SATURATED

Pixel saturated during exposure

2

4

JUMP_DET

Jump detected during exposure

3

8

DROPOUT

Data lost in transmission

4

16

OUTLIER

Flagged by outlier detection

5

32

RESERVED

6

64

RESERVED

7

128

RESERVED

8

256

UNRELIABLE_ERROR

Uncertainty exceeds quoted error

9

512

NON_SCIENCE

Pixel not on science portion of detector

10

1024

11

2048

HOT

Hot pixel

12

4096

WARM

Warm pixel

13

8192

LOW_QE

Low quantum efficiency

14

16384

RC

RC pixel

15

32768

TELEGRAPH

Telegraph pixel

16

65536

NONLINEAR

Pixel highly nonlinear

17

131072

Reference pixel cannot be used

18

262144

NO_FLAT_FIELD

Flat field cannot be measured

19

524288

NO_GAIN_VALUE

Gain cannot be measured

20

1048576

NO_LIN_CORR

Linearity correction not available

21

2097152

NO_SAT_CHECK

Saturation check not available

22

4194304

UNRELIABLE_BIAS

Bias variance large

23

8388608

UNRELIABLE_DARK

Dark variance large

24

16777216

UNRELIABLE_SLOPE

Slope variance large (i.e., noisy pixel)

25

33554432

UNRELIABLE_FLAT

Flat variance large

26

67108864

OPEN

Open pixel (counts move to adjacent pixels)

27

134217728

28

268435456

UNRELIABLE_RESET

Sensitive to reset anomaly

29

536870912

MSA_FAILED_OPEN

Pixel sees light from failed-open shutter

30

1073741824

A catch-all flag

31

2147483648

REFERENCE_PIXEL

Pixel is a reference pixel

Note: Words like “highly” and “large” will be defined by each instrument team. They are likely to vary from one detector to another – or even from one observing mode to another.

Parameter Specification¶

There are a number of steps, such as OutlierDetectionStep or SkyMatchStep, that define what data quality flags a pixel is allowed to have to be considered in calculations. Such parameters can be set in a number of ways.

First, the flag can be defined as the integer sum of all the DQ bit values from the input images DQ arrays that should be considered “good”. For example, if pixels in the DQ array can have combinations of 1, 2, 4, and 8 and one wants to consider DQ flags 2 and 4 as being acceptable for computations, then the parameter value should be set to “6” (2+4). In this case a pixel having DQ values 2, 4, or 6 will be considered a good pixel, while a pixel with a DQ value, e.g., 1+2=3, 4+8=”12”, etc. will be flagged as a “bad” pixel.

Alternatively, one can enter a comma-separated or ‘+’ separated list of integer bit flags that should be summed to obtain the final “good” bits. For example, both “4,8” and “4+8” are equivalent to a setting of “12”.

Finally, instead of integers, the JWST mnemonics, as defined above, may be used. For example, all the following specifications are equivalent:

"12" == "4+8" == "4, 8" == "JUMP_DET, DROPOUT"

Note

• The default value (0) will make all non-zero pixels in the DQ mask be considered “bad” pixels and the corresponding pixels will not be used in computations.

• Setting to None will turn off the use of the DQ array for computations.

• In order to reverse the meaning of the flags from indicating values of the “good” DQ flags to indicating the “bad” DQ flags, prepend ‘~’ to the string value. For example, in order to exclude pixels with DQ flags 4 and 8 for computations and to consider as “good” all other pixels (regardless of their DQ flag), use a value of ~4+8, or ~4,8. A string value of ~0 would be equivalent to a setting of None.