Processing Levels and Product Stages

Here we describe the structure and content of the most frequently used forms of files for JWST science data products, the vast majority of which are in FITS format. Each type of FITS file is the result of serialization of a corresponding data model. All JWST pipeline input and output products, with the exception of a few reference files and catalogs, are serialized as FITS files. The ASDF representation of the data model is serialized as a FITS BINTABLE extension within the FITS file, with EXTNAME=”ASDF”. The ASDF extension is essentially a text character serialization in YAML format of the data model. The ASDF representation is read from the extension when a FITS file is loaded into a data model to provide the initial instance of the data model. Values in the other FITS extensions then either override this initial model or are added to it.

Within the various STScI internal data processing and archiving systems that are used for routine processing of JWST data, there are some different uses of terminology to refer to different levels or stages of processing and products. For those who are interested or need to know, the table below gives high-level translations between those naming conventions.

Data Processing Levels

User Data Product Stages

MAST/CAOM Data Levels

N/A

N/A

-1 = Planned, but not yet executed

Level 0 = Science telemetry

Not available to users

Not available to users

Level 0.5 = POD files

Not available to users

Not available to users

Level 1a = Original FITS file

Stage 0 = Original FITS file

0 = raw (not available to users)

Level 1b = Uncal FITS file

Stage 0 = Fully-populated FITS file

1 = uncalibrated

Level 2a = Countrate exposure

Stage 1 = Countrate FITS file

2 = calibrated

Level 2b = Calibrated exposure

*Level 2c = CR-flagged exposure

Stage 2 = Calibrated exposure

2 = calibrated

Level 3 = Combined data

Stage 3 = Combined data

3 = Science product

*Note that Level 2c files are intermediate files produced during pipeline Stage 3 processing, and are not final products (as opposed to all the other product types that are listed here). Therefore, Level 2c files are not a final product of any pipeline stage, but are produced within the pipeline Stage 3 processing. Level 2c files (identified by the ‘crf’ extension) are in the same format as Level 2b products, with the difference being that their data quality flags have been updated after running outlier detection in pipeline Stage 3 processing.

Throughout this document, we will use the “Stage” terminology to refer to data products. Stage 0, 1, and 2 products are always files containing the data from a single exposure and a single detector. A NIRCam exposure that uses all 10 detectors will therefore result in 10 separate FITS files for each of the Stage 0, 1, and 2 products. Because these stages contain the data for a single exposure, they are refered to as “exposure-based” products and use an “exposure-based” file naming syntax. Stage 3 products, on the other hand, are constructed from the combined data of multiple exposures for a given source or target. They are referred to as “source-based” products and use a “source-based” file naming syntax. Observing modes that include multiple defined sources within a single exposure or observation, such as NIRSpec MOS and NIRCam/NIRISS WFSS, will result in multiple Stage 3 products, one for each defined or identifiable source.