Input and Output File Conventions

Input Files

There are two general types of input to any step or pipeline: references files and data files. The references files, unless explicitly overridden, are provided through CRDS.

Data files are the science input, such as exposure FITS files and association files. All files are assumed to be co-resident in the directory where the primary input file is located. This is particularly important for associations: JWST associations contain file names only. All files referred to by an association are expected to be located in the directory in which the association file is located.

Output Files

Output files will be created either in the current working directory, or where specified by the output_dir parameter.

File names for the outputs from pipelines and steps come from three different sources:

  • The name of the input file

  • The product name defined in an association

  • As specified by the output_file parameter

Regardless of the source, each pipeline/step uses the name as a base name, onto which several different suffixes are appended, which indicate the type of data in that particular file. A list of the main suffixes can be found below.

The pipelines do not file manage versions. When re-running a pipeline, previous files will be overwritten.

Output Files and Associations

Stage 2 pipelines can take an individual file or an association as input. Nearly all Stage 3 pipelines require an association as input. Normally, the output file is defined in each association’s “product name”, which defines the basename that will be used for output file naming.

Often, one may reprocess the same set of data multiple times, such as to change reference files or parameters. When doing so, it is highly suggested to use output_dir to place the results in a different directory instead of using output_file to rename the output files. Most pipelines and steps create sets of output files. Separating runs by directory may be much easier to manage.

Individual Step Outputs

If individual steps are executed without an output file name specified via the output_file parameter, the stpipe infrastructure automatically uses the input file name as the root of the output file name and appends the name of the step as an additional suffix to the input file name. If the input file name already has a known suffix, that suffix will be replaced. For example:

$ strun jwst.dq_init.DQInitStep jw00017001001_01101_00001_nrca1_uncal.fits

produces an output file named jw00017001001_01101_00001_nrca1_dq_init.fits.

See Pipeline/Step Suffix Definitions for a list of the more common suffixes used.