Running the JWST pipeline: Command Line Interface (strun)¶
Note
For seasoned users who are familiar with using collect_pipeline_cfgs and
running pipelines by the default configuration (CFG) files, please note that
this functionality has been deprecated. Please read CFG Usage
Deprecation Notice.
Individual steps and pipelines (consisting of a series of steps) can be run
and configured from the command line using the strun command.
strun is one of two options for running the pipeline. See
here for an overview of the alternative Python
interface.
CRDS Environment Variables¶
The CRDS environment variables need to be defined before running a pipeline
or step with strun to allow the pipeline to access reference and parameter
files. See CRDS for more information.
Overview of Running the Pipeline with strun¶
The first argument to strun must be one of either a pipeline name, Python
class of the step or pipeline to be run, or the name of a parameter file for the
desired step or pipeline (see Parameter Files). The second argument to
strun is the name of the input data file to be processed.
$ strun <pipeline_name, class_name, or parameter_file> <input_file>
Pipeline classes also have a pipeline name, or alias, that can be used
instead of the full class specification. For example, jwst.pipeline.Detector1Pipeline
has the alias calwebb_detector1 and can be run as
$ strun calwebb_detector1 jw00017001001_01101_00001_nrca1_uncal.fits
A full list of pipeline aliases can be found in Pipeline Stages.
Exit Status¶
strun produces the following exit status codes:
0: Successful completion of the step/pipeline
1: General error occurred
64: No science data found
The “No science data found” condition is returned by the assign_wcs step of
the calwebb_spec2 pipeline when, after successfully determining the WCS
solution for a file, the WCS indicates that no science data will be found. This
condition most often occurs with NIRSpec’s Multi-object Spectroscopy (MOS) mode:
There are certain optical and MSA configurations in which dispersion will not
cross one or the other of NIRSpec’s detectors.
Configuring a Pipeline/Step with strun¶
By default, pipeline parameters and reference files are chosen by CRDS based on instrument, observing mode, date, etc. If set to the most current Reference Files Mappings (CRDS Context), these represent the ‘best’ set of parameters and reference files for the pipeline as determined by the JWST instrument teams.
A Pipeline/Step can be configured for custom processing. Pipeline-level and
step-level parameters can be changed, output file behavior can be set, references
files can be overridden, and pipeline steps can be skipped if desired. This
section will be a general overview on how to configure the pipeline when running with strun,
and the following sections will elaborate on each of these possible customizations
and demonstrate usage.
When running command line with ``strun``, there are two ways two configure a Pipeline/Step.
By passing in arguments to a pipeline/step on the command line
By using a parameter file and passing this in as an argument on the command line
A combination of arguments and custom parameter files can be used for configuration, but keep in mind the hierarchy of parameter precedence to keep track of which value will get used if set in multiple locations.
Setting Step Parameters on a Pipeline or Individual Step¶
All pipelines and steps have parameters that can be set to change various aspects
of how they execute (e.g switching on and off certain options in a step,
setting thresholds). By default, the values of these parameters are set in
the CRDS-chosen parameter file, but they can be overridden when running the
pipeline with strun. As mentioned, this can either be done by passing in command line
arguments or by passing in a custom parameter file - both methods will be described in this
section.
Using Command Line Arguments
When running a pipeline, step-level parameters can be changed by passing in a command
line argument to that step. For example, to change the rejection_threshold parameter of
the jump detection step when running the full Detector1Pipeline:
$ strun calwebb_detector1 jw00017001001_01101_00001_nrca1_uncal.fits
--steps.jump.rejection_threshold=12.0
When running a standalone step, command line arguments do not need to be nested within
steps. For example, to change the parameter rejection_threshold for the jump detection
step when running the step individually:
$ strun jump jw00017001001_01101_00001_nrca1_uncal.fits --rejection_threshold=12.0
Using a Parameter File
Alternatively, if using a parameter file, edit the file to add the following snippet (in this example, to a file named ‘my_config_file.asdf’ in the current working directory):
steps:
- class: jwst.jump.jump_step.JumpStep
name: jump
parameters:
rejection_threshold : 12
And pass in the modified file to strun:
$ strun my_config_file.asdf jw00017001001_01101_00001_nrca1_uncal.fits
Overriding Reference Files¶
By default, when the pipeline or step is run, CRDS will determine the best set of reference files based on file metadata and the current CRDS mapping (also known as ‘context’). It is possible to override these files and use a custom reference file, or one not chosen by CRDS.
Using Command Line Arguments
For any step that uses a calibration reference file you always have the
option to override the automatic selection of a reference file from CRDS and
specify your own file to use. Parameters for this are of the form
--override_<ref_type>, where ref_type is the name of the reference file
type, such as mask, dark, gain, or linearity. When in doubt as to
the correct name, just use the -h argument to strun to show you the list
of available override parameters.
To override the use of the default linearity reference file selection with a custom
file in the current working directory called my_lin.fits, for example,
you would do:
$ strun calwebb_detector1 jw00017001001_01101_00001_nrca1_uncal.fits
--steps.linearity.override_linearity='my_lin.fits'
Or, if running the step individually, to override the reference file:
$ strun linearity jw00017001001_01101_00001_nrca1_uncal.fits
--override_linearity='my_lin.fits'
Using a Parameter File
If using a parameter file for configuration, to override a reference edit the file to add the following snippet (in this example, to a file named ‘my_config_file.asdf’ in the current working directory):
steps:
- class: jwst.saturation.saturation_step.SaturationStep
name: saturation
parameters:
override_saturation: '/path/to/new_saturation_ref_file.fits'
And pass in the modified file to strun:
$ strun my_config_file.asdf jw00017001001_01101_00001_nrca1_uncal.fits
To use an entire set of past reference files from a previous CRDS mapping, see here.
Skipping a Pipeline Step¶
Note
Some steps in a pipeline expect certain previous steps to have been run beforehand, and therefore won’t run if that expected previous correction has not been applied. Proceed with caution when skipping steps.
When running a pipeline with strun, one or several steps within that pipeline
can be skipped.
Using Command Line Arguments
Every step in a pipeline has a skip parameter that when set to true, will entirely
skip that step. For example, to skip the saturation step in the Detector1Pipeline:
$ strun calwebb_detector1 jw00017001001_01101_00001_nrca1_uncal.fits
--steps.saturation.skip=True
Using a Parameter File
The equivalent to the above example can be done by adding the following snippet to your parameter file (in this example, to a file named ‘my_config_file.asdf’ in the current working directory):
steps:
- class: jwst.saturation.saturation_step.SaturationStep
parameters:
skip: true
And pass in the modified file to the config_file argument:
result = Detector1Pipeline.call('jw00017001001_01101_00001_nrca1_uncal.fits',
config_file='my_config_file.asdf')
Controlling Output File Behavior with strun¶
By default, when running the pipeline with strun, the final outputs of a pipeline
(or final outputs when running an individual step) will be written out to a file
in the current working directory. The base name of these final output files is
derived from the input file name, by default. Additionally, no intermediate step
results will be saved. This behavior can be modified to change output file names,
locations, and specify that intermediate results from a step in a pipeline should
be written out to a file.
Saving Intermediate Pipeline Results to a File¶
The stpipe infrastructure automatically passes the output data model from
one step to the input of the next step, without saving any intermediate results
to disk. If you want to save the results from individual steps, you have two options:
Specify
save_resultson an individual step within the pipeline. This option will save the results of the step, using a filename created by the step.Specify a file name using
output_file <basename>for an individual step. This option indicated that results should be saved, and to use the name specified.
For example, to save the result from the dark current step of Detector1Pipeline
(using the alias name calwebb_detector1):
$ strun calwebb_detector1 jw00017001001_01101_00001_nrca1_uncal.fits
--steps.dark_current.save_results=true
This will create the file jw00017001001_01101_00001_dark_current.fits in the
current working directory.
Setting Output File Name¶
As demonstrated in the section above, the output_file
parameter is used to specify the desired name for output files. When done at the
step-level as shown in those examples, the intermediate output files from steps
within a pipeline are saved with the specified name.
You can also specify a particular file name for saving the end result of
the entire pipeline using the --output_file parameter:
$ strun calwebb_detector1 jw00017001001_01101_00001_nrca1_uncal.fits
--output_file='stage1_processed'
In this situation, using the default configuration, three files are created:
stage1_processed_trapsfilled.fits
stage1_processed_rate.fits
stage1_processed_rateints.fits
When running a standalone step, setting --output_file at the top-level
will determine the name of the final output product for that step, overriding
the default based on input name:
$ strun linearity jw00017001001_01101_00001_nrca1_uncal.fits
--output_file='intermediate_linearity'
Similarly, to save the result from a step within a pipeline (for example,
the dark current step of calwebb_detector1) with a different file name:
$ strun calwebb_detector1 jw00017001001_01101_00001_nrca1_uncal.fits
--steps.dark_current.output_file='intermediate_result'
A file, intermediate_result_dark_current.fits, will then be created. Note
that the name of the step will be appended as the file name suffix
Setting Output File Directory¶
To change the output directory of the final pipeline products from the default of the
current working directory, use the output_dir option.
$ strun calwebb_detector1 jw00017001001_01101_00001_nrca1_uncal.fits
--steps.dark_current.output_dir='calibrated'
When this is run, all three final output products of Detector1Pipeline will
be saved within the subdirectory calibrated.
Setting output_dir at the step-level indicates that the step’s result should
be saved (so, also setting save_results is redundant), and that the files
should be saved in the directory specified instead of the current working directory.
For example, to save the intermediate results of DarkCurrentStep when running
Detector1Pipeline in a subdirectory /calibrated:
$ strun calwebb_detector1 jw00017001001_01101_00001_nrca1_uncal.fits
--steps.dark_current.output_dir='calibrated'
Similarly, when output_dir is set on an individual step class, this will indicate
that the result from that step should be saved to the specified directory:
$ strun dark_current jw00017001001_01101_00001_nrca1_uncal.fits --output_dir='calibrated'