CFG Usage Deprecation Notice¶
As of March 18, 2021, a significant change to how JWST pipelines operate was completed and pushed to the JWST master branch on github. Theoretically the change should be transparent. However, we are all familiar with the difference between theory and practice and hence we want to alert all users.
Originally, how the pipelines operated was determined by a set of configuration
(CFG) files that were delivered as part of the JWST package. These configuration
files were retrieved using the
collect_pipeline_cfgs command. The
configuration files were used to run each of the different pipelines using the
strun command. For example:
$ collect_pipeline_cfgs ./ $ strun calwebb_spec2.cfg an_exposure_file.fits
The issue with the above process is that any changes, as determined by INS and the Calibration Working Group, to the default operation of the pipeline requires a code release. A better solution would be if the pipeline configurations could come from reference files retrieved from CRDS.
As of the version of master introduced on March 18th, 2021, in conjunction with
CRDS context jwst_0712, the default pipeline configurations no longer depend on
the package-delivered configuration files. Instead, all default configuration
relies on settings in the pipeline code itself, using CRDS-retrieved parameter
reference files to modify any parameters that are data-dependent. There is no
longer any need to run
collect_pipeline_cfgs and specify a configuration
file for the
strun command. One only needs to specify a simplified pipeline
name. In most cases, this simple name, or alias, is the same as the name of the
old configuration file, but without the suffix
Taking the example above, to get the same operation, the single command would become:
$ strun calwebb_spec2 an_exposure_file.fits
The JWST documentation has been updated to account for this change in usage. To get familiarized, it is best to start with the Introduction
A list of the available pipeline aliases can be found in the Pipeline Stages section.
An added benefit to removing the dependency on package-delivered configuration files is that users, under normal circumstances, no longer need to be concerned with configuration files and whether they are up-to-date. One only needs to install the JWST package and start using the pipelines out-of-the-box.
Does this mean that everyone has to immediately change their behavior and code
if using the default configuration files? Short answer is “No”. If one wishes to
continue using the package-delivered configuration files from
collect_pipeline_cfgs, one may do so. However, these configuration files no
longer contain any parameter settings; only the class name of the pipeline to be
run. This allows the code-plus-CRDS-retrieved parameter reference files to
Since the configuration settings have simply been moved to CRDS, the results one obtains should not change. If a change in behavior is noted, please report the issue to the Help Desk, file a Github issue on the JWST Github repository, or file a Jira issue against the JP project.
In the meantime, please consider deprecating the use of
and the .cfg files in favor of simply specifying pipeline aliases, as the
documentation now describes.
For users that use their own, custom configuration files, there is no change to functionality. However, there are changes to both how these files are documented and their format.
Concerning documentation, there is a change of terminology. No longer are these files referred to as “configuration files”, but are called “parameter files” or “parameter reference files” when retrieved from CRDS.
In order to simplify integration with CRDS, the format of parameter files have changed from the “cfg”, init-like format, to the ASDF format. All parameter files in CRDS are in this format. Similarly, the tools provided by the JWST package to create parameter files will create them in ASDF. “cfg”-formatted files are still supported, but it is strongly suggested that users change to using the ASDF form. For more information, please to refer to ASDF Parameter Files
As always, if anyone finds any discrepancies or other issues with the documentation, or actual operation of the pipelines, please contact the Help Desk, or file issues directly against the Github repository or the JIRA “JP” project.