Step

class jwst.stpipe.Step(name=None, parent=None, config_file=None, _validate_kwds=True, **kws)[source]

Bases: object

Create a Step instance.

Parameters
  • name (str, optional) – The name of the Step instance. Used in logging messages and in cache filenames. If not provided, one will be generated based on the class name.

  • parent (Step instance, optional) – The parent step of this step. Used to determine a fully-qualified name for this step, and to determine the mode in which to run this step.

  • config_file (str path, optional) – The path to the config file that this step was initialized with. Use to determine relative path names of other config files.

  • **kws (dict) – Additional parameters to set. These will be set as member variables on the new Step instance.

Attributes Summary

input_dir

make_output_path

Return function that creates the output path

prefetch_references

reference_file_types

spec

Methods Summary

__call__(*args)

Run handles the generic setup and teardown that happens with the running of each step.

call(*args, **kwargs)

Creates and runs a new instance of the class.

closeout([to_close, to_del])

Close out step processing

default_output_file([input_file])

Create a default filename based on the input name

default_suffix()

Return a default suffix based on the step

from_cmdline(args)

Create a step from a configuration file.

from_config_file(config_file[, parent, name])

Create a step from a configuration file.

from_config_section(config[, parent, name, …])

Create a step from a configuration file fragment.

get_config_from_reference(dataset[, …])

Retrieve step parameters from reference database

get_pars()

Retrieve the configuration parameters of a step

get_pars_model()

Return Step parameters as StepParsModel

get_ref_override(reference_file_type)

Determine and return any override for reference_file_type.

get_reference_file(input_file, …)

Get a reference file from CRDS.

load_as_level2_asn(obj)

Load object as an association

load_as_level3_asn(obj)

Load object as an association

load_spec_file([preserve_comments])

make_input_path(file_path)

Create an input path for a given file path

merge_config(config, config_file)

open_model(obj)

Open a datamodel

prefetch(*args)

Prefetch reference files, nominally called when self.prefetch_references is True.

print_configspec()

process(*args)

This is where real work happens.

record_step_status(datamodel, cal_step[, …])

Record whether or not a step completed in meta.cal_step

reference_uri_to_cache_path(reference_uri)

Convert an abstract CRDS reference URI to an absolute file path in the CRDS cache.

resolve_file_name(file_name)

Resolve a file name expressed relative to this Step’s configuration file.

run(*args)

Run handles the generic setup and teardown that happens with the running of each step.

save_model(model[, suffix, idx, …])

Saves the given model using the step/pipeline’s naming scheme

search_attr(attribute[, default, parent_first])

Return first non-None attribute in step heirarchy

set_primary_input(obj[, exclusive])

Sets the name of the master input file and input directory.

Attributes Documentation

input_dir
make_output_path

Return function that creates the output path

prefetch_references = True
reference_file_types = []
spec = "\n pre_hooks = string_list(default=list())\n post_hooks = string_list(default=list())\n output_file = output_file(default=None) # File to save output to.\n output_dir = string(default=None) # Directory path for output files\n output_ext = string(default='.fits') # Default type of output\n output_use_model = boolean(default=False) # When saving use `DataModel.meta.filename`\n output_use_index = boolean(default=True) # Append index.\n save_results = boolean(default=False) # Force save results\n skip = boolean(default=False) # Skip this step\n suffix = string(default=None) # Default suffix for output files\n search_output_file = boolean(default=True) # Use outputfile define in parent step\n input_dir = string(default=None) # Input directory\n "

Methods Documentation

__call__(*args)

Run handles the generic setup and teardown that happens with the running of each step. The real work that is unique to each step type is done in the process method.

classmethod call(*args, **kwargs)[source]

Creates and runs a new instance of the class.

Gets a config file from CRDS if one is available

To set configuration parameters, pass a config_file path or keyword arguments. Keyword arguments override those in the specified config_file.

Any positional *args will be passed along to the step’s process method.

Note: this method creates a new instance of Step with the given config_file if supplied, plus any extra *args and **kwargs. If you create an instance of a Step, set parameters, and then use this call() method, it will ignore previously-set parameters, as it creates a new instance of the class with only the config_file, *args and **kwargs passed to the call() method.

If not used with a config_file or specific *args and **kwargs, it would be better to use the run method, which does not create a new instance but simply runs the existing instance of the Step class.

closeout(to_close=None, to_del=None)[source]

Close out step processing

Parameters
  • to_close ([object(, ..)]) – List of objects with a close method to execute The objects will also be deleted

  • to_del ([object(, ..)]) – List of objects to simply delete

Notes

Other operations, such as forced garbage collection will also be done.

default_output_file(input_file=None)[source]

Create a default filename based on the input name

default_suffix()[source]

Return a default suffix based on the step

static from_cmdline(args)[source]

Create a step from a configuration file.

Parameters

args (list of str) – Commandline arguments

Returns

step – If the config file has a class parameter, the return value will be as instance of that class.

Any parameters found in the config file will be set as member variables on the returned Step instance.

Return type

Step instance

classmethod from_config_file(config_file, parent=None, name=None)[source]

Create a step from a configuration file.

Parameters
  • config_file (path or readable file-like object) – The config file to load parameters from

  • parent (Step instance, optional) – The parent step of this step. Used to determine a fully-qualified name for this step, and to determine the mode in which to run this step.

  • name (str, optional) – If provided, use that name for the returned instance. If not provided, the following are tried (in order): - The name parameter in the config file - The filename of the config file - The name of returned class

Returns

step – If the config file has a class parameter, the return value will be as instance of that class. The class parameter in the config file must specify a subclass of cls. If the configuration file has no class parameter, then an instance of cls is returned.

Any parameters found in the config file will be set as member variables on the returned Step instance.

Return type

Step instance

classmethod from_config_section(config, parent=None, name=None, config_file=None)[source]

Create a step from a configuration file fragment.

Parameters
  • config (configobj.Section instance) – The config file fragment containing parameters for this step only.

  • parent (Step instance, optional) – The parent step of this step. Used to determine a fully-qualified name for this step, and to determine the mode in which to run this step.

  • name (str, optional) – If provided, use that name for the returned instance. If not provided, try the following (in order): - The name parameter in the config file fragment - The name of returned class

  • config_file (str, optional) – The path to the config file that created this step, if any. This is used to resolve relative file name parameters in the config file.

Returns

step – Any parameters found in the config file fragment will be set as member variables on the returned Step instance.

Return type

instance of cls

classmethod get_config_from_reference(dataset, observatory=None, disable=None)[source]

Retrieve step parameters from reference database

Parameters
  • cls (jwst.stpipe.Step) – Either a class or instance of a class derived from Step.

  • dataset (jwst.datamodels.ModelBase) – A model of the input file. Metadata on this input file will be used by the CRDS “bestref” algorithm to obtain a reference file.

  • observatory (str) – telescope name used with CRDS, e.g. ‘jwst’.

  • disable (bool or None) – Do not retrieve parameters from CRDS. If None, check global settings.

Returns

step_parameters – The parameters as retrieved from CRDS. If there is an issue, log as such and return an empty config obj.

Return type

configobj

get_pars()

Retrieve the configuration parameters of a step

Parameters
  • step (Step-derived class or instance) – The class or instance to retrieve the parameters for.

  • full_spec (bool) – Return all parameters, including parent-specified parameters. If False, return only parameters specific to the class/instance.

Returns

pars – Keys are the parameters and values are the values.

Return type

dict

get_pars_model()

Return Step parameters as StepParsModel

Parameters
  • step (Step-derived class or instance) – The Step or Step instance to retrieve the parameters model for.

  • full_spec (bool) – Return all parameters, including parent-specified parameters. If False, return only parameters specific to the class/instance.

Returns

model – The StepParsModel.

Return type

StepParsModel

get_ref_override(reference_file_type)[source]

Determine and return any override for reference_file_type.

Returns

Return type

override_filepath or None.

get_reference_file(input_file, reference_file_type)[source]

Get a reference file from CRDS.

If the configuration file or commandline parameters override the reference file, it will be automatically used when calling this function.

Parameters
  • input_file (jwst.datamodels.ModelBase instance) – A model of the input file. Metadata on this input file will be used by the CRDS “bestref” algorithm to obtain a reference file.

  • reference_file_type (string) – The type of reference file to retrieve. For example, to retrieve a flat field reference file, this would be ‘flat’.

Returns

reference_file

Return type

path of reference file, a string

load_as_level2_asn(obj)[source]

Load object as an association

Loads the specified object into a Level2 association. If necessary, prepend Step.input_dir to all members.

Parameters

obj (object) – Object to load as a Level2 association

Returns

association – Association

Return type

jwst.associations.lib.rules_level2_base.DMSLevel2bBase

load_as_level3_asn(obj)[source]

Load object as an association

Loads the specified object into a Level3 association. If necessary, prepend Step.input_dir to all members.

Parameters

obj (object) – Object to load as a Level3 association

Returns

association – Association

Return type

jwst.associations.lib.rules_level3_base.DMS_Level3_Base

classmethod load_spec_file(preserve_comments=False)[source]
make_input_path(file_path)[source]

Create an input path for a given file path

If file_path has no directory path, use self.input_dir as the directory path.

Parameters

file_path (str or obj) – The supplied file path to check and modify. If anything other than str, the object is simply passed back.

Returns

full_path – File path using input_dir if the input had no directory path.

Return type

str or obj

classmethod merge_config(config, config_file)[source]
open_model(obj)[source]

Open a datamodel

Primarily a wrapper around DataModel.open to handle Step peculiarities

Parameters

obj (object) – The object to open

Returns

datamodel – Object opened as a datamodel

Return type

DataModel

prefetch(*args)[source]

Prefetch reference files, nominally called when self.prefetch_references is True. Can be called explictly when self.prefetch_refences is False.

classmethod print_configspec()[source]
process(*args)[source]

This is where real work happens. Every Step subclass has to override this method. The default behaviour is to raise a NotImplementedError exception.

record_step_status(datamodel, cal_step, success=True)[source]

Record whether or not a step completed in meta.cal_step

Parameters
  • datamodel (Datamodel instance) – This is the datamodel or container of datamodels to modify in place

  • cal_step (str) – The attribute in meta.cal_step for recording the status of the step

  • success (bool) – If True, then ‘COMPLETE’ is recorded. If False, then ‘SKIPPED’

classmethod reference_uri_to_cache_path(reference_uri)[source]

Convert an abstract CRDS reference URI to an absolute file path in the CRDS cache. Reference URI’s are typically output to dataset headers to record the reference files used.

e.g. ‘crds://jwst_miri_flat_0177.fits’ –>

‘/grp/crds/cache/references/jwst/jwst_miri_flat_0177.fits’

The CRDS cache is typically located relative to env var CRDS_PATH with default value /grp/crds/cache. See also https://jwst-crds.stsci.edu

resolve_file_name(file_name)[source]

Resolve a file name expressed relative to this Step’s configuration file.

run(*args)[source]

Run handles the generic setup and teardown that happens with the running of each step. The real work that is unique to each step type is done in the process method.

save_model(model, suffix=None, idx=None, output_file=None, force=False, format=None, **components)[source]

Saves the given model using the step/pipeline’s naming scheme

Parameters
  • model (jwst.datamodels.Model instance) – The model to save.

  • suffix (str) – The suffix to add to the filename.

  • idx (object) – Index identifier.

  • output_file (str) – Use this file name instead of what the Step default would be.

  • force (bool) – Regardless of whether save_results is False and no output_file is specified, try saving.

  • format (str) – The format of the file name. This is a format string that defines where suffix and the other components go in the file name. If False, it will be presumed output_file will have all the necessary formatting.

  • components (dict) – Other components to add to the file name.

Returns

output_paths – List of output file paths the model(s) were saved in.

Return type

[str[, ..]]

search_attr(attribute, default=None, parent_first=False)[source]

Return first non-None attribute in step heirarchy

Parameters
  • attribute (str) – The attribute to retrieve

  • default (obj) – If attribute is not found, the value to use

  • parent_first (bool) – If True, allow parent definition to override step version

Returns

value – Attribute value or default if not found

Return type

obj

set_primary_input(obj, exclusive=True)[source]

Sets the name of the master input file and input directory. Used to generate output file names.

Parameters
  • obj (str or DataModel) – The object to base the name on. If a datamodel, use Datamodel.meta.filename.

  • exclusive (bool) – If True, only set if an input name is not already used by a parent Step. Otherwise, always set.