Configuration¶

This page describes the configuration management scheme used within the Fermipy package and documents the configuration parameters that can be set in the configuration file.

Class Configuration¶

Classes in the Fermipy package own a configuration state dictionary that is initialized when the class instance is created. Elements of the configuration dictionary can be scalars (str, int, float) or dictionaries containing groups of parameters. The settings in this dictionary are used to control the runtime behavior of the class.

When creating a class instance, the configuration is initialized by passing either a configuration dictionary or configuration file path to the class constructor. Keyword arguments can be passed to the constructor to override configuration parameters in the input dictionary. In the following example the config dictionary defines values for the parameters emin and emax. By passing a dictionary for the selection keyword argument, the value of emax in the keyword argument (10000) overrides the value of emax in the input dictionary.

config = {
'selection' : { 'emin' : 100,
                'emax' : 1000 }
}

gta = GTAnalysis(config,selection={'emax' : 10000})

The first argument can also be the path to a YAML configuration file rather than a dictionary:

gta = GTAnalysis('config.yaml',selection={'emax' : 10000})

Configuration File¶

Fermipy uses YAML files to read and write its configuration in a persistent format. The configuration file has a hierarchical structure that groups parameters into dictionaries that are keyed to a section name (data, binning, etc.).

Sample Configuration¶

data:
  evfile : ft1.lst
  scfile : ft2.fits
  ltcube : ltcube.fits

binning:
  roiwidth   : 10.0
  binsz      : 0.1
  binsperdec : 8

selection :
  emin : 100
  emax : 316227.76
  zmax    : 90
  evclass : 128
  evtype  : 3
  tmin    : 239557414
  tmax    : 428903014
  filter  : null
  target : 'mkn421'

gtlike:
  edisp : True
  edisp_bins : -1
  irfs : 'P8R3_SOURCE_V2'
  edisp_disable : ['isodiff','galdiff']

model:
  src_roiwidth : 15.0
  galdiff  : '$FERMI_DIFFUSE_DIR/gll_iem_v07.fits'
  isodiff  : 'iso_P8R3_SOURCE_V2_v1.txt'
  catalogs : ['4FGL']

The configuration file has the same structure as the configuration dictionary such that one can read/write configurations using the load/dump methods of the yaml module :

import yaml
# Load a configuration
config = yaml.load(open('config.yaml'))
# Update a parameter and write a new configuration
config['selection']['emin'] = 1000.
yaml.dump(config, open('new_config.yaml','w'))

Most of the configuration parameters are optional and if not set explicitly in the configuration file will be set to a default value. The parameters that can be set in each section are described below.

binning¶

Options in the binning section control the spatial and spectral binning of the data.

Sample binning Configuration¶

binning:

  # Binning
  roiwidth   : 10.0
  npix       : null
  binsz      : 0.1 # spatial bin size in deg
  binsperdec : 8   # nb energy bins per decade
  projtype   : WCS

*binning* Options¶
Option	Default	Description
`binsperdec`	8	Number of energy bins per decade.
`binsz`	0.1	Spatial bin size in degrees.
`coordsys`	CEL	Coordinate system of the spatial projection (CEL or GAL).
`enumbins`	None	Number of energy bins. If none this will be inferred from energy range and `binsperdec` parameter.
`hpx_ebin`	True	Include energy binning
`hpx_order`	10	Order of the map (int between 0 and 12, included)
`hpx_ordering_scheme`	RING	HEALPix Ordering Scheme
`npix`	None	Number of pixels in the x and y direction. If none then this will be set from `roiwidth` and `binsz`.
`proj`	AIT	Spatial projection for WCS mode.
`projtype`	WCS	Projection mode (WCS or HPX).
`roiwidth`	10.0	Width of the ROI in degrees. The number of pixels in each spatial dimension will be set from `roiwidth` / `binsz` (rounded up).

components¶

The components section can be used to define analysis configurations for independent subselections of the data. Each subselection will have its own binned likelihood instance that is combined in a global likelihood function for the ROI (implemented with the SummedLikelihood class in pyLikelihood). The components section is optional and when set to null (the default) only a single likelihood component will be created with the parameters of the root analysis configuration.

The component section is defined as a list of dictionaries where each element sets analysis parameters for a different subcomponent of the analysis. The component configurations follow the same structure and accept the same parameters as the root analysis configuration. Parameters not defined in a given element will default to the values set in the root analysis configuration.

The following example illustrates how to define a Front/Back analysis with two components. Files associated to each component will be given a suffix according to their order in the list (e.g. file_00.fits, file_01.fits, etc.).

# Component section for Front/Back analysis
  - { selection : { evtype : 1 } } # Front
  - { selection : { evtype : 2 } } # Back

data¶

The data section defines the input data files for the analysis (FT1, FT2, and livetime cube). evfile and scfile can either be individual files or group of files. The optional ltcube option can be used to choose a pre-generated livetime cube. If ltcube is null a livetime cube will be generated at runtime with gtltcube.

Sample data Configuration¶

data :
  evfile : ft1.lst
  scfile : ft2.fits
  ltcube : null

*data* Options¶
Option	Default	Description
`cacheft1`	True	Cache FT1 files when performing binned analysis. If false then only the counts cube is retained.
`evfile`	None	Path to FT1 file or list of FT1 files.
`ltcube`	None	Path to livetime cube. If none a livetime cube will be generated with `gtmktime`.
`scfile`	None	Path to FT2 (spacecraft) file.

extension¶

The options in extension control the default behavior of the extension method. For more information about using this method see the Extension Fitting page.

*extension* Options¶
Option	Default	Description
`fit_ebin`	False	Perform a fit for the angular extension in each analysis energy bin.
`fit_position`	False	Perform a simultaneous fit to the source position and extension.
`fix_shape`	False	Fix spectral shape parameters of the source of interest. If True then only the normalization parameter will be fit.
`free_background`	False	Leave background parameters free when performing the fit. If True then any parameters that are currently free in the model will be fit simultaneously with the source of interest.
`free_radius`	None	Free normalizations of background sources within this angular distance in degrees from the source of interest. If None then no sources will be freed.
`make_plots`	False	Generate diagnostic plots.
`make_tsmap`	True	Make a TS map for the source of interest.
`psf_scale_fn`	None	Tuple of two vectors (logE,f) defining an energy-dependent PSF scaling function that will be applied when building spatial models for the source of interest. The tuple (logE,f) defines the fractional corrections f at the sequence of energies logE = log10(E/MeV) where f=0 corresponds to no correction. The correction function f(E) is evaluated by linearly interpolating the fractional correction factors f in log(E). The corrected PSF is given by P’(x;E) = P(x/(1+f(E));E) where x is the angular separation.
`save_model_map`	False	Save model counts cubes for the best-fit model of extension.
`spatial_model`	RadialGaussian	Spatial model that will be used to test the sourceextension. The spatial scale parameter of the model will be set such that the 68% containment radius of the model is equal to the width parameter.
`sqrt_ts_threshold`	None	Threshold on sqrt(TS_ext) that will be applied when `update` is True. If None then nothreshold is applied.
`tsmap_fitter`	tsmap	Set the method for generating the TS map. Valid options are tsmap or tscube.
`update`	False	Update this source with the best-fit model for spatial extension if TS_ext > `tsext_threshold`.
`width`	None	Sequence of values in degrees for the likelihood scan over spatial extension (68% containment radius). If this argument is None then the scan points will be determined from width_min/width_max/width_nstep.
`width_max`	1.0	Maximum value in degrees for the likelihood scan over spatial extent.
`width_min`	0.01	Minimum value in degrees for the likelihood scan over spatial extent.
`width_nstep`	21	Number of scan points between width_min and width_max. Scan points will be spaced evenly on a logarithmic scale between `width_min` and `width_max`.
`write_fits`	True	Write the output to a FITS file.
`write_npy`	True	Write the output dictionary to a numpy file.

fileio¶

The fileio section collects options related to file bookkeeping. The outdir option sets the root directory of the analysis instance where all output files will be written. If outdir is null then the output directory will be automatically set to the directory in which the configuration file is located. Enabling the usescratch option will stage all output data files to a temporary scratch directory created under scratchdir.

Sample fileio Configuration¶

fileio:
   outdir : null
   logfile : null
   usescratch : False
   scratchdir  : '/scratch'

*fileio* Options¶
Option	Default	Description
`logfile`	None	Path to log file. If None then log will be written to fermipy.log.
`outdir`	None	Path of the output directory. If none this will default to the directory containing the configuration file.
`outdir_regex`	[’.fits$\|.fit$\|.xml$\|.npy$\|.png$\|.pdf$\|.yaml$’]	Stage files to the output directory that match at least one of the regular expressions in this list. This option only takes effect when `usescratch` is True.
`savefits`	True	Save intermediate FITS files.
`scratchdir`	/scratch	Path to the scratch directory. If `usescratch` is True then a temporary working directory will be created under this directory.
`usescratch`	False	Run analysis in a temporary working directory under `scratchdir`.
`workdir`	None	Path to the working directory.
`workdir_regex`	[’.fits$\|.fit$\|.xml$\|.npy$’]	Stage files to the working directory that match at least one of the regular expressions in this list. This option only takes effect when `usescratch` is True.

gtlike¶

Options in the gtlike section control the setup of the likelihood analysis include the IRF name (irfs). The edisp_bin option has been recently added to implement the latest handling of the energy dispersion (see FSSC for further details).

*gtlike* Options¶
Option	Default	Description
`bexpmap`	None
`bexpmap_base`	None	Set the basline all-sky expoure map file. This will be used to generate a scaled source map.
`bexpmap_roi`	None
`bexpmap_roi_base`	None	Set the basline ROI expoure map file. This will be used to generate a scaled source map.
`convolve`	True
`edisp`	True	Enable the correction for energy dispersion.
`edisp_bins`	-1	Number of bins to use for energy correction.
`edisp_disable`	None	Provide a list of sources for which the edisp correction should be disabled.
`expscale`	None	Exposure correction that is applied to all sources in the analysis component. This correction is superseded by `src_expscale` if it is defined for a source.
`irfs`	None	Set the IRF string.
`llscan_npts`	20	Number of evaluation points to use when performing a likelihood scan.
`minbinsz`	0.05	Set the minimum bin size used for resampling diffuse maps.
`resample`	True
`rfactor`	2
`src_expscale`	None	Dictionary of exposure corrections for individual sources keyed to source name. The exposure for a given source will be scaled by this value. A value of 1.0 corresponds to the nominal exposure.
`srcmap`	None	Set the source maps file. When defined this file will be used instead of the local source maps file.
`srcmap_base`	None	Set the baseline source maps file. This will be used to generate a scaled source map.
`use_external_srcmap`	False	Use an external precomputed source map file.
`use_scaled_srcmap`	False	Generate source map by scaling an external srcmap file.
`wmap`	None	Likelihood weights map.

lightcurve¶

The options in lightcurve control the default behavior of the lightcurve method. For more information about using this method see the Light Curves page.

*lightcurve* Options¶
Option	Default	Description
`binsz`	86400.0	Set the lightcurve bin size in seconds.
`free_background`	False	Leave background parameters free when performing the fit. If True then any parameters that are currently free in the model will be fit simultaneously with the source of interest.
`free_params`	None	Set the parameters of the source of interest that will be re-fit in each time bin. If this list is empty then all parameters will be freed.
`free_radius`	None	Free normalizations of background sources within this angular distance in degrees from the source of interest. If None then no sources will be freed.
`free_sources`	None	List of sources to be freed. These sources will be added to the list of sources satisfying the free_radius selection.
`make_plots`	False	Generate diagnostic plots.
`max_free_sources`	5	Maximum number of sources that will be fit simultaneously with the source of interest.
`multithread`	False	Split the calculation across number of processes set by nthread option.
`nbins`	None	Set the number of lightcurve bins. The total time range will be evenly split into this number of time bins.
`nthread`	None	Number of processes to create when multithread is True. If None then one process will be created for each available core.
`outdir`	None	Store all data in this directory (e.g. “30days”). If None then use current directory.
`save_bin_data`	True	Save analysis directories for individual time bins. If False then only the analysis results table will be saved.
`shape_ts_threshold`	16.0	Set the TS threshold at which shape parameters of sources will be freed. If a source is detected with TS less than this value then its shape parameters will be fixed to values derived from the analysis of the full time range.
`systematic`	0.02	Systematic correction factor for TS:subscript:`var`. See Sect. 3.6 in 2FGL for details.
`time_bins`	None	Set the lightcurve bin edge sequence in MET. This option takes precedence over binsz and nbins.
`use_local_ltcube`	True	Generate a fast LT cube.
`use_scaled_srcmap`	False	Generate approximate source maps for each time bin by scaling the current source maps by the exposure ratio with respect to that time bin.
`write_fits`	True	Write the output to a FITS file.
`write_npy`	True	Write the output dictionary to a numpy file.

model¶

The model section collects options that control the inclusion of point-source and diffuse components in the model. galdiff and isodiff set the templates for the Galactic IEM and isotropic diffuse respectively. catalogs defines a list of catalogs that will be merged to form a master analysis catalog from which sources will be drawn. Valid entries in this list can be FITS files or XML model files. sources can be used to insert additional point-source or extended components beyond those defined in the master catalog. src_radius and src_roiwidth set the maximum distance from the ROI center at which sources in the master catalog will be included in the ROI model.

Sample model Configuration¶

model :

  # Diffuse components
  galdiff  : '$FERMI_DIR/refdata/fermi/galdiffuse/gll_iem_v06.fits'
  isodiff  : '$FERMI_DIR/refdata/fermi/galdiffuse/iso_P8R2_SOURCE_V6_v06.txt'

  # List of catalogs to be used in the model.
  catalogs :
    - '3FGL'
    - 'extra_sources.xml'

  sources :
    - { 'name' : 'SourceA', 'ra' : 60.0, 'dec' : 30.0, 'SpectrumType' : PowerLaw }
    - { 'name' : 'SourceB', 'ra' : 58.0, 'dec' : 35.0, 'SpectrumType' : PowerLaw }

  # Include catalog sources within this distance from the ROI center
  src_radius  : null

  # Include catalog sources within a box of width roisrc.
  src_roiwidth : 15.0

*model* Options¶
Option	Default	Description
`assoc_xmatch_columns`	[‘3FGL_Name’]	Choose a set of association columns on which to cross-match catalogs.
`catalogs`	None
`diffuse`	None
`diffuse_dir`	None
`diffuse_xml`	None
`extdir`	None	Set a directory that will be searched for extended source FITS templates. Template files in this directory will take precendence over catalog source templates with the same name.
`extract_diffuse`	False	Extract a copy of all mapcube components centered on the ROI.
`galdiff`	None	Set the path to one or more galactic IEM mapcubes. A separate component will be generated for each item in this list.
`isodiff`	None	Set the path to one or more isotropic templates. A separate component will be generated for each item in this list.
`limbdiff`	None
`merge_sources`	True	Merge properties of sources that appear in multiple source catalogs. If merge_sources=false then subsequent sources with the same name will be ignored.
`sources`	None
`src_radius`	None	Radius of circular region in degrees centered on the ROI that selects sources for inclusion in the model. If this parameter is none then no selection is applied. This selection is ORed with the `src_roiwidth` selection.
`src_radius_roi`	None	Half-width of `src_roiwidth` selection. This parameter can be used in lieu of `src_roiwidth`.
`src_roiwidth`	None	Width of square region in degrees centered on the ROI that selects sources for inclusion in the model. If this parameter is none then no selection is applied. This selection will be ORed with the `src_radius` selection.

optimizer¶

*optimizer* Options¶
Option	Default	Description
`init_lambda`	0.0001	Initial value of damping parameter for step size calculation when using the NEWTON fitter. A value of zero disables damping.
`max_iter`	100	Maximum number of iterations for the Newtons method fitter.
`min_fit_quality`	2	Set the minimum fit quality.
`optimizer`	MINUIT	Set the optimization algorithm to use when maximizing the likelihood function.
`retries`	3	Set the number of times to retry the fit when the fit quality is less than `min_fit_quality`.
`tol`	0.001	Set the optimizer tolerance.
`verbosity`	0

plotting¶

*plotting* Options¶
Option	Default	Description
`catalogs`	None
`cmap`	magma	Set the colormap for 2D plots.
`cmap_resid`	RdBu_r	Set the colormap for 2D residual plots.
`figsize`	[8.0, 6.0]	Set the default figure size.
`format`	png
`graticule_radii`	None	Define a list of radii at which circular graticules will be drawn.
`interactive`	False	Enable interactive mode. If True then plots will be drawn after each plotting command.
`label_ts_threshold`	0.0	TS threshold for labeling sources in sky maps. If None then no sources will be labeled.
`loge_bounds`	None

residmap¶

The options in residmap control the default behavior of the residmap method. For more information about using this method see the Residual Map page.

*residmap* Options¶
Option	Default	Description
`exclude`	None	List of sources that will be removed from the model when computing the residual map.
`loge_bounds`	None	Restrict the analysis to an energy range (emin,emax) in log10(E/MeV) that is a subset of the analysis energy range. By default the full analysis energy range will be used. If either emin/emax are None then only an upper/lower bound on the energy range wil be applied.
`make_plots`	False	Generate diagnostic plots.
`model`	None	Dictionary defining the spatial/spectral properties of the test source. If model is None the test source will be a PointSource with an Index 2 power-law spectrum.
`use_weights`	False	Used weighted version of maps in making plots.
`write_fits`	True	Write the output to a FITS file.
`write_npy`	True	Write the output dictionary to a numpy file.

roiopt¶

The options in roiopt control the default behavior of the optimize method. For more information about using this method see the ROI Optimization and Fitting page.

*roiopt* Options¶
Option	Default	Description
`max_free_sources`	5	Maximum number of sources that will be fit simultaneously in the first optimization step.
`npred_frac`	0.95
`npred_threshold`	1.0
`shape_ts_threshold`	25.0	Threshold on source TS used for determining the sources that will be fit in the third optimization step.
`skip`	None	List of str source names to skip while optimizing.

sed¶

The options in sed control the default behavior of the sed method. For more information about using this method see the SED Analysis page.

*sed* Options¶
Option	Default	Description
`bin_index`	2.0	Spectral index that will be use when fitting the energy distribution within an energy bin.
`cov_scale`	3.0	Scale factor that sets the strength of the prior on nuisance parameters that are free. Setting this to None disables the prior.
`free_background`	False	Leave background parameters free when performing the fit. If True then any parameters that are currently free in the model will be fit simultaneously with the source of interest.
`free_pars`	None	Set the parameters of the source of interest that will be freed when performing the global fit. By default all parameters will be freed.
`free_radius`	None	Free normalizations of background sources within this angular distance in degrees from the source of interest. If None then no sources will be freed.
`make_plots`	False	Generate diagnostic plots.
`ul_confidence`	0.95	Confidence level for flux upper limit.
`use_local_index`	False	Use a power-law approximation to the shape of the global spectrum in each bin. If this is false then a constant index set to `bin_index` will be used.
`write_fits`	True	Write the output to a FITS file.
`write_npy`	True	Write the output dictionary to a numpy file.

selection¶

The selection section collects parameters related to the data selection and target definition. The majority of the parameters in this section are arguments to gtselect and gtmktime. The ROI center can be set with the target parameter by providing the name of a source defined in one of the input catalogs (defined in the model section). Alternatively the ROI center can be defined by giving explicit sky coordinates with ra and dec or glon and glat.

selection:

  # gtselect parameters
  emin    : 100
  emax    : 100000
  zmax    : 90
  evclass : 128
  evtype  : 3
  tmin    : 239557414
  tmax    : 428903014

  # gtmktime parameters
  filter : 'DATA_QUAL>0 && LAT_CONFIG==1'
  roicut : 'no'

  # Set the ROI center to the coordinates of this source
  target : 'mkn421'

*selection* Options¶
Option	Default	Description
`convtype`	None	Conversion type selection.
`dec`	None
`emax`	None	Maximum Energy (MeV)
`emin`	None	Minimum Energy (MeV)
`evclass`	None	Event class selection.
`evtype`	None	Event type selection.
`filter`	None	Filter string for `gtmktime` selection.
`glat`	None
`glon`	None
`logemax`	None	Maximum Energy (log10(MeV))
`logemin`	None	Minimum Energy (log10(MeV))
`phasemax`	None	Maximum pulsar phase
`phasemin`	None	Minimum pulsar phase
`ra`	None
`radius`	None	Radius of data selection. If none this will be automatically set from the ROI size.
`roicut`	no
`target`	None	Choose an object on which to center the ROI. This option takes precendence over ra/dec or glon/glat.
`tmax`	None	Maximum time (MET).
`tmin`	None	Minimum time (MET).
`zmax`	None	Maximum zenith angle.

sourcefind¶

The options in sourcefind control the default behavior of the find_sources method. For more information about using this method see the Source Finding page.

*sourcefind* Options¶
Option	Default	Description
`free_params`	None
`max_iter`	5	Maximum number of source finding iterations. The source finder will continue adding sources until no additional peaks are found or the number of iterations exceeds this number.
`min_separation`	1.0	Minimum separation in degrees between sources detected in each iteration. The source finder will look for the maximum peak in the TS map within a circular region of this radius.
`model`	None	Dictionary defining the spatial/spectral properties of the test source. If model is None the test source will be a PointSource with an Index 2 power-law spectrum.
`multithread`	False	Split the calculation across number of processes set by nthread option.
`nthread`	None	Number of processes to create when multithread is True. If None then one process will be created for each available core.
`sources_per_iter`	4	Maximum number of sources that will be added in each iteration. If the number of detected peaks in a given iteration is larger than this number, only the N peaks with the largest TS will be used as seeds for the current iteration.
`sqrt_ts_threshold`	5.0	Source threshold in sqrt(TS). Only peaks with sqrt(TS) exceeding this threshold will be used as seeds for new sources.
`tsmap_fitter`	tsmap	Set the method for generating the TS map. Valid options are tsmap or tscube.

tsmap¶

The options in tsmap control the default behavior of the tsmap method. For more information about using this method see the TS Map page.

*tsmap* Options¶
Option	Default	Description
`exclude`	None	List of sources that will be removed from the model when computing the TS map.
`loge_bounds`	None	Restrict the analysis to an energy range (emin,emax) in log10(E/MeV) that is a subset of the analysis energy range. By default the full analysis energy range will be used. If either emin/emax are None then only an upper/lower bound on the energy range wil be applied.
`make_plots`	False	Generate diagnostic plots.
`max_kernel_radius`	3.0	Set the maximum radius of the test source kernel. Using a smaller value will speed up the TS calculation at the loss of accuracy.
`model`	None	Dictionary defining the spatial/spectral properties of the test source. If model is None the test source will be a PointSource with an Index 2 power-law spectrum.
`multithread`	False	Split the calculation across number of processes set by nthread option.
`nthread`	None	Number of processes to create when multithread is True. If None then one process will be created for each available core.
`write_fits`	True	Write the output to a FITS file.
`write_npy`	True	Write the output dictionary to a numpy file.

tscube¶

The options in tscube control the default behavior of the tscube method. For more information about using this method see the TS Cube page.

*tscube* Options¶
Option	Default	Description
`cov_scale`	-1.0	Scale factor to apply to broadband fitting cov. matrix in bin-by-bin fits ( < 0 -> fixed )
`cov_scale_bb`	-1.0	Scale factor to apply to global fitting cov. matrix in broadband fits. ( < 0 -> no prior )
`do_sed`	True	Compute the energy bin-by-bin fits
`exclude`	None	List of sources that will be removed from the model when computing the TS map.
`init_lambda`	0	Initial value of damping parameter for newton step size calculation. A value of zero disables damping.
`max_iter`	30	Maximum number of iterations for the Newtons method fitter.
`model`	None	Dictionary defining the spatial/spectral properties of the test source. If model is None the test source will be a PointSource with an Index 2 power-law spectrum.
`nnorm`	10	Number of points in the likelihood v. normalization scan
`norm_sigma`	5.0	Number of sigma to use for the scan range
`remake_test_source`	False	If true, recomputes the test source image (otherwise just shifts it)
`st_scan_level`	0	Level to which to do ST-based fitting (for testing)
`tol`	0.001	Critetia for fit convergence (estimated vertical distance to min < tol )
`tol_type`	0	Absoulte (0) or relative (1) criteria for convergence.