fermipy.diffuse subpackage¶
The fermipy.diffuse sub-package is a collection of standalone utilities that allow the user to parallelize the data and template preparation for all-sky analysis.
The tools described here perform a number of functions:
- Making binned counts maps and expousre maps over the whole sky.
- Managing model components for all-sky analysis; including both diffuse emission and point source contributions. This includes make spatial-spectral templates and expected counts maps for various components.
- Building intergrated models for a collection of model components.
- Fitting those models.
Overview¶
This package implements an analysis pipeline prepare data and templates for analysis. This involves a lot of bookkeeping and loops over various things.It is probably easiest to first describe this with a bit of pseudo-code that represents the various analysis steps.
The various loop variables are:
Input data files
For practical reasons, the input photon event files (FT1) files are split into monthly files. In the binning step of the analysis we loop over those files.
binning components
We split the data into several “binning components” and make seperate binned counts maps for each components. A binning component is defined by energy range and data sub-selection (such as PSF event type and zenith angle cuts).
Diffuse model components
The set of all the model components that represent diffuse emission, such as contributions for cosmic ray interactions with
Catalog model components
The set of all the catalog sources (both point sources and extended source), merged into a few distinct contributions.
Diffuse emission model definitions
A set of user defined models that merge the various model components with specific spectral models.
# Data Binning, prepare the analysis directories and precompute the DM spectra
# First we loop over all the input input files and split up the
# data by binning component and bin the data using the command
# fermipy-split-and-bin-sg, which is equivalent to:
for file in input_data_files:
fermipy-split-and-bin(file)
# Then we loop over the binning components and coadd the binned
# data from all the input files using the command
# fermipy-coadd-split-sf, which is equivalent to:
for comp in binning_components:
fermipy-coadd-split(comp)
# We also loop over the binning components and compute the
# exposure maps for each binning component using the command
# fermipy-gtexpcube2-sg, which is equivalent to:
for comp in binned_components:
gtexpcube2(comp)
# We loop over the diffuse components that come from GALProp
# templates and refactor them using the command
# fermipy-sum-ring-gasmaps-sg, which is equivalent to
for galprop_comp in diffuse_galprop_components:
fermipy-coadd(galprop_comp)
# We do a triple loop over all of the diffuse components, all the
# binning components and all the energy bins and convolve the
# emission template with the instrument response using the command
# fermipy-srcmaps-diffuse-sg, which is equivalent to
for diffuse_comp in diffuse_components:
for binning_comp in binned_components:
for energy in energy_bins:
fermipy-srcmap-diffuse(diffuse_comp, binning_comp, energy)
# We then do a double loop over all the diffuse components and all
# the binning components and stack the template maps into single
# files using the command
# fermipy-vstack-diffuse-sg, which is equivalent to
for diffuse_comp in diffuse_components:
for binning_comp in binned_components:
fermipy-vstack-diffuse(diffuse_comp, binning_comp)
# We then do a double loop over source catalogs and binning
# components and compute the templates for each source using the
# command
# fermipy-srcmaps-catalog-sg, which is equivalent to
for catalog in catalogs:
for binning_comp in binned_components:
fermipy-srcmaps-catalog(catalog, binning_comp)
# We then loop over the catalog components (essentially
# sub-sets of catalog sources that we want to merge)
# and merge those sources into templates using the command
# fermipy-merge-srcmaps-sg, which is equivalent to
for catalog_comp in catalog_components:
for binning_comp in binned_components:
fermipy-merge-srcmaps(catalog_comp, binning_comp)
# At this point we have a library to template maps for all the
# emision components that we have defined.
# Now we want to define specific models. We do this
# using the commands
# fermipy-init-model and fermipy-assemble-model-sg, which is equivalent to
for model in models:
fermipy-assemble-model(model)
# At this point we, for each model under consideration we have an
# analysis directory that is set up for fermipy
Configuration¶
This section describes the configuration management scheme used within the fermipy.diffuse package and documents the configuration parameters that can be set in the configuration file.
Analysis classes in the dmpipe package all inherit from the fermipy.jobs.Link
class, which allow user to invoke the class either interactively within python
or from the unix command line.
From the command line
$ fermipy-srcmaps-diffuse-sg --comp config/binning.yaml --data config/dataset_source.yaml --library models/library.yaml
From python there are a number of ways to do it, we recommend this:
from fermipy.diffuse.gt_srcmap_partial import SrcmapsDiffuse_SG
link = SrcmapsDiffuse_SG( )
link.update_args(dict(comp='config/binning.yaml', data='config/dataset_source.yaml',library='models/library.yaml'))
link.run()
Top Level Configuration¶
We use a yaml file to define the top-level analysis parameters.
# The binning components
comp : config/binning.yaml
# The dataset
data : config/dataset_source.yaml
# Library with the fitting components
library : models/library.yaml
# Yaml file with the list of models to prepare
models : models/modellist.yaml
# Input FT1 file
ft1file : P8_P305_8years_source_zmax105.lst
# HEALPix order for counts cubes
hpx_order_ccube : 9
# HEALPix order for exposure cubes
hpx_order_expcube : 6
# HEALPix order fitting models
hpx_order_fitting : 7
# Build the XML files for the diffuse emission model components
make_diffuse_comp_xml : True
# Build the XML files for the catalog source model components
make_catalog_comp_xml : True
# Name of the directory for the merged GALProp gasmaps
merged_gasmap_dir : merged_gasmap
# Number of catalog sources per batch job
catalog_nsrc : 500
Binning Configuration¶
We use a yaml file to define the binning components.
coordsys : 'GAL'
E0:
log_emin : 1.5
log_emax : 2.0
enumbins : 2
zmax : 80.
psf_types :
PSF3 :
hpx_order : 5
E1:
log_emin : 2.0
log_emax : 2.5
enumbins : 2
zmax : 90.
psf_types :
PSF23 :
hpx_order : 6
E2:
log_emin : 2.5
log_emax : 3.0
enumbins : 3
zmax : 100.
psf_types :
PSF123 :
hpx_order : 8
E3:
log_emin : 3.0
log_emax : 6.0
enumbins : 9
zmax : 105.
psf_types :
PSF0123 :
hpx_order : 9
- coordsys : ‘GAL’ or ‘CEL’
- Coordinate system to use
- log_emin, log_emax: float
- Energy bin boundries in log10(MeV)
- enumbins: int
- Number of energy bins for this binning component
- zmax : float
- Maximum zenith angle (in degrees) for this binning component
- psf_types: dict
- Sub-dictionary of binning components by PSF event type, PSF3 means PSF event type 3 events only. PSF0123 means all four PSF event types.
- hpx_order: int
- HEALPix order to use for binning. The more familiar nside parameter is nside = 2**order
Dataset Configuration¶
We use a yaml file to define the data set we are using. The example below specifies using a pre-defined 8 year dataset, selecting the “SOURCE” event class and using the V2 version of the corresponding IRFs (specifically P8R3_SOURCE_V2).
basedir : '/gpfs/slac/kipac/fs1/u/dmcat/data/flight/diffuse_dev'
data_pass : 'P8'
data_ver : 'P305'
evclass : 'source'
data_time : '8years'
irf_ver : 'V2'
The basedir parameter should point at the analysis directory. For the most part the other parameter are using to make the names of the various files produced by the pipeline. The evclass parameter defines the event selection, and the IRF version is defined by a combination of the data_ver, evclass and irf_ver parameters.
GALProp Rings Configuration¶
We use a yaml file to define the how we combine GALProp emission templates. The example below specifies how to construct as series of ‘merged_CO’ rings by combining GALProp intensity template predictions.
galprop_run : 56_LRYusifovXCO5z6R30_QRPD_150_rc_Rs8
ring_limits : [1, 2, 3, 4, 6, 8, 9, 10, 11, 12, 13, 15]
diffuse_comp_dict :
merged_CO : ['pi0_decay_H2R', 'bremss_H2R']
remove_rings : ['merged_CO_7']
- galprop_run : string
- Define the GALProp run to use for this component. This is used to make the filenames for input template maps.
- ring_limits : list of int
- This specfies how to combine the GALProp rings into a smaller set of rings.
- diffuse_comp_dict : dict
- This specifies how to make GALProp components into merged components for the diffuse analysis
- remove_rings: list of str
- This allow use to remove certain rings from the model
Catalog Component Configuration¶
We use a yaml file to define the how we split up the catalog source components. The example below specifies using the FL8Y source list, and to split out the faint sources (i.e., those with the Signif_Avg value less that 100.), and the extended source, and to keep all the remaining sources (i.e., the bright, pointlike, sources) as individual sources.
catalog_name : FL8Y
catalog_file : /nfs/slac/kipac/fs1/u/dmcat/ancil/catalogs/official/4FGLp/gll_psc_8year_v4.fit
catalog_extdir : /nfs/slac/kipac/fs1/u/dmcat/ancil/catalogs/official/extended/Extended_archive_v18
catalog_type : FL8Y
rules_dict :
faint :
cuts :
- { cut_var: Signif_Avg, max_val : 100. }
- mask_extended
extended :
cuts :
- select_extended
remainder :
merge : False
Model Component Library¶
We use a yaml file to define a “library” of model components. The comprises a set of named emission components, and a set one or more versions for each named component. Here is an example library defintion file.
# Catalog Components
FL8Y :
model_type : catalog
versions : [v00]
# Diffuse Components
galprop_rings :
model_type : galprop_rings
versions : [p8-ref_IC_thin, p8-ref_HI_150, p8-ref_CO_300_mom, p8-ref_dnm_300hp]
dnm_merged:
model_type : MapCubeSource
versions : ['like_4y_300K']
gll-iem :
model_type : MapCubeSource
versions : [v06]
loopI :
model_type : MapCubeSource
versions : [haslam]
bubbles :
model_type : MapCubeSource
versions : [v00, v01]
iso_map :
model_type : MapCubeSource
versions : [P8R3_SOURCE_V2]
patches :
model_type : MapCubeSource
versions : [v09]
selection_dependent : True
no_psf : True
edisp_disable : True
unresolved :
model_type : MapCubeSource
versions : [strong]
sun-ic :
model_type : MapCubeSource
versions : [v2r0, P8R3-v2r0]
moving : True
edisp_disable : True
sun-disk :
model_type : MapCubeSource
versions : [v3r1, P8R3-v3r1]
moving : True
edisp_disable : True
moon :
model_type : MapCubeSource
versions : [v3r2, P8R3-v3r2]
moving : True
edisp_disable : True
- model_type: ‘MapCubeSource’ or ‘catalog’ or ‘galprop_rings’ or ‘SpatialMap’
- Specifies how this model should be constructed. See more below in the versions parameters.
- versions: list of str
- Specifies different versions of this model component. How this string is used depend on the model type. for ‘MapCubeSource’ and ‘SpatialMap’ sources it is used to construct the expected filename for the intensity template. For ‘catalog’ and ‘galprop_rings’ it is used to construct the filename for the yaml file that defines the sub-components for that component.
- moving: bool If true, then will use source-specific livetime cubes to constuct source templates for each zenith angle cut.
- selection_dependent : bool If true, then will used different source templates for each binning component.
- no_psf : bool Turns of PSF convolution for this source. Useful for data-driven components.
- edisp_disable : bool Turns off energy dispersion for the source. Useful for data-driven components.
Spectral Model Configuration¶
We use a yaml file to define the spectral models and default parameters. This file is simply a dictionary mapping names to sub-dictionaries defining spectral models and default model parameters.
Model Defintion¶
We use a yaml file to define each overall model, which combine library components and spectral models.
library : models/library.yaml
spectral_models : models/spectral_models.yaml
sources :
galprop_rings_p8-ref_IC_thin :
model_type : galprop_rings
version : p8-ref_IC_150
SpectrumType :
default : Constant_Correction
galprop_rings-p8-ref_HI_300:
model_type : galprop_rings
version : p8-ref_HI_300
SpectrumType :
default : Powerlaw_Correction
merged_HI_2_p8-ref_HI_300 : BinByBin_5
merged_HI_3_p8-ref_HI_300 : BinByBin_5
merged_HI_4_p8-ref_HI_300 : BinByBin_9
merged_HI_5_p8-ref_HI_300 : BinByBin_9
merged_HI_6_p8-ref_HI_300 : BinByBin_9
merged_HI_8_p8-ref_HI_300 : BinByBin_5
merged_HI_9_p8-ref_HI_300 : BinByBin_5
galprop_rings-p8-ref_CO_300:
model_type : galprop_rings
version : p8-ref_CO_300_mom
SpectrumType :
default : Powerlaw_Correction
merged_CO_2_p8-ref_CO_300_mom : BinByBin_5
merged_CO_3_p8-ref_CO_300_mom : BinByBin_5
merged_CO_4_p8-ref_CO_300_mom : BinByBin_9
merged_CO_5_p8-ref_CO_300_mom : BinByBin_9
merged_CO_6_p8-ref_CO_300_mom : BinByBin_9
merged_CO_8_p8-ref_CO_300_mom : BinByBin_5
merged_CO_9_p8-ref_CO_300_mom : BinByBin_5
dnm_merged :
version : like_4y_300K
SpectrumType : BinByBin_5
iso_map :
version : P8R3_SOURCE_V2
SpectrumType : Iso
sun-disk :
version : v3r1
SpectrumType : Constant_Correction
edisp_disable : True
sun-ic :
version : v2r0
SpectrumType : Constant_Correction
edisp_disable : True
moon :
version : v3r2
SpectrumType : Constant_Correction
edisp_disable : True
patches :
version : v09
SpectrumType : Patches
edisp_disable : True
unresolved :
version : strong
SpectrumType : Constant_Correction
FL8Y :
model_type : Catalog
version : v00
SpectrumType :
default : Constant_Correction
FL8Y_v00_remain : Catalog
FL8Y_v00_faint : BinByBin_9
- model_type: ‘MapCubeSource’ or ‘catalog’ or ‘galprop_rings’ or ‘SpatialMap’
- Specifies how this model should be constructed. See more below.
- version: str
- Specifies version of this model component.
- edisp_disable : bool Turns off energy dispersion for the source. Useful for data-driven components. Needed for model XML file construction.
- SpectrumType: str or dictionary This specifies the Spectrum type to use for this model component. For ‘catalog’ and ‘galprop_rings’ model types it can be a dictionary mapping model sub-components to spectrum types. Note that the spectrum types should be defined in the spectral model configuration described above.
Examples¶
Module contents¶
Configuration, binning, default options, etc…¶
Small helper class to represent the binning used for a single component of a summed likelihood in diffuse analysis
-
class
fermipy.diffuse.binning.
Component
(**kwargs)[source]¶ Bases:
object
Small helper class to represent the binning used for a single component of a summed likelihood in diffuse analysis
Parameters: - log_emin (float) – Log base 10 of minimum energy for this component
- log_emax (float) – Log base 10 of maximum energy for this component
- enumbins (int) – Number of energy bins for this component
- zmax (float) – Maximum zenith angle cube for this component in degrees
- mktimefilters (list) – Filters for gtmktime.
- hpx_order (int) – HEALPix order to use for this component
- coordsys (str) – Coodinate system, ‘CEL’ or ‘GAL’
-
classmethod
build_from_energy_dict
(ebin_name, input_dict)[source]¶ Build a list of components from a dictionary for a single energy range
-
emax
¶ Maximum energy for this component
-
emin
¶ Minimum energy for this component
-
evtype
¶ Event type bit mask for this component
Analysis framework for all-sky diffuse emission fitting
Handle the naming conventions for composite likelihood analysis
-
class
fermipy.diffuse.name_policy.
NameFactory
(**kwargs)[source]¶ Bases:
object
Helper class to define file names and keys consistently.
-
angprofile_format
= 'templates/profile_{sourcekey}.fits'¶
-
bexpcube_format
= 'bexp_cubes/bexcube_{dataset}_{mktime}_{component}_{coordsys}_{irf_ver}.fits'¶
-
bexpcubemoon_format
= 'bexp_cubes/bexcube_{dataset}_{mktime}_{component}_{irf_ver}_moon.fits'¶
-
bexpcubesun_format
= 'bexp_cubes/bexcube_{dataset}_{mktime}_{component}_{irf_ver}_sun.fits'¶
-
catalog_split_yaml_format
= 'models/catalog_{sourcekey}.yaml'¶
-
ccube_format
= 'counts_cubes/ccube_{dataset}_{mktime}_{component}_{coordsys}.fits'¶
-
comp_srcmdl_xml_format
= 'analysis/model_{modelkey}/srcmdl_{modelkey}_{component}.xml'¶
-
component_format
= '{zcut}_{ebin}_{psftype}'¶
-
dataset_format
= '{data_pass}_{data_ver}_{data_time}_{evclass}'¶
-
diffuse_template_format
= 'templates/template_{sourcekey}.fits'¶
-
ft1file_format
= '{dataset}_{zcut}.lst'¶
-
ft2file_format
= 'ft2_files/ft2_{data_time}.lst'¶
-
fullpath_format
= '{basedir}/{localpath}'¶
-
galprop_gasmap_format
= 'gasmap/{sourcekey}_{projtype}_{galprop_run}.gz'¶
-
galprop_ringkey
(**kwargs)[source]¶ return the sourcekey for galprop input maps : specifies the component and ring
-
galprop_ringkey_format
= '{source_name}_{ringkey}'¶
-
galprop_rings_yaml_format
= 'models/galprop_rings_{galkey}.yaml'¶
-
galprop_sourcekey
(**kwargs)[source]¶ return the sourcekey for merged galprop maps : specifies the merged component and merging scheme
-
galprop_sourcekey_format
= '{source_name}_{galpropkey}'¶
-
generic
(input_string, **kwargs)[source]¶ return a generic filename for a given dataset and component
-
ltcube_format
= 'lt_cubes/ltcube_{data_time}_{mktime}_{zcut}.fits'¶
-
ltcubemoon_format
= 'sunmoon/ltcube_{data_time}_{mktime}_{zcut}_moon.fits'¶
-
ltcubesun_format
= 'sunmoon/ltcube_{data_time}_{mktime}_{zcut}_sun.fits'¶
-
master_srcmdl_xml_format
= 'analysis/model_{modelkey}/srcmdl_{modelkey}_master.xml'¶
-
mcube_format
= 'model_cubes/mcube_{sourcekey}_{dataset}_{mktime}_{component}_{coordsys}_{irf_ver}.fits'¶
-
merged_gasmap_format
= 'merged_gasmaps/{sourcekey}_{projtype}.fits'¶
-
merged_sourcekey
(**kwargs)[source]¶ return the sourcekey for merged sets of point sources : specifies the catalog and merging rule
-
merged_sourcekey_format
= '{catalog}_{rulekey}'¶
-
merged_srcmaps_format
= 'analysis/model_{modelkey}/srcmaps_{dataset}_{mktime}_{component}_{coordsys}_{irf_ver}.fits'¶
-
mktime_format
= 'counts_cubes/mktime_{dataset}_{mktime}_{component}.fits'¶
-
model_yaml_format
= 'models/model_{modelkey}.yaml'¶
-
nested_srcmdl_xml
(**kwargs)[source]¶ return the file name for source model xml files of nested sources
-
nested_srcmdl_xml_format
= 'srcmdls/{sourcekey}_sources.xml'¶
-
residual_cr_format
= 'residual_cr/residual_cr_{dataset}_{mktime}_{component}_{coordsys}_{irf_ver}.fits'¶
-
select_format
= 'counts_cubes/select_{dataset}_{component}.fits'¶
-
sourcekey
(**kwargs)[source]¶ Return a key that specifies the name and version of a source or component
-
sourcekey_format
= '{source_name}_{source_ver}'¶
-
spectral_template_format
= 'templates/spectral_{sourcekey}.txt'¶
-
srcmaps_format
= 'srcmaps/srcmaps_{sourcekey}_{dataset}_{mktime}_{component}_{coordsys}_{irf_ver}.fits'¶
-
srcmdl_xml_format
= 'srcmdls/{sourcekey}.xml'¶
-
stamp_format
= 'stamps/{linkname}.stamp'¶
-
templatesunmoon_format
= 'templates/template_{sourcekey}_{zcut}.fits'¶
-
Utilities and tools¶
Small helper class to represent the selection of mktime filters used in the analysis
-
class
fermipy.diffuse.timefilter.
MktimeFilterDict
(aliases, selections)[source]¶ Bases:
object
Small helper class toselection of mktime filters used in the analysis
Classes and utilities that create fermipy source objects
-
class
fermipy.diffuse.source_factory.
SourceFactory
[source]¶ Bases:
object
Small helper class to build and keep track of sources
-
classmethod
copy_selected_sources
(roi, source_names)[source]¶ Build and return a
fermipy.roi_model.ROIModel
object by copying selected sources from another such object
-
static
make_fermipy_roi_model_from_catalogs
(cataloglist)[source]¶ Build and return a
fermipy.roi_model.ROIModel object from a list of fermipy.catalog.Catalog
objects
-
classmethod
make_roi
(sources=None)[source]¶ Build and return a
fermipy.roi_model.ROIModel
object from a dict with information about the sources
-
source_info_dict
¶ Return the dictionary of source_info objects used to build sources
-
sources
¶ Return the dictionary of sources
-
classmethod
-
fermipy.diffuse.source_factory.
make_catalog_sources
(catalog_roi_model, source_names)[source]¶ Construct and return dictionary of sources that are a subset of sources in catalog_roi_model.
Parameters: - catalog_roi_model (dict or
fermipy.roi_model.ROIModel
) – Input set of sources - source_names (list) – Names of sourcs to extract
- dict mapping source_name to fermipy.roi_model.Source object (Returns) –
- catalog_roi_model (dict or
-
fermipy.diffuse.source_factory.
make_composite_source
(name, spectrum)[source]¶ Construct and return a
fermipy.roi_model.CompositeSource
object
-
fermipy.diffuse.source_factory.
make_isotropic_source
(name, Spectrum_Filename, spectrum)[source]¶ Construct and return a
fermipy.roi_model.IsoSource
object
-
fermipy.diffuse.source_factory.
make_mapcube_source
(name, Spatial_Filename, spectrum)[source]¶ Construct and return a
fermipy.roi_model.MapCubeSource
object
-
fermipy.diffuse.source_factory.
make_point_source
(name, src_dict)[source]¶ Construct and return a
fermipy.roi_model.Source
object
-
fermipy.diffuse.source_factory.
make_sources
(comp_key, comp_dict)[source]¶ Make dictionary mapping component keys to a source or set of sources
Parameters:
-
fermipy.diffuse.source_factory.
make_spatialmap_source
(name, Spatial_Filename, spectrum)[source]¶ Construct and return a
fermipy.roi_model.Source
object
Prepare data for diffuse all-sky analysis
Helper classes to manage model building¶
-
class
fermipy.diffuse.model_component.
ModelComponentInfo
(**kwargs)[source]¶ Bases:
object
Information about a model component
Parameters: - source_name (str) – The name given to the component, e.g., loop_I or moon
- source_ver (str) – Key to indentify the model version of the source, e.g., v00
- sourcekey (str) – Key that identifies this component, e.g., loop_I_v00 or moon_v00
- model_type (str) – Type of model, ‘MapCubeSource’ | ‘IsoSource’ | ‘CompositeSource’ | ‘Catalog’ | ‘PointSource’
- srcmdl_name (str) – Name of the xml file with the xml just for this component
- moving (bool) – Flag for moving sources (i.e., the sun and moon)
- selection_dependent (bool) – Flag for selection dependent sources (i.e., the residual cosmic ray model)
- no_psf (bool) – Flag to indicate that we do not smear this component with the PSF
- components (dict) – Sub-dictionary of
ModelComponentInfo
objects for moving and selection_dependent sources - comp_key (str) – Component key for this component of moving and selection_dependent sources
-
add_component_info
(compinfo)[source]¶ Add sub-component specific information to a particular data selection
Parameters: compinfo ( ModelComponentInfo
object) – Sub-component being added
-
clone_and_merge_sub
(key)[source]¶ Clones self and merges clone with sub-component specific information
Parameters: - key (str) – Key specifying which sub-component
- ModelComponentInfo object (Returns) –
-
class
fermipy.diffuse.model_component.
CatalogInfo
(**kwargs)[source]¶ Bases:
object
Information about a source catalog
Parameters: - catalog_name (str) – The name given to the merged component, e.g., merged_CO or merged_HI
- catalog_file (str) – Fits file with catalog data
- catalog_extdir (str) – Directory with extended source templates
- catalog_type (str) – Identifies the format of the catalog fits file: e.g., ‘3FGL’ or ‘4FGLP’
- catalog (
fermipy.catalog.Catalog
) – Catalog object - roi_model (
fermipy.roi_model.ROIModel
) – Fermipy object describing all the catalog sources - srcmdl_name (str) – Name of xml file with the catalog source model
-
class
fermipy.diffuse.model_component.
GalpropMergedRingInfo
(**kwargs)[source]¶ Bases:
object
Information about a set of Merged Galprop Rings
Parameters: - source_name (str) – The name given to the merged component, e.g., merged_CO or merged_HI
- ring (int) – The index of the merged ring
- sourcekey (str) – Key that identifies this component, e.g., merged_CO_1, or merged_HI_3
- galkey (str) – Key that identifies how to merge the galprop rings, e.g., ‘ref’
- galprop_run (str) – Key that idenfifies the galprop run used to make the input rings
- files (str) – List of files of the input gasmap files
- merged_gasmap (str) – Filename for the merged gasmap
-
class
fermipy.diffuse.model_component.
ModelComponentInfo
(**kwargs)[source] Bases:
object
Information about a model component
Parameters: - source_name (str) – The name given to the component, e.g., loop_I or moon
- source_ver (str) – Key to indentify the model version of the source, e.g., v00
- sourcekey (str) – Key that identifies this component, e.g., loop_I_v00 or moon_v00
- model_type (str) – Type of model, ‘MapCubeSource’ | ‘IsoSource’ | ‘CompositeSource’ | ‘Catalog’ | ‘PointSource’
- srcmdl_name (str) – Name of the xml file with the xml just for this component
- moving (bool) – Flag for moving sources (i.e., the sun and moon)
- selection_dependent (bool) – Flag for selection dependent sources (i.e., the residual cosmic ray model)
- no_psf (bool) – Flag to indicate that we do not smear this component with the PSF
- components (dict) – Sub-dictionary of
ModelComponentInfo
objects for moving and selection_dependent sources - comp_key (str) – Component key for this component of moving and selection_dependent sources
-
add_component_info
(compinfo)[source] Add sub-component specific information to a particular data selection
Parameters: compinfo ( ModelComponentInfo
object) – Sub-component being added
-
clone_and_merge_sub
(key)[source] Clones self and merges clone with sub-component specific information
Parameters: - key (str) – Key specifying which sub-component
- ModelComponentInfo object (Returns) –
-
get_component_info
(comp)[source] Return the information about sub-component specific to a particular data selection
Parameters: - comp (
binning.Component
object) – Specifies the sub-component - ModelComponentInfo object (Returns) –
- comp (
-
update
(**kwargs)[source] Update data members from keyword arguments
-
class
fermipy.diffuse.model_component.
IsoComponentInfo
(**kwargs)[source]¶ Bases:
fermipy.diffuse.model_component.ModelComponentInfo
Information about a model component represented by a IsoSource
Parameters: Spectral_Filename (str) – Name of the template file for the spatial model
-
class
fermipy.diffuse.model_component.
PointSourceInfo
(**kwargs)[source]¶ Bases:
fermipy.diffuse.model_component.ModelComponentInfo
Information about a model component represented by a PointSource
-
class
fermipy.diffuse.model_component.
CompositeSourceInfo
(**kwargs)[source]¶ Bases:
fermipy.diffuse.model_component.ModelComponentInfo
Information about a model component represented by a CompositeSource
Parameters: - source_names (list) – The names of the nested sources
- catalog_info (
model_component.CatalogInfo
or None) – Information about the catalog containing the nested sources - roi_model (
fermipy.roi_model.ROIModel
) – Fermipy object describing the nested sources
-
class
fermipy.diffuse.model_component.
CatalogSourcesInfo
(**kwargs)[source]¶ Bases:
fermipy.diffuse.model_component.ModelComponentInfo
Information about a model component consisting of sources from a catalog
Parameters: - source_names (list) – The names of the nested sources
- catalog_info (
model_component.CatalogInfo
or None) – Information about the catalog containing the nested sources - roi_model (
fermipy.roi_model.ROIModel
) – Fermipy object describing the nested sources
-
class
fermipy.diffuse.diffuse_src_manager.
GalpropMapManager
(**kwargs)[source]¶ Bases:
object
Small helper class to keep track of Galprop gasmaps
This keeps track of two types of dictionaries. Both are keyed by: key = {source_name}_{ring}_{galkey}
Where: {source_name} is something like ‘merged_C0’ {ring} is the ring index {galkey} is a key specifying which version of galprop rings to use.
The two dictionaries are: ring_dict[key] =
model_component.GalpropMergedRingInfo
diffuse_comp_info_dict[key] ]model_component.ModelComponentInfo
The dictionaries are defined in files called. models/galprop_rings_{galkey}.yaml
-
diffuse_comp_info_dicts
(galkey)[source]¶ Return the components info dictionary for a particular galprop key
-
make_diffuse_comp_info
(merged_name, galkey)[source]¶ Make the information about a single merged component
Parameters:
-
make_diffuse_comp_info_dict
(galkey)[source]¶ Make a dictionary maping from merged component to information about that component
Parameters: galkey (str) – A short key identifying the galprop parameters
-
make_merged_name
(source_name, galkey, fullpath)[source]¶ Make the name of a gasmap file for a set of merged rings
Parameters:
-
make_ring_dict
(galkey)[source]¶ Make a dictionary mapping the merged component names to list of template files
Parameters: - galkey (str) – Unique key for this ring dictionary
- model_component.GalpropMergedRingInfo (Returns) –
-
make_ring_filelist
(sourcekeys, rings, galprop_run)[source]¶ Make a list of all the template files for a merged component
Parameters: - sourcekeys (list-like of str) – The names of the componenents to merge
- rings (list-like of int) – The indices of the rings to merge
- galprop_run (str) – String identifying the galprop parameters
-
make_ring_filename
(source_name, ring, galprop_run)[source]¶ Make the name of a gasmap file for a single ring
Parameters:
-
-
class
fermipy.diffuse.diffuse_src_manager.
DiffuseModelManager
(**kwargs)[source]¶ Bases:
object
Small helper class to keep track of diffuse component templates
This keeps track of the ‘diffuse component infomation’ dictionary
This keyed by: key = {source_name}_{source_ver} Where: {source_name} is something like ‘loopI’ {source_ver} is somthinng like v00
The dictioary is diffuse_comp_info_dict[key] - >
model_component.ModelComponentInfo
Note that some components ( those that represent moving sources or are selection depedent ) will have a sub-dictionary of diffuse_comp_info_dict object for each sub-component
The compoents are defined in a file called config/diffuse_components.yaml
-
make_diffuse_comp_info
(source_name, source_ver, diffuse_dict, components=None, comp_key=None)[source]¶ Make a dictionary mapping the merged component names to list of template files
Parameters: - source_name (str) – Name of the source
- source_ver (str) – Key identifying the version of the source
- diffuse_dict (dict) – Information about this component
- comp_key (str) – Used when we need to keep track of sub-components, i.e., for moving and selection dependent sources.
- model_component.ModelComponentInfo or (Returns) –
- model_component.IsoComponentInfo –
-
make_diffuse_comp_info_dict
(diffuse_sources, components)[source]¶ Make a dictionary maping from diffuse component to information about that component
Parameters: Returns: ret_dict – Dictionary mapping sourcekey to
model_component.ModelComponentInfo
Return type:
-
make_template_name
(model_type, sourcekey)[source]¶ Make the name of a template file for particular component
Parameters:
-
-
class
fermipy.diffuse.catalog_src_manager.
CatalogSourceManager
(**kwargs)[source]¶ Bases:
object
Small helper class to keep track of how we deal with catalog sources
This keeps track of two dictionaries
One of the dictionaries is keyed by catalog name, and contains information about complete catalogs catalog_comp_info_dicts[catalog_name] :
model_component.CatalogInfo
The other dictionary is keyed by [{catalog_name}_{split_ver}][{split_key}] Where: {catalog_name} is something like ‘3FGL’ {split_ver} is somthing like ‘v00’ and specifes how to divide sources in the catalog {split_key} refers to a specific sub-selection of sources
split_comp_info_dicts[splitkey] :
model_component.ModelComponentInfo
-
catalog_components
(catalog_name, split_ver)[source]¶ Return the set of merged components for a particular split key
-
make_catalog_comp_info
(full_cat_info, split_key, rule_key, rule_val, sources)[source]¶ Make the information about a single merged component
Parameters: - full_cat_info (
_model_component.CatalogInfo
) – Information about the full catalog - split_key (str) – Key identifying the version of the spliting used
- rule_key (str) – Key identifying the specific rule for this component
- rule_val (list) – List of the cuts used to define this component
- sources (list) – List of the names of the sources in this component
- CompositeSourceInfo or CatalogSourcesInfo (Returns) –
- full_cat_info (
-
make_catalog_comp_info_dict
(catalog_sources)[source]¶ Make the information about the catalog components
Parameters: catalog_sources (dict) – Dictionary with catalog source defintions Returns: - catalog_ret_dict (dict) – Dictionary mapping catalog_name to
model_component.CatalogInfo
- split_ret_dict (dict) – Dictionary mapping sourcekey to
model_component.ModelComponentInfo
- catalog_ret_dict (dict) – Dictionary mapping catalog_name to
-
split_comp_info
(catalog_name, split_ver, split_key)[source]¶ Return the info for a particular split key
-
-
class
fermipy.diffuse.model_manager.
ModelComponent
(**kwargs)[source]¶ Bases:
object
Small helper class to tie a ModelComponentInfo to a spectrum
-
class
fermipy.diffuse.model_manager.
ModelInfo
(**kwargs)[source]¶ Bases:
object
Small helper class to keep track of a single fitting model
-
component_names
¶ Return the list of name of the components
-
edisp_disable_list
()[source]¶ Return the list of source for which energy dispersion should be turned off
-
make_model_rois
(components, name_factory)[source]¶ Make the fermipy roi_model objects for each of a set of binning components
-
make_srcmap_manifest
(components, name_factory)[source]¶ Build a yaml file that specfies how to make the srcmap files for a particular model
Parameters: - components (list) – The binning components used in this analysis
- name_factory (
NameFactory
) – Object that handles naming conventions - a dictionary that contains information about where to find the (Returns) –
- maps for each component of the model (source) –
-
-
class
fermipy.diffuse.model_manager.
ModelManager
(**kwargs)[source]¶ Bases:
object
Small helper class to create fitting models and manager XML files for fermipy
This class contains a ‘library’, which is a dictionary of all the source components:
specifically it maps:
sourcekey :
model_component.ModelComponentInfo
-
csm
¶ Return the CatalogSourceManager
-
dmm
¶ Return the DiffuseModelManager
-
static
get_sub_comp_info
(source_info, comp)[source]¶ Build and return information about a sub-component for a particular selection
-
gmm
¶ Return the GalpropMapManager
-
make_fermipy_config_yaml
(modelkey, components, data, **kwargs)[source]¶ Build a fermipy top-level yaml configuration file
Parameters:
-
make_library
(diffuse_yaml, catalog_yaml, binning_yaml)[source]¶ Build up the library of all the components
Parameters:
-
make_model_info
(modelkey)[source]¶ Build a dictionary with the information for a particular model.
Parameters: - modelkey (str) – Key used to identify this particular model
- ModelInfo (Return) –
-
Trivial Link Sub-classes¶
Standalone Analysis Links¶
-
class
fermipy.diffuse.gt_assemble_model.
InitModel
(**kwargs)[source]¶ Bases:
fermipy.jobs.link.Link
Small class to preprate files fermipy analysis.
Specifically this create the srcmap_manifest and fermipy_config_yaml files
-
appname
= 'fermipy-init-model'¶
-
default_options
= {'comp': ('config/binning.yaml', 'Path to yaml file defining binning.', <class 'str'>), 'data': ('config/dataset_sourceveto.yaml', 'Path to yaml file defining dataset.', <class 'str'>), 'hpx_order': (7, 'Maximum HEALPIX order for model fitting.', <class 'int'>), 'library': ('models/library.yaml', 'Path to yaml file defining model components.', <class 'str'>), 'models': ('models/modellist.yaml', 'Path to yaml file defining models.', <class 'str'>)}¶
-
description
= 'Initialize model fitting directory'¶
-
linkname_default
= 'init-model'¶
-
usage
= 'fermipy-init-model [options]'¶
-
-
class
fermipy.diffuse.gt_assemble_model.
AssembleModel
(**kwargs)[source]¶ Bases:
fermipy.jobs.link.Link
Small class to assemple source map files for fermipy analysis.
This is useful for re-merging after parallelizing source map creation.
-
static
append_hdus
(hdulist, srcmap_file, source_names, hpx_order)[source]¶ Append HEALPix maps to a list
Parameters:
-
appname
= 'fermipy-assemble-model'¶
-
static
assemble_component
(compname, compinfo, hpx_order)[source]¶ Assemble the source map file for one binning component
Parameters:
-
static
copy_ccube
(ccube, outsrcmap, hpx_order)[source]¶ Copy a counts cube into outsrcmap file reducing the HEALPix order to hpx_order if needed.
-
default_options
= {'compname': (None, 'Component name.', <class 'str'>), 'hpx_order': (7, 'Maximum HEALPIX order for model fitting.', <class 'int'>), 'input': (None, 'Input yaml file', <class 'str'>)}¶
-
description
= 'Assemble sourcemaps for model fitting'¶
-
linkname_default
= 'assemble-model'¶
-
usage
= 'fermipy-assemble-model [options]'¶
-
static
Batch job dispatch classes¶
-
class
fermipy.diffuse.gt_assemble_model.
AssembleModel_SG
(link, **kwargs)[source]¶ Bases:
fermipy.jobs.scatter_gather.ScatterGather
Small class to generate configurations for this script
Parameters: - --compname (binning component definition yaml file) –
- --data (datset definition yaml file) –
- --models (model definitino yaml file) –
- args (Names of models to assemble source maps for) –
-
appname
= 'fermipy-assemble-model-sg'¶
-
clientclass
¶ alias of
AssembleModel
-
default_options
= {'comp': ('config/binning.yaml', 'Path to yaml file defining binning.', <class 'str'>), 'data': ('config/dataset_sourceveto.yaml', 'Path to yaml file defining dataset.', <class 'str'>), 'hpx_order': (7, 'Maximum HEALPIX order for model fitting.', <class 'int'>), 'models': ('models/modellist.yaml', 'Path to yaml file defining models.', <class 'str'>)}¶
-
description
= 'Copy source maps from the library to a analysis directory'¶
-
job_time
= 300¶
-
usage
= 'fermipy-assemble-model-sg [options]'¶
Analysis chain classes¶
-
class
fermipy.diffuse.gt_assemble_model.
AssembleModelChain
(**kwargs)[source]¶ Bases:
fermipy.jobs.chain.Chain
Small class to split, apply mktime and bin data according to some user-provided specification
-
appname
= 'fermipy-assemble-model-chain'¶
-
default_options
= {'comp': ('config/binning.yaml', 'Path to yaml file defining binning.', <class 'str'>), 'data': ('config/dataset_sourceveto.yaml', 'Path to yaml file defining dataset.', <class 'str'>), 'dry_run': (False, 'Print commands but do not run them', <class 'bool'>), 'hpx_order': (7, 'Maximum HEALPIX order for model fitting.', <class 'int'>), 'library': ('models/library.yaml', 'Path to yaml file defining model components.', <class 'str'>), 'models': ('models/modellist.yaml', 'Path to yaml file defining models.', <class 'str'>)}¶
-
description
= 'Run init-model and assemble-model'¶
-
linkname_default
= 'assemble-model-chain'¶
-
usage
= 'fermipy-assemble-model-chain [options]'¶
-