Reading in Files

Input handlers govern how the observational data is read in. This includes both images and catalogs. The class InputFiles is the default, so if you are using that (which is typical), you can omit the :type: specification.

class piff.Input[source]

The base class for handling inputs for building a Piff model.

This is essentially an abstract base class intended to define the methods that should be implemented by any derived class.

getBandpass(logger=None)[source]

Get the observing bandpass for the input images.

Parameters:

logger – A logger object for logging debug info. [default: None]

Returns:

a galsim.Bandpass or None.

getPointing(logger=None)[source]

Get the pointing coordinate of the (noinal) center of the field of view.

Parameters:

logger – A logger object for logging debug info. [default: None]

Returns:

a galsim.CelestialCoord of the pointing direction.

getWCS(logger=None)[source]

Get the WCS solutions for all the chips in the field of view.

Parameters:

logger – A logger object for logging debug info. [default: None]

Returns:

a dict of WCS solutions (galsim.BaseWCS instances) indexed by chipnum

makeStars(logger=None)[source]

Process the input images and star data, cutting out stamps for each star along with other relevant information.

The base class implementation expects the derived class to have appropriately set the following attributes:

stamp_size:

The size of the postage stamp to use for the cutouts

x_col:

The name of the column in the catalogs to use for the x position.

y_col:

The name of the column in the catalogs to use for the y position.

Parameters:

logger – A logger object for logging debug info. [default: None]

Returns:

a list of Star instances

classmethod process(config, base=None, logger=None)[source]

Parse the input field of the config dict.

Parameters:
  • config – The configuration dict for the input field.

  • base – The base configuration dict.

  • logger – A logger object for logging debug info. [default: None]

Returns:

stars, wcs, pointing, bandpass

stars is a list of Star instances with the initial data. wcs is a dict of WCS solutions indexed by chipnum. pointing is either a galsim.CelestialCoord or None. bandpass is either a galsim.Bandpass or None.

class piff.InputFiles(config, base=None, logger=None)[source]

An Input handler that just takes a list of image files and catalog files.

Parse the input config dict (Normally the ‘input’ field in the overall configuration dict).

The two required fields in the input dict are:

image_file_name:

The file name(s) of the input image(s).

cat_file_name:

The file name(s) of the input catalog(s).

There are a number of ways to specify these file names.

  1. A string giving a single file name. e.g.:

    image_file_name: image.fits
    cat_file_name: input_cat.fits
    
  2. A list of several file names. e.g.:

    image_file_name: [image_00.fits, image_01.fits, image_02.fits]
    cat_file_name: [input_cat_00.fits, input_cat_01.fits, input_cat_02.fits]
    
  3. A string that glob can recognize to list several file names. e.g.:

    image_file_name: image_*.fits
    cat_file_name: input_cat_*.fits
    
  4. A dict parseable as a string value according to the GalSim configuration parsing types. In this case, you also must specify nimages to say how many file names to generate in this way. e.g.:

        nimages: 20
        image_file_name:
            type: FormattedStr
            format: image_%03d_%02d.fits.fz
            items:
                - { type : Sequence, first: 0, repeat: 4 }  # Exposure number
                - { type : Sequence, first: 1, last: 4 }    # Chip number
        cat_file_name:
            type: Eval
            str: "image_file_name.replace('image','input_cat')"
            simage_file_name: '@input.image_file_name'
    
    See the description of the GalSim config parser for more details about the various
    types that are valid here.
    
        `https://github.com/GalSim-developers/GalSim/wiki/Config-Values`_
    

There are many other optional parameters, which help govern how the input files are read or interpreted:

chipnum:

The id number of this chip used to reference this image [default: image_num]

image_hdu:

The hdu to use in the image files. [default: None, which means use either 0 or 1 as typical given the compression sceme of the file]

weight_hdu:

The hdu to use for weight images. [default: None, which means a weight image with all 1’s will be automatically created]

badpix_hdu:

The hdu to use for badpix images. Pixels with badpix != 0 will be given weight == 0. [default: None]

noise:

Rather than a weight image, provide the noise variance in the image. (Useful for simulations where this is a known value.) [default: None]

weight_file_name:

If the weight image is in a different file than the main image, this gives the name of the file to use. [default: None]

badpix_file_name:

If the badpix image is in a different file than the main image, this gives the name of the file to use. [default: None]

badpix_zeros:

Whether to treat all zeros in the input image as bad pixels. [default: False]

cat_hdu:

The hdu to use in the catalog files. [default: 1]

x_col:

The name of the X column in the input catalogs. [default: ‘x’]

y_col:

The name of the Y column in the input catalogs. [default: ‘y’]

ra_col:

(Alternative to x_col, y_col) The name of a right ascension column in the input catalogs. Will use the WCS to find (x,y) [default: None]

dec_col:

(Alternative to x_col, y_col) The name of a declination column in the input catalogs. Will use the WCS to find (x,y) [default: None]

flag_col:

The name of a flag column in the input catalogs. [default: None] By default, this will skip any objects with flag != 0, but see skip_flag and use_flag for other possible meanings for how the flag column can be used to select stars.

skip_flag:

The flag indicating which items to not use. [default: -1] Items with flag & skip_flag != 0 will be skipped.

use_flag:

The flag indicating which items to use. [default: None] Items with flag & use_flag == 0 will be skipped.

property_cols:

A list of column names of star properties (e.g. star colors). [default: None]

properties:

A dict of constant star properties to attach to every star from each input image. The keys are property names. Values are parsed using the GalSim config parser for each image_num. Plain strings that begin with $ or @ are treated as config expressions; other strings are left as literal strings. The explicit type/value dict form is also supported. [default: None]

sed_col:

The name of a column with per-star SED selectors. If sed_file_name is not provided, selectors are interpreted as SED file names. If sed_file_name is provided, selectors are treated as keys in the SED mapping. [default: None]

sed_file_name:

A single SED file name to use for all stars (if sed_col is not provided), or a dict / YAML file mapping sed_col selectors to SED file names. [default: None]

sed_wave_type:

Wavelength unit for SED files (e.g. nm, Angstrom). [required if sed_col or sed_file_name is not None]

sed_flux_type:

Flux convention for SED files (e.g. flambda, fnu, fphotons or units parseable by astropy.Unit). [required if sed_col or sed_file_name is not None]

sed_wave_key:

FITS-table column name to use for wavelength when loading SEDs from FITS tables. [default: WAVE]

sed_flux_key:

FITS-table column name to use for flux when loading SEDs from FITS tables. [default: FLUX]

sed_tol:

Relative tolerance for reducing the number of samples of input SEDs based on the calculated flux over the given bandpass. The SEDs are each reduced on first access, and the reduced SED object is then shared by all matching stars. Values <= 0 disable this step. [default: 0]

sed_max_samples:

Maximum number of samples to keep in the effective SED. If set, use a greedy knot-selection algorithm to choose up to this many samples, then refit their values for the best piecewise-linear approximation of the original effective SED. Values <= 0 disable this step. [default: 0]

bandpass:

GalSim bandpass config dict describing the observing bandpass for the input image(s). Required when using sed_col or sed_file_name. [default: None]

Note

For chromatic PSF fitting, you must define the SED of each star in one of the following ways:

  1. Use a single SED for each every star.

    In this case, the SED should probably be some kind of median SED of the stars in your sample. You won’t be capturing the variation in the chromaticity among the various stars, but at least you will be using the correct average chromatic effect of the stellar population. For this case, give the SED file name as sed_file_name.

    Sample config file excerpt:

    input:
        [...]
        sed_file_name: 'stellar_sed.fits'
        sed_wave_type: 'nm'
        sed_flux_type: 'erg cm**(-2) s**(-1) angstrom**(-1)'
    
  2. Specify a file name for the SED of each star.

    You may instead want to specify a (potentially) different SED file for each star. In this case, you can give that file name in the input catalog as one of the columns. Set sed_col to the name of the catalog column containing the file names.

    Sample config file excerpt:

    input:
        [...]
        sed_col: 'sed_file'
        sed_wave_type: 'nm'
        sed_flux_type: 'erg cm**(-2) s**(-1) angstrom**(-1)'
    
  3. Specify the SED of each star by some label.

    Another way to give each star its own SED is to have the values in the input catalog be some kind of label indicating which file to use. The mapping of these labels to file names would then be in a YAML file, which should parse as a dict giving sed_file_name[label]. In this case, specify sed_col to be the input column with the labels and sed_file_name to be the YAML file with the mapping.

    Sample config file excerpt:

    input:
        [...]
        sed_col: 'sed_label'
        sed_file_name: 'sed_mapping.yaml'
        sed_wave_type: 'nm'
        sed_flux_type: 'erg cm**(-2) s**(-1) angstrom**(-1)'
    

    If you want, you can also put the mapping dict directly in the config file:

    input:
        [...]
        sed_col: 'stellar_type'
        sed_file_name:
            'F': 'xsl_spectrum_X0529_merged.fits'
            'G': 'xsl_spectrum_X0203_merged.fits'
            'K': 'xsl_spectrum_X0066_merged_scl.fits'
            'M': 'xsl_spectrum_X0527_merged.fits'
        sed_wave_type: 'nm'
        sed_flux_type: 'erg cm**(-2) s**(-1) angstrom**(-1)'
    

In each case, sed_wave_type and sed_flux_type are required values, specifying the units to use for the wavelengths and flux values in the SED file. For SED files that are FITS tables, you may also want to set sed_wave_key and sed_flux_key to specify which columns correspond to the wave and flux arrays.

sky_col:

The name of a column with sky values. [default: None]

gain_col:

The name of a column with gain values. [default: None]

sky:

The sky level to subtract from the image values. [default: None] Note: It is an error to specify both sky and sky_col. If both are None, no sky level will be subtracted off.

Note

The special value sky = ‘median’ means to compute the median of the image and use that as the sky level. Any other string value (rather than a float) indicates to get the value from the FITS header.

gain:

The gain to use for adding Poisson noise to the weight map. [default: None] It is an error for both gain and gain_col to be specified. If both are None, then no additional noise will be added to account for the Poisson noise from the galaxy flux.

satur:

The staturation level. If any pixels for a star exceed this, then the star is skipped. [default: None]

trust_pos:

Whether to trust the input position as essentially exact (True), or to let the stars move slightly from their nominal positions (False). [default: False]

use_partial:

Whether to use stars whose postage stamps are only partially on the full image. [default: False]

nstars:

Stop reading each input catalog at this many stars. This legacy option is deprecated; prefer select.nstars, which applies globally after the selection cuts and keeps the highest-S/N stars. [default: None]

nproc:

How many multiprocessing processes to use for reading in data from multiple files at once. [default: 1]

wcs:

Normally, the wcs is automatically read in when reading the image. However, this parameter allows you to optionally provide a different WCS. It should be defined using the same style as a wcs object in GalSim config files. [defulat: None]

Note

Some values that are nominally floats may be given as FITS header key words instead. These include sky, gain, and satur. If these are strings (other than sky=median, which is special), then they should be a valid key word in the FITS header giving the value to use.

The above values are parsed separately for each input image/catalog. In addition, there are a couple other parameters that are just parsed once:

stamp_size:

The size of the postage stamps to use for the cutouts. Note: some stamps may be smaller than this if the star is near a chip boundary. [default: 32]

ra, dec:

The RA, Dec of the telescope pointing. [default: None; See setPointing for details about how this can be specified]

Parameters:
  • config – The configuration dict used to define the above parameters.

  • base – The base configuration dict if needed for value parsing. [default: None]

  • logger – A logger object for logging debug info. [default: None]

getRawImageData(image_num, logger=None)[source]

Read an image and catalog.

Parameters:
  • image_num – The index of the image to read.

  • logger – A logger object for logging debug info. [default: None]

Returns:

image, weight, image_pos, extra_props.

load_images(stars, logger=None)[source]

Load the image data into a list of Stars.

We don’t store the image data for Stars when we write them to a file, since that would take up a lot of space and is usually not desired. However, we do store the bounds in the original image where the star was cutout, so if you want to load back in the original data from the image file(s), you can do so with this function.

Parameters:
  • stars – A list of Star instances.

  • logger – A logger object for logging debug info. [default: None]

Returns:

a new list of Stars with the images information loaded.

static readImage(image_file_name, image_hdu, weight_file_name, weight_hdu, badpix_file_name, badpix_hdu, badpix_zeros, sky_file_name, sky_hdu, image_num, config, base, logger)[source]

Read in the image and weight map (or make one if no weight information is given

Parameters:
  • image_file_name – The name of the file to read.

  • image_hdu – The hdu of the main image.

  • weight_file_name – The name if the file with the weight image (if any).

  • weight_hdu – The hdu of the weight image (if any).

  • badpix_file_name – The name of the file with the bad pixel mask (if any).

  • badpix_hdu – The hdu of the bad pixel mask (if any).

  • badpix_zeros – Whether to treat all zeros in the input image as bad pixels. [default: False]

  • sky_file_name – A file to use for a sky background to subtract from the image (if any).

  • sky_hdu – The hdu to use in the sky_file_name (if any).

  • config – The input section of the config dict.

  • base – The base config dict.

  • logger – A logger object for logging debug info.

Returns:

image, weight

static readStarCatalog(cat_file_name, cat_hdu, x_col, y_col, ra_col, dec_col, ra_units, dec_units, image, flag_col, skip_flag, use_flag, property_cols, sed_col, sed_wave_type, sed_file_name, sed_flux_type, sed_wave_key, sed_flux_key, sed_tol, sed_max_samples, properties, sky_col, gain_col, nstars, stamp_size, config, base, logger)[source]

Read in the star catalogs and return lists of positions for each star in each image.

Parameters:
  • cat_file_name – The name of the catalog file to read in.

  • cat_hdu – The hdu to use.

  • x_col – The name of the column with x values.

  • y_col – The name of the column with y values.

  • ra_col – The name of a column with RA values.

  • dec_col – The name of a column with Dec values.

  • ra_units – The units of the ra column.

  • dec_units – The units of the dec column.

  • image – The image that was already read in (mostly for the wcs).

  • flag_col – The name of a column with flag values.

  • skip_flag – The flag indicating which items to not use. [default: -1] Items with flag & skip_flag != 0 will be skipped.

  • use_flag – The flag indicating which items to use. [default: None] Items with flag & use_flag == 0 will be skipped.

  • property_cols – A list of column names with star properties (e.g. star colors).

  • properties – A dict of additional star properties to attach to every star.

  • sed_col – A column with per-star SED file names.

  • sed_file_name – A single SED file name for all stars (if sed_col is None).

  • sed_wave_type – Wave type for SED files.

  • sed_flux_type – Flux type for SED files.

  • sed_wave_key – FITS-table wavelength column name for SED files.

  • sed_flux_key – FITS-table flux column name for SED files.

  • sed_tol – Relative tolerance for reducing loaded SEDs.

  • sed_max_samples – Maximum number of retained SED samples.

  • sky_col – A column with sky (background) levels.

  • gain_col – A column with gain values.

  • nstars – Optionally a maximum number of stars to use from this catalog.

  • stamp_size – The stamp size being used for the star stamps.

  • config – The input section of the config dict.

  • base – The base config dict.

  • logger – A logger object for logging debug info. [default: None]

Returns:

lists image_pos, extra_props

setPointing(ra, dec, logger=None)[source]

Set the pointing attribute based on the input ra, dec (given in the initializer)

There are a number of ways the pointing can be specified. Even this is probably not sufficiently generic for all applications, but it’s a start.

  1. numerical values (in Hours, Degrees respective) for ra, dec

  2. hh:mm:ss.ssss, dd:mm:ss.ssss strings giving hours/degrees, minutes, seconds for each

  3. FITS header key words to read to get the ra, dec values

  4. None, which will attempt to find the spatial center of all the input images using the midpoint of the min/max ra and dec values of the image corners according to their individual WCS functions. [Not implemented currently.]

  5. dict with GalSim-style config types to be parsed.