panoptes-utils 0.2.42.dev0 documentation

Raises:

Exception – Raised if the config server is not available.

panoptes.utils.config.client.server_is_running(*args, **kwargs)[source]¶: Thin-wrapper to check server.

panoptes.utils.config.client.set_config(key, new_value, host=None, port=None, parse=True)[source]¶

Set config item in config server.

Given a key entry, update the config to match. The key is a dot accessible string, as given by scalpl. See Examples in get_config() for details.

Examples:

>>> from astropy import units as u

>>> # Can use astropy units.
>>> set_config('location.horizon', 35 * u.degree)
{'location.horizon': <Quantity 35. deg>}

>>> get_config(key='location.horizon')
<Quantity 35. deg>

>>> # String equivalent works for 'deg', 'm', 's'.
>>> set_config('location.horizon', '30 deg')
{'location.horizon': <Quantity 30. deg>}

Parameters:

key (str) – The key to update, see Examples in get_config() for details.
new_value (scalar|object) – The new value for the key, can be any serializable object.
host (str, optional) – The config server host. First checks for PANOPTES_CONFIG_HOST env var, defaults to ‘localhost’.
port (str or int, optional) – The config server port. First checks for PANOPTES_CONFIG_HOST env var, defaults to 6563.
parse (bool, optional) – If response should be parsed by panoptes.utils.serializers.from_json(), default True.

Returns:

The updated config entry.

Return type:

Raises:

Exception – Raised if the config server is not available.

panoptes.utils.config.helpers module¶

panoptes.utils.config.helpers.load_config(config_files: Path | List | None = None, parse: bool = True, load_local: bool = True)[source]¶

Load configuration information.

Note

This function is used by the config server and normal config usage should be via a running config server.

This function supports loading of a number of different files. If no options are passed to config_files then the default $PANOPTES_CONFIG_FILE will be loaded.

config_files is a list and loaded in order, so the second entry will overwrite any values specified by similarly named keys in the first entry.

config_files should be specified by an absolute path, which can exist anywhere on the filesystem.

Local versions of files can override built-in versions and are automatically loaded if they exist alongside the specified config path. Local files have a <>_local.yaml name, where <> is the built-in file.

Given the following path:

/path/to/dir
|- my_conf.yaml
|- my_conf_local.yaml

You can do a load_config('/path/to/dir/my_conf.yaml') and both versions of the file will be loaded, with the values in the local file overriding the non-local. Typically the local file would also be ignored by git, etc.

For example, the panoptes.utils.config.server.config_server will always save values to a local version of the file so the default settings can always be recovered if necessary.

Local files can be ignored (mostly for testing purposes or for recovering default values) with the load_local=False parameter.

Parameters:

config_files (list, optional) – A list of files to load as config, see Notes for details of how to specify files.
parse (bool, optional) – If the config file should attempt to create objects such as dates, astropy units, etc.
load_local (bool, optional) – If local files should be used, see Notes for details.

Returns:

A dictionary of config items.

Return type:

panoptes.utils.config.helpers.parse_config_directories(directories: Dict[str, str])[source]¶

Parse the config dictionary for common objects.

Given a base entry that corresponds to the absolute path of a directory, prepend the base to all other relative directory entries.

The base directory must exist or an exception is rasied.

If the base entry is not given the current working directory is used.

>>> dirs_config = dict(base='/tmp', foo='bar', baz='bam', app='/app')
>>> parse_config_directories(dirs_config)
{'base': '/tmp', 'foo': '/tmp/bar', 'baz': '/tmp/bam', 'app': '/app'}

>>> # If base doesn't exist an exception is raised.
>>> dirs_config = dict(base='/panoptes', foo='bar', baz='bam', app='/app')
>>> parse_config_directories(dirs_config)
Traceback (most recent call last):
...
panoptes.utils.error.NotFound: NotFound: Base directory does not exist: /panoptes

Parameters:: directories (dict) – The dictionary of directory information. Usually comes from the “directories” entry in the config.
Returns:: The same directory but with relative directories resolved.
Return type:: dict
Raises:: panoptes.utils.error.NotFound – if the ‘base’ entry is given but does not exist.

panoptes.utils.config.helpers.save_config(save_path: Path, config: dict, overwrite: bool = True)[source]¶

Save config to local yaml file.

Parameters:

save_path (str) – Path to save, can be relative or absolute. See Notes in load_config.
config (dict) – Config to save.
overwrite (bool, optional) – True if file should be updated, False to generate a warning for existing config. Defaults to True for updates.

Returns:

If the save was successful.

Return type:

bool

Raises:

FileExistsError – If the local path already exists and overwrite=False.

panoptes.utils.config.server module¶

panoptes.utils.config.server.config_server(config_file, host=None, port=None, load_local=True, save_local=False, auto_start=True, access_logs=None, error_logs='logger')[source]¶

Start the config server in a separate process.

A convenience function to start the config server.

Parameters:

config_file (str or None) – The absolute path to the config file to load.
host (str, optional) – The config server host. First checks for PANOPTES_CONFIG_HOST env var, defaults to ‘localhost’.
port (str or int, optional) – The config server port. First checks for PANOPTES_CONFIG_HOST env var, defaults to 6563.
load_local (bool, optional) – If local config files should be used when loading, default True.
save_local (bool, optional) – If setting new values should auto-save to local file, default False.
auto_start (bool, optional) – If server process should be started automatically, default True.
access_logs (‘default’ or logger or File-like or None, optional) – Controls access logs for the gevent WSGIServer. The default string will cause access logs to go to stderr. The string logger will use the panoptes logger. A File-like will write to file. The default None will turn off all access logs.
error_logs (‘default’ or ‘logger’ or File-like or None, optional) – Same as access_logs except we use our logger as the default.

Returns:

The process running the config server.

Return type:

multiprocessing.Process

panoptes.utils.config.server.get_config_entry()[source]¶

Get config entries from server.

Endpoint that responds to GET and POST requests and returns configuration item corresponding to provided key or entire configuration. The key entries should be specified in dot-notation, with the names corresponding to the entries stored in the configuration file. See the scalpl documentation for details on the dot-notation.

The endpoint should receive a JSON document with a single key named "key" and a value that corresponds to the desired key within the configuration.

For example, take the following configuration:

{
    'location': {
        'elevation': 3400.0,
        'latitude': 19.55,
        'longitude': 155.12,
    }
}

To get the corresponding value for the elevation, pass a JSON document similar to:

'{"key": "location.elevation"}'

Returns:: The json string for the requested object if object is found in config. Otherwise a json string with status and msg keys will be returned.
Return type:: str

panoptes.utils.config.server.heartbeat()[source]¶

A simple echo service to be used for a heartbeat.

Defaults to looking for the ‘config_server.running’ bool value, although a different key can be specified in the POST.

panoptes.utils.config.server.reset_config()[source]¶

Reset the configuration.

An endpoint that accepts a POST method. The json request object must contain the key reset (with any value).

The method will reset the configuration to the original configuration files that were used, skipping the local (and saved file).

Note

If the server was originally started with a local version of the file, those will be skipped upon reload. This is not ideal but hopefully this method is not used too much.

Returns:: A json string object containing the keys success and msg that indicate success or failure.
Return type:: str

panoptes.utils.config.server.set_config_entry()[source]¶

Sets an item in the config.

Endpoint that responds to GET and POST requests and sets a configuration item corresponding to the provided key.

The key entries should be specified in dot-notation, with the names corresponding to the entries stored in the configuration file. See the scalpl documentation for details on the dot-notation.

The endpoint should receive a JSON document with a single key named "key" and a value that corresponds to the desired key within the configuration.

For example, take the following configuration:

{
    'location': {
        'elevation': 3400.0,
        'latitude': 19.55,
        'longitude': 155.12,
    }
}

To set the corresponding value for the elevation, pass a JSON document similar to:

'{"location.elevation": "1000 m"}'

Returns:: If method is successful, returned json string will be a copy of the set values. On failure, a json string with status and msg keys will be returned.
Return type:: str

Module contents¶

panoptes.utils.database package¶

Submodules¶

panoptes.utils.database.base module¶

class panoptes.utils.database.base.AbstractPanDB(db_name=None, **kwargs)[source]¶

Bases: object

abstract clear_current(type)[source]¶

Clear the current record of a certain type

Parameters:: type (str) – The type of entry in the current collection that should be cleared.

abstract find(collection, obj_id)[source]¶

Find an object by it’s identifier.

Parameters:

collection (str) – Collection to search for object.
obj_id (ObjectID|str) – Record identifier returned earlier by insert or insert_current.

Returns:

Object matching identifier or None.

Return type:

dict|None

abstract get_current(collection)[source]¶

Returns the most current record for the given collection

Parameters:: collection (str) – Name of the collection to get most current from
Returns:: Current object of the collection or None.
Return type:: dict|None

abstract insert(collection, obj)[source]¶

Insert an object into the collection provided.

The obj to be stored in a collection should include the type and date metadata as well as a data key that contains the actual object data. If these keys are not provided then obj will be wrapped in a corresponding object that does contain the metadata.

Parameters:

collection (str) – Name of valid collection within the db.
obj (dict or str) – Object to be inserted.

Returns:

identifier of inserted record in collection.: Returns None if unable to insert into the collection.

Return type:

abstract insert_current(collection, obj, store_permanently=True)[source]¶

Insert an object into both the current collection and the collection provided.

Parameters:

collection (str) – Name of valid collection within the db.
obj (dict or str) – Object to be inserted.
store_permanently (bool) – Whether to also update the collection, defaults to True.

Returns:

identifier of inserted record. If store_permanently is True, will: be the identifier of the object in the collection, otherwise will be the identifier of object in the current collection. These may or may not be the same. Returns None if unable to insert into the collection.

Return type:

class panoptes.utils.database.base.PanDB(db_type='memory', db_name=None, *args, **kwargs)[source]¶

Bases: object

Simple class to load the appropriate DB type based on the config.

We don’t actually create instances of this class, but instead create an instance of the ‘correct’ type of db.

classmethod permanently_erase_database(db_type, db_name, storage_dir=None, really=False, dangerous=False, *args, **kwargs)[source]¶: Permanently delete the contents of the identified database.

panoptes.utils.database.base.create_storage_obj(collection, data, obj_id)[source]¶: Wrap the data in a dict along with the id and a timestamp.

panoptes.utils.database.base.get_db_class(module_name='file')[source]¶

Load the main DB class for the module of the given name.

Note

This is used by the PanDB constructor to determine the correct database type. Normal DB instantiation should be done via the PanDB() class with the desired db_type parameter set. See example in PanDB below.

Parameters:: module_name (str) – Name of module, one of: file (default), ‘memory’.
Returns:: An instance of the db class for the correct database type.
Return type:: panoptes.utils.database.PanDB
Raises:: Exception – If an unsupported database type string is passed.

panoptes.utils.database.file module¶

class panoptes.utils.database.file.PanFileDB(db_name='panoptes', storage_dir='json_store', **kwargs)[source]¶

Bases: AbstractPanDB

Stores collections as files of JSON records.

clear_current(record_type)[source]¶

Clears the current record of the given type.

Parameters:: record_type (str) – The record type, e.g. ‘weather’, ‘environment’, etc.

find(collection, obj_id)[source]¶

Find an object by it’s identifier.

Parameters:

collection (str) – Collection to search for object.
obj_id (ObjectID|str) – Record identifier returned earlier by insert or insert_current.

Returns:

Object matching identifier or None.

Return type:

dict|None

get_current(collection)[source]¶

Returns the most current record for the given collection

Parameters:: collection (str) – Name of the collection to get most current from
Returns:: Current object of the collection or None.
Return type:: dict|None

insert(collection, obj)[source]¶

Insert an object into the collection provided.

The obj to be stored in a collection should include the type and date metadata as well as a data key that contains the actual object data. If these keys are not provided then obj will be wrapped in a corresponding object that does contain the metadata.

Parameters:

collection (str) – Name of valid collection within the db.
obj (dict or str) – Object to be inserted.

Returns:

identifier of inserted record in collection.: Returns None if unable to insert into the collection.

Return type:

insert_current(collection, obj, store_permanently=True)[source]¶

Insert an object into both the current collection and the collection provided.

Parameters:

collection (str) – Name of valid collection within the db.
obj (dict or str) – Object to be inserted.
store_permanently (bool) – Whether to also update the collection, defaults to True.

Returns:

identifier of inserted record. If store_permanently is True, will: be the identifier of the object in the collection, otherwise will be the identifier of object in the current collection. These may or may not be the same. Returns None if unable to insert into the collection.

Return type:

classmethod permanently_erase_database(db_name, storage_dir=None)[source]¶

panoptes.utils.database.memory module¶

class panoptes.utils.database.memory.PanMemoryDB(**kwargs)[source]¶

Bases: AbstractPanDB

In-memory store of serialized objects.

We serialize the objects in order to test the same code path used when storing in an external database.

active_dbs = <WeakValueDictionary>¶

clear_current(entry_type)[source]¶

Clear the current record of a certain type

Parameters:: type (str) – The type of entry in the current collection that should be cleared.

find(collection, obj_id)[source]¶

Find an object by it’s identifier.

Parameters:

collection (str) – Collection to search for object.
obj_id (ObjectID|str) – Record identifier returned earlier by insert or insert_current.

Returns:

Object matching identifier or None.

Return type:

dict|None

get_current(collection)[source]¶

Returns the most current record for the given collection

Parameters:: collection (str) – Name of the collection to get most current from
Returns:: Current object of the collection or None.
Return type:: dict|None

classmethod get_or_create(db_name=None, **kwargs)[source]¶

Returns the named db, creating if needed.

This method exists because PanDB gets called multiple times for the same database name. With mongo or a file store where the storage is external from the instance, that is not a problem, but with PanMemoryDB the instance is the store, so the instance must be shared.

insert(collection, obj)[source]¶

Insert an object into the collection provided.

The obj to be stored in a collection should include the type and date metadata as well as a data key that contains the actual object data. If these keys are not provided then obj will be wrapped in a corresponding object that does contain the metadata.

Parameters:

collection (str) – Name of valid collection within the db.
obj (dict or str) – Object to be inserted.

Returns:

identifier of inserted record in collection.: Returns None if unable to insert into the collection.

Return type:

insert_current(collection, obj, store_permanently=True)[source]¶

Insert an object into both the current collection and the collection provided.

Parameters:

collection (str) – Name of valid collection within the db.
obj (dict or str) – Object to be inserted.
store_permanently (bool) – Whether to also update the collection, defaults to True.

Returns:

identifier of inserted record. If store_permanently is True, will: be the identifier of the object in the collection, otherwise will be the identifier of object in the current collection. These may or may not be the same. Returns None if unable to insert into the collection.

Return type:

classmethod permanently_erase_database(*args, **kwargs)[source]¶

Module contents¶

panoptes.utils.images package¶

Submodules¶

panoptes.utils.images.bayer module¶

class panoptes.utils.images.bayer.RGB(value)[source]¶

Bases: IntEnum

Helper class for array index access.

B = 2¶

BLUE = 2¶

G = 1¶

G1 = 1¶

GREEN = 1¶

R = 0¶

RED = 0¶

panoptes.utils.images.bayer.get_pixel_color(x, y)[source]¶

Given a zero-indexed x,y position, return the corresponding color.

Note

See get_rgb_data() for a description of the RGGB pattern.

Returns:: one of ‘R’, ‘G1’, ‘G2’, ‘B’
Return type:: str

panoptes.utils.images.bayer.get_rgb_background(data, box_size=(79, 84), filter_size=(11, 11), estimator='mmm', interpolator='zoom', sigma=5, iters=10, exclude_percentile=100, return_separate=False, *args, **kwargs)[source]¶

Get the background for each color channel.

Note: This funtion does not perform any additional calibration, such as flat, bias, or dark correction. It is expected you have performed any necessary pre-processing to data before passing to this function.

By default this uses a box size of (79, 84), which gives an integer number of boxes. The size of the median filter box for the low resolution background is on the order of the stamp size.

Most of the options are described in the photutils.Background2D page: https://photutils.readthedocs.io/en/stable/background.html#d-background-and-noise-estimation

>>> from panoptes.utils.images.bayer import RGB
>>> from panoptes.utils.images import fits as fits_utils
>>> # Get our data and pre-process (basic bias subtract here).
>>> fits_fn = getfixture('solved_fits_file')
>>> camera_bias = 2048
>>> data = fits_utils.getdata(fits_fn) - camera_bias

>> The default is to return a single array for the background. >>> rgb_back = get_rgb_background(data) >>> rgb_back.mean() 136… >>> rgb_back.std() 36…

>>> # Can also return the Background2D objects, which is the input to save_rgb_bg_fits
>>> rgb_backs = get_rgb_background(data, return_separate=True)
>>> rgb_backs[RGB.RED]
<photutils.background.background_2d.Background2D...>

>>> {color.name:int(rgb_back[color].mean()) for color in RGB}
{'RED': 145, 'GREEN': 127, 'BLUE': 145}

Parameters:

data (np.array) – The data to use if no fits_fn is provided.
box_size (tuple, optional) – The box size over which to compute the 2D-Background, default (79, 84).
filter_size (tuple, optional) – The filter size for determining the median, default (11, 12).
estimator (str, optional) – The estimator object to use, default ‘mmm’.
interpolator (str, optional) – The interpolater object to user, default ‘zoom’.
sigma (int, optional) – The sigma on which to filter values, default 5.
iters (int, optional) – The number of iterations to sigma filter, default 10.
exclude_percentile (int, optional) – The percentage of the data (per channel) that can be masked, default 100 (i.e. all).
return_separate (bool, optional) – If the function should return a separate array for color channel, default False.
*args – Description
**kwargs – Description

Returns:

Either a single numpy array representing the entire: background, or a list of masked numpy arrays in RGB order. The background for each channel has full interploation across all pixels, but the mask covers them.

Return type:

`numpy.array`|list(Background2D)

panoptes.utils.images.bayer.get_rgb_data(data, separate_green=False)[source]¶

Get the data split into separate channels for RGB.

data can be a 2D (W x H) or 3D (N x W x H) array where W=width and H=height of the data, with N=number of frames.

The return array will be a 3 x W x H or 3 x N x W x H array.

The Bayer array defines a superpixel as a collection of 4 pixels set in a square grid:

R G
G B

ds9 and other image viewers define the coordinate axis from the lower left corner of the image, which is how a traditional x-y plane is defined and how most images would expect to look when viewed. This means that the (0, 0) coordinate position will be in the lower left corner of the image.

When the data is loaded into a numpy array the data is flipped on the vertical axis in order to maintain the same indexing/slicing features. This means the the (0, 0) coordinate position is in the upper-left corner of the array when output. When plotting this array one can use the origin='lower' option to view the array as would be expected in a normal image although this does not change the actual index.

Image dimensions:

----------------------------
x | width  | i | columns |  5208
y | height | j | rows    |  3476

Bayer pattern as seen in ds9:

                             x / j

             0     1    2     3 ... 5204 5205 5206 5207
           --------------------------------------------
      3475 |  R   G1    R    G1        R   G1    R   G1
      3474 | G2    B   G2     B       G2    B   G2    B
      3473 |  R   G1    R    G1        R   G1    R   G1
      3472 | G2    B   G2     B       G2    B   G2    B
         . |
y / i    . |
         . |
         3 |  R   G1    R    G1        R   G1    R   G1
         2 | G2    B   G2     B       G2    B   G2    B
         1 |  R   G1    R    G1        R   G1    R   G1
         0 | G2    B   G2     B       G2    B   G2    B

The RGGB super-pixels thus start in the upper-left.

Bayer pattern as seen in a numpy array:

                             x / j

             0     1    2     3 ... 5204 5205 5206 5207
           --------------------------------------------
         0 | G2    B   G2     B       G2    B   G2    B
         1 |  R   G1    R    G1        R   G1    R   G1
         2 | G2    B   G2     B       G2    B   G2    B
         3 |  R   G1    R    G1        R   G1    R   G1
         . |
y / i    . |
         . |
      3472 | G2    B   G2     B       G2    B   G2    B
      3473 |  R   G1    R    G1        R   G1    R   G1
      3474 | G2    B   G2     B       G2    B   G2    B
      3475 |  R   G1    R    G1        R   G1    R   G1

Here the RGGB super-pixels are flipped upside down.

In both cases the data is in the following format:

    | row (y) |  col (x)
--------------| ------
 R  |  odd i, |  even j
 G1 |  odd i, |   odd j
 G2 | even i, |  even j
 B  | even i, |   odd j

And a mask can therefore be generated as:

bayer[1::2, 0::2] = 1 # Red
bayer[1::2, 1::2] = 1 # Green
bayer[0::2, 0::2] = 1 # Green
bayer[0::2, 1::2] = 1 # Blue

panoptes.utils.images.bayer.get_rgb_masks(data, separate_green=False)[source]¶

Get the RGGB Bayer pattern for the given data.

Note

See get_rgb_data() for a description of the RGGB pattern.

Parameters:

data (np.array) – An array of data representing an image.
separate_green (bool, optional) – If the two green channels should be separated, default False.

Returns:

A 3-tuple of numpy arrays of bool type.

Return type:

tuple(np.array, np.array, np.array)

panoptes.utils.images.bayer.get_stamp_slice(x, y, stamp_size=(14, 14), ignore_superpixel=False, as_slices=True)[source]¶

Get the slice around a given position with fixed Bayer pattern.

Given an x,y pixel position, get the slice object for a stamp of a given size but make sure the first position corresponds to a red-pixel. This means that x,y will not necessarily be at the center of the resulting stamp.

>>> from panoptes.utils.images import bayer
>>> # Make a super-pixel as represented in numpy (see full stamp below).
>>> superpixel = np.array(['G2', 'B', 'R', 'G1']).reshape(2, 2)
>>> superpixel
array([['G2', 'B'],
       ['R', 'G1']], dtype='<U2')
>>> # Tile it into a 5x5 grid of super-pixels, i.e. a 10x10 stamp.
>>> stamp0 = np.tile(superpixel, (5, 5))
>>> stamp0
array([['G2', 'B', 'G2', 'B', 'G2', 'B', 'G2', 'B', 'G2', 'B'],
       ['R', 'G1', 'R', 'G1', 'R', 'G1', 'R', 'G1', 'R', 'G1'],
       ['G2', 'B', 'G2', 'B', 'G2', 'B', 'G2', 'B', 'G2', 'B'],
       ['R', 'G1', 'R', 'G1', 'R', 'G1', 'R', 'G1', 'R', 'G1'],
       ['G2', 'B', 'G2', 'B', 'G2', 'B', 'G2', 'B', 'G2', 'B'],
       ['R', 'G1', 'R', 'G1', 'R', 'G1', 'R', 'G1', 'R', 'G1'],
       ['G2', 'B', 'G2', 'B', 'G2', 'B', 'G2', 'B', 'G2', 'B'],
       ['R', 'G1', 'R', 'G1', 'R', 'G1', 'R', 'G1', 'R', 'G1'],
       ['G2', 'B', 'G2', 'B', 'G2', 'B', 'G2', 'B', 'G2', 'B'],
       ['R', 'G1', 'R', 'G1', 'R', 'G1', 'R', 'G1', 'R', 'G1']],
      dtype='<U2')
>>> stamp1 = np.arange(100).reshape(10, 10)
>>> stamp1
array([[ 0,  1,  2,  3,  4,  5,  6,  7,  8,  9],
       [10, 11, 12, 13, 14, 15, 16, 17, 18, 19],
       [20, 21, 22, 23, 24, 25, 26, 27, 28, 29],
       [30, 31, 32, 33, 34, 35, 36, 37, 38, 39],
       [40, 41, 42, 43, 44, 45, 46, 47, 48, 49],
       [50, 51, 52, 53, 54, 55, 56, 57, 58, 59],
       [60, 61, 62, 63, 64, 65, 66, 67, 68, 69],
       [70, 71, 72, 73, 74, 75, 76, 77, 78, 79],
       [80, 81, 82, 83, 84, 85, 86, 87, 88, 89],
       [90, 91, 92, 93, 94, 95, 96, 97, 98, 99]])
>>> x = 7
>>> y = 5
>>> pixel_index = (y, x)  # y=rows, x=columns
>>> stamp0[pixel_index]
'G1'
>>> stamp1[pixel_index]
57
>>> slice0 = bayer.get_stamp_slice(x, y, stamp_size=(6, 6))
>>> slice0
(slice(2, 8, None), slice(4, 10, None))
>>> stamp0[slice0]
array([['G2', 'B', 'G2', 'B', 'G2', 'B'],
       ['R', 'G1', 'R', 'G1', 'R', 'G1'],
       ['G2', 'B', 'G2', 'B', 'G2', 'B'],
       ['R', 'G1', 'R', 'G1', 'R', 'G1'],
       ['G2', 'B', 'G2', 'B', 'G2', 'B'],
       ['R', 'G1', 'R', 'G1', 'R', 'G1']], dtype='<U2')
>>> stamp1[slice0]
array([[24, 25, 26, 27, 28, 29],
       [34, 35, 36, 37, 38, 39],
       [44, 45, 46, 47, 48, 49],
       [54, 55, 56, 57, 58, 59],
       [64, 65, 66, 67, 68, 69],
       [74, 75, 76, 77, 78, 79]])
>>> # Return y_min, y_max, x_min, x_max
>>> bayer.get_stamp_slice(x, y, stamp_size=(6, 6), as_slices=False)
(2, 8, 4, 10)

The original index had a value of 57, which is within the center superpixel.

Notice that the resulting stamp has a super-pixel in the center and is bordered on all sides by a complete superpixel. This is required by default and an invalid size

We can use ignore_superpixel=True to get an odd-sized stamp.

>>> slice1 = bayer.get_stamp_slice(x, y, stamp_size=(5, 5), ignore_superpixel=True)
>>> slice1
(slice(3, 8, None), slice(5, 10, None))
>>> stamp0[slice1]
array([['G1', 'R', 'G1', 'R', 'G1'],
       ['B', 'G2', 'B', 'G2', 'B'],
       ['G1', 'R', 'G1', 'R', 'G1'],
       ['B', 'G2', 'B', 'G2', 'B'],
       ['G1', 'R', 'G1', 'R', 'G1']], dtype='<U2')
>>> stamp1[slice1]
array([[35, 36, 37, 38, 39],
       [45, 46, 47, 48, 49],
       [55, 56, 57, 58, 59],
       [65, 66, 67, 68, 69],
       [75, 76, 77, 78, 79]])

This puts the requested pixel in the center but does not offer any guarantees about the RGGB pattern.

Parameters:

x (float) – X pixel position.
y (float) – Y pixel position.
stamp_size (tuple, optional) – The size of the cutout, default (14, 14).
ignore_superpixel (bool) – If superpixels should be ignored, default False.
as_slices (bool) – Return slice objects, default True. Otherwise returns: y_min, y_max, x_min, x_max

Returns:

A list of row and: column slice objects or a list defining the bounding box: y_min, y_max, x_min, x_max. Return type depends on the as_slices parameter and defaults to a list of two slices.

Return type:

list(slice, slice) or list(int, int, int, int)

panoptes.utils.images.bayer.save_rgb_bg_fits(rgb_bg_data, output_filename, header=None, fpack=True, overwrite=True)[source]¶

Save a FITS file containing a combined background as well as separate channels.

Parameters:

rgb_bg_data (list[photutils.Background2D]) – The RGB background data as returned by calling panoptes.utils.images.bayer.get_rgb_background with return_separate=True.
output_filename (str) – The output name for the FITS file.
header (astropy.io.fits.Header) – FITS header to be saved with the file.
fpack (bool) – If the FITS file should be compressed, default True.
overwrite (bool) – If FITS file should be overwritten, default True.

panoptes.utils.images.cr2 module¶

Convert a CR2 file to FITS.

This is a convenience function that first converts the CR2 to PGM via ~cr2_to_pgm. Also adds keyword headers to the FITS file.

Note

The intermediate PGM file is automatically removed

Parameters:

cr2_fname (str) – Name of the CR2 file to be converted.
fits_fname (str, optional) – Name of the FITS file to output. Default is None, in which case the cr2_fname is used as the base.
overwrite (bool, optional) – Overwrite existing FITS, default False.
headers (dict, optional) – Header data added to the FITS file.
fits_headers (dict, optional) – Header data added to the FITS file without filtering.
remove_cr2 (bool, optional) – If CR2 should be removed after processing, default False.
**kwargs – Description

Returns:

The full path to the generated FITS file.

Return type:

panoptes.utils.images.cr2.cr2_to_jpg(cr2_fname: Path, jpg_fname: str | None = None, title: str = '', overwrite: bool = False, remove_cr2: bool = False) → Path | None[source]¶: Extract a JPG image from a CR2, return the new path name.

panoptes.utils.images.cr2.cr2_to_pgm(cr2_fname, pgm_fname=None, overwrite=True, *args, **kwargs)[source]¶

Convert CR2 file to PGM

Converts a raw Canon CR2 file to a netpbm PGM file via dcraw. Assumes dcraw is installed on the system

Note

This is a blocking call

Parameters:

convert (cr2_fname {str} -- Name of CR2 file to) –
script (**kwargs {dict} -- Additional keywords to pass to) –

Keyword Arguments:

None (pgm_fname {str} -- Name of PGM file to output, if) – use same name as CR2 (default: {None})
(default (dcraw {str} -- Path to installed dcraw) – {‘dcraw’})
overwritten (overwrite {bool} -- A bool indicating if existing PGM should be) – (default: {True})

Returns:

str – Filename of PGM that was created

panoptes.utils.images.cr2.read_exif(fname, exiftool='exiftool')[source]¶

Read the EXIF information

Gets the EXIF information using exiftool

Note

Assumes the exiftool is installed

Parameters:: file (fname {str} -- Name of) –
Keyword Arguments:: (default (exiftool {str} -- Location of exiftool) – {‘/usr/bin/exiftool’})
Returns:: dict – Dictionary of EXIF information

panoptes.utils.images.cr2.read_pgm(fname, byteorder='>', remove_after=False)[source]¶

Return image data from a raw PGM file as numpy array.

Note

Format Spec: http://netpbm.sourceforge.net/doc/pgm.html Source: http://stackoverflow.com/questions/7368739/numpy-and-16-bit-pgm

Note

This is correctly processed as a Big endian even though the CR2 itself marks it as a Little endian. See the notes in Source page above as well as the comment about significant bit in the Format Spec

Parameters:

fname (str) – Filename of PGM to be converted
byteorder (str) – Big endian
remove_after (bool) – Delete fname file after reading, defaults to False.
overwrite (bool) – overwrite existing PGM or not, defaults to True

Returns:

The raw data from the PGMx

Return type:

numpy.array

panoptes.utils.images.fits module¶

Bases: object

Parse the location path for an image.

This is a small dataclass that offers some convenience methods for dealing with a path based on the image id.

This would usually be instantiated via path:

>>> from panoptes.utils.images.fits import ObservationPathInfo  # noqa
>>> bucket_path = 'gs://panoptes-images-background/PAN012/Hd189733/358d0f/20180824T035917/20180824T040118.fits'
>>> path_info = ObservationPathInfo(path=bucket_path)

>>> path_info.id
'PAN012_358d0f_20180824T035917_20180824T040118'

>>> path_info.unit_id
'PAN012'

>>> path_info.sequence_id
'PAN012_358d0f_20180824T035917'

>>> path_info.image_id
'PAN012_358d0f_20180824T040118'

>>> path_info.as_path(base='/tmp', ext='jpg')
PosixPath('/tmp/PAN012/358d0f/20180824T035917/20180824T040118.jpg')

>>> ObservationPathInfo(path='foobar')
Traceback (most recent call last):
  ...
ValueError: Invalid path received: self.path='foobar'

>>> # Works from a fits file directly, which reads header.
>>> fits_fn = getfixture('unsolved_fits_file')
>>> path_info = ObservationPathInfo.from_fits(fits_fn)
>>> path_info.unit_id
'PAN001'

as_path(base: str | Path | None = None, ext: str | None = None) → Path[source]¶: Return a Path object.

camera_id: str = None¶

field_name: str = None¶

classmethod from_fits(fits_file)[source]¶

classmethod from_fits_header(header)[source]¶

get_full_id(sep='_') → str[source]¶: Returns the full path id with the given separator.

property id¶: Full path info joined with underscores

property image_id: str¶: The matched image id.

image_time: str | datetime | Time = None¶

path: str | Path = None¶

property sequence_id: str¶: The sequence id.

sequence_time: str | datetime | Time = None¶

unit_id: str = None¶

panoptes.utils.images.fits.extract_metadata(header: Header) → dict[source]¶

Get the metadata from a FITS image.

This function parses some of the more common headers (some from the update_observation_headers but others as well) and puts them into a dict with the obvious data types converted into objects (e.g. dates and times).

>>> # Check the headers
>>> from panoptes.utils.images import fits as fits_utils
>>> fits_fn = getfixture('unsolved_fits_file')
>>> header = fits_utils.getheader(fits_fn)
>>> metadata = extract_metadata(header)
>>> metadata['unit']['name']
'PAN001'

Parameters:: header (astropy.io.fits.Header) – The Header object from a FITS file.

panoptes.utils.images.fits.fits_to_jpg(fname=None, title=None, figsize=(10, 7.547169811320755), dpi=150, alpha=0.2, number_ticks=7, clip_percent=99.9, **kwargs)[source]¶

panoptes.utils.images.fits.fpack(fits_fname, unpack=False, overwrite=True)[source]¶

Compress/Decompress a FITS file

Uses fpack (or funpack if unpack=True) to compress a FITS file

Parameters:

fits_fname ({str}) – Name of a FITS file that contains a WCS.
unpack ({bool}, optional) – file should decompressed instead of compressed, default False.

Returns:

Filename of compressed/decompressed file.

Return type:

panoptes.utils.images.fits.funpack(*args, **kwargs)[source]¶

Unpack a FITS file.

Note

This is a thin-wrapper around the ~fpack function with the unpack=True option specified. See ~fpack documentation for details.

Parameters:

*args – Arguments passed to ~fpack.
**kwargs – Keyword arguments passed to ~fpack.

Returns:

Path to uncompressed FITS file.

Return type:

panoptes.utils.images.fits.get_solve_field(fname: str | Path, replace: bool = True, overwrite: bool = True, timeout: float = 30, **kwargs) → Dict[source]¶

Convenience function to wait for solve_field to finish.

This function merely passes the fname of the image to be solved along to solve_field, which returns a subprocess.Popen object. This function then waits for that command to complete, populates a dictonary with the EXIF informaiton and returns. This is often more useful than the raw solve_field function.

Example:

>>> from panoptes.utils.images import fits as fits_utils

>>> # Get our fits filename.
>>> fits_fn = getfixture('unsolved_fits_file')

>>> # Perform the solve.
>>> solve_info = fits_utils.get_solve_field(fits_fn)  

>>> # Show solved filename.
>>> solve_info['solved_fits_file']  
'.../unsolved.fits'

>>> # Pass a suggested location.
>>> ra = 15.23
>>> dec = 90
>>> radius = 5 # deg
>>> solve_info = fits_utils.solve_field(fits_fn, ra=ra, dec=dec, radius=radius)  

>>> # Pass kwargs to `solve-field` program.
>>> solve_kwargs = {'--pnm': '/tmp/awesome.bmp'}
>>> solve_info = fits_utils.get_solve_field(fits_fn, skip_solved=False, **solve_kwargs) 
>>> assert os.path.exists('/tmp/awesome.bmp') 

Parameters:

fname ({str}) – Name of FITS file to be solved.
replace (bool, optional) – Saves the WCS back to the original file, otherwise output base filename with .new extension. Default True.
overwrite (bool, optional) – Clobber file, default True. Required if replace=True.
timeout (int, optional) – The timeout for solving, default 30 seconds.
**kwargs ({dict}) – Options to pass to solve_field should start with –.

Returns:

Keyword information from the solved field.

Return type:

panoptes.utils.images.fits.get_wcsinfo(fits_fname, **kwargs)[source]¶

Returns the WCS information for a FITS file.

Uses the wcsinfo astrometry.net utility script to get the WCS information from a plate-solved file.

Parameters:

fits_fname ({str}) – Name of a FITS file that contains a WCS.
**kwargs – Args that can be passed to wcsinfo.

Returns:

Output as returned from wcsinfo.

Return type:

Raises:

error.InvalidCommand – Raised if wcsinfo is not found (part of astrometry.net)

panoptes.utils.images.fits.getdata(fn, *args, **kwargs)[source]¶

Get the FITS data.

Small wrapper around astropy.io.fits.getdata to auto-determine the FITS extension. This will return the data associated with the image.

>>> fits_fn = getfixture('solved_fits_file')
>>> d0 = getdata(fits_fn)
>>> d0
array([[2215, 2169, 2200, ..., 2169, 2235, 2168],
       [2123, 2191, 2133, ..., 2212, 2127, 2217],
       [2208, 2154, 2209, ..., 2159, 2233, 2187],
       ...,
       [2120, 2201, 2120, ..., 2209, 2126, 2195],
       [2219, 2151, 2199, ..., 2173, 2214, 2166],
       [2114, 2194, 2122, ..., 2202, 2125, 2204]], dtype=uint16)
>>> d1, h1 = getdata(fits_fn, header=True)
>>> (d0 == d1).all()
True
>>> h1['FIELD']
'KIC 8462852'

Parameters:

fn (str) – Path to FITS file.
*args – Passed to astropy.io.fits.getdata.
**kwargs – Passed to astropy.io.fits.getdata.

Returns:

The FITS data.

Return type:

np.ndarray

panoptes.utils.images.fits.getheader(fn, *args, **kwargs)[source]¶

Get the FITS header.

Small wrapper around astropy.io.fits.getheader to auto-determine the FITS extension. This will return the header associated with the image. If you need the compression header information use the astropy module directly.

>>> fits_fn = getfixture('tiny_fits_file')
>>> os.path.basename(fits_fn)
'tiny.fits'
>>> header = getheader(fits_fn)
>>> header['IMAGEID']
'PAN001_XXXXXX_20160909T081152'

>>> # Works with fpacked files
>>> fits_fn = getfixture('solved_fits_file')
>>> os.path.basename(fits_fn)
'solved.fits.fz'
>>> header = getheader(fits_fn)
>>> header['IMAGEID']
'PAN001_XXXXXX_20160909T081152'

Parameters:

fn (str) – Path to FITS file.
*args – Passed to astropy.io.fits.getheader.
**kwargs – Passed to astropy.io.fits.getheader.

Returns:

The FITS header for the data.

Return type:

astropy.io.fits.header.Header

panoptes.utils.images.fits.getval(fn, *args, **kwargs)[source]¶

Get a value from the FITS header.

Small wrapper around astropy.io.fits.getval to auto-determine the FITS extension. This will return the value from the header associated with the image (not the compression header). If you need the compression header information use the astropy module directly.

>>> fits_fn = getfixture('tiny_fits_file')
>>> getval(fits_fn, 'IMAGEID')
'PAN001_XXXXXX_20160909T081152'

Parameters:: fn (str) – Path to FITS file.
Returns:: Value from header (with no type conversion).
Return type:: str or float

panoptes.utils.images.fits.getwcs(fn, *args, **kwargs)[source]¶

Get the WCS for the FITS file.

Small wrapper around astropy.wcs.WCS.

>>> from panoptes.utils.images import fits as fits_utils
>>> fits_fn = getfixture('solved_fits_file')
>>> wcs = fits_utils.getwcs(fits_fn)
>>> wcs.is_celestial
True
>>> fits_fn = getfixture('unsolved_fits_file')
>>> wcs = fits_utils.getwcs(fits_fn)
>>> wcs.is_celestial
False

Parameters:

fn (str) – Path to FITS file.
*args – Passed to astropy.io.fits.getheader.
**kwargs – Passed to astropy.io.fits.getheader.

Returns:

The World Coordinate System information.

Return type:

astropy.wcs.WCS

panoptes.utils.images.fits.solve_field(fname, timeout=15, solve_opts=None, *args, **kwargs)[source]¶

Plate solves an image.

Note: This is a low-level wrapper around the underlying solve-field: program. See get_solve_field for more typical usage and examples.

Parameters:

fname (str, required) – Filename to solve in .fits extension.
timeout (int, optional) – Timeout for the solve-field command, defaults to 60 seconds.
solve_opts (list, optional) – List of options for solve-field.

panoptes.utils.images.fits.update_observation_headers(file_path, info)[source]¶

Update FITS headers with items from the Observation status.

>>> # Check the headers
>>> from panoptes.utils.images import fits as fits_utils
>>> fits_fn = getfixture('unsolved_fits_file')
>>> # Show original value
>>> fits_utils.getval(fits_fn, 'FIELD')
'KIC 8462852'

>>> info = {'field_name': 'Tabbys Star'}
>>> update_observation_headers(fits_fn, info)
>>> # Show new value
>>> fits_utils.getval(fits_fn, 'FIELD')
'Tabbys Star'

Parameters:

file_path (str) – Path to a FITS file.
info (dict) – The return dict from pocs.observatory.Observation.status, which includes basic information about the observation.

panoptes.utils.images.fits.write_fits(data, header, filename, exposure_event=None, **kwargs)[source]¶

Write FITS file to requested location.

>>> from panoptes.utils.images import fits as fits_utils
>>> data = np.random.normal(size=100)
>>> header = { 'FILE': 'delete_me', 'TEST': True }
>>> filename = str(getfixture('tmpdir').join('temp.fits'))
>>> fits_utils.write_fits(data, header, filename)
>>> assert os.path.exists(filename)

>>> fits_utils.getval(filename, 'FILE')
'delete_me'
>>> data2 = fits_utils.getdata(filename)
>>> assert np.array_equal(data, data2)

Parameters:

data (array_like) – The data to be written.
header (dict) – Dictionary of items to be saved in header.
filename (str) – Path to filename for output.
exposure_event (None|`threading.Event`, optional) – A threading.Event that can be triggered when the image is written.
kwargs (dict) – Options that are passed to the astropy.io.fits.PrimaryHDU.writeto method.

panoptes.utils.images.focus module¶

panoptes.utils.images.focus.focus_metric(data, merit_function='vollath_F4', **kwargs)[source]¶

Compute the focus metric.

Computes a focus metric on the given data using a supplied merit function. The merit function can be passed either as the name of the function (must be defined in this module) or as a callable object. Additional keyword arguments for the merit function can be passed as keyword arguments to this function.

Parameters:

data (numpy array) –
merit_function (str/callable) – panoptes.utils.images) or a callable object.

Returns:

result of calling merit function on data

Return type:

scalar

panoptes.utils.images.focus.vollath_F4(data, axis=None)[source]¶

Compute F4 focus metric

Computes the F_4 focus metric as defined by Vollath (1998) for the given 2D numpy array. The metric can be computed in the y axis, x axis, or the mean of the two (default).

Parameters:

data (numpy array) –
axis (str, optional, default None) – be ‘Y’/’y’, ‘X’/’x’ or None, which will calculate the F4 value for both axes and return the mean.

Returns:

Calculated F4 value for y, x axis or both

Return type:

float64

panoptes.utils.images.misc module¶

panoptes.utils.images.misc.crop_data(data, box_width=200, center=None, data_only=True, wcs=None, **kwargs)[source]¶

Return a cropped portion of the image.

Shape is a box centered around the middle of the data

>>> from matplotlib import pyplot as plt
>>> from astropy.wcs import WCS
>>> from panoptes.utils.images.misc import crop_data
>>> from panoptes.utils.images.plot import add_colorbar, get_palette
>>> from panoptes.utils.images.fits import getdata
>>>
>>> fits_url = 'https://github.com/panoptes/panoptes-utils/raw/develop/tests/data/solved.fits.fz'
>>> data, header = getdata(fits_url, header=True)
>>> wcs = WCS(header)
>>> # Crop a portion of the image by WCS and get Cutout2d object.
>>> cropped = crop_data(data, center=(600, 400), box_width=100, wcs=wcs, data_only=False)
>>> fig, ax = plt.subplots()
>>> im = ax.imshow(cropped.data, origin='lower', cmap=get_palette())
>>> add_colorbar(im)
>>> plt.show()

Parameters:

data (numpy.array) – Array of data.
box_width (int, optional) – Size of box width in pixels, defaults to 200px.
center (tuple(int, int), optional) – Crop around set of coords, default to image center.
data_only (bool, optional) – If True (default), return only data. If False return the Cutout2D object.
wcs (None|`astropy.wcs.WCS`, optional) – A valid World Coordinate System (WCS) that will be cropped along with the data if provided.

Returns:

A clipped (thumbnailed) version of the data if data_only=True, otherwise: a astropy.nddata.Cutout2D object.

Return type:

np.array

panoptes.utils.images.misc.make_timelapse(directory, fn_out=None, glob_pattern='20[1-9][0-9]*T[0-9]*.jpg', overwrite=False, timeout=60, **kwargs)[source]¶

Create a timelapse.

A timelapse is created from all the images in given directory

Parameters:

directory (str) – Directory containing image files.
fn_out (str, optional) – Full path to output file name, if not provided, defaults to directory basename.
glob_pattern (str, optional) – A glob file pattern of images to include, default ‘20[1-9][0-9]*T[0-9]*.jpg’, which corresponds to the observation images but excludes any pointing images. The pattern should be relative to the local directory.
overwrite (bool, optional) – Overwrite timelapse if exists, default False.
timeout (int) – Timeout for making movie, default 60 seconds.
**kwargs (dict) –

Returns:

Name of output file

Return type:

Raises:

error.InvalidSystemCommand – Raised if ffmpeg command is not found.
FileExistsError – Raised if fn_out already exists and overwrite=False.

panoptes.utils.images.misc.mask_saturated(data, saturation_level=None, threshold=0.9, bit_depth=None, dtype=None)[source]¶

Convert data to a masked array with saturated values masked.

>>> from matplotlib import pyplot as plt
>>> from astropy.wcs import WCS
>>> from panoptes.utils.images.misc import crop_data, mask_saturated
>>> from panoptes.utils.images.plot import add_colorbar, get_palette
>>> from panoptes.utils.images.fits import getdata
>>>
>>> fits_url = 'https://github.com/panoptes/panoptes-utils/raw/develop/tests/data/solved.fits.fz'
>>> data, header = getdata(fits_url, header=True)
>>> wcs = WCS(header)
>>> # Crop a portion of the image by WCS and get Cutout2d object.
>>> cropped = crop_data(data, center=(600, 400), box_width=100, wcs=wcs, data_only=False)
>>> masked = mask_saturated(cropped.data, saturation_level=11535)
>>> fig, ax = plt.subplots()
>>> im = ax.imshow(masked, origin='lower', cmap=get_palette())
>>> add_colorbar(im)
>>> fig.show()

Parameters:

data (array_like) – The numpy data array.
saturation_level (scalar, optional) – The saturation level. If not given then the saturation level will be set to threshold times the maximum pixel value.
threshold (float, optional) – The fraction of the maximum pixel value to use as the saturation level, default 0.9.
bit_depth (astropy.units.Quantity or int, optional) – The effective bit depth of the data. If given the maximum pixel value will be assumed to be 2**bit_depth, otherwise an attempt will be made to infer the maximum pixel value from the data type of the data. If data is not an integer type the maximum pixel value cannot be inferred and an IllegalValue exception will be raised.
dtype (numpy.dtype, optional) – The requested dtype for the masked array. If not given the dtype of the masked array will be same as data.

Returns:

The masked numpy array.

Return type:

numpy.ma.array

Raises:

error.IllegalValue – Raised if bit_depth is an astropy.units.Quantity object but the units are not compatible with either bits or bits/pixel.
error.IllegalValue – Raised if neither saturation level or bit_depth are given, and data has a non integer data type.

panoptes.utils.images.plot module¶

panoptes.utils.images.plot.add_colorbar(axes_image, size='5%', pad=0.05, orientation='vertical')[source]¶

Add a colorbar to the image.

This is a simple convenience function to add a colorbar to a plot generated by matplotlib.pyplot.imshow.

>>> from matplotlib import pyplot as plt
>>> import numpy as np
>>> from panoptes.utils.images.plot import add_colorbar
>>>
>>> x = np.arange(0.0, 100.0)
>>> y = np.arange(0.0, 100.0)
>>> X, Y = np.meshgrid(x, y)
>>>
>>> func = lambda x, y: x**2 + y**2
>>> z = func(X, Y)
>>>
>>> fig, ax = plt.subplots()
>>> im1 = ax.imshow(z, origin='lower')
>>>
>>> # Add the colorbar to the Image object (not the Axes).
>>> add_colorbar(im1)
>>>
>>> fig.show()

_images/panoptes-utils-images-3.png — A colorbar with sane settings.¶

Parameters:: axes_image (matplotlib.image.AxesImage) – A matplotlib AxesImage.

panoptes.utils.images.plot.add_pixel_grid(ax1, grid_height, grid_width, show_axis_labels=True, show_superpixel=False, major_alpha=0.5, minor_alpha=0.25)[source]¶

Adds a pixel grid to a plot, including features for the Bayer array superpixel.

>>> from matplotlib import pyplot as plt
>>> import numpy as np
>>> from panoptes.utils.images.plot import add_pixel_grid
>>>
>>> x = np.arange(-5, 5)
>>> y = np.arange(-5, 5)
>>> X, Y = np.meshgrid(x, y)
>>> func = lambda x, y: x**2 - y**2
>>>
>>> fig, ax = plt.subplots()
>>> im1 = ax.imshow(func(X, Y), origin='lower', cmap='Greys')
>>>
>>> # Add the grid to the Axes object.
>>> add_pixel_grid(ax, grid_height=10, grid_width=10, show_superpixel=True, show_axis_labels=False)
>>>
>>> fig.show()