HealSparseMap File Specification v1.1.2

A HealSparseMap file is a standard FITS file with two extensions. The primary (zeroth) extension is an integer image that describes the coverage map, and the first extension is an image or binary table that describes the sparse map. This document describes the file format specification of these two extensions in the FITS file.

Terminology

This is a list of terminology used in a HealSparseMap file:

  • nside_sparse: The HEALPix nside for the fine-grained (sparse) map

  • nside_coverage: The HEALPix nside for the coverage map

  • bit_shift: The number of bits to shift to convert from nside_sparse to nside_coverage in the HEALPix NEST scheme. bit_shift = 2*log_2(nside_sparse/nside_coverage).

  • valid_pixels: The list of pixels with defined values (> sentinel) in the sparse map.

  • sentinel: The sentinel value that notes if a pixel is not a valid pixel. Default is healpy.UNSEEN for floating-point maps, -MAXINT for integer maps, and 0 for wide mask maps.

  • nfine_per_cov: The number of fine (sparse) pixels per coverage pixel. nfine_per_cov = 2**bit_shift.

  • wide_mask_width: The width of a wide mask, in bytes.

Pixel Lookups

The HealSparseMap file format is derived from the method of encoding fast look-ups of arbitary pixel values using the HEALPix nest pixel scheme.

Given a nest-encoded sparse (high resolution) pixel value, pix_nest, we can compute the coverage (low resolution) pixel value with a simple bit shift operation: ipnest_cov = right_shift(pix_nest, bit_shift), where bit_shift is defined in Terminology.

The sparse map itself is stored in blocks of data, each of which includes nfine_per_cov contiguous pixels for each coverage pixel that contains valid data. To find the location within a given a coverage block, we need to subtract the first fine (sparse) nest pixel value for the given coverage pixel. Therefore, first_pixel = nfine_per_cov*ipnest_cov.

Next, if we have a map of offsets which points to the location of the proper sparse map block, cov_map_raw_offset, we find the look-up index is index = pix_nest - nfine_per_cov*ipnest_cov + cov_map_raw_offset[ipnest_cov]. In practice, we can combine the final terms here. The look-up index is then index = pix_nest + cov_map[ipnest_cov] where cov_map = cov_map_raw_offset[ipnest_cov] - nfine_per_cov*ipnest_cov.

As described in Sparse Map, the first block in the sparse map is special, and is always filled with sentinel values. All cov_map indices for coverage pixels outside the coverage map point to this sentinel block.

Coverage Map

The coverage map encodes the mapping from raw pixel number to location within the sparse map. It is an integer (numpy.int64) map with 12*nside_coverage*nside_coverage values, all of which are filled. The structure of the coverage map is as follows.

Coverage Map Header

The coverage map header must contain the following keywords:

  • EXTNAME must be "COV"

  • PIXTYPE must be "HEALSPARSE"

  • NSIDE is equal to nside_coverage

Coverage Map Image

As described in Pixel Lookups, the coverage map image contains indices that are offset pointers to the block in the sparse map with associated values for that coverage pixel. An empty HealSparseMap is intialized with the following coverage pixel values, which all point to the first sentinel block in the sparse map.

import numpy as np
import healpy as hp

cov_map[:] = -1*np.arange(hp.nside2npix(nside_coverage), dtype=np.int64)*nfine_per_cov

Sparse Map

The sparse map contains the map data, split into blocks, each of which is nfine_per_cov elements long. The first block is special, and is always filled with sentinel values.

The following datatypes are allowed:

  • 1-byte unsigned integer (numpy.uint8)

  • 1-byte signed integer (numpy.int8)

  • 2-byte unsigned integer (numpy.uint16)

  • 2-byte signed integer (numpy.int16)

  • 4-byte unsigned integer (numpy.uint32)

  • 4-byte signed integer (numpy.uint32)

  • 8-byte signed integer (numpy.int64)

  • 4-byte floating point (numpy.float32)

  • 8-byte floating point (numpy.float64)

  • Numpy record array of numeric types that can be serialized with FITS

  • The WIDE_MASK special encoding

Sparse Map Header

The sparse map header must contain:

  • EXTNAME must be "SPARSE"

  • PIXTYPE must be "HEALSPARSE"

  • SENTINEL is equal to sentinel

If the sparse map is a numpy record array, it must contain:

  • PRIMARY is equal to the name of the “primary” field which defines the valid pixels.

If the sparse map is a wide mask, it must contain:

  • WIDEMASK must be True

  • WWIDTH must be the width (in bytes) of the wide mask.

Sparse Map Image

If the sparse map is not of a numpy record array type, it is stored as a one dimensional image array. The first block of nfine_per_cov values are set to sentinel. Each additional block of nfine_per_cov is associated with a single element in the coverage map. These blocks may be in any arbitrary order, allowing for easy appending of new coverage pixels. All invalid pixels must be set to sentinel. If the image is an integer type with 32 bits or fewer, it may be stored with FITS tile compression, with the tile size set to the block size (nfine_per_cov). If the image is a floating-point image, it may be stored with FITS tile compression, with quantization_level=0 and GZIP_2 (lossless gzip compression), with the tile size set to the block size (nfine_per_cov).

Sparse Map Wide Mask

If the sparse map is a wide mask map, the sparse map is stored as a flattened version of the in-memory wide_mask_width * npix array. This should be flattened on storage, and reshaped on read, using the default numpy memory ordering. The sentinel value for wide masks must be 0, and all invalid pixels must be set to 0. The wide mask image may be stored with FITS tile compression, with the tile size set to the block size times with width (wide_mask_width * nfine_per_cov).

Sparse Map Table

If the sparse map is a numpy record array type, it is stored as a one dimensional table array. The first block of nfine_per_cov values are set such that the primary field must be set to sentinel. As with the sparse map image, each additional block of nfine_per_cov is associated with a single element in the coverage map. These blocks may be in any arbitrary order, allowing for easy appending of new coverage pixels. All invalid pixels must have the primary field set to sentinel.