Introduction

The Hypervision SDK is built around the concept of declarative pipelines that are constructed by applying operations on the hsi.HSImage type. Despite its name, the type can represent either an image file, an in-memory hyperspectral image, a hyperspectral camera connection, or an operation composed of other hsi.HSImage elements.

In general, we can group the kinds of hsi.HSImage into two groups:

Inputs

Any type that either holds its own data or represents an interface to a data producer like a file or camera.

Operations

Any type that is composed of other hsi.HSImage elements and performs some operation or transformation of their data.

Important

In this documentation, instances of the hsi.HSImage type are usually referred to as objects or elements.

Let us look at an example. We start by opening a file using the hsi.open() function:

img: hsi.HSImage = hsi.open("<path-to-file>")

hv_hs_image_t* img = hv_hsi_file_open("<path-to-file>");

The returned image object represents a file, but the data has not yet been loaded. Using the API, it is possible to fetch the data in a number of ways:

# All of these operations are blocking
array = img.to_numpy() # read the entire file into a NumPy array
plane = img.array_plane(200, hsi.bands) # read a single 2d plane
img = img.resolve() # read the file into an in-memory buffer

# This operation does not block
stream = img.stream() # create a stream object

// All of these operations are blocking

// read the entire file into an array
hv_array_t* array;
hv_hs_image_full(img, &array);

// read a single 2d plane
hv_array_t* plane;
hv_hs_image_array_plane(img, 200, HV_AXIS_BANDS, &plane);

// read the file into an in-memory buffer
hv_hs_image_t* img_array;
hv_hs_image_resolve(img, &img_array);

// This operation does not block

// create a stream object
hv_stream_t* stream;
hv_hs_image_stream(img, &stream);

Not that all of these are blocking, except the hsi.HSImage.stream() method, which returns immediately with an hsi.Stream object that represents a running computation. This computation runs asynchronously and therefore does not block the main thread.

Note

The other operations use hsi.Stream internally to perform their function.

Important

A running hsi.Stream object will queue up its results in a temporary buffer. This is less efficient than gathering the result in an array using hsi.HSImage.to_numpy() .

Only use hsi.HSImage.stream() when you intend to use the data immediately, for example, by iterating over the streamed planes. This method is useful if you want to operate on individual planes instead of full data cubes.

Warning

The hsi.Stream is currently in an incomplete state for the external API. At some point, a dedicated section of the guide will be added to explain how to use this type appropriately.

You can expect the type to be particularly useful for interfacing the SDK with your own custom processes.

A complete example

To better understand how pipelines work in the SDK, let’s take a look at a complete example of a calculation that is used in hyperspectral image analysis, reflectance calibration:

import hsi

# Open the input files
img = hsi.open("image.pam")
white_img = hsi.open("white_ref.pam")
dark_img = hsi.open("dark_ref.pam")

# Convert the main image to float32
img = img.ensure_dtype(hsi.float32)

# Pre-calculate the white reference
white_ref = white_img
    .ensure_dtype(hsi.float32)
    .mean_axis(hsi.lines)
    .resolve()

# Pre-calculate the dark reference
dark_ref = dark_img
    .ensure_dtype(hsi.float32)
    .mean_axis(hsi.lines)
    .resolve()

# Perform the calibration itself
calibrated = (img - dark_ref) / (white_ref - dark_ref)

# Write the output to disk
hsi.write(calibrated, "calibrated.pam")

int err = 0;

// Open files and get HSImage handles
hv_hsi_file_t* f_img = NULL;
hv_hsi_file_t* f_white = NULL;
hv_hsi_file_t* f_dark = NULL;

err = hv_hsi_file_open("image.pam", &f_img);
if (err) return err;
err = hv_hsi_file_open("white_ref.pam", &f_white);
if (err) return err;
err = hv_hsi_file_open("dark_ref.pam", &f_dark);
if (err) return err;

hv_hs_image_t* img = NULL;
hv_hs_image_t* white_img = NULL;
hv_hs_image_t* dark_img = NULL;

img = hv_hsi_file_to_image(&f_img);
white_img = hv_hsi_file_to_image(&f_white);
dark_img = hv_hsi_file_to_image(&f_dark);

// Create a float32_t version of the image
hv_hs_image_t* img_f32 = NULL;
err = hv_hs_image_as_dtype(img, HV_DTYPE_F32, &img_f32);
if (err) return err;
hv_hs_image_free(&img);


// Create the white reference using the following steps:

// 1. Convert the data type to float32_t
hv_hs_image_t* white_f32 = NULL;
err = hv_hs_image_as_dtype(white_img, HV_DTYPE_F32, &white_f32);
if (err) return err;

// 2. Compute the mean image along the lines axis
hv_hs_image_t* white_mean = NULL;
err = hv_hs_image_mean_axis(white_f32, HV_AXIS_LINES, &white_mean);
if (err) return err;

// 3. Resolve the computation to an in-memory array
hv_hs_image_t* white_ref = NULL;
err = hv_hs_image_resolve(white_mean, &white_ref);
if (err) return err;
hv_hs_image_free(&white_img);
hv_hs_image_free(&white_f32);
hv_hs_image_free(&white_mean);

// Create the dark reference using the same steps.
hv_hs_image_t* dark_f32 = NULL;
err = hv_hs_image_as_dtype(dark_img, HV_DTYPE_F32, &dark_f32);
if (err) return err;
hv_hs_image_t* dark_mean = NULL;
err = hv_hs_image_mean_axis(dark_f32, HV_AXIS_LINES, &dark_mean);
if (err) return err;
hv_hs_image_t* dark_ref = NULL;
err = hv_hs_image_resolve(dark_mean, &dark_ref);
if (err) return err;
hv_hs_image_free(&dark_img);
hv_hs_image_free(&dark_f32);
hv_hs_image_free(&dark_mean);


// Calculate the reflectance using the formula (img - dark) / (white - dark)

// 1. Subtract the dark reference from the image
hv_hs_image_t* img_minus_dark = NULL;
err = hv_hs_image_sub(img_f32, dark_ref, &img_minus_dark);
if (err) return err;

// 2. Subtract the dark reference from the white reference
hv_hs_image_t* white_minus_dark = NULL;
err = hv_hs_image_sub(white_ref, dark_ref, &white_minus_dark);
if (err) return err;

// 3. Divide the two intermediary results
hv_hs_image_t* calibrated = NULL;
err = hv_hs_image_div(img_minus_dark, white_minus_dark, &calibrated);
if (err) return err;


// Write the result to disk (adjust path/format as needed)
err = hv_hs_image_write(calibrated_buf, "calibrated.pam");
if (err) return err;

// Cleanup
hv_hs_image_free(&img_f32);
hv_hs_image_free(&white_ref);
hv_hs_image_free(&dark_ref);
hv_hs_image_free(&img_minus_dark);
hv_hs_image_free(&white_minus_dark);
hv_hs_image_free(&calibrated);
hv_hs_image_free(&calibrated_buf);

The example combines multiple source images and operations to achieve a result. Much of the actual computation is spent generating each 2d reference from a reference data cube. Note how hsi.HSImage.resolve() is used to cache the reference result. If this hadn’t been done, any future operations using white_ref or dark_ref would run the computation again.

Knowing when to use hsi.HSImage.resolve() is critical for using the SDK effectively. Caching ensures that results can be stored instead of being rerun, but the result also takes up space. In the example above, the hsi.HSImage.mean_axis() call flattens the image to two dimensions, which means that the result’s memory usage is negligible.

Tip

Use caching whenever you have to reuse the result of a computation, especially when that computation is expensive to run.

Warning

Be mindful about caching large images if memory is limited.

For example, a \(1000\times 1296\) image with \(920\) bands taken using the camera’s 8-bit mode, takes up \(\sim 1.2 \text{gb}\). If the data type is changed to hsi.float64 , the image now takes up \(\sim 9.6 \text{gb}\).