Usage Guide
This guide covers the key functions and concepts needed to work effectively with the HV SDK: loading and manipulating hyperspectral datacubes, building lazy processing pipelines, calibrating raw captures to reflectance, and streaming data from a live or simulated camera. For a complete API reference, see the official documentation linked below.
Refer to the Getting Started and Guide, as well as Recipes from the official documentation, for a complete overview of the HV SDK's capabilities.
Also check the Examples section for more complete examples using the HV SDK.
One of the most important SDK concepts is lazy loading. This means that data is only loaded into memory when it is strictly necessary (and only the required part of the datacube), which keeps the RAM usage low.
However, this also requires the user to think about when they want to trigger a data load:
- Converting to a NumPy array (
HSImage.to_numpy()) will trigger a load. - Forcing a cache with
HSImage.resolve()will trigger calculations and store results in RAM. - In camera pipelines, data capture is only triggered when the data is actually accessed.
This guide requires a basic understanding of data ordering/interleave. See the Interleave Optimization section below.
Interleave Optimization
Hyperspectral images are inherently three-dimensional, structured by lines (height), samples (width), and bands (spectral channels). The way this 3D data is stored in memory or on disk is known as its interleave type. The HV SDK supports three primary interleave types: BIP (Band-Interleaved-by-Pixel), BIL (Band-Interleaved-by-Line), and BSQ (Band-Sequential).
Understanding and strategically choosing the interleave type is crucial for optimizing the performance of your hyperspectral processing workflows. This is because modern CPUs and memory systems operate most efficiently when accessing data that is stored contiguously in memory (cache locality). If an operation frequently accesses data along a specific dimension, aligning the data's interleave with that access pattern can significantly improve processing speed by maximizing cache hits and minimizing memory fetches.
Here's a breakdown of common interleave types and the operations they typically optimize:
| Interleave | Layout | Typically best for |
|---|---|---|
| BIP | L x S x B | Pixel-wise spectral workflows (classification, unmixing) |
| BIL | L x B x S | Line-wise processing across width and/or bands |
| BSQ | B x L x S | Per-band 2D image operations |
-
Performance: Accessing data contiguously (e.g., all elements of a row in a C-contiguous array) is significantly faster due to CPU caching. If your algorithm iterates over bands for each pixel, BIP is generally more efficient. If it processes entire bands as 2D images, BSQ is usually better.
-
Memory Access Patterns: When working with NumPy arrays derived from
HSImage, understanding the interleave helps you choose the correct axis order for array indexing (arr[lines, samples, bands]vs.arr[bands, lines, samples]), avoiding performance penalties from non-contiguous memory access and preventing logical errors. -
SDK Capabilities: The HV SDK can transform interleave efficiently when needed, but it is still beneficial to align your chosen interleave with your primary operations.
HSImage.to_interleave()andimg.to_numpy_with_interleave()let you control this explicitly.
By being mindful of interleave type, you can write more efficient and correct hyperspectral data processing code.
BIP (Band-Interleaved-by-Pixel)
- Memory Layout:
[lines, samples, bands](L x S x B) - Optimized for: Operations that require accessing all bands for a single pixel or all bands for a specific spatial location sequentially. This means spectral information for a pixel is stored contiguously.
- Examples: Pixel-wise spectral classification, spectral unmixing, analyzing or comparing individual spectral curves, point-based spectral feature extraction.
- SDK Slicing:
HSImageslicingimg[line, sample, :]is inherently optimized for BIP-like access.
BIL (Band-Interleaved-by-Line)
- Memory Layout:
[lines, bands, samples](L x B x S) - Optimized for: Operations that process all samples for a single band within a line or all bands for an entire line sequentially. This means a full line of spectral data is stored together.
- Examples: Line-based spatial filtering applied independently to each band, transformations that operate across the spatial width of a line for all bands, and processing where line-wise access across multiple bands is primary.
BSQ (Band-Sequential)
- Memory Layout:
[bands, lines, samples](B x L x S) - Optimized for: Operations that process an entire 2D image for a single band at a time. This means all spatial data for one spectral band is stored contiguously.
- Examples: Image processing techniques applied to individual bands (e.g., image segmentation, object detection on a specific spectral band), creating grayscale visualizations for individual bands, and band-ratio calculations across the entire image.
Basics
This section covers basic functions for reading, writing, and accessing data. It also introduces core data handling patterns used throughout this guide.
Reading and writing datacubes
The SDK supports reading and writing PAM, ENVI and TIFF formats. The format is determined by the file extension.
import hsi
# Opens an ENVI file
# Data is not yet loaded into memory due to lazy loading
# The data will only be loaded when accessed (e.g., for processing or conversion)
img = hsi.open("path/to/file.hdr")
# Writes PAM format
# Data is read from the source as it is written to the file.
img.write("path/to/output.pam")
Interacting with NumPy
Data can be easily moved between the SDK and NumPy. Note that exporting to NumPy triggers a memory load for the necessary data.
import hsi
# All 3 methods below cause the necessary data to be read into memory
# Get the full 3D cube
array = img.to_numpy()
# Get the full 3D cube, with the desired memory layout
array = img.to_numpy_with_interleave(hsi.bil)
# Get a 2D grayscale image corresponding to a single wavelength band (index 200).
slice_2d = img.array_plane(200, hsi.bands)
# Create HSImage from numpy array
img = hsi.HSImage.from_numpy(array, hsi.bsq)
The axis order of the resulting ndarray depends on the native interleave of
the HSImage. See the Slicing section for details on how this
interacts with indexing, and use to_numpy_with_interleave() for predictable
axis ordering.
Note that explicitly converting the interleave incurs a reordering cost, but
may actually improve downstream performance if your subsequent operations are
better suited to the target interleave — see Interleave Optimization
for guidance on choosing the right layout for your workload.
Data access
Slicing
Use HSImage.slice() or [] to operate on a specified subset of data.
Note: Slicing indices are always in BIP format: [lines, samples, bands].
# Note that the ordering is always BIP: lines, samples, bands
img[:, :, :] # The full image
img[:20, :20, :] # The first 20 lines and samples
img[:, :, ::4] # Take only every 4th band
Slice indices are always in BIP format [lines, samples, bands], regardless
of the image's native interleave. However, to_numpy() returns an array whose
axis order does reflect the native interleave — this mismatch is a common
source of bugs.
To avoid this, prefer to_numpy_with_interleave(hsi.bip) for consistent
indexing. If you use to_numpy(), you must check img.header.interleave
and index accordingly. Explicitly converting the interleave via
to_numpy_with_interleave() incurs a reordering cost, but can be a net
gain if your subsequent operations align well with the target layout. See
Interleave Optimization for guidance.
arr = img.to_numpy()
# the index order depends on the original interleave type
if img.header.interleave == hsi.Interleave.BIL:
px = arr[line, band, sample] # (Lines, Bands, Samples)
elif img.header.interleave == hsi.Interleave.BSQ:
px = arr[band, line, sample] # (Bands, Lines, Samples)
elif img.header.interleave == hsi.Interleave.BIP:
px = arr[line, sample, band] # (Lines, Samples, Bands)
Changing the interleave type
The SDK provides two ways to work with a specific interleave: converting the
HSImage itself, or specifying the layout only at the point of NumPy export.
-
to_interleave()— converts theHSImageto a new interleave in the SDK pipeline. Use this when subsequent SDK operations will benefit from a specific memory layout (see Interleave Optimization), or when writing to disk in a particular format (see exporting datacubes). -
to_numpy_with_interleave()— exports to NumPy with a specified axis order without modifying the underlyingHSImage. Use this when you only need a consistent layout for NumPy indexing and don't need the SDK pipeline to reflect the change.
import hsi
img = hsi.open("path/to/file.hdr")
# Convert the HSImage itself to BIL — subsequent SDK operations will use this layout
new_img = img.to_interleave(hsi.Interleave.BIL)
# Export to NumPy with a specific layout, without changing the HSImage
arr = img.to_numpy_with_interleave(hsi.bil)
Type conversion
The SDK operations preserve the input dtype — conversions are never applied implicitly. This means you should convert before operations that require a specific type, such as division in reflectance calibration (which requires a float type to avoid integer truncation).
Use ensure_dtype() to convert to a target type, or no-op if already correct.
# Raw captures are typically uint8 or uint16.
# Dividing without conversion causes integer truncation — most values become 0.
img = hsi.open("image.pam") # dtype: uint8
wrong = img / 255 # still uint8 — all values 0 or 1
# ensure_dtype() converts only if needed — safe to use even if type is uncertain
correct = img.ensure_dtype(hsi.float32) / 255.0 # dtype: float32, values in [0, 1]
as_dtype() is also available and behaves identically, except it always
converts even if the type already matches.
Forcing cache
By default, all SDK operations are lazy — nothing is computed until data is
actually needed. HSImage.resolve() explicitly triggers computation and stores
the result in memory, returning a new cached HSImage.
This is useful for intermediate results that are expensive to compute and
reused multiple times in a pipeline. Without resolve(), such results would
be recomputed from scratch each time they are accessed.
The example below shows a manual reflectance calibration pipeline to
illustrate why caching matters. White and dark references are reduced once and
cached with resolve(), then reused in the calibration expression.
For production usage, prefer the SDK's built-in calibration helpers in the
Reflectance calibration section
(make_reference() and reflectance_calibration()), which are clearer and
less error-prone.
import hsi
img = hsi.open("image.pam")
white_img = hsi.open("white_ref.pam")
dark_img = hsi.open("dark_ref.pam")
# Without resolve(), these reductions would be recomputed every time
# the references are used.
white_ref = white_img.ensure_dtype(hsi.float32).mean_axis(hsi.lines).resolve()
dark_ref = dark_img.ensure_dtype(hsi.float32).mean_axis(hsi.lines).resolve()
img = img.ensure_dtype(hsi.float32)
calibrated = (img - dark_ref) / (white_ref - dark_ref)
hsi.write(calibrated, "calibrated.pam")
Operations
Elementwise operations
The standard arithmetic operators (+, -, *, /) work directly on
HSImage objects and are applied lazily across the datacube. These are
the building blocks for operations like reflectance calibration and band
arithmetic.
Scalar operations preserve the input dtype. Use ensure_dtype() before
division to avoid integer truncation — see Type Conversion.
# Subtract two bands to highlight spectral differences
diff = img[:, :, 500] - img[:, :, 810]
# Scale image to (0, 1) — convert to float first to avoid truncation
scaled = img.ensure_dtype(hsi.float32) / 254.
Reduction operations
Reductions collapse the datacube along a single axis, returning an HSImage
with that dimension removed. Supported operations are mean, standard deviation,
variance, and sum.
# Collapse the bands axis → 2D spatial image (mean across spectrum)
img.mean_axis(hsi.bands)
# Collapse lines then samples → 1D mean spectrum
img.mean_axis(hsi.lines).mean_axis(hsi.samples)
Matrix multiplication
The dot() method applies a vector or matrix projection along a specified
axis. This is useful for spectral weighting, dimensionality reduction, and
applying learned spectral transforms.
-
A 1D vector operand of shape
(bands,)produces a single-band output — equivalent to a weighted sum across the spectrum per pixel. -
A 2D matrix operand of shape
(n, bands)produces ann-band output, where each output band is one row of the matrix dotted with the spectrum.
# 1D case: weighted sum of bands 2 and 200
operand = np.zeros(920)
operand[2] = 0.5
operand[200] = 0.5
res = img.dot(operand, hsi.bands)
# 2D case: two-band output
# Band 0: weighted sum of bands 2 and 200 (same as above)
# Band 1: passthrough of band 25
operand = np.zeros((2, 920))
operand[0, 2] = 0.5
operand[0, 200] = 0.5
operand[1, 25] = 1.0
res = img.dot(operand, hsi.bands)
Other operations
The SDK provides several additional pointwise and reduction utilities:
-
binning(n, axis)— averages everynelements along the given axis. Note that the output dtype is not automatically promoted, so useensure_dtype()beforehand if overflow is a risk. -
clip(min, max)— clamps all values to the provided range, useful for removing sensor artifacts or outliers before further processing. -
nan_to_num(value)— replacesNaNvalues with a scalar, typically needed after reflectance calibration where division by zero can occur.
# Apply mean-binning (beware that the dtype will not be changed automatically)
# Use `ensure_dtype()` first if the input type is not big enough to hold the binning results
condensed = img.ensure_dtype(hsi.float32).binning(8, hsi.bands)
# Clip image to the provided range
clipped = img.clip(0, 1)
# Convert NaN values to the provided value
non_nan = img.nan_to_num(0)
Custom operations
The SDK provides facilities for injecting custom code directly into the lazy pipeline, meaning your functions benefit from the same streaming and lazy evaluation as built-in operations — without loading the full cube into memory.
Use this when the built-in operations don't cover your use case. The two main
utilities are @hsi.util.operation for custom spectral transforms, and
hsi.util.predictor() for wrapping ML model inference.
Reflectance calibration
Raw hyperspectral data contains sensor-specific artifacts that make direct comparison between pixels or datasets unreliable. Reflectance calibration corrects for these by normalizing against a white reference (a uniformly reflective surface) and a dark reference (captured with the lens covered), converting raw intensity values into physically meaningful reflectance values in the range [0, 1].
The SDK provides make_reference() and reflectance_calibration() in
hsi.preprocessing to streamline this process.
import hsi
from hsi.preprocessing import make_reference, reflectance_calibration
img = hsi.open("path/to/file.hdr")
dark = hsi.open("path/to/dark_file.hdr")
# inline white reference
white_ref = make_reference(img[:100, :, :])
dark_ref = make_reference(dark)
reflectance = reflectance_calibration(img, white_ref, dark_ref)
# Note that the data is now of float 32 type
# Consider scaling and converting to 'uint8' to save memory
reflectance_uint8 = (255*reflectance).ensure_dtype(hsi.uint8)
Camera interface
The SDK connects directly to a Hypervision camera over Ethernet, providing
control over exposure, framerate, spatial crop, and spectral bands. Once
configured, the camera is represented as an HSImage, meaning the same lazy
pipeline operations used for file-based data apply equally to live capture.
import hsi
from datetime import datetime
def current_datetime_filename():
return datetime.now().strftime("%Y-%m-%d_%H-%M-%S")
N_IMGS = 10
filename = f"/tmp/_HSI_{current_datetime_filename()}"
SAVE_CUBE = True
# Desired settings
EXP = 1000
FPS = 100
# Horizontal crop
H_START = 200
H_END = 300
# Bands
V_START = 0
V_END = 920
# ETH_B interface
cam = hsi.HSCamera("10.100.10.100")
# Get information
print(f"{cam.get_config()=}")
print(f"{cam.get_settings()=}")
print(f"{cam.get_crop()=}")
print(f"{cam.get_exposure()=}")
print(f"{cam.get_framerate()=} {cam.get_framerate_list()=}")
#print(f"Binning: {cam.get_horizontal_binning()}x{cam.get_vertical_binning()}")
print(f"{cam.get_bands()=}")
#print(f"{cam.get_wavelengths()=}")
# Set parameters
print(f"{cam.set_exposure(EXP)=}")
print(f"{cam.set_framerate(FPS)=}")
#cam.set_horizontal_binning(1)
#cam.set_vertical_binning(1)
print(f"{cam.set_horizontal_crop((H_START, H_END))=}")
print(f"{cam.set_bands([(V_START, V_END)])=}")
# Multiple band intervals (up to 8 regions):
#print(f"{cam.set_bands([(V_START1, V_END1), (V_START2, V_END2)])=}")
######### Datacube Capture
# Create a stream object
img = cam.to_hs_image()
# Configure the datacube size (N_IMGS)
datacube = img[:N_IMGS, :, :]
# Write to file or convert to numpy:
# triggers the streaming start
if SAVE_CUBE:
#hsi.write(datacube, filename + ".pam")
hsi.write(datacube, filename + ".hdr")
#hsi.write(datacube, filename + ".tif")
else:
array = datacube.to_numpy()
# Also triggers the streaming start
#datacube.resolve()
######### Datacube Processing
# For processing the captured datacube, see the Operations section above.
See also the more complete example under the Quick Start section and be aware of data throughput limitations.
Simulated Camera
From version 0.7.3, the SDK can also simulate the output of a HSI camera by providing an input datacube.
The simulated camera streams frames in the interleave of the source file.
If the source file does not match the camera you are simulating — for example,
a push-broom camera like the HV1700 outputs BIL —
convert the source image before passing it to HSCamera.
# Convert to BIL to match the native output of a push-broom camera (e.g. HV1700)
src_img = hsi.open("path/to/file.hdr")
cam = hsi.HSCamera(src_img.to_interleave(hsi.bil))
# Create a stream object
img = cam.to_hs_image()
# Configure the datacube size (N_IMGS)
datacube = img[:N_IMGS, :, :]
# Trigger the streaming start
datacube.resolve()
#array = datacube.to_numpy()
Streaming individual frames
Available from version 0.10.0.
import hsi
cam = hsi.HSCamera("10.100.10.100")
# Use as an iterator with the context manager protocol
with cam.stream() as stream:
for i in range(100):
meta, frame = next(stream) # Get the next frame
print(frame.shape)
# Manual usage
stream = cam.stream() # Start streaming
meta, frame = stream.get_frame() # Get a single frame
stream.stop() # Stop the stream
Checking for dropped frames
Frame metadata is also available from the camera, which makes it possible to check if any frames were dropped.
import hsi
import numpy as np
import logging
logger = logging.getLogger("main")
cam = hsi.HSCamera("10.100.10.100")
with cam.stream() as stream:
for i in range(100):
meta, frame = next(stream) # Get the next frame
if meta.dropped:
print(f"Dropped frame {meta.seq} at time {meta.timestamp}")
# Handle dropped frame
Alternatively, dropped frames can be detected inside an @operation function
when working within the SDK pipeline, allowing warnings to propagate lazily
alongside the data:
import hsi
import numpy as np
import logging
from hsi.util import operation
logger = logging.getLogger("main")
cam = hsi.HSCamera("10.100.10.100")
@operation(hsi.bil)
def op(meta: hsi.FrameMeta, frame: np.ndarray) -> np.ndarray:
if meta.dropped:
logger.warning(f"Dropped frame {meta.seq} at time {meta.timestamp}")
# Do some processing here
# ...
return frame
out_img = op(cam)
# Any method calls that fetches data from the pipeline will now issue warnings if a dropped frame is encountered.
out_img.array_plane(250, hsi.bands)
out_img.to_numpy()
Buteo interface
In its normal configuration, the Buteo has a host-PC which controls the whole system (screen, camera, lights and belt) in order to make it simple to use. However, more flexibility is desired in some situations. Therefore, from version 0.9.1, the SDK also implements functionality that allows the camera to control the belt and lights of the Buteo directly.
See https://docs.qtec.com/hv-sdk/guide/buteo.html for more information.
This functionality is still being tested and the interface may therefore change.
Note that in order to give the camera direct access to the belt and light controller (instead of having the host-PC controlling them) it is required to switch some cables around inside the Buteo cabinet. Contact qtec for more information on the procedure if you wish to explore this option.
The process also requires providing the camera with a proper software image to boot from. See qtecOS Image for more information on how to create the required bootable media (CFAST/USB drive).
Note that the Buteo screen is not functional while the camera is in control of the belt and lights. All configuration and control needs to be done via a Python script (using HV SDK) running on an external PC (or the camera itself via the terminal).
This process is also easily reversible if it is desired to go back to original Buteo functionality.
import hsi
from hsi import StageController
from datetime import datetime
def current_datetime_filename():
return datetime.now().strftime("%Y-%m-%d_%H-%M-%S")
# Settings
BIT_MODE = 8
WIDTH = 1280
N_BANDS = 920
EXP_TIME = 3000 # max exposure ~10.000 us at 100fps
TARGET_DIR = "/tmp/"
filename = f"_HSI_{current_datetime_filename()}"
# Camera IP
buteo = StageController("10.100.10.100")
# Can only set fps OR velocity not both
# So we recommend setting the max fps for the available 1Gbs connection
# The actual throughput seen experimentally for the 1Gbs connection is 12-15% lower than the max theoretical value
MAX_FPS = pow(2,30) / (WIDTH*N_BANDS*BIT_MODE) * 0.85
#print(f"{MAX_FPS=}")
# Set conveyor belt velocity in mm/s
#buteo.velocity = 20.0
# max fps for the 1Gbs connection ~100 (for image size of 1280px x 900bands)
buteo.framerate = MAX_FPS
# Set conveyor move distance in mm.
buteo.distance = 50.0
# Set oversampling (lines)
# An oversampling of 4x gives a correct aspect ratio visually
buteo.oversampling = 4.0
# Adjust Camera Settings
exp_time = buteo.hs_camera.set_exposure(EXP_TIME)
# Gain is not available at the moment so it is fixed at 1x
#gain = buteo.hs_camera.set_gain(0)
# Change crop top in config to match the one from the Buteo
config = buteo.hs_camera.get_config()
print(f"{config=}")
#cal = config.calibration
#cal.crop_top = 20 # must be multiples of 4
#config.calibration = cal
#config = buteo.hs_camera.set_config(config)
#print(f"{config=}")
# Adjust spatial cropping
#crop = buteo.hs_camera.get_crop()
#crop = buteo.hs_camera.set_horizontal_crop(0, crop.max_width)
#crop = buteo.hs_camera.set_horizontal_crop((H_START, H_END))
h_crop = buteo.hs_camera.set_horizontal_crop(0, WIDTH)
print(f"{h_crop=}")
# Adjust number of bands or band intervals
#bands = buteo.hs_camera.get_bands()
# Multiple band intervals (up to 8 regions):
#bands = buteo.hs_camera.set_bands([(V_START1, V_END1), (V_START2, V_END2)])
bands = buteo.hs_camera.set_bands([(0, N_BANDS)])
print(f"{bands=}")
# Convert to HSImage object
image = buteo.to_hs_image()
# Don't change settings after this point
# Start streaming plus belt/lights
image.resolve()
# Save to file: the SDK supports PAM, ENVI and TIFF
hsi.write(image, TARGET_DIR + filename + ".hdr")
#hsi.write(image, TARGET_DIR + filename + ".pam")
#hsi.write(image, TARGET_DIR + filename + ".tiff")
# Save the current settings to a txt file as well if desired
try:
with open(TARGET_DIR + filename + "_settings.txt", 'w') as f:
f.write("--- Camera Configuration ---\n")
f.write(f"{filename=}\n")
f.write(f"{config=}\n")
f.write(f"{bands=}\n")
f.write(f"{exp_time=}\n")
f.write(f"{h_crop=}\n")
f.write(f"{buteo.framerate=}\n")
f.write(f"{buteo.distance=}\n")
f.write(f"{buteo.oversampling=}\n")
f.write(f"{buteo.velocity=}\n")
print(f"Successfully saved settings to {filename}")
except Exception as e:
print(f"An error occurred: {e}")
The hsi.StageController communicates directly with a REST API interface
running on the camera at <camera_ip>:5001.
Information about the available endpoints (for controlling the belt and lights)
is available in the live documentation at <camera_ip>:5001/docs.
PCA
The HV SDK integrates with scikit-learn's PCA via the pca_helper utility,
which wraps a fitted sklearn PCA model so it can be applied directly inside
the SDK's lazy pipeline. This means the projection runs band-by-band without
loading the full cube into memory at once.
The example below demonstrates a memory-efficient workflow: a random spectral
subsample is drawn from the image using ufunc, used to fit the PCA model,
and the trained projection is then applied lazily across the full datacube.
import hsi
from hsi.ml import pca_helper
import numpy as np
from sklearn.decomposition import PCA
img = hsi.open("path/to/file").as_dtype(hsi.float32)
def gen_select(img, n_samples_per_line=10):
"""Sample randomly (with the same number of samples per line) from the image."""
# Assumes BIL (doesn't work for BSQ/BIP)
def select(plane):
sample = np.random.choice(np.arange(plane.shape[1]), size=n_samples_per_line)
sample = plane[:, sample]
return sample
return img.ufunc(select) # Any Python function can be passed here
# Convert interleave type
img = img.to_interleave(hsi.Interleave.BIL)
# Get subsample from image in memory-efficient manner
s_out = gen_select(img).to_numpy()
s_out = s_out.transpose((0, 2, 1))
s_out = s_out.reshape((-1, s_out.shape[2]))
# Number of PCA components to retain
n_components = 10
# Fit model
model = PCA(n_components)
model.fit(s_out)
hs_model = pca_helper(model)
# Here, the prediction function is created.
out = hs_model(img)
# The calculation is only applied when requested, similar to other operations
result = out.to_numpy()
See also the more complete example under the Examples section.