Camera Integration#

The SDK provides multiple ways to interact with the Qtechnology Hypervision line of hyperspectral cameras. The pipeline API integrates directly with the cameras and as a result makes it easy to create complex processing pipelines without having to worry about handling individual frames. Alternatively, the cameras can be operated like regular cameras using a stream of two-dimensional frames.

Warning

The camera functionality is in its early stages of development and may change significantly.

Creating a camera object#

The base type for camera interoperability is hs.control.Camera and it may represent either a real hyperspectral camera or an image-based mock camera. The latter is particularly useful for testing systems without having to deal with the complexities of controlling the physical camera.

Let us look at an example of how to create a new camera object and perform some basic operations on it:

python

from qtec_hv_sdk import control

cam = control.Camera("<camera_ip>")
# Alternatively, you can create a simulated camera from an existing Image
# Note that this will convert BSQ images to BIL internally, since the Camera assumes a push-broom architecture
cam = control.Camera(<img_object>)

# Printing properties of the camera is easy
print(cam.get_wavelengths())
print(cam.get_framerate())

c/c++

{
	// Create the camera object
	hv_camera_t* cam;
	hv_camera_new("192.168.2.229", &cam);

	// Alternatively, the camera can be initialized from a hv_image_t object.
	// hv_camera_from_image(<img>, &cam);

	// Get the wavelengths
	slice_boxed_float_t* wavelengths;
	hv_camera_get_wavelengths(cam, &wavelengths);

	// Print the wavelength values
	printf("Wavelengths:\n");
	for (size_t i = 0; i < wavelengths->len; i++) {
		printf("%fnm\n", wavelengths->ptr[i]);
	}

	// Print the framerate
	const float fps = hv_camera_get_framerate(cam);

The camera object supports setting many common camera settings, including exposure time, framerate, and binning.

Tip

We provide a guide on how to capture high-quality hyperspectral images on our main documentation site.

Note

The simulated camera is useful for testing pipelines that are meant to run on live data.

The simulated camera allows setting/reading the same settings as the real camera, but many settings don’t have any impact on the image output. Only the band-selection currently affects the output image.

The following additional settings will be implemented soon: - Horizontal cropping - Binning - Bit-mode

Image capture#

The hs.control.Camera object integrates directly with the pipeline functionality of hs.Image. The camera object transforms into a pipeline element that then captures images, when the pipeline is executed.

The process is more easily explained with an example:

python

img = cam.to_hs_image()

# capture 10 frames
array_img = img[:10, :, :].resolve()

# build pipeline and then capture 10 frames
pl = img.ensure_dtype(hs.float32) / 254
array_img = pl[:10, :, :].resolve()

# trying to resolve the raw image results in an error because the stream is infinite
try:
    array_img = img.resolve()
except ValueError as e:
    print(e)

c/c++

// Get image object
hv_image_t* img;
hv_camera_get_image(cam, &img);

// Capture 10 frames
hv_slice_desc_t slice_first10 = {
	.lines = hv_slice_elem_new_range(0, 10, 1),
	.samples = hv_slice_elem_new_range(0, -1, 1),
	.bands = hv_slice_elem_new_range(0, -1, 1)
};
hv_image_t* first10 = hv_image_slice(img, slice_first10);
hv_image_t* array_img;
hv_image_resolve(first10, &array_img);

hv_image_free(&array_img);
hv_image_free(&first10);


// Build pipeline and then capture 10 frames

// Create calculation (convert to float32 and divide by 254)
hv_image_t* float_img;
hv_image_ensure_dtype(img, HV_DTYPE_F32, &float_img);
hv_image_t* img_result = hv_image_div_scalar_f32(float_img, 254);

// Slice and resolve the image
hv_image_t* first10_pipeline = hv_image_slice(img_result, slice_first10);
hv_image_t* array_img_pipeline;
hv_image_resolve(first10_pipeline, &array_img_pipeline);

hv_image_free(&first10_pipeline);
hv_image_free(&array_img_pipeline);


// Trying to resolve the raw image results in an error because the
// stream is infinite
hv_image_t* img_infinite;
int err = hv_image_resolve(img, &img_infinite);

// The error value will be set
assert(err != 0);
// and the image pointer will still be NULL
assert(img_infinite == NULL);

To capture images, you call a function on the image object that executes the pipeline. It doesn’t have to be hs.Image.resolve(), but could just as well be hs.Image.to_numpy().

Note

A future version of the SDK will support streaming images from a pipeline. This feature will make it easy to integrate a live camera stream with any processing system or library.

As you can see in the example, it is necessary to, at some point, slice the streaming dimension of the pipeline. Since the Hypervision camera is a pushbroom camera, the streaming dimension is the lines, i.e., the first slicing dimension.

Hyperspectral camera settings#

There is no classical cropping functionality. Instead, you set the desired bands using the hs.control.Camera.set_bands() method.

python

# Set bands
bands = cam.set_bands([
    (0, 50),
    (152, 180)
])
print(bands)

# Set horizontal cropping
crop = cam.set_horizontal_crop(200, 500)
print(crop)

c/c++

// Set bands

hv_band_list_t* bands;
hv_band_list_new(&bands);

// The API allows for setting a step value, but this is currently ignored.
hv_band_list_append(bands, 0, 50, -1);
hv_band_list_append(bands, 152, 180, -1);

hv_band_list_t* out_bands;
hv_camera_set_bands(cam, bands, &out_bands);

const char* str = hv_band_list_debug_fmt(out_bands);
printf("%s\n", str);
free(str);
hv_band_list_free(&out_bands);


// Set horizontal cropping

hv_horizontal_crop_t* out_crop;
hv_camera_set_horizontal_crop(camera, 200, 500, out_crop);

const char* str2 = hv_horizontal_crop_debug_fmt(out_crop);
printf("%s\n", str2);
free(str2);
hv_horizontal_crop_free(&out_crop);

Warning

The camera interface currently only supports band ranges since it is tied to the underlying sensor functionality. The Hypervision 1700 sensors support up to 8 distinct ranges. Additionally, the ranges y-values will always be clamped to numbers divisible by 4 - this is a sensor limitation. The camera interface will always extend a provided range to include the selection. For example, the range (5, 10) will result in a returned band selection of (4, 11).

Important

In the future, we will implement the ability to select arbitrary bands.

Streaming individual frames #

It is possible to stream raw frames from the camera by calling hs.control.Camera.stream(). The stream can be used as an iterator and also supports the context manager protocol to automatically stop streaming.

from qtec_hv_sdk import control

cam = control.Camera("10.100.10.100")

# Use as an iterator with the context manager protocol
with cam.stream() as stream:
    for i in range(100):
        meta, frame = next(stream) # Get the next frame
        print(frame.shape)

# Manual usage
stream = cam.stream() # Start streaming
meta, frame = stream.get_frame() # Get a single frame
stream.stop() # Stop the stream

stream() returns only once the camera has accepted the connection and is streaming, so a camera that cannot start raises there rather than yielding an empty stream. A camera that has just stopped a stream needs a moment to release the video device before it will accept a new one, which stream() waits out for you; back-to-back stop()/stream() pairs therefore work without a delay in between, at the cost of the call occasionally taking a second or so to return.

Buffering and backpressure #

Frames arrive faster than most consumers process them, so a camera keeps a local buffer of capacity frames. Buffering is configured on the camera, not per stream, through the stream_options attribute. The configuration therefore applies to every stream that camera starts - including the ones driven by an Image pipeline, which have no other way to pass options.

from qtec_hv_sdk import control

cam = control.Camera("10.100.10.100")
cam.stream_options.capacity = 64

# Guarantee that no frame is lost on this machine.
with cam.stream(n_frames=1000) as stream:
    for meta, frame in stream:
        slow_processing(frame)

assert stream.dropped() == 0

The attribute is a live view of the camera’s configuration, so assigning a field takes effect immediately. To change several at once, assign a whole StreamOptions; its constructor takes each setting as a keyword-only argument and leaves omitted ones at their default:

cam.stream_options = control.StreamOptions(capacity=64, on_full="block")

There are four settings:

buffered: Deliver frames oldest-first (FIFO). Defaults to True, which is what you want for capture. When False, each read returns the newest frame and discards the backlog, which suits live preview. The discarded frames count towards dropped().
capacity: Number of frames the buffer holds. Defaults to 20. Must be at least 1; 0 raises ValueError.
wait_timeout: How long a read waits for a frame before raising TimeoutError, which usually means the camera has stalled. Must exceed the longest expected gap between frames. Iterating stops the stream on a timeout; get_frame leaves it running so the call can be retried. Defaults to 1 s.
on_full: What to do when the buffer is full, either "block" or "evict". "block" (the default) stalls the camera until the consumer frees a slot, so dropped() stays at 0. "evict" discards the oldest buffered frame so the camera never stalls, at the cost of silently losing frames.

So a live-preview camera, which wants the most recent frame rather than a complete record, is configured like this:

cam.stream_options = control.StreamOptions(buffered=False, capacity=2, on_full="evict")

Warning

on_full="block" only guarantees that nothing is lost after the frame reaches the host. If the consumer stays behind for long enough, the backpressure propagates to the camera, which then drops frames on its own. Those show up as meta.dropped on later frames - see below. Give the buffer enough capacity to absorb bursts, and keep the per-frame work short.

Checking for dropped frames#

All streaming functionalities forward frame metadata. For frames captured from a camera, this includes a timestamp and a dropped flag. These can be used to handle uneven frame pacing and missing frames.

Note

meta.dropped marks a frame the camera failed to deliver to the host. It is unrelated to dropped(), which counts frames discarded locally because the consumer fell behind.

A frame the camera failed to deliver is replaced by a black frame with meta.dropped set, so the stream stays gapless and a pipeline still sees the line it expected. These stand-ins carry the meta.seq of the line that went missing, but the meta.timestamp of the frame that follows the gap. So meta.seq is contiguous across a camera-side loss, and a jump in it means frames were discarded locally - see Buffering and backpressure.

When using pipelines, this can be used in the @hs.util.operation decorator as shown below:

import qtec_hv_sdk as hs
from qtec_hv_sdk import control
import numpy as np
import logging

logger = logging.get_logger("main")

cam = control.Camera("10.100.10.100")

@operation(hs.bil)
def op(meta: hs.FrameMeta, frame: np.ndarray) -> np.ndarray:
    if meta.dropped:
        logger.warning(f"Dropped frame {meta.seq} at time {meta.timestamp}")

    # Do some processing here
    # ...

    return frame

out_img = op(cam)

# Any method calls that fetches data from the pipeline will now issue warnings if a dropped frame is encountered.
out_img.array_plane(250, hs.bands)
out_img.to_numpy()

When using a plain camera CameraStream object or a pipeline Stream object, the metadata is passed with each new frame, making custom processing simple:

import qtec_hv_sdk as hs
from qtec_hv_sdk import control
import numpy as np
import logging

logger = logging.get_logger("main")

cam = control.Camera("10.100.10.100")

with cam.stream() as stream:
    for i in range(100):
        meta, frame = next(stream) # Get the next frame
        if meta.dropped:
            print(f"Dropped frame {meta.seq} at time {meta.timestamp}")
            # Handle dropped frame