Camera Integration

The SDK provides multiple ways to interact with the Qtechnology Hypervision line of hyperspectral cameras. The pipeline API integrates directly with the cameras and as a result makes it easy to create complex processing pipelines without having to worry about handling individual frames. Alternatively, the cameras can be operated like regular cameras using a stream of two-dimensional frames.

Warning

The camera functionality is in its early stages of development and may change significantly.

Creating a camera object

The base type for camera interoperability is hsi.HSCamera and it may represent either a real hyperspectral camera or an image-based mock camera. The latter is particularly useful for testing systems without having to deal with the complexities of controlling the physical camera.

Let us look at an example of how to create a new camera object and perform some basic operations on it:

cam = hsi.HSCamera("<camera_ip>")
# Alternatively, you can create a simulated camera from an existing HSImage
cam = hsi.HSCamera(<img_object>)

# Printing properties of the camera is easy
print(cam.get_wavelengths())
print(cam.get_framerate())

{
	// Create the camera object
	hv_camera_t* cam;
	hv_camera_new("192.168.2.229", &cam);

	// Alternatively, the camera can be initialized from a hv_image_t object.
	// hv_camera_from_hs_image(<img>, &cam);

	// Get the wavelengths
	slice_boxed_float_t* wavelengths;
	hv_camera_get_wavelengths(cam, &wavelengths);

	// Print the wavelength values
	printf("Wavelengths:\n");
	for (size_t i = 0; i < wavelengths->len; i++) {
		printf("%fnm\n", wavelengths->ptr[i]);
	}

	// Print the framerate
	const float fps = hv_camera_get_framerate(cam);

The camera object supports setting many common camera settings, including exposure time, framerate, and binning.

Tip

We provide a guide on how to capture high-quality hyperspectral images on our main documentation site.

Note

The simulated camera is useful for testing pipelines that are meant to run on live data.

The simulated camera allows setting/reading the same settings as the real camera, but many settings don’t have any impact on the image output. Only the band-selection currently affects the output image.

The following additional settings will be implemented soon: - Horizontal cropping - Binning - Bit-mode

Image capture

The hsi.HSCamera object integrates directly with the pipeline functionality of hsi.HSImage . The camera object transforms into a pipeline element that then captures images, when the pipeline is executed.

The process is more easily explained with an example:

img = cam.to_hs_image()

# capture 10 frames
array_img = img[:10, :, :].resolve()

# build pipeline and then capture 10 frames
pl = img.ensure_dtype(hsi.float32) / 254
array_img = pl[:10, :, :].resolve()

# trying to resolve the raw image results in an error because the stream is infinite
try:
    array_img = img.resolve()
except ValueError as e:
    print(e)

// Get image object
hv_hs_image_t* img;
hv_camera_get_hs_image(cam, &img);

// Capture 10 frames
hv_slice_desc_t slice_first10 = {
	.lines = hv_slice_elem_new_range(0, 10, 1),
	.samples = hv_slice_elem_new_range(0, -1, 1),
	.bands = hv_slice_elem_new_range(0, -1, 1)
};
hv_hs_image_t* first10 = hv_hs_image_slice(img, slice_first10);
hv_hs_image_t* array_img;
hv_hs_image_resolve(first10, &array_img);

hv_hs_image_free(&array_img);
hv_hs_image_free(&first10);


// Build pipeline and then capture 10 frames

// Create calculation (convert to float32 and divide by 254)
hv_hs_image_t* float_img;
hv_hs_image_ensure_dtype(img, HV_DTYPE_F32, &float_img);
hv_hs_image_t* img_result = hv_hs_image_div_scalar_f32(float_img, 254);

// Slice and resolve the image
hv_hs_image_t* first10_pipeline = hv_hs_image_slice(img_result, slice_first10);
hv_hs_image_t* array_img_pipeline;
hv_hs_image_resolve(first10_pipeline, &array_img_pipeline);

hv_hs_image_free(&first10_pipeline);
hv_hs_image_free(&array_img_pipeline);


// Trying to resolve the raw image results in an error because the
// stream is infinite
hv_hs_image_t* img_infinite;
int err = hv_hs_image_resolve(img, &img_infinite);

// The error value will be set
assert(err != 0);
// and the image pointer will still be NULL
assert(img_infinite == NULL);

To capture images, you call a function on the image object that executes the pipeline. It doesn’t have to be hsi.HSImage.resolve() , but could just as well be hsi.HSImage.to_numpy() .

Note

A future version of the SDK will support streaming images from a pipeline. This feature will make it easy to integrate a live camera stream with any processing system or library.

As you can see in the example, it is necessary to, at some point, slice the streaming dimension of the pipeline. Since the Hypervision camera is a pushbroom camera, the streaming dimension is the lines, i.e., the first slicing dimension.

Hyperspectral camera settings

There is no classical cropping functionality. Instead, you set the desired bands using the hsi.HSCamera.set_bands() method.

# Set bands
bands = cam.set_bands([
    (0, 50),
    (152, 180)
])
print(bands)

# Set horizontal cropping
crop = cam.set_horizontal_crop(200, 500)
print(crop)

// Set bands

hv_band_list_t* bands;
hv_band_list_new(&bands);

// The API allows for setting a step value, but this is currently ignored.
hv_band_list_append(bands, 0, 50, -1);
hv_band_list_append(bands, 152, 180, -1);

hv_band_list_t* out_bands;
hv_camera_set_bands(cam, bands, &out_bands);

const char* str = hv_band_list_debug_fmt(out_bands);
printf("%s\n", str);
free(str);
hv_band_list_free(&out_bands);


// Set horizontal cropping

hv_horizontal_crop_t* out_crop;
hv_camera_set_horizontal_crop(camera, 200, 500, out_crop);

const char* str2 = hv_horizontal_crop_debug_fmt(out_crop);
printf("%s\n", str2);
free(str2);
hv_horizontal_crop_free(&out_crop);

Warning

The camera interface currently only supports band ranges since it is tied to the underlying sensor functionality. The Hypervision 1700 sensors support up to 8 distinct ranges. Additionally, the ranges y-values will always be clamped to numbers divisible by 4 - this is a sensor limitation. The camera interface will always extend a provided range to include the selection. For example, the range (5, 10) will result in a returned band selection of (4, 11).

Important

In the future, we will implement the ability to select arbitrary bands.