Skip to main content

Sensor Specific Functions

Some of the sensors supported in the qtec C-series platform have some specific functionality which is listed and explained below.

Sony IMX​

Refers to the IMX42x/25x series used in the RGB AI Cameras and IMX99x series used in the Hypervision 1700 HSI Camera.

Light Sensitivity​

The Sony IMX42x/25x/99x families have 3 available bit modes (8, 10 and 12 bit). The internal Vsat is adjusted to 1/4 when in 8 bit mode compared to the 10/12 bit modes. Therefore the light sensitivity is 4x higher in 8 bit mode when compared to the other modes, which makes the 8 bit mode the recommended mode for applications where the available light intensity is limited. For example fast moving objects or hyperspectral imaging.

Gpixel GSENSE​

Refers to the GSENSE2020 used in the Hypervision 1000 HSI Camera.

Shutter modes​

The Gpixel GSENSE2020 is a quite unique sensor which offers the possibility of choosing between rolling shutter and global shutter. Each of the shutter modes offers some specific features and advantages: for example it is possible to adjust the exposure time for each individual line when using the rolling shutter mode and to select individually which lines should be captured.

Rolling shutter vs Global shutter​

The shutter type refers to the image capture process in the sensor: cameras can either have a rolling shutter or a global shutter.

In a rolling shutter setup each sensor line is exposed and read-out sequentially (in waves), while with a global shutter all pixels are exposed simultaneously. Global shutter sensors require therefore more internal memory (to store all the pixel values until they can be read out). Rolling shutter sensors are normally simpler and therefore cheaper and they also tend to have less noise and bigger sensitivity. But because of the sequential nature of the rolling shutter it is not recommended for fast moving objects since it can result in severe distortions in the image (see below).

Rolling shutter distortion effect Rolling shutter effect (cmglee, CC BY-SA 3.0, via Wikimedia Commons)

Qtec focuses on industrial cameras which normally involve fast moving objects, therefore most of the sensors that qtec works with use global shutter, with the exception of the GSENSE2020 which is a scientific CMOS with the option to select the shutter mode.

Shutter mode selection​

The different shutter modes and their specific features can be selected by choosing the appropriate bitstream from the qtec-c-paris series in conjuction with toggling the global shutter V4L2 control:

tip

Note that by default both bitstreams start in Rolling Shutter mode. Switching between the two shutter modes can be achieved by toggling the global shutter V4L2 control.

info

pisces-paris is the default bitstream for the GSENSE2020 and is what is used in the Hypervision 1000 HSI Camera because of it's ability to control the exposure individually for each line.

Rolling shutter mode​

The rolling shutter on the GSENSE2020 is fully programmable. This means that the exposure time of lines can be selected individually. This can be achieved by programming the sensor via the Line Exposure Sequence V4L2 control, which is effectively an array of 20482048 values, one for each line.

While global shutter sensors normally allow for a single continuous cropping area (or in some cases up to 8 areas), the fully programmable rolling shutter makes it possible to select individually which lines should be captured (as well in what order). In this way the frame capture process and speed can be greatly optimized (the maximum framerate is directly proportional to the amount of lines captured).

If a single cropping area is desired the standard V4L2 crop/selection interface can be used, just like for all other sensors supported by qtec. However if selection of individual lines is desired, then the Line Order V4L2 control should be used. The Line Order control is an array with the same size as the height of the cropping area (set via the standard V4L2 crop/selection interface). It's contents reflect the position of the lines to be captured, in the desired capture order.

Setting the exposure per line​
Bitstream specific

Only available in the pisces-paris bitstream

This is a small example showing how to change the exposure per line in rolling shutter mode, with qamlib.

import qamlib

cam = qamlib.Camera()

# Get the current exposure sequence
exposures = cam.get_ext_control("line exposure sequence").value

# Set the exposure for all lines to 5000
exposures[:] = 5000

# Set a different exposure on the bottom half of the lines
n = int(len(exposures)/2)
exposures[n:] = 2000

# Set the new exposure sequence
try:
        cam.set_ext_control("line exposure sequence", qamlib.ArrayControlValue(exposures))
except V4L2Exception as e:
        print(e)
tip

quark-ctl can also be used as a quick way to set all lines to the same exposure value:

quark-ctl exposure-sequence --exposure 1000
exposure time limitations

Because of limited RAM in the FPGA, we have a limit on how complex/long the exposures can be. E.g. for a full frame the maximum exposure time is ≈68000us\approx 68000 us.

The requested change in the exposure sequence will be rejected if the requested parameters don't fit within the available memory. In this case qamlib will throw an V4L2Exception and an error will be visible in dmesg:

V4L2Exception: Failed to set extended control : (22) Invalid argument

dmesg: gsense-0: Line exposure sequence is too long or complex

framerate

Changing Line Exposure Sequence will adjust the framerate if necessary (if the current fps is higher than the new maximum fps)

Changing the selected lines​
Bitstream specific

Only available in the pisces-paris bitstream

This is a small example showing how to change the line order in rolling shutter mode, with qamlib.

import qamlib

cam = qamlib.Camera()

# Start by adjusting the crop region to the desired size
cam.set_crop(left=0, top=0, width=2048, height=100)

# Get the current line order array
line_order = cam.get_ext_control("line order").value

# Adjust the desired lines:
# For example use every 5th line starting from 50
line = 50
for i, pos in enumerate(line_order):
        line_order[i] = line
        line += 5

# Set the new line order
try:
        cam.set_ext_control("line order", qamlib.ArrayControlValue(line_order))
except V4L2Exception as e:
        print(e)
framerate

Changing Line Order will adjust the framerate if necessary (if the current fps is higher than the new max fps)

Global Shutter mode​

With the GSENSE2020 in global shutter mode, it acts a lot more like all the other supported sensors, except that for each image two frames are generated. The first is a reference frame that needs to be subtracted from the second one in order to generate the actual image.

Enabling Global shutter

Note that by default both bitstreams start in Rolling Shutter mode. In order to switch to the Global Shutter mode it is necessary to enable the global shutter V4L2 control.

Bitstream differences​

Note that both the legacy (capricorn-paris) and the default (pisces-paris) bitstreams support global shutter mode. There are however some small differences between their behaviour because of different implementations.

exposure time limitations

When using the pisces-paris bitstream, because of limited RAM in the FPGA, we have a limit on how complex/long the exposures can be. For example, for a full frame the maximum exposure time is ≈16000us\approx 16000 us.

This limit is visible in the max parameter of the exposure time absolute V4L2 control, and it is updated when cropping is applied. Values higher than the maximum get clamped (as happens with all normal V4L2 controls).

tip

This limitation doesn't exist in the capricorn-paris because it is implemented in a different way. However, that bitstream can present some image "artifacts" in some situations (see below), therefore pisces-paris is the reccomended (and default) bitstream.

Image "artifacts"

When using the capricorn-paris legacy bitstream in global shutter mode there is no limitation on the maximum exposure time, however some combinations of framerate and exposure time can result in "artifacts" in the image.

An intensity change might occur in the image when changing the framerate while keeping the exposure time fixed (which should normally not occur).

Moreover, a horizontal "line" might be visible separating areas with higher and lower intensities. Changing the exposure time might cause the line to move up and down. The effect seems to happen only in the "real image", and not in the reference frame.

info

This behavior has been generally seen with framerate values under 40fps and exposure values around 2-5ms.

Framerate values above 40fps do not seem to produce issues.

tip

The global shutter in the pisces-paris bitstream does not have the same issue, but as mentioned above, the maximum exposure time there is limited to 16ms for a full frame.

Frame read-out in global shutter mode​

This is a small example (using qamlib) of how to do the frame read-out from the GSENSE2020 in global shutter mode and to combine the 2 frames into the resulting image.

import qamlib

try:
        # throw exception on dropped frames
        cam = qamlib.Camera(overflow_exception=True)
except Exception as e:
        print(e)
        exit(-1)

# Set to global shutter (default is rolling shutter)
cam.set_control("global shutter", 1)

# Frame capture
i2 = 0
try:
        with cam:
                while True:
                        # Using 'buffered=True' will return the next frame in the queue
                        # and throw a 'DroppedFrameException' if the queue gets filled up
                        # set 'overflow_exception=False' in the Camera constructor
                        # if you want to disable this exception, but then it will be necessary
                        # to check for dropped frames using the sequence nr from the metadata

                        ref_meta, ref_frame = cam.get_frame(timeout=1, buffered=True)
                        meta, frame = cam.get_frame(timeout=1, buffered=True)

                        # the GSENSE2020 produces 2 frames "per frame" when on global shutter
                        # the 1st is a reference frame that should be subtracted from the 2nd
                        # in order to create the resulting image
                        res_frame = frame - ref_frame

                        # all this sequence nr checks shouldn't be necessary
                        # since we will get a 'DroppedFrameException' if frames are dropped
                        # but are present to illustrate how to check for dropped frames under other circumstances

                        # reference frames must have even seq nr
                        if ref_meta.sequence % 2 != 0:
                                print(f"cam: reference frame out of sequence")
                                break

                        # secondary frames must have odd seq nr
                        if meta.sequence % 2 == 0:
                                print(f"cam: secondary frame out of sequence")
                                break

                        # both frames seq nrs must be sequential
                        n3 = meta.sequence - ref_meta.sequence
                        if n3 != 1:
                                print(f"cam skipped {n3} frames")
                                break

                        n2 = ref_meta.sequence - i2
                        if n2 > 0:
                                print(f"cam skipped {n2} frames")
                                break
                        i2 = meta.sequence + 1
except Exception as e:
        template = "An exception of type {0} occurred. Arguments:\n{1!r}"
        message = template.format(type(e).__name__, e.args)
        print(message)
HG and LG images​

The GSENSE2020 sensor has functionality available for creating HDR images, and therefore produces two images per frame: a high gain (HG) and a low gain (LG). By default only the 'high gain' image is returned, however, it is possible to retrieve the 'low gain' image instead by enabling the Rolling Shutter LG Image V4L2 control.

Gain values

Using the 'high gain' image with low values (under 1x) in the Gain control can result in wrong pixel saturation values (under 255). In that case enable the 'low gain' image instead.

Invalid frames on stream start​

The first X frames received from the GSENSE when it starts streaming are unstable. Some are very dark and some are very bright and some are in between. This is related to the sensor needing time to stabilize.

A delay has been added in the driver in order to address this issue, so that a certain amount of frames will be discarted (not passed to the user). Use the start up delay V4L2 control to adjust the delay if necessary.