From robotics to webcams, the number and variety of applications reliant on image data for their safe operation or core functionality is continuing to increase. For the sensors themselves, there are a number of challenges created by modern applications that must be met if image accuracy is to be guaranteed.

In implementations such as vehicles or security cameras, sensors need high performance in dark or high dynamic scenes, as well as the precise tracking of moving objects and the ability to accurately function in the presence of LED lighting.

Here, advanced techniques, such as HDR on-chip intelligent linearization, are required to ensure accurate images are captured even in the most challenging of scenes.

Challenges in Performing HDR Merging

The process of combining images is known as ‘linearization’ and, with modern high-resolution sensors, this can require significant processing power and memory.

As the images are captured at slightly different times, moving objects or flickering lighting can cause issues with image quality, such as artifacts in the final combined image. However, by applying intelligence to the linearization process, these issues can be mitigated, or overcome entirely. One issue in linearization occurs during the transition from long to short-exposure images. Due to a change in signal-to-noise ratio between the images, there can be a line on the final image reflecting an abrupt change.

Figure 1: Intelligent linearization can mitigate issues with transitions in HDR processing

Intelligent linearization can calculate a weighted signal based upon the signal levels within the long and short exposures which smooths the transition, improving the final image.

Motion is another challenging issue that is a common occurrence in applications such as surveillance, body cameras and industrial robotics. As the images are captured at different times the moving object will be in slightly different places in each image.

The situation can be simulated by using a rotating fan. The  AR0822 image sensor from onsemi can sense the amount of motion between the long and short exposure. Using this information, a combined signal is produced that eliminates color artifacts due to motion.

Figure 2: The effects of motion can be mitigated using intelligent linearization

Zooming in on the result without intelligent linearization a line can be seen – this is removed by intelligent linearization.

One of the most common challenges for image sensing is the popularity of LED lighting – whether there is an LED light in the scene or it is being used to illuminate the scene. As the lighting is not constant and can change significantly between the long and short image, this may appear as a change in the hue of an image – or a LED may appear as "off" while a human eye can clearly see it is "on".

Figure 3: LED lighting can be challenging for image sensors without intelligent linearization

Again, by sensing the difference in the linearized signal between captures, a combined signal can be generated that mitigates the issue of LEDs appearing to be "off" and any resulting shifts in color.

System Level Advantages

As image sensing systems become more sophisticated, employing multiple high-resolution cameras, the level of processing and data throughput is rising dramatically putting a strain on an external image signal processor (ISP) or system-on-chip (SoC).

However, using techniques such as eHDR where the linearization is moved from the ISP to an intelligent image sensor, such as the AR0822,  the bandwidth requirement is significantly reduced as only the final image is sent to the ISP. This also reduces the computation required within the ISP, allowing for higher frame rates without a significant increase in bandwidth.

With advanced intelligent linearization algorithms built-in to their image sensors, onsemi can mitigate or eliminate any artifacts created while combining multi-exposure HDR images. As a result, it is far easier to integrate one or more sensors onto an external ISP/SoC.