Low-Latency and Low-Power Image Predistortion

Classical head-up displays require a flat transparent surface onto which an image is projected. Such a flat surface only requires simple key-stone correction for displaying a clean image. In newer architectures the windscreen itself is to be used as the projection target.

Without correction, a clean, projected image might look like this on the windscreen:

With the three-dimensional surface of the windscreen, a more sophisticated arbitrary predistortion is required beyond keystone or pin-cushion correction. This is needed considering the physical shape of the windscreen, the position of the projector plus the possible head position of the driver.

Such a predistortion requires a transformation or warping of the image where every single pixel is considered in its required position.

Microchip has partnered with TES Electronic Solutions GmbH (TES) for an implementation of such a warping engine. The warping engine is based on the following simple concept:

For every output-pixel the sub-pixel-exact position of the required source pixel is read from a Look Up Table (LUT), the required source pixels surrounding this position are read from the source image and the output pixel is computed by weighting these source pixels according to the sub-pixel position by bilinear filtering.

The LUT is highly compressed by TES’ proprietary Adaptive Delta Encoding algorithm allowing compression factors from 40- up to 60-times depending on configurable maximum error values for the sub-pixel coordinates.

Based on this approach, arbitrary shapes can be implemented for warping. The warping engine creates the inverse curvature of the three-dimensional image which needs to be projected.

A generic setup of the warping IP is as follows:

Display data enters the PolarFire® FPGA, is handled by the warping engine and is then transferred to the projector image. Local dimming can be performed as part of the warping process; however, this requires an add-on feature to the standard warping engine.

By default, the warping process works based on temporary storage of the incoming video frames in DDR memory and adds a latency of four image frames:

This approach allows full flexibility of warping and rotating the image as well as processing the image size.

In situations where external DDR memory is not available or should be avoided, the warping process can also be adapted to work without external DDR memory but only on partial input frames and a streaming output interface:

The compressed warping LUT is stored inside the FPGA and several lines of the incoming video data are buffered within the FPGA. The internal RAM requirement depends on the width of the incoming data frame and the number of lines that need to be stored, the latter depending on the maximum image distortion.

Processing the output pixels can start when sufficient input lines are in memory. The output pixels are created based on available data from the stored input image lines.

Frame n-2:

Frame n-1:

Frame n:

The output of the warping will then be available line by line via a streaming interface directly to the display controller:

This approach optimizes for the lowest latency when not using external DDR memory; however, it reduces the flexibility on the warping capabilities. For relatively even surfaces where only a little warping is required, significant savings can be achieved by reducing memory requirements. A positive side-effect is reduced latency. If greater image distortions are required, such as for fish-eye lens corrections, the approach is still applicable but with less memory savings and increased latency.

As an additional benefit, the removal of the DDR memory also removes the risk of frozen frames caused by the memory, which can be a safety concern in the system.

Local dimming is a feature often required in projection applications. Local dimming counteracts image gradients caused by the illumination heatmap of LED matrices used for backlighting:

The pixels of the image are individually adjusted in color and brightness so that the illumination heatmap of the backlight matrix produces an image with the intended color and brightness.

As every pixel of the output image is touched by the warping engine, the local dimming can be applied at the same time without adding any additional latency. The granularity of the local dimming down to individual pixels depends on the memory available for storing the dimming table.

In cases where only local dimming is needed without warping, the by-pixel approach can also be utilized in stand-alone mode. For this setup, only memory for the heatmap table with information about the dimming-areas is required. The dimming is implemented in streaming mode, without storing full image frames. In this approach, the latency for the dimming is kept to only two image lines and the risk of frozen images in external memories is removed.

Based on the flexibility of the FPGA, different communication interfaces can be supported. This allows for re-use of the design across multiple platforms a customer may have. When using an FPGA in a design, a moderately complex FPGA is sufficient to run the warping engine so a very space-efficient small package can be used for simple and cost-effective routing on the PCB. Additionally, using an FPGA in challenging thermal environments is possible with minimal effort as Microchip’s PolarFire FPGAs, for example, are the industry’s lowest power devices at mid-range densities.

For more detailed information on the generic warping solution please visit: Warping Engine

For detailed information on PolarFire FPGAs please visit: PolarFire SoC FPGA Family

Additionally, you can contact:

- Martin Kellermann, Manager Marketing, Microchip

Martin.Kellermann@microchip.com

- Thomas Hase, Senior Business Development Manager, TES Electronic Solutions GmbH

Thomas.Hase@tes-dst.com