Processing Bayer pixels

Scripts on this page use dcraw and exiftool. dcraw has been slightly hacked by me; see dcraw and WB. The IM version should be Q16 or larger. It does not need to be HDRI.

References

Introduction

When processing raw camera images, we may want to operate on the raw camera values before they are de-Bayered (aka de-mosaiced). For example, when a pixel in a camera sensor records noise, the de-Bayer process will spread that noise to other pixels. Hence de-noising is easier and more effective if we apply it to raw pixels, before de-Bayering.

Inserting our own ImageMagick processing into a program such as dcraw is a non-trivial task. Instead, a simple process enables us to extract the raw pixel data for each channel, then process that data however we want, then put that data into a DNG file that can be de-Bayered in the usual way.

On this page, a Bayer image contains one channel. Each pixel represents the intensity of light received by that photo sensor, either red or green or blue, in a repeating 2x2 pattern known as a Color Filter Array (CFA). We assume the pattern is:

With this arrangement, the camera will sample green at twice the spatial frequency of red and blue, which mimics the human visual system's greater sensitivity to green.

An image viewer will show a grayscale image with pixels varying in a chequer-board pattern.

Some camera-raw images have a different repeating pattern, perhaps larger than 2x2. We ignore those.

A TIFF file contains one or more images, and metadata (known as "tags") for each image. The TIFF format was originally defined by Aldus, and now by Adobe. The extension is often ".TIFF" or ".TIF", but other extensions are used.

A NEF file is a TIFF file made by a Nikon camera. It contains a Bayer image (the "raw" camera image) and some JPEG versions of that image that we ignore. Some metadata is specific or proprietary to Nikon. The extension is usually ".NEF".

A DNG ("Digital Negative") file is a TIFF file with images and metadata that conform to an open standard, defined by Adobe. However, it may also contain metadata that is proprietary to a manufacturer, and it may not contain all proprietary metadata converted to the open format. It may also contain an exact copy of the raw file it was derived from. In theory, it provides a manufacturer-neutral format for raw images. Most raw processing software can read DNG files. The extension is usually ".DNG".

In addition, we do a final transformation from a DNG to an sRGB image using dcraw so we can see what we have done.

We also use ImageMagick scripts to transform between a Bayer image and individual images that represent the raw red, green and blue channels:

We can do any processing we want on the raw R, G0, G1 and B images. They can be stored in any image format we want such as MIFF or TIFF or PNG.

Most processing is with linear values, proportional to the intensity of light received by the sensor pixels.

Simple dcraw process

For comparison purposes, we make an sRGB image using dcraw in the conventional way.

NEF to Bayer

-W tells dcraw not to auto-brighten the result. The camera records 14 bits/channel/pixel, so the maximum possible value that can be recorded is 2¹⁴-1 = 16383.

The maximum possible has actually been recorded, suggesting the sensor saturated when the photograph was taken.

For viewing, we need to divide by 16383 and multiply by QuantumRange of this version of IM, then apply the sRGB transfer curve to the linear values.

We can multiply the image by QMULT and then view the Bayer image with any image viewer. For the web, we make a small version and a crop that is scaled up.

In the scaled-up crop we clearly see the chequer-board pattern that is characteristic of a Bayer image.

Bayer to DNG

We use exiftool to write a DNG file containing the Bayer image. This changes metadata, while simply copying the image without changing pixel values. The Bayer file must contain one grayscale image, with no alpha channel, and no compression. We do not write any JPEG versions of the image to the DNG.

DNG to sRGB

We have made a DNG file. As a check, we use dcraw to convert it to a TIFF in sRGB colorspace, then IM to make versions for the web.

Split Bayer into channel images

A Bayer file is not a convenient format for image processing. The script gray2rggb.bat reads a Bayer image and makes four images, representing the R, G0, G1 and B components.

We show each component. As before, we show a resized version and an enlarged crop.

This is sufficient to check that we haven't swapped channels. The colours are inaccurate mostly because we have not applied the camera matrix. We can do that:

Combine channel images into Bayer image

This is the inverse of the previous process. The script rggb2gray.bat reads four images, representing the R, G0, G1 and B components, and combines them to make a Bayer image.

As expected, all corresponding pixels are identical. As shown above, we can then convert the Bayer image into a DNG file.

The script works by replicating each input pixel into four pixels (using "-scale"), and using a mask to make three of those pixels transparent. The command does a lot of processing. A custom program or process module could simply read the four input files, then walk though each output pixel, picking the value from the appropriate input pixel. This would be much faster.

So, we can start with a camera raw file, make a Bayer image, split that into four channels, then convert those back to a Bayer image which we can convert to a DNG file which we can read with any raw processor to convert to sRGB or whatever we want. But why would we want to? What can we do with this?

Application: denoising

This noisy image is from a Nikon D800 with 85/1.8 lens, ISO "H2" (256000), auto-exposure with -2 stops bias to reduce burnt highlights, 1/320s f/2.8.

Notice the heavy noise in the 1:1 crop. The noise is clumped, in areas a handful of pixels wide or high or both.

Create a Bayer image with no auto-brighten, but multiplying to stretch from 14 bits to 16 bits. We will show just the 1:1 crop of the red channel.

This noise is different: it seems to affect each pixel independently of other pixels, and any clumping seems coincidental.

A simple denoising filter smooths outliers by ensuring every pixel is within the local minimum and maximum of a small blur. See Limit min-max. The full effect on this image makes the flesh look unpleasantly blotchy, so we apply only 50% of the effect.

The difference between these images shows where noise is removed. The difference is black where nothing is removed, and lighter where more noise is removed. Ideally, the difference will be essentially random, and will not show any "signal", any detail that we want to retain.

This is generally lighter where the inputs are darker, which means that more noise is removed from shadow areas, which is reasonable. The denoising process also removes a small amount of detail in the near-vertical specular highlight on the left. I can live with that. If we wanted, we could mask the effect with an edge-detector.

Process the DNG, de-Bayering and converting to sRGB, and show the 1:1 crop of the colour result:

Application: de-Bayering artificial images

Make a DNG. We declare the camera "Make" to be one that dcraw doesn't know about, so it will apply the identity camera matrix. We use a linear version of the TIFF so raw converters can convert it back to sRGB.

We use dcraw to convert to a conventional image. "-r 1 1 1 1" overides any multipliers set in the file metadata.

As the detail approaches the size of the Bayer pattern, the interpolation introduces false colours. This also happens at the edge of the gray ring, though not so obviously. The edge of the gray ring has become less smooth, more broken.

Application: removing hot pixels

In some conditions (low light, high ISO, longish exposures, hot sensor), cameras sometimes create "hot" pixels. These can be regarded as noise and treated as above.

A red pixel on the sensor is "hot": far brighter than adjacent pixels of any colour. The de-Bayering has spread this brightness to other pixels. If the hot pixel represented green instead of red, the effect would not be so obvious on this image.

In this 3x3 crop, we see that the central pixel is more than twenty times the value of any other pixel. We could use an outlier detection and fix. In practice, I so rarely experience hot pixels that I don't treat them differently to other noise.

If we want to remove hot pixels without denoising the rest of the image, see Hot pixels.

Application: evaluating de-Bayer algorithms

A Bayer image contains only one channel in each pixel. The de-Bayering process interpolates the other two channels at each pixel. We might say it "invents" two-thirds of the data in the final image. There is usually no ground-truth version of the image, so evaluating different de-Bayer algorithms is often subjective. By constructing an image that is then mosaiced, removing two-thirds of the data, we can then objectively compare the results with the ground-truth.

For useful testing, the input DNG image must be well-designed. It should not be derived from a raw image that has been de-Bayered as artefacts from the de-Bayering could bias results. It could be a crop from a raw image that has not been de-Bayered.

Scripts

For convenience, .bat scripts are also available in a single zip file. See Zipped BAT files.

gray2rggb.bat

rggb2gray.bat

rggb2gray2.bat

All images on this page were created by the commands shown, using:

%IMG7%magick identify -version

Version: ImageMagick 7.1.0-4 Q16 x64 2021-07-18 https://imagemagick.org
Copyright: (C) 1999-2021 ImageMagick Studio LLC
License: https://imagemagick.org/script/license.php
Visual C++: 192930038
Features: Cipher DPC HDRI Modules OpenCL OpenMP(2.0) 
Delegates (built-in): bzlib cairo flif freetype gslib heic jng jp2 jpeg jxl lcms lqr lzma openexr pangocairo png ps raqm raw rsvg tiff webp xml zip zlib

%DCRAW% |head -n 4

 Raw photo decoder "dcraw" v9.27
by Dave Coffin, dcoffin a cybercom o net
slightly hacked by Alan Gibson v2.0

exiftool -ver

12.13

%DTPATH%darktable-cli --version

this is darktable-cli 3.0.2
copyright (c) 2012-2020 johannes hanika, tobias ellinghaus

To improve internet download speeds, some images may have been automatically converted (by ImageMagick, of course) from PNG or TIFF or MIFF to JPG.

Source file for this web page is procbay.h1. To re-create this web page, execute "procH1 procbay".

This page, including the images, is my copyright. Anyone is permitted to use or adapt any of the code, scripts or images for any purpose, including commercial use.

Anyone is permitted to re-publish this page, but only for non-commercial use.

Anyone is permitted to link to this page, including for commercial use.

%IMG7%magick ^ pbpx_one_R.miff ^ -set colorspace RGB ^ -colorspace sRGB ^ ( +clone ^ %WEBSIZE% ^ +write pbpx_one_R_sm.miff ^ +delete ^ ) ^ %WEBCROP2% ^ pbpx_one_R_cr.miff
%IMG7%magick ^ pbpx_one_G0.miff ^ -set colorspace RGB ^ -colorspace sRGB ^ ( +clone ^ %WEBSIZE% ^ +write pbpx_one_G0_sm.miff ^ +delete ^ ) ^ %WEBCROP2% ^ pbpx_one_G0_cr.miff
%IMG7%magick ^ pbpx_one_G1.miff ^ -set colorspace RGB ^ -colorspace sRGB ^ ( +clone ^ %WEBSIZE% ^ +write pbpx_one_G1_sm.miff ^ +delete ^ ) ^ %WEBCROP2% ^ pbpx_one_G1_cr.miff
%IMG7%magick ^ pbpx_one_B.miff ^ -set colorspace RGB ^ -colorspace sRGB ^ ( +clone ^ %WEBSIZE% ^ +write pbpx_one_B_sm.miff ^ +delete ^ ) ^ %WEBCROP2% ^ pbpx_one_B_cr.miff