Putting OOG back in the box

Some operations result in colours that are out of gamut ("OOG"), that is, some pixels have values outside the range of 0 to 100% of QuantumRange. If we are using integer IM, or saving to an integer file format, then these values will be clamped (or "clipped") to 0 or 100%, and any detail in those pixels is lost, and cannot be reconstructed. We won't even know which pixels were OOG, though we might guess that any values at 0 or 100% were OOG, and we might attempt reconstruction from nearby pixels that are not at the limits.

Being out of gamut is not, of itself, a problem. We can continue processing with floating-point. But, eventually, we usually want to bring all the colours within gamut, if only because output devices can't display OOG colors.

This page isn't about recovering data that has been lost due to clipping. It is about preventing the loss in the first place.

Sample input

For example, from the Colours to matrix and polynomials page, where the intention is to change toes.png to look more like toes_x_holed.png:

We have shown the range of values in each channel as multiples of QuantumRange, and saved the image to a floating-point miff (which you can download: ob_src.miff). For display on the web, the image has been clamped and converted to JPEG. Notice how the skin-tone at the right-edge bleeds into the white page background.

Working in floating-point, we can readily isolate pixels where a channel is more than 100%, and where a channel is less than zero.

In the first result, most pixels are black. In those positions, pixels have not exceeded 100%. The red pixels are where only the red channel exceeded 100%. Yellow pixels are where both red and green exceeded 100%. The small number of white pixels are where all three channels exceeded 100%.

The second result works in a similar way: it is blue where the blue channel was less than zero, magenta where both red and blue were less zero, and so on.

Clamp

This has clamped values beyond 0 or 100% to those limits. Detail is lost, especially in parts of the sandal top-right of the image. (Many pixels that were different have become the same, pure white.)

Auto-level

Instead of clamping, we can "-auto-level". This applies a gain and bias that modifies all pixels, probably reducing contrast, squishing values together so they fit in the range. The same transformation is usually applied to all the channels, though they can be processed separately (but this will create a colour cast).

This has retained all the detail in all the pixels. Applying the same transformation to all channels has reduced contrast heavily. Processing the channels separately hasn't reduced contrast as much, but has created a colour cast.

For some purposes, the mean of clamped and auto-levelled versions is satisfactory, though it does lower the contrast even in the mid-tones:

Linear modulation

Another option is to modulate all values outside certain limits, say all values above 0.9 or below 0.1. The modulation is by a linear function (the usual gain-and-bias) where a value at 0.9 is unchanged but greater values are progressively reduced so the highest value is transformed to 1.0. We modulate values below 0.1 in a similar way. The overall transformation is shown in green on this diagram:

In the diagram, in-gamut colours are in the square between (0,0) and (1,1). Input values are between x0 and x1, where x0<0 and x1>1. We want the output values to be between 0 and 1. Most input values, those between P0 and P1 (where P0<P1), are unchanged. Input values between x0 and P0 will be transformed linearly by y=a*x+b, and values between P1 and x1 will be transformed by y=c*x+d, where a, b, c and d are calculated from x0, x1, P0 and P1:

Highlights and shadows are processed independently. OOG may occur in highlights, or shadows, or both.

This will reduce contrast where the input is less than P0 or more than P1, and will cause hue-shift for RGB pixels that have channels straddling P0 or P1. Setting P0 to 0.0 will clip shadows; setting P1 to 1.0 will clip highlights.

Three methods are shown. The first operates on all colour channels (usually red, green and blue), applying the same transformation to each channel. The second operates on all colour channels, applying independent transformations to each channel. The third operates on the L channel of HCLp.

All colour channels, same transformation

All colour channels, independently

The script oogLinear3.bat separates the image, makes three independent calls to oogLinear.bat, and combines the results.

Lightness only

In HCLp colorspace, only the L (Luma) channel of this image has OOG values. (In general, the C channel may also be OOG.)

Power modulation

Instead of a linear transformation at each end, we can use power curves (sometimes called filmic curves, as they mimic the response of photographic film). These will have slopes of zero at (x0,0) and (x1,1), and one at P0 and P1.

... and we will use the equation y = A0*(x-x0)^B0. The gradient of the toe curve is y' = B0*A0*(x-x0)^(B0-1).

At the shoulder, the origin is at (x1,1), and the slope at P1 is one, and both axes are negated, so:

... and we will use the equation y = 1 - A1*(x1-x)^B1. The gradient of the shoulder curve is y' = B1*A1*(x1-x)^(B1-1).

As for the linear transformation, this method can be applied to all channels together or independently, or just to a lightness channel. Here, we use only one method: all channels together. The script oogPower.bat implements this, using "-fx". (See below Process module below for a faster method.)

The above creates power curves that have zero gradient at y=0 (at the shadows) and y=1 (at the highlights). This compresses contrast in the shadows (which is generally okay) and in the highlights (which can look bad). It would be useful if non-zero gradients could be specified, with values between zero and one.

Considering just the shadow region, we want a power curve that passes through (x0,0) and P0, with gradient G0 at (x0,0) and 1.0 at P0.

The general equation will be:

y = A0*(x-OffsX)^B0 + OffsY

... where the gradient is zero at (OffsX,OffsY).

We need expressions for OffsX, OffsY, A0 and B0, all in terms of x0, P0 and G0.

Passing through (x0,0) and (P0,P0):

0  = A0*(x0-OffsX)^B0 + OffsY
P0 = A0*(P0-OffsX)^B0 + OffsY

Subtracting:

P0 = A0*(P0-OffsX)^B0 - A0*(x0-OffsX)^B0
P0/A0 = (P0-OffsX)^B0 - (x0-OffsX)^B0 ... but this isn't used?

Differentiating the general equation:

y' = B0*A0*(x-OffsX)^(B0-1)

At (OffsX,OffsY), y'=0, so:

0 = B0*A0*(OffsX-OffsX)^(B0-1)
  = 0 as required.

At (x0,0), y'=G0, so:

G0 = B0*A0*(x0-OffsX)^(B0-1)

At (P0,P0), y'=1, so:

1 = B0*A0*(P0-OffsX)^(B0-1)

Divide:

G0 = (x0-OffsX)^(B0-1) / (P0-OffsX)^(B0-1)
   = [ (x0-OffsX) / (P0-OffsX) ] ^ (B0-1)

... Or we do this with splines, each with two control points.

... Or we do this with cubic curves.

y = a*x^3 + b*x^2 + c*x + d

y' = 3*a*x^2 + 2*b*x + c

[[ y'' = 6*a*x + 2*b ]]

Passing through (x0,0) and (P0,P0):

0  = a*x0^3 + b*x0^2 + c*x0 + d   ... (1)
P0 = a*P0^3 + b*P0^2 + c*P0 + d   ... (2)

Subtracting (2)-(1):

P0 = a*P0^3 + b*P0^2 + c*P0 - a*x0^3 - b*x0^2 - c*x0
   = a*(P0^3-x0^3) + b*(P0^2-x0^2) + c*(P0-x0)    ... (3)

At (x0,0), y'=G0, so:

G0 = 3*a*x0^2 + 2*b*x0 + c   ... (4)

At (P0,P0), y'=1, so:

1 = 3*a*P0^2 + 2*b*P0 + c   ... (5)

c = 1 - 3*a*P0^2 - 2*b*P0

Subtracting (5)-(4):

1 - G0 = 3*a*P0^2 + 2*b*P0 - 3*a*x0^2 - 2*b*x0
       = 3*a*(P0^2-x0^2) + 2*b*(P0-x0)

b = [ (1 - G0) - 3*a*(P0^2-x0^2) ] / [ 2*(P0-x0) ]

Substitute c:

G0 = 3*a*x0^2 + 2*b*x0 + c
   = 3*a*x0^2 + 2*b*x0 + 1 - 3*a*P0^2 - 2*b*P0
   = 3*a*(x0^2 - P0^2) + 2*b*(x0 - P0) + 1   This is not new.

[[ At (P0,P0), y''=0, so:

0 = 6*a*P0 + 2*b
b = -3*a*P0

]]

Substitute b:

[[
1 - G0 = 3*a*(P0^2-x0^2) -2*3*a*P0*(P0-x0)
       = a * (3*P0^2 - 3*x0^2 - 6*P0*(P0-x0) )
       = a * (3*P0^2 - 3*x0^2 - 6*P0^2 + 6*P0*x0)
       = 3*a * (P0^2 - x0^2 - 2*P0^2 + 2*P0*x0)
       = 3*a * (-P0^2 - x0^2 + 2*P0*x0)
       = -3*a * (P0^2 + x0^2 - 2*P0*x0)
       = -3*a * (P0 - x0)^2

a = (1-G0) / [ -3 * (P0 - x0)^2 ]
]]

[[
b = -3*a*P0
  = -3 * P0 * (1-G0) / [ -3 * (P0 - x0)^2 ]
  = P0 * (1-G0) / (P0 - x0)^2

c = 1 - 3*a*P0^2 - 2*b*P0

0 = a*x0^3 + b*x0^2 + c*x0 + d
d = -a*x0^3 - b*x0^2 - c*x0
]]

... Or a quadratic:

y = a*x^2 + b*x + c

y' = 2*a*x + b

At the shadows:

Passing through (x0,0) and (P0,P0):

0 = a*x0^2 + b*x0 + c
P0 = a*P0^2 + b*P0 + c

Subract:

P0 = a*(P0^2-x0^2) + b*(P0-x0)

b = [ P0 - a*(P0^2-x0^2) ] / (P0-x0)

Gradient at (x0,0) is G0, and at (P0,P0) is 1.0:

G0 = 2*a*x0 + b
1 = 2*a*P0 + b

Subtract:

1 - G0 = 2*a*(P0-x0)

a = (1-G0) / [ 2*(P0-x0) ]

At x=x0, y=0:

0 = a*x0^2 + b*x0 + c

c = -a*x0^2 - b*x0

At the highlights:

Passing through (x1,1) and (P1,P1):

1 = a*x1^2 + b*x1 + c
P1 = a*P1^2 + b*P1 + c

Subract:

P1 - 1 = a*(P1^2-x1^2) + b*(P1-x1)

b = [ P1 - 1 - a*(P1^2-x1^2) ] / (P1-x1)

Gradient at (x1,1) is G1, and at (P1,P1) is 1:

G1 = 2*a*x1 + b
1 = 2*a*P1 + b

Subtract:

1 - G1 = 2*a*(P1-x1)

a = (1-G1) / [ 2*(P1-x1) ]

b = 1 - 2*a*P1

1 = a*x1^2 + b*x1 + c
c = 1 - a*x1^2 - b*x1

... Or solve simultaneous equations by GaussJordan elimination.

... Or consider case where origin is at gradient=0, and adjust later.

y = a*x^3 + b*x^2 + c*x
y' = 3*a*x^2 + 2*b*x

Passes through (xA,yA) and (xB,yB):

yA = a*xA^3 + b*xA^2 + c*xA   ... (1)
yB = a*xB^3 + b*xB^2 + c*xB   ... (2)

At (xA,yA) gradient is GA:

GA = 3*a*xA^2 + 2*b*xA   ... (3)

At (xB,yB) gradient is 1:

1 = 3*a*xB^2 + 2*b*xB   ... (4)

b = (1 - 3*a*xB^2) / (2*xB)

Cross-multiply (3) and (4):

2*xB*Ga = 2*xB*3*a*xA^2 + 2*xB*2*b*xA   ... (5)
2*xA*1  = 2*xA*3*a*xB^2 + 2*xA*2*b*xB   ... (6)

(6)-(5):

2*xA*1 - 2*xB*Ga  = 2*xA*3*a*xB^2 - 2*xB*3*a*xA^2
xA - xB*Ga  = xA*3*a*xB^2 - xB*3*a*xA^2
a = (xA - xB*Ga) / (xA*3*xB^2 - xB*3*xA^2)
  = (xA - xB*Ga) / [ 3*xA*xB * (xB-xA) ]

From (4) we know b. From (1) or (2) we know c.

However, we don't know xA, xB, yA or yB. We know only the relative values xB-xA and yB-yA.

... Or do a power curve, blended with a linear transformation.

Comparison

Most of the results have differences that are subtle. For comparison purposes, we make a GIF of the results that are similar.

Process module

The process module oogbox is faster, especially for the power curve, and can be incorporated into "convert" or "magick" commands.

Option	Description
Short form	Long form
`m string`	`method string`	Method for adjusting the shadows and highlights, one of: `Linear` `Power` Default = `Power`.
`fmn number`	`forceMin number`	Force minimum value (override calculation). Default: use calculated value.
`fmx number`	`forceMax number`	Force maximum value (override calculation). Default: use calculated value.
`l number`	`loLimit number`	Low limit for shadow processing. Default: `0.1`.
`h number`	`hiLimit number`	High limit for highlight processing. Default: `0.9`.
`lg number`	`loGradient number`	Gradient for shadows. Default: use calculated value.
`hg number`	`hiGradient number`	Gradient for highlights. Default: use calculated value.
`v`	`verbose`	Write some text output to stderr.
	`version`	Write version information to stdout.

The parameters loLimit and hiLimit take the place of P0 and P1 in the description above. To prevent any shadow processing, set loLimit to 0.0 or less. Likewise to prevent any highlight processing, set hiLimit to 1.0 or more.

By default, minimum and maximum statistics are taken from all channels, and all channels are adjusted, by the same amount. To use the statistics from only certain channels, and adjust only those channels, use the IM -channel setting. (Before December 2022, the module had a "channel" setting, and explicily processed red, green and blue channels. Now it uses the general IM "-channel" setting, and processes all channels that have the "update" trait.)

Hence, to adjust RGB channels independently, run the module three times with different -channel settings.

The module calculates the minimum and maximum values. To override these, use forceMin and/or forceMax. For method power, values outside the range will not be changed. For method linear and method blend, values outside the range will be changed, but not far enough to bring them inside the box.

Those results from the process module should be the same as the corresponding results from the scripts:

As expected, the differences are virtually zero, which confirms the process module has the same effect as the scripts.

The process module is also useful when converting camera raw files into colorspaces. See Squishing xyY into shape.

Graphed examples of oogbox

To see the effect more easily, we create a Nx1 gradient image with values outside 0 to 100%:

For "method blend", verbose output will state the calculated gradient of the linear part of the blend. This can be overridden, but larger values may result in tonal reversal. For example, with a very large value:

Retinex

Future

When an image is encoded HCLp, and we convert to sRGB, is the sRGB image guaranteed to be in gamut? No. Hence, if we use HCLp to try and bring an sRGB OOG back into gamut, we should then test the sRGB result and apply a suitable process to that.

Can we invert the process? We haven't lost any pixel data, but we have lost the knowledge of how far OOG the image was. We can't reverse the process without this knowledge. If we know the original x0, x1, P0 and P1, and hence a, b, c and d, we can calculate the inverse transformation:

Currently loLimit and hiLimit are parameters from the user, defaulting to 0.1 and 0.9. Perhaps sensible values could be calculated somehow.

An adaptive (aka localised) oogbox would be useful, eg for increasing local contrast that can push cvalues outside range of 0..100%. Instead of applying a global curve, it would apply a series local curves. Implementation? Treat shadows and highlights separately. Find and apply the strongest curve needed for shadows, then blend with the input via a mask that defines how much of the effect we need.

Scripts

For convenience, .bat scripts are also available in a single zip file. See Zipped BAT files.

oogLinear.bat

oogLinear3.bat

oogLinear1ch.bat

oogPower.bat

All images on this page were created by the commands shown, using:

%IMG7%magick -version

Version: ImageMagick 7.1.1-20 Q16-HDRI x86 98bb1d4:20231008 https://imagemagick.org
Copyright: (C) 1999 ImageMagick Studio LLC
License: https://imagemagick.org/script/license.php
Features: Cipher DPC HDRI OpenCL OpenMP(2.0) 
Delegates (built-in): bzlib cairo freetype gslib heic jng jp2 jpeg jxl lcms lqr lzma openexr pangocairo png ps raqm raw rsvg tiff webp xml zip zlib
Compiler: Visual Studio 2022 (193532217)

%IM7DEV%magick -version

Version: ImageMagick 7.1.0-33 beta Q32-HDRI x86_64 64b5fe68a:20220501 https://imagemagick.org
Copyright: (C) 1999 ImageMagick Studio LLC
License: https://imagemagick.org/script/license.php
Features: Cipher DPC HDRI Modules OpenMP(4.5) 
Delegates (built-in): bzlib fpx jbig jpeg lcms ltdl lzma tiff webp x xml zip zlib
Compiler: gcc (11.3)

To improve internet download speeds, some images may have been automatically converted (by ImageMagick, of course) from PNG or TIFF or MIFF to JPG.

Source file for this web page is oogbox.h1. To re-create this web page, execute "procH1 oogbox".

This page, including the images, is my copyright. Anyone is permitted to use or adapt any of the code, scripts or images for any purpose, including commercial use.

Anyone is permitted to re-publish this page, but only for non-commercial use.

Anyone is permitted to link to this page, including for commercial use.

All channels, same transformation %IM7DEV%magick ^ "%SRC%" ^ -process 'oogbox m Linear' ^ ob_linsamp.png
All channels, idenpendent transformations %IM7DEV%magick ^ "%SRC%" ^ -channel 0 -process 'oogbox m Linear' ^ -channel 1 -process 'oogbox m Linear' ^ -channel 2 -process 'oogbox m Linear' ^ +channel ^ ob_linindp.png
Lightness only %IM7DEV%magick ^ "%SRC%" ^ -colorspace HCLp ^ -channel 2 -process 'oogbox m Linear' ^ -colorspace sRGB ^ -depth 32 ^ -define "quantum:format=floating-point" ^ ob_hclp2_linp.miff
Power modulation %IM7DEV%magick ^ "%SRC%" ^ -process 'oogbox m Power' ^ ob_powsamp.png

%IMG7%magick ^ %HDR_SRC% ^ -clamp ^ ob_hdr_clamp.tiff call %PICTBAT%graphLineNeg ^ ob_hdr_clamp.tiff "`-clamp`" zeros highlight and shadow contrast.
%IMG7%magick ^ %HDR_SRC% ^ -auto-level ^ ob_hdr_autolev.tiff call %PICTBAT%graphLineNeg ^ ob_hdr_autolev.tiff "`-auto-level`" reduces all contrast.
%IM7DEV%magick ^ %HDR_SRC% ^ -process 'oogbox method Linear' ^ ob_hdr_lin.tiff call %PICTBAT%graphLineNeg ^ ob_hdr_lin.tiff Linear method retains mid-tone contrast; reduces highlight and shadow contrast; second-order discontinuity may be obvious.
%IM7DEV%magick ^ %HDR_SRC% ^ -process 'oogbox method Power' ^ ob_hdr_pow.tiff call %PICTBAT%graphLineNeg ^ ob_hdr_pow.tiff Power method has no second-order discontinuity; reduces highlight and shadow contrast, with zero contrast at the extremes.
%IM7DEV%magick ^ %HDR_SRC% ^ -process 'oogbox method Blend' ^ ob_hdr_blnd.tiff call %PICTBAT%graphLineNeg ^ ob_hdr_blnd.tiff Blend method doesn't zero contrast anywhere.
%IM7DEV%magick ^ %HDR_SRC% ^ -process 'oogbox method Blend hiLimit 0.7' ^ ob_hdr_blnd2.tiff call %PICTBAT%graphLineNeg ^ ob_hdr_blnd2.tiff Any `oogbox` method may benefit from limits closer to 0.5.

%IM7DEV%magick ^ %SRC% ^ -auto-level ^ ob_al.jpg
%IM7DEV%magick ^ %SRC% ^ -channel RGB ^ -auto-level ^ +channel ^ ob_al2.jpg