%IMG7%magick ^ dcr_wbdef0.tiff ^ %WEBSIZE% ^ dcr_wbdef0.jpg
... (white balance) and clipping and contrast.
dcraw, Dave Coffin's raw image converter, is a splendid piece of work. It must count as (probably) the greatest and most influential little program of the last 20 years. It isn't a library, and it's difficult to pull apart to tweak it in the directions I want, or even to understand. But what it does, it does well.
This page shows how we can use dcraw to get accurately white-balanced images, with good contrast and brightness, without clipping, automatically.
Scripts on this page assume that the version of ImageMagick in %IM7DEV% has been built with various process modules. See Process modules.
Some versions of IM come with a pre-built binary of dcraw, or you can build it yourself.
wget http://cybercom.net/~dcoffin/dcraw/dcraw.cOr you can download my slightly hacked version, dcrawimg.c. Or download the official source, and apply the differences file dcrawag.diff between it and my hacked version.
gcc -o dcrawag.exe dcraw/dcrawimg.c -DNO_JASPER -I. \ -Ofast -m64 -march=native -mtune=native -Wall -llcms2 -ljpeg cp dcrawag.exe $PICTBAT\dcrawag.exe
echo %DCRAW%
F:\pictures\dcrawag.exe
%DCRAW%
Raw photo decoder "dcraw" v9.27 by Dave Coffin, dcoffin a cybercom o net slightly hacked by Alan Gibson v2.0 Usage: /cygdrive/f/pictures/dcrawag [OPTION]... [FILE]... -v Print verbose messages -c Write image data to standard output -e Extract embedded thumbnail image -i Identify files without decoding them -i -v Identify files and show metadata -z Change file dates to camera timestamp -w Use camera white balance, if possible -a Average the whole image for white balance -A <x y w h> Average a grey box for white balance -r <r g b g> Set custom white balance -R n Divisor for white balance multipliers +M/-M Use/don't use an embedded color matrix -C <r b> Correct chromatic aberration -O <file> Write output to this file -P <file> Fix the dead pixels listed in this file -K <file> Subtract dark frame (16-bit raw PGM) -k <num> Set the darkness level -S <num> Set the saturation level -n <num> Set threshold for wavelet denoising -H [0-9] Highlight mode (0=clip, 1=unclip, 2=blend, 3+=rebuild) -t [0-7] Flip image (0=none, 3=180, 5=90CCW, 6=90CW) -o [0-6] Output colorspace (raw,sRGB,Adobe,Wide,ProPhoto,XYZ,ACES) -o <file> Apply output ICC profile from file -p <file> Apply camera ICC profile from file or "embed" -d Document mode (no color, no interpolation) -D Document mode without scaling (totally raw) -j Don't stretch or rotate raw pixels -W Don't automatically brighten the image -B <num> Clipping threshold for auto-brighten (default = 0.0001) -b <num> Adjust brightness (default = 1.0) -g <p ts> Set custom gamma curve (default = 2.222 4.5) -q [0-3] Set the interpolation quality -h Half-size color image (twice as fast as "-q 0") -f Interpolate RGGB as four colors -m <num> Apply a 3x3 median filter to R-G and B-G -s [0..N-1] Select one raw image or "all" from each file -6 Write 16-bit instead of 8-bit -4 Linear 16-bit, same as "-6 -W -g 1 1" -T Write TIFF instead of PPMNote the -O and -B options.
dcraw has a "automatically brighten the image" operation. This stretches the pixel values by multiplying channel values. The idea is that if the maximum value is 75%, multiplying by 1.3333 will use more of the range between 0 and 100%. (dcraw won't also stretch the bottom, so if the minimum value was 10% it might become 13.333%. Why won't it stretch both ends? I don't know.) This happens by default; if we don't want it, we can disable it with "-W", and we can then use IM to "-auto-level" or whatever we want.
But this auto-brighten feature has a problem: it doesn't merely stretch to use values up to 100%, but over-stretches, clipping the top 1% of pixels. This is a lot of pixels. An area one-tenth of the image width and one-tenth of the height amounts to 1% of the image area. The whole point of using raw images (for me) is to get good pixels from the camera, giving me more flexibility than the in-camera JPEG provides, and hence better quality. Wrecking 1% of the pixels before I even begin the creative stuff is a bad start.
So I've also created a patch for dcraw. This adds a variable, BrightThr initialised to 0.0001, and an option "-B <num>" to set the variable. The variable is used in a line that used to read...
perc = width * height * 0.01;
... but I have changed to ...
perc = width * height * BrightThr;
So, by default "auto-brighten" will now clip 0.0001 of the pixels, which is 0.01% of the pixels. If that is too much, -B 0 will auto-brighten without clipping any pixels. As before, "-W" will not auto-brighten at all.
dcraw applies a white balance by simple channel multiplication in linear colorspace, before de-Bayering and conversion to sRGB or whatever. It can get the multipliers from metadata in the raw file ("camera" white balance), or from averaging a rectangle of pixels that should be neutral gray, or from the "-r" command line option. Multiplying channels can result in clipping.
dcraw's -b option, to set brightness, also multiplies in linear colorspace. See Brightness below.
We will show three methods for white balancing:
[Embed linear images with a profile from Elle.]
First, some basic statistics about the raw Bayer data. "-D" give us "Document", which isn't de-Bayered or scaled. We don't auto-brighten or white-balance or adjust gamma.
set SRCNEF=%PICTLIB%20171029\AGA_3437.nef set WEBDIMS=600x600 set WEBSIZE=-resize %WEBDIMS% %DCRAW% -v -6 -W -D -r 1 1 1 1 -g 1 1 -o 0 -T -O dcr_basic.tiff %SRCNEF%
What are the minimum, mean and maximum Bayer values?
%IMG7%magick dcr_basic.tiff ^ -format "%%[fx:minima*QuantumRange] %%[fx:mean*QuantumRange] %%[fx:maxima*QuantumRange]\n" ^ info:
0 1148.18 16383
We are using IM Q16 but the camera records only 14 bits per pixel, so the maximum value possible is 2^14-1 = 16383.
How many Bayer values are at the extremes?
call %PICTBAT%propIntClip dcr_basic.tiff
f:\prose\PICTURES>rem From image dcr_basic.tiff, f:\prose\PICTURES>rem finds proportion that have 0 or 100 in any channel. f:\prose\PICTURES>c:\im\ImageMagick-7.1.0-42-Q16-HDRI\magick dcr_basic.tiff -separate -fill White -opaque Black -fill Black +opaque White -background Black -compose Lighten -layers Merge -format "clipped=%[fx:mean]\n" info: clipped=0.500001 f:\prose\PICTURES>c:\im\ImageMagick-7.1.0-42-Q16-HDRI\magick dcr_basic.tiff -separate -fill White +opaque Black +write x.tiff -negate +write x2.tiff -background Black -compose Lighten -layers Merge +write x3.tiff -format "shadowClipped=%[fx:mean]\n" info: shadowClipped=0.500001
A very small number of sensor pixels recorded no signal at all. A larger number recorded the maximum signal, so the sensors were saturated (the amount of light was at or beyond the capacity of the sensors). (The camera has 35 M pixels, so the proportion of pixels that are clipped, 0.05%, isn't horribly bad.) We want to avoid increasing the numbers at either end.
We tell dcraw to make a 16-bit TIFF sRGB image, using the camera WB, with an auto brighten that doesn't clip.
First, we use -H 0. This normalises the WB channel multipliers so the smallest multiplier is 1.0, so values in other channels can increase, so this can cause clipping in those other channels. However, this clipping means that pixels recorded at or near 100% in all three channels will become exactly or nearly white after the balancing, which is normally the correct result.
%DCRAW% -v -6 -w -o 1 -H 0 -B 0 -T -O dcr_wbdef0.tiff %SRCNEF%
What are the minimum, mean and maximum channel values?
%IMG7%magick dcr_wbdef0.tiff ^ -format "%%[fx:minima] %%[fx:mean] %%[fx:maxima]\n" ^ info:
0 0.186095 1
How many pixel channel values are at the extremes?
call %PICTBAT%propIntClipRGB dcr_wbdef0.tiff
shadowClipped.0=11489 shadowClipped.1=24 shadowClipped.2=1448 highlightClipped.0=882805 highlightClipped.1=13256 highlightClipped.2=1.3762e+06
Many pixels (about 3%) have clipped in the blue channel.
The image looks like this:
%IMG7%magick ^ dcr_wbdef0.tiff ^ %WEBSIZE% ^ dcr_wbdef0.jpg |
|
Using a GUI editor, we can see that most of the sky has burnt out in one or more channels.
We repeat, using -H 1. Modes from this upwards normalise the WB channel multipliers so the largest is 1.0. It also does other processing that I don't understand.
%DCRAW% -v -6 -w -o 1 -H 1 -B 0 -T -O dcr_wbdef1.tiff %SRCNEF%
What are the minimum, mean and maximum channel values?
%IMG7%magick dcr_wbdef1.tiff ^ -format "%%[fx:minima] %%[fx:mean] %%[fx:maxima]\n" ^ info:
0 0.119695 1
How many pixel channel values are at the extremes?
call %PICTBAT%propIntClipRGB dcr_wbdef1.tiff
shadowClipped.0=11580 shadowClipped.1=23 shadowClipped.2=1538 highlightClipped.0=195181 highlightClipped.1=1 highlightClipped.2=1
That's better. Far fewer pixels have clipped.
The image looks like this:
%IMG7%magick ^ dcr_wbdef1.tiff ^ %WEBSIZE% ^ dcr_wbdef1.jpg |
|
Eek! The sky has gone stupid. Other parts of the image are darker than the previous version, and this is expected as the multiplier normalisation has reduced instead of increasing values.
We repeat, using -H 2:
%DCRAW% -v -6 -w -o 1 -H 2 -B 0 -T -O dcr_wbdef2.tiff %SRCNEF%
What are the minimum, mean and maximum channel values?
%IMG7%magick dcr_wbdef2.tiff ^ -format "%%[fx:minima] %%[fx:mean] %%[fx:maxima]\n" ^ info:
0 0.14441 1
How many pixel channel values are at the extremes?
call %PICTBAT%propIntClipRGB dcr_wbdef2.tiff
shadowClipped.0=11422 shadowClipped.1=23 shadowClipped.2=1538 highlightClipped.0=70589 highlightClipped.1=70498 highlightClipped.2=70498
That's better. Far fewer pixels have clipped, about 0.2%.
The image looks like this:
%IMG7%magick ^ dcr_wbdef2.tiff ^ %WEBSIZE% ^ dcr_wbdef2.jpg |
|
That looks better.
Settings of -H 3 and above look bad.
%DCRAW% -v -4 -o 0 -T -H 1 -O x.tiff %SRCNEF% %IMG7%magick identify -format %[fx:maxima] x.tiff %DCRAW% -v -4 -o 0 -T -H 0 -R 0.25745 -O x.tiff %SRCNEF% %IMG7%magick x.tiff -set colorspace RGB -colorspace sRGB s.tiff
set RNum=2 %DCRAW% -v -4 -w -W -o 0 -H 0 -R %RNum% -T -O dcr_wbrn0.tiff %SRCNEF%
What are the minimum, mean and maximum channel values?
%IMG7%magick dcr_wbrn0.tiff ^ -format "%%[fx:minima] %%[fx:mean] %%[fx:maxima]\n" ^ info:
0 0.045132 0.958984
Calculate the maximum multiplied by the RNum:
for /F "usebackq" %%L in (`%IMG7%magick dcr_wbrn0.tiff ^ -format "newRNum=%%[fx:maxima*%RNum%]\n" ^ info:`) do set %%L echo newRNum=%newRNum%
newRNum=1.91797
Re-run dcraw, using newRNum:
%DCRAW% -v -4 -w -W -o 0 -H 0 -R %newRNum% -T -O dcr_wbrn0.tiff %SRCNEF%
How many pixel channel values are at the extremes?
call %PICTBAT%propIntClipRGB dcr_wbrn0.tiff
shadowClipped.0=62 shadowClipped.1=2 shadowClipped.2=12 highlightClipped.0=70446 highlightClipped.1=5 highlightClipped.2=1
Blah.
The image looks like this:
%IMG7%magick ^ dcr_wbrn0.tiff ^ %WEBSIZE% ^ dcr_wbrn0.jpg |
|
When using -R n, you should usually also use -W or -6.
This method calculates white balance from gray patches on a colour card. During the photography session, we include a standard colour card in one of the photos. We automatically find the patches, and the lightest patch should be a neutral gray, so this gives us the channel multipliers.
First, we process the raw image in a "flat" manner, with no white balancing or conversion to sRGB. [[ The "-h" option reduces the time to one-third. ]] We allow a small proportion of pixels to clip in "auto brighten".
In the -r option, the four numbers are multipliers for the Bayer RGBG channels. Whatever numbers we give, the standard version of dcraw will normalise to make the lowest or the highest multiplier 1.0. If "-H 0" is used (the default), the lowest multiplier is made 1.0, so this will never lower values in any channels (and it can cause clipping). For higher "-H" settings, the highest multiplier is made 1.0, so this will never raise values in any channels (and it can't cause clipping). In either case, we can't use -r to change the overall lightness of the image.
(My patched version of dcraw has an extra option, -R n, which prevents normalisation of the -r white balance multipliers.)
set SRCNEF=%PICTLIB%20171029\AGA_3431.nef set WEBSIZE=-resize 600x600 set dcrParams=-v -B 0.0001 -6 -T %DCRAW% %dcrParams% -r 1 1 1 1 -o 0 -g 1 1 -O dcr_flat.tiff %SRCNEF%
|
Out of interest, we take a look at the image.
%IMG7%magick ^ dcr_flat.tiff ^ -set colorspace sRGB ^ -gravity SouthWest -crop 35x35%%+0+0 ^ %WEBSIZE% ^ -quality 40 ^ dcr_flat.jpg |
|
The image looks dark because the pixels values are linear but we are displaying as if they were sRGB.
Next, extract the colour card. See Finding and analysing colour charts for more detail.
call %PICTBAT%24card dcr_flat.tiff dcr_flat_XX.png
This gives a 6x4 pixel image, and a scaled-up version for viewing:
|
dcr_flat_scl.png |
|
The bottom row of the 6x4 image should be neutral shades of gray. We could choose any pixel on that row to give us the multipliers for red and blue, compared to green, but the lightest patch will show a colour-cast most easily, so we choose it. We are not concerned with the absolute values in the channels (eg whether the patch is pure white or 90% white), but only with their relative values.
set FMT=^ r=%%[fx:mean.r]\n^ g=%%[fx:mean.g]\n^ b=%%[fx:mean.b]\n^ mr=%%[fx:mean.g/mean.r]\n^ mb=%%[fx:mean.g/mean.b]\n for /F "usebackq" %%L in (`%IMG7%magick ^ -precision 16 ^ dcr_flat_mat.png ^ -gravity SouthWest -crop 1x1+0+0 +repage ^ -format "%FMT%" ^ info:`) do set %%L echo r=%r% g=%g% b=%b% echo mr=%mr% mb=%mb%
r=0.4866712443732357 g=0.977630273899443 b=0.6918745708400091 mr=2.008810434564495 mb=1.413016629173834
For convenience, we put this in a script, whiteMult.bat.
Re-run dcraw with those multipliers. This time, we include the conversion to sRGB.
%DCRAW% %dcrParams% -r %mr% 1 %mb% 1 -o 1 -O dcr_real.tiff %SRCNEF%
|
Out of interest, we take a look at the image.
%IMG7%magick ^ dcr_real.tiff ^ -gravity SouthWest -crop 35x35%%+0+0 ^ %WEBSIZE% ^ -quality 40 ^ dcr_real.jpg |
|
The white balance of the result looks good. We can check it by finding the card and examining the bottom row. (If we hadn't used "-h" for the first dcraw, we could re-use the coordinates from the first 24card.bat.)
call %PICTBAT%24card dcr_real.tiff dcr_real_XX.png
This gives a 6x4 pixel image, and a scaled-up version for viewing:
|
dcr_real_scl.png |
|
Rather than looking at numbers, we will graph the gray patches:
%IMG7%magick ^
dcr_real_mat.png ^
-gravity South ^
-crop x1+0+0 +repage ^
-scale "600x1^!" ^
dcr_real_mat_gr.png
call %PICTBAT%graphLineCol ^
dcr_real_mat_gr.png
|
|
As expected, the lightest patch is neutral. The others are fairly close, though not exact. We have ignored the camera's WB metadata, and used dcraw features to balance a stepped gray card. We can use the same settings to balance all photos (or video frames) taken under the same conditions.
If we want a closer match to gray throughout the range, we can find the polynomials that most closely transform each channel to match the grayed patches, and hence to match each other. See Colour checker charts for more detail.
set mypolyr=
for /F "usebackq tokens=1,2 delims==" %%A in (`%IM7DEV%magick ^
dcr_real_mat.png ^
-gravity South -crop x1+0+0 +repage ^
^( +clone ^
-colorspace Gray ^
^) ^
-precision 16 ^
-process 'cols2mat method NoCrossPoly noTrans f stdout' ^
NULL:`) do (
if "%%A"=="PolyRed" set mypolyr=%%B
if "%%A"=="PolyGreen" set mypolyg=%%B
if "%%A"=="PolyBlue" set mypolyb=%%B
)
if "%mypolyr%"=="" goto error
echo mypolyr=%mypolyr%
echo mypolyg=%mypolyg%
echo mypolyb=%mypolyb%
mypolyr=0.1237015048600939,0.8764969348923533,0.01229858749638352 mypolyg=-0.01700277760391167,1.015004353649351,-0.0002309841390380485 mypolyb=-0.1875444578742101,1.202348214008359,-0.02983195546404099
For convenience, we put this in a script, grayPoly.bat.
We can apply these polynomials directly to the _mat image, and graph it.
%IMG7%magick ^
dcr_real_mat.png ^
-gravity South ^
-crop x1+0+0 +repage ^
-channel R -function Polynomial %mypolyr% ^
-channel G -function Polynomial %mypolyg% ^
-channel B -function Polynomial %mypolyb% ^
+channel ^
-scale "600x1^!" ^
dcr_real_mat_gr2.png
call %PICTBAT%graphLineCol ^
dcr_real_mat_gr2.png
|
|
The graph is greatly improved (though the difference in the patches is not visible).
Apply the polynomials to the output from dcraw, and also ensure the standard deviation is at least 0.166667. For the setmnsd module, see Set mean and stddev.
%IM7DEV%magick ^
dcr_real.tiff ^
-channel R -function Polynomial %mypolyr% ^
-channel G -function Polynomial %mypolyg% ^
-channel B -function Polynomial %mypolyb% ^
+channel ^
-auto-level ^
-depth 32 ^
-define "quantum:format=floating-point" ^
+write dcr_rp.tiff ^
( +clone ^
%WEBSIZE% ^
-quality 40 ^
+write dcr_rp_sm.jpg ^
+delete ^
) ^
-process 'setmnsd sd 0.166667 d incOnly v' ^
+write dcr_rp_sd.tiff ^
%WEBSIZE% ^
-quality 40 ^
dcr_rp_sd_sm.jpg
We show the result before and after setting the SD. |
|
The setmnsd module has sent this to stdout:
setmnsd options: verbose sdGoal 0.166667 tolerance 1e-05 mid mean initCon0 1e-07 initCon2 30 direction incOnlydecOnlyboth input: mean=0.20839228 sd=0.14062557 nIter=19 result: mean=0.25034501 sd=0.16666933 setmnsd command: -sigmoidal-contrast 2.5030329,20.839228%
A purist (like me) might say we shouldn't allow dcraw to clip at all. What happens then? We repeat the above with auto-brighten but without any clipping.
set dcrParams=-v -B 0 -6 -T %DCRAW% %dcrParams% -r 1 1 1 1 -o 0 -g 1 1 -O dcr_flat_nc.tiff %SRCNEF% call %PICTBAT%24card dcr_flat_nc.tiff dcr_flat_nc_XX.png call %PICTBAT%whiteMult dcr_flat_nc_mat.png dcr_wm set dcr_wm
dcr_wm_b=0.4306858930342565 dcr_wm_g=0.6093537804226749 dcr_wm_mb=1.414844995571302 dcr_wm_mr=2.011383096605218 dcr_wm_r=0.3029526207370107
%DCRAW% %dcrParams% -r %dcr_wm_mr% 1 %dcr_wm_mb% 1 -o 1 -O dcr_real_nc.tiff %SRCNEF% call %PICTBAT%24card dcr_real_nc.tiff dcr_real_nc_XX.png call %PICTBAT%grayPoly dcr_real_nc_mat.png dcr_gp set dcr_gp
dcr_gp_polyb=-0.2980922587311937,1.176119490546361,-0.01102737607666592 dcr_gp_polyg=-0.02826531796778715,1.013959754170565,0.0003155726938555597 dcr_gp_polyr=0.2010445813695857,0.8889097628070177,0.003474310135753049
%IM7DEV%magick ^ dcr_real_nc.tiff ^ -channel R -function Polynomial %dcr_gp_polyr% ^ -channel G -function Polynomial %dcr_gp_polyg% ^ -channel B -function Polynomial %dcr_gp_polyb% ^ +channel ^ -auto-level ^ -process 'setmnsd sd 0.166667 d incOnly v' ^ +write dcr_rpnc_sd.tiff ^ %WEBSIZE% ^ -quality 40 ^ dcr_rpnc_sd_sm.jpg
setmnsd options: verbose sdGoal 0.166667 tolerance 1e-05 mid mean initCon0 1e-07 initCon2 30 direction incOnlydecOnlyboth input: mean=0.091012126 sd=0.073283111 nIter=20 result: mean=0.21448637 sd=0.16666103 setmnsd command: -sigmoidal-contrast 6.2427924,9.1012126%
Here is the result:
|
dcr_rpnc_sd_sm.jpg |
|
Because we haven't allowed dcraw to clip, the output from dcraw is darker, with lower contrast (ie lower standard deviation). So setmnsd needed a steeper curve to make the SD 0.166667, which has visibly increased the saturation. In addition, setmnsd applies a sigmoidal, "S", curve instead of the dcraw linear multiplication, so this increases the contrast around the mid-point and decreases it at the extremes. In my opinion, both results are satisfactory.
Downsizing for the web needs some sharpening. I like the halo minimization technique, although this also slightly increases saturation.
call %PICTBAT%resampHM ^ dcr_rpnc_sd.tiff ^ %WEBDIMS% d 100 . . ^ dcr_web.png |
|
dcraw has an option, -b, for setting the brightness. This multiples all the channels by the same number, after de-Bayering, but still in linear RGB space. If we have auto-brightened, some pixel values will already be at 100%, and -b will increase them further, causing more clipping.
I don't usually want clipping, so I don't use -b.
For basic "developing" of raw images by dcraw, I do the equivalent of IM "-auto-level" but no other tonal adjustment.
As an indicator of clipping, the script propIntClipRGB.bat counts how many pixels in each channel are at the lowest value (shadow clipping) or highest value (highlight clipping). As the script does an "-auto-level" to each channel independently, we expect a value of at least one for each clipped.
call %PICTBAT%propIntClipRGB dcr_flat.tiff
shadowClipped.0=210352 shadowClipped.1=13709 shadowClipped.2=358816 highlightClipped.0=2 highlightClipped.1=3750 highlightClipped.2=7
call %PICTBAT%propIntClipRGB dcr_real.tiff
shadowClipped.0=268174 shadowClipped.1=26163 shadowClipped.2=1.15202e+06 highlightClipped.0=3671 highlightClipped.1=5 highlightClipped.2=8
Why do we have over a million pixels shadow-clipped? Experiments show that dcraw's "o 1" creates this.
call %PICTBAT%propIntClipRGB dcr_rp.tiff
shadowClipped.0=268174 shadowClipped.1=26163 shadowClipped.2=1.15202e+06 highlightClipped.0=3671 highlightClipped.1=5 highlightClipped.2=8
call %PICTBAT%propIntClipRGB dcr_rp_sd.tiff
shadowClipped.0=268174 shadowClipped.1=26163 shadowClipped.2=1.15202e+06 highlightClipped.0=3682 highlightClipped.1=5 highlightClipped.2=8
call %PICTBAT%propIntClipRGB dcr_flat_nc.tiff
shadowClipped.0=210352 shadowClipped.1=13709 shadowClipped.2=358816 highlightClipped.0=1 highlightClipped.1=1 highlightClipped.2=1
call %PICTBAT%propIntClipRGB dcr_real_nc.tiff
shadowClipped.0=268050 shadowClipped.1=26187 shadowClipped.2=1.15033e+06 highlightClipped.0=1 highlightClipped.1=1 highlightClipped.2=1
call %PICTBAT%propIntClipRGB dcr_rpnc_sd.tiff
shadowClipped.0=268050 shadowClipped.1=26187 shadowClipped.2=1.15033e+06 highlightClipped.0=1 highlightClipped.1=1 highlightClipped.2=1
The above development works well, except for the shadow clipping. This seems to be caused by dcraw's conversion to sRGB. So, we can do that in IM instead. This changes the process from the second dcraw onwards, but we show the complete process for clarity.
set dcrParams=-v -B 0 -6 -T -o 0 -g 1 1 %DCRAW% %dcrParams% -r 1 1 1 1 -O dcr_flat_nc2.tiff %SRCNEF% call %PICTBAT%24card dcr_flat_nc2.tiff dcr_flat_nc2_XX.png call %PICTBAT%whiteMult dcr_flat_nc2_mat.png dcr_wm2 set dcr_wm2
dcr_wm2_b=0.4306858930342565 dcr_wm2_g=0.6093537804226749 dcr_wm2_mb=1.414844995571302 dcr_wm2_mr=2.011383096605218 dcr_wm2_r=0.3029526207370107
%DCRAW% %dcrParams% -r %dcr_wm2_mr% 1 %dcr_wm2_mb% 1 -O dcr_real_nc2.tiff %SRCNEF% call %PICTBAT%24card dcr_real_nc2.tiff dcr_real_nc2_XX.png call %PICTBAT%grayPoly dcr_real_nc2_mat.png dcr_gp2 set dcr_gp2
dcr_gp2_polyb=-0.1660481056765132,1.084244140875546,-0.001469736138722338 dcr_gp2_polyg=-0.01858332659971874,1.0087915315815,0.0001250062521560949 dcr_gp2_polyr=0.1198562568555971,0.940842675243889,0.0003073658570344404
%IM7DEV%magick ^ dcr_real_nc2.tiff ^ -channel R -function Polynomial %dcr_gp2_polyr% ^ -channel G -function Polynomial %dcr_gp2_polyg% ^ -channel B -function Polynomial %dcr_gp2_polyb% ^ +channel ^ -set colorspace RGB ^ -colorspace sRGB ^ -auto-level ^ -process 'setmnsd sd 0.166667 d incOnly v' ^ +write dcr_rpnc2_sd.tiff ^ %WEBSIZE% ^ -quality 40 ^ dcr_rpnc2_sd_sm.jpg
setmnsd options: verbose sdGoal 0.166667 tolerance 1e-05 mid mean initCon0 1e-07 initCon2 30 direction incOnlydecOnlyboth input: mean=0.19257008 sd=0.11061397 nIter=19 result: mean=0.28864945 sd=0.1666758 setmnsd command: -sigmoidal-contrast 4.167876,19.257008%
Here is the result:
|
dcr_rpnc2_sd_sm.jpg |
|
call %PICTBAT%propIntClipRGB dcr_flat_nc2.tiff
shadowClipped.0=210352 shadowClipped.1=13709 shadowClipped.2=358816 highlightClipped.0=1 highlightClipped.1=1 highlightClipped.2=1
call %PICTBAT%propIntClipRGB dcr_real_nc2.tiff
shadowClipped.0=84011 shadowClipped.1=15090 shadowClipped.2=220208 highlightClipped.0=1 highlightClipped.1=2 highlightClipped.2=1
call %PICTBAT%propIntClipRGB dcr_rpnc2_sd.tiff
shadowClipped.0=84011 shadowClipped.1=15090 shadowClipped.2=220208 highlightClipped.0=1 highlightClipped.1=2 highlightClipped.2=1
The "-B 0" option works well, stretching values upwards to use the full 16-bit range without clipping. Or we can allow any proportion of pixels to clip. But what proportion is reasonable? I think 1% is far too high, but how can we determine the right amount? One possibility: any pixel that is already at maximum in at least one channel, and has high values in the other channels, can be safely clipped.
Aside: dcraw contains code, in function colorcheck(), that does some processing with pixels of a 24-colour card in an image. The coordinates of each patch (w, h, x, y) should be put in the array cut[24][4], but the only way of doing this is to modify the dcraw source code. Weird.
For convenience, .bat scripts are also available in a single zip file. See Zipped BAT files.
rem From bottom-left pixel of image %1, rem calculates multipliers green/red and green/blue. rem Sets environment variables prefixed by %2. @rem @rem Updated: @rem 18-August-2022 for IM v7. @rem @if "%1"=="" findstr /B "rem @rem" %~f0 & exit /B 1 @setlocal enabledelayedexpansion @call echoOffSave call %PICTBAT%setInOut %1 wm set PREF=%2 if "%PREF%"=="." set PREF= if "%PREF%"=="" set PREF=whiteMult set r= set FMT=^ r=%%[fx:mean.r]\n^ g=%%[fx:mean.g]\n^ b=%%[fx:mean.b]\n^ mr=%%[fx:mean.g/mean.r]\n^ mb=%%[fx:mean.g/mean.b]\n for /F "usebackq" %%L in (`%IMG7%magick ^ -precision 16 ^ %INFILE% ^ -gravity SouthWest -crop 1x1+0+0 +repage ^ -format "%FMT%" ^ info:`) do set %%L if "%r%"=="" exit /B 1 echo %PREF%_r=%r% %PREF%_g=%g% %PREF%_b=%b% echo %PREF%_mr=%mr% %PREF%_mb=%mb% call echoRestore @endlocal & set %PREF%_r=%r%&set %PREF%_g=%g%&set %PREF%_b=%b%& set %PREF%_mr=%mr%&set %PREF%_mb=%mb%
rem From botom row of image %1, assuming pixels should be neutral grays,
rem calculates channel polynomials.
rem Sets environment variables prefixed by %2.
@rem
@rem Updated:
@rem 20-August-2022 for IM v7.
@rem
@if "%1"=="" findstr /B "rem @rem" %~f0 & exit /B 1
@setlocal enabledelayedexpansion
@call echoOffSave
call %PICTBAT%setInOut %1 gp
set PREF=%2
if "%PREF%"=="." set PREF=
if "%PREF%"=="" set PREF=grayPoly
set polyr=
for /F "usebackq tokens=1,2 delims==" %%A in (`%IM7DEV%magick ^
%INFILE% ^
-gravity South -crop x1+0+0 +repage ^
^( +clone ^
-colorspace Gray ^
^) ^
-precision 16 ^
-process 'cols2mat method NoCrossPoly noTrans f stdout' ^
NULL:`) do (
if "%%A"=="PolyRed" set polyr=%%B
if "%%A"=="PolyGreen" set polyg=%%B
if "%%A"=="PolyBlue" set polyb=%%B
)
if "%mypolyr%"=="" exit /B 1
echo %PREF%_polyr=%polyr%
echo %PREF%_polyg=%polyg%
echo %PREF%_polyb=%polyb%
call echoRestore
@endlocal & set %PREF%_polyr=%polyr%&set %PREF%_polyg=%polyg%&set %PREF%_polyb=%polyb%
@rem From image %1,
@rem after auto-levelling each channel independently,
@rem finds number of pixels that have 0 or 100% in each channel.
@rem
@rem Updated:
@rem 20-August-2022 for IM v7.
@rem
@%IMG7%magick ^
%1 ^
-channel RGB ^
-auto-level ^
-separate ^
+channel ^
( -clone 0-2 ^
-fill White +opaque Black ^
-negate ^
-format "shadowClipped.%%s=%%[fx:int(mean*w*h+0.5)]\n" ^
+write info: ^
-delete 0-2 ^
) ^
-fill Black +opaque White ^
-format "highlightClipped.%%s=%%[fx:int(mean*w*h+0.5)]\n" ^
info:
--- dcraw.c 2017-11-06 23:02:40.011105100 +0000
+++ dcrawimg.c 2018-05-17 10:50:13.685703600 +0100
@@ -2,6 +2,8 @@
dcraw.c -- Dave Coffin's raw photo decoder
Copyright 1997-2016 by Dave Coffin, dcoffin a cybercom o net
+ [This file has been slightly hacked by Alan Gibson.]
+
This is a command-line ANSI C program to convert raw photos from
any digital camera on any computer running any operating system.
@@ -21,6 +23,9 @@
$Revision: 1.477 $
$Date: 2016/05/10 21:30:43 $
+
+ Updates (by Alan Gibson):
+ 17-March-2018 Added "-R" option.
*/
#define DCRAW_VERSION "9.27"
@@ -28,6 +33,8 @@
#ifndef _GNU_SOURCE
#define _GNU_SOURCE
#endif
+#define WIN32_LEAN_AND_MEAN
+#define VC_EXTRALEAN
#define _USE_MATH_DEFINES
#include <ctype.h>
#include <errno.h>
@@ -42,7 +49,7 @@
#include <time.h>
#include <sys/types.h>
-#if defined(DJGPP) || defined(__MINGW32__)
+#if defined(DJGPP) || defined(__MINGW32__) || defined(WIN32) || defined(WIN64)
#define fseeko fseek
#define ftello ftell
#else
@@ -68,6 +75,10 @@
typedef unsigned long long UINT64;
#endif
+#if !defined(M_PI)
+# define M_PI 3.14159265358979323846
+#endif
+
#ifdef NODEPS
#define NO_JASPER
#define NO_JPEG
@@ -104,6 +115,7 @@
FILE *ifp, *ofp;
short order;
const char *ifname;
+const char *outfile = NULL;
char *meta_data, xtrans[6][6], xtrans_abs[6][6];
char cdesc[5], desc[512], make[64], model[64], model2[64], artist[64];
float flash_used, canon_ev, iso_speed, shutter, aperture, focal_len;
@@ -140,6 +152,9 @@
void (*write_thumb)(), (*write_fun)();
void (*load_raw)(), (*thumb_load_raw)();
jmp_buf failure;
+double BrightThr=0.0001;
+int noNormWBM=0;
+float wbmDiv=1.0;
struct decode {
struct decode *branch[2];
@@ -4247,11 +4262,18 @@
dmax = pre_mul[c];
}
if (!highlight) dmax = dmin;
+ if (noNormWBM) dmax = wbmDiv;
+ if (verbose) {
+ fprintf (stderr,
+ _("Multipliers pre-norm: ") );
+ FORCC fprintf (stderr, " %f", pre_mul[c]);
+ fputc ('\n', stderr);
+ }
FORC4 scale_mul[c] = (pre_mul[c] /= dmax) * 65535.0 / maximum;
if (verbose) {
fprintf (stderr,
_("Scaling with darkness %d, saturation %d, and\nmultipliers"), dark, sat);
- FORC4 fprintf (stderr, " %f", pre_mul[c]);
+ FORCC fprintf (stderr, " %f", pre_mul[c]);
fputc ('\n', stderr);
}
if (filters > 1000 && (cblack[4]+1)/2 == 1 && (cblack[5]+1)/2 == 1) {
@@ -9538,6 +9560,21 @@
for (out_cam[i][j] = k=0; k < 3; k++)
out_cam[i][j] += out_rgb[output_color-1][i][k] * rgb_cam[k][j];
}
+
+ if (verbose) {
+ fprintf (stderr, "rgb_cam: ");
+ for (i=0; i < 3; i++)
+ for (j=0; j < colors; j++)
+ fprintf (stderr, " %g", rgb_cam[i][j]);
+ fprintf (stderr, "\n");
+
+ fprintf (stderr, "out_cam: ");
+ for (i=0; i < 3; i++)
+ for (j=0; j < colors; j++)
+ fprintf (stderr, " %g", out_cam[i][j]);
+ fprintf (stderr, "\n");
+ }
+
if (verbose)
fprintf (stderr, raw_color ? _("Building histograms...\n") :
_("Converting to %s colorspace...\n"), name[output_color-1]);
@@ -9794,7 +9831,7 @@
int c, row, col, soff, rstep, cstep;
int perc, val, total, white=0x2000;
- perc = width * height * 0.01; /* 99th percentile white level */
+ perc = width * height * BrightThr; /* eg 0.01 for 99th percentile white level */
if (fuji_width) perc /= 2;
if (!((highlight & ~2) || no_auto_bright))
for (white=c=0; c < colors; c++) {
@@ -9802,6 +9839,12 @@
if ((total += histogram[c][val]) > perc) break;
if (white < val) white = val;
}
+
+ if (verbose)
+ fprintf (stderr,_("bright=%f, gamma_curve imax=%i\n"),
+ bright,
+ (int)floor((white << 3)/bright));
+
gamma_curve (gamm[0], gamm[1], 2, (white << 3)/bright);
iheight = height;
iwidth = width;
@@ -9862,6 +9905,7 @@
if (argc == 1) {
printf(_("\nRaw photo decoder \"dcraw\" v%s"), DCRAW_VERSION);
printf(_("\nby Dave Coffin, dcoffin a cybercom o net\n"));
+ printf(_("slightly hacked by Alan Gibson v2.0\n"));
printf(_("\nUsage: %s [OPTION]... [FILE]...\n\n"), argv[0]);
puts(_("-v Print verbose messages"));
puts(_("-c Write image data to standard output"));
@@ -9873,8 +9917,10 @@
puts(_("-a Average the whole image for white balance"));
puts(_("-A <x y w h> Average a grey box for white balance"));
puts(_("-r <r g b g> Set custom white balance"));
+ puts(_("-R n Divisor for white balance multipliers"));
puts(_("+M/-M Use/don't use an embedded color matrix"));
puts(_("-C <r b> Correct chromatic aberration"));
+ puts(_("-O <file> Write output to this file"));
puts(_("-P <file> Fix the dead pixels listed in this file"));
puts(_("-K <file> Subtract dark frame (16-bit raw PGM)"));
puts(_("-k <num> Set the darkness level"));
@@ -9891,6 +9937,7 @@
puts(_("-D Document mode without scaling (totally raw)"));
puts(_("-j Don't stretch or rotate raw pixels"));
puts(_("-W Don't automatically brighten the image"));
+ puts(_("-B <num> Clipping threshold for auto-brighten (default = 0.0001)"));
puts(_("-b <num> Adjust brightness (default = 1.0)"));
puts(_("-g <p ts> Set custom gamma curve (default = 2.222 4.5)"));
puts(_("-q [0-3] Set the interpolation quality"));
@@ -9918,6 +9965,10 @@
case 'b': bright = atof(argv[arg++]); break;
case 'r':
FORC4 user_mul[c] = atof(argv[arg++]); break;
+ case 'R':
+ noNormWBM = 1;
+ wbmDiv = atof(argv[arg++]);
+ break;
case 'C': aber[0] = 1 / atof(argv[arg++]);
aber[2] = 1 / atof(argv[arg++]); break;
case 'g': gamm[0] = atof(argv[arg++]);
@@ -9943,6 +9994,7 @@
#endif
break;
case 'P': bpfile = argv[arg++]; break;
+ case 'O': outfile = argv[arg++]; break;
case 'K': dark_frame = argv[arg++]; break;
case 'z': timestamp_only = 1; break;
case 'e': thumbnail_only = 1; break;
@@ -9965,6 +10017,7 @@
case '4': gamm[0] = gamm[1] =
no_auto_bright = 1;
case '6': output_bps = 16; break;
+ case 'B': BrightThr = atof(argv[arg++]); break;
default:
fprintf (stderr,_("Unknown option \"-%c\".\n"), opt);
return 1;
@@ -10226,7 +10279,10 @@
if (write_to_stdout)
strcpy (ofname,_("standard output"));
else {
- strcpy (ofname, ifname);
+ if (outfile)
+ strcpy (ofname,outfile);
+ else {
+ strcpy (ofname, ifname);
if ((cp = strrchr (ofname, '.'))) *cp = 0;
if (multi_out)
sprintf (ofname+strlen(ofname), "_%0*d",
@@ -10234,6 +10290,7 @@
if (thumbnail_only)
strcat (ofname, ".thumb");
strcat (ofname, write_ext);
+ }
ofp = fopen (ofname, "wb");
if (!ofp) {
status = 1;
All images on this page were created by the commands shown, using:
%IMG7%magick -version
Version: ImageMagick 7.1.0-42 Q16-HDRI x64 396d87c:20220709 https://imagemagick.org Copyright: (C) 1999 ImageMagick Studio LLC License: https://imagemagick.org/script/license.php Features: Cipher DPC HDRI OpenCL Delegates (built-in): bzlib cairo freetype gslib heic jng jp2 jpeg jxl lcms lqr lzma openexr pangocairo png ps raqm raw rsvg tiff webp xml zip zlib Compiler: Visual Studio 2022 (193231332)
%DCRAW% | head -n 4
Raw photo decoder "dcraw" v9.27 by Dave Coffin, dcoffin a cybercom o net slightly hacked by Alan Gibson v2.0
To improve internet download speeds, some images may have been automatically converted (by ImageMagick, of course) from PNG or TIFF or MIFF to JPG.
Source file for this web page is dcrawwb.h1. To re-create this web page, execute "procH1 dcrawwb".
This page, including the images, is my copyright. Anyone is permitted to use or adapt any of the code, scripts or images for any purpose, including commercial use.
Anyone is permitted to re-publish this page, but only for non-commercial use.
Anyone is permitted to link to this page, including for commercial use.
Page version v1.0 9-November-2017.
Page created 21-Aug-2022 06:54:45.
Copyright © 2022 Alan Gibson.