snibgo's ImageMagick pages

Seamless photomontage

Poisson image editing, and related techniques.

This page describes the Poisson image editing technique for blending two or more images. I implement the basic Poisson process as a fairly simple script, relaxFillMS.bat. However, this can be used in a number of ways.

This page is concerned with images, not mathematics. (I am a picture maker, not a mathematician.) Interested readers may feast on the mathematics in the references.

CAUTION: In v1.0 of this page (23-October-2014), slope and divergence images were offset so zero was represented by pixel values of 50%.

However, this caused problems due to the Fifty percent issue, so now zero slopes are represented by pixel values of 0%, and negative values are common, and HDRI should be used.

After I wrote v2 of this page in 2017, IM has implemented operations "-compose seamless-blend -composite" and "-compose saliency-blend -composite". See Seamless-blend and Saliency-blend below.

References

This page draws heavily on:

Other related papers, though not directly considered on this page:

This page uses a version of IM assumed to be HDRI.

Sample inputs

We use two images from the following paper:

set BEAR=perez_bear.jpg
perez_bear.jpg
set WATER=perez_water.jpg
perez_water.jpg

The images are not my copyright. I think the animal is a bear, but I could be wrong.

The goal is to cut-out the bear and paste it over the water. The water in the foreground bear photo is generally darker than the required background.

For the cut-out, we will use a mask that I made with Gimp, roughly following the example in the Pérez paper. It is white where we want the bear image; otherwise black.

set MASK=bear_mask.png
bear_mask.png

We will offset the bear image:

set sOFFSET=+40+30

set sGEOM=-geometry %sOFFSET%

Get the dimensions of the original bear image:

for /F "usebackq" %%L in (`%IMG7%magick identify ^
  -format "WW=%%w\nHH=%%h\nWm1=%%[fx:w-1]\nHm1=%%[fx:h-1]" ^
  %BEAR%`) do set %%L

echo WW=%WW% HH=%HH% 
WW=297 HH=163 

For this page, we will try to use OpenCL, to do some processing on the GPU.

set MAGICK_OCL_DEVICE=true

Simple montage

We can simply cut out the bear and composite over the water.

%IMG7%magick ^
  %WATER% ^
  ( %BEAR% %MASK% -alpha off ^
    -compose CopyOpacity -composite ^
  ) ^
  %sGEOM% ^
  -compose Over -composite ^
  spm_simpmont.jpg
spm_simpmont.jpg

The boundary between the images is obvious. We want a seamless boundary. We could blur the boundary, but the difference in the water would still be obvious.

Poisson pasting

The basic method in the Pérez paper is called "Poisson interpolation", "seamless boundary by Poisson image editing" or even "seamless boundary by Poisson equation with Dirichlet boundary conditions". For short, I call it simply "Poisson pasting".

The simple version of the method (with no explicit "guidance vector field") adds an adjustment (or "correction") image to the foreground bear image before compositing it over the water background. The adjustment image has pixels corresponding to the mask boundary exactly equal to the water image minus the bear image. Hence, when added to the bear image, those pixels will be equal to the water background.

The pixels in the adjustment image outside the mask boundary won't be used so we don't care what values they have. The pixels inside the mask boundary do matter. Values should change smoothly, interpolating between values at the boundary. The smoother they change, the better the result. Put it another way: we should smooth the pixels of the adjustment image, and keep smoothing until they no longer change. This is exactly what relaxFillMS.bat does. (See Filling holes: relaxation.)

Calculate background minus foreground.
Make a hole where the bear is.

%IMG7%magick ^
  %WATER% ^
  %BEAR% ^
  %sGEOM% ^
  -compose Mathematics ^
    -define compose:args=0,-0.5,0.5,0.5 ^
    -composite ^
  -crop %WW%x%HH%%sOFFSET% +repage ^
  +depth ^
  +write spm_minus.png ^
  ( %MASK% -negate ) ^
  -alpha off -compose CopyOpacity -composite ^
  spm_minus_mskd.png
spm_minus.pngjpg spm_minus_mskd.pngjpg

Fill the hole by relaxation.

call %PICTBAT%relaxFillMS ^
  spm_minus_mskd.png . spm_add1.png
spm_add1.pngjpg

Add the result to the bear image.
Cut this out wth the mask and composite over the water.

%IMG7%magick ^
  %BEAR% ^
  spm_add1.png ^
  -compose Mathematics ^
    -define compose:args=0,2,1,-1 ^
    -composite ^
  +depth ^
  +write spm_bear_adj.png ^
  %MASK% -alpha off ^
  -compose CopyOpacity -composite ^
  %WATER% ^
  +swap ^
  %sGEOM% ^
  -compose Over -composite ^
  spm_s2.png
spm_bear_adj.pngjpg spm_s2.pngjpg

The result is good. We have the bear's reflection in the water, and the water's colour is taken from the correct image. Note that the water has lightened, but so has the bear. If we want to prevent the bear from becoming lightened, see Inner Dirichlet boundary below.

We can express the pixels that fill the hole as:

fill_pixels = bear + relax (water - bear)

At the boundary:

relax (water - bear) = water - bear

Hence, at the boundary:

fill_pixels = bear + water - bear
            = water

ASIDE

Instead of relaxFillMS.bat, we could use blurFill.bat. But this creates a discontinuity at the centre. For this example, the discontinuity is a roughly horizontal line that coincides with the base of the bear, so the problem it creates isn't visible.

call %PICTBAT%blurFill ^
  spm_minus_mskd.png . spm_blr_add.png
spm_blr_add.pngjpg

ASIDE 2

For the two mathematics composites, we could use compose:args=0,-1,1,0.5 and compose:args=0,1,1,-0.5 respectively. But if we don't use HDRI, this can clip the first result. The parameters used above have the same effect but ensure we have no clipping (at the expense of one bit of precision).

Above, we have adjusted the bear image to match the required background water. This has visibly lightened the bear's fur. Instead, we can adjust the water to match the bear.

%IMG7%magick ^
  %WATER% ^
  %BEAR% ^
  %sGEOM% ^
  -compose Mathematics ^
    -define compose:args=0,0.5,-0.5,0.5 ^
    -composite ^
  +depth ^
  +write spm_minus_r.png ^
  -alpha transparent ^
  ( %MASK%  -alpha off ) ^
  %sGEOM% ^
  -compose CopyOpacity -composite ^
  spm_minus_mskd_r.png
spm_minus_r.pngjpg spm_minus_mskd_r.pngjpg
call %PICTBAT%relaxFillMS ^
  spm_minus_mskd_r.png . ^
  spm_add1_r.png 1e-6 1000
spm_add1_r.pngjpg
%IMG7%magick ^
  %WATER% ^
  spm_add1_r.png ^
  -compose Mathematics ^
    -define compose:args=0,2,1,-1 ^
    -composite ^
  +depth ^
  +write spm_bear_adj_r.png ^
  ( %MASK% -negate ) -alpha off ^
  %sGEOM% ^
  -set option:compose:outside-overlay false ^
  -compose CopyOpacity -composite ^
  %BEAR% ^
  %sGEOM% ^
  -compose DstOver -composite ^
  +depth ^
  spm_s2_r.png
spm_bear_adj_r.pngjpg spm_s2_r.pngjpg

So we can paste a foreground to a background with seamless boundary, either by adjusting the foreground or background. Of course, we could blend the two results.

The code above has used a boundary as exactly specified by a mask. Instead, a mask could define a range of possible boundaries, eg by describing a ring 20 pixels thick. Code could then find the darkest path around the ring, and we use that as the actual boundary.

%IMG7%magick ^
  %MASK% ^
  -alpha off ^
  -morphology EdgeOut disk:20 ^
  spm_ring_mask.png
spm_ring_mask.pngjpg

We use shapeDp.bat to find the darkest path in the difference between the images, around this ring. (See Awkward boundaries with dark paths.) This needs two input images and the ring mask, all the same size.

%IMG7%magick ^
  %WATER% ^
  -crop %WW%x%HH%%sOFFSET% +repage ^
  spm_back_crp.png
spm_back_crp.pngjpg
set sdpBLUR_SIG=0
set sdpMASK_FILE=spm_dp_mask.png

call %PICTBAT%shapeDp ^
  spm_back_crp.png %BEAR% spm_ring_mask.png
spm_back_crp_sdp.pngjpg spm_dp_mask.pngjpg

shapeDp.bat outputs the second image composited over the first, via a calculated mask. We use that calculated mask.

set MASK2=spm_dp_mask.png

%IMG7%magick ^
  %WATER% ^
  %BEAR% ^
  %sGEOM% ^
  -compose Mathematics ^
    -define compose:args=0,-0.5,0.5,0.5 ^
    -composite ^
  -crop %WW%x%HH%%sOFFSET% +repage ^
  +depth ^
  +write spm_minus_rng.png ^
  ( %MASK2% -negate ) ^
  -alpha off ^
  -compose CopyOpacity -composite ^
  spm_minus_mskd_rng.png
spm_minus_rng.pngjpg spm_minus_mskd_rng.pngjpg
call %PICTBAT%relaxFillMS ^
  spm_minus_mskd_rng.png . spm_add1_rng.png
spm_add1_rng.pngjpg
%IMG7%magick ^
  %BEAR% ^
  spm_add1_rng.png ^
  -compose Mathematics ^
    -define compose:args=0,2,1,-1 ^
    -composite ^
  +depth ^
  +write spm_bear_adj_rng.png ^
  %MASK2% -alpha off ^
  -compose CopyOpacity -composite ^
  %WATER% ^
  +swap ^
  %sGEOM% ^
  -compose Over -composite ^
  +depth ^
  spm_s2_rng.png
spm_bear_adj_rng.pngjpg spm_s2_rng.pngjpg

The only noticable difference is that the water around the bear seems calmer. This is simply because the boundary is larger.

poissonPaste script

We implement the above in the script poissonPaste.bat. For simplicity, it requires the two input images and the mask to be the same size, with no offsets.

The script will adjust part of the second input image to match the first at the mask boundary. Using the mask, it then cuts out the second image and composites that over the first.

(Thus, for the common case where we adjust part of a small image to paste over a larger image, the relaxFillMS.bat wastes time blurring a large area that will be overwritten. The script could be optimised, eg by trimming the mask. Or we could have a wrapper that finds the trim of the mask, crops all three images to that, calls poissonPaste.bat, and finally reassembles the pieces.)

In this example, we extend the bear and mask to be the same size as the water.

%IMG7%magick ^
  %WATER% ^
  +depth ^
  -fill Black -colorize 100 ^
  ( -clone 0 ^
    %BEAR% ^
    %sGEOM% ^
    -compose Over -composite ^
    +write spm_bear_ext.png ^
    +delete ^
  ) ^
  ( -clone 0 ^
    %MASK% ^
    %sGEOM% ^
    -compose Over -composite ^
    +write spm_mask_ext.png ^
    +delete ^
  ) ^
  NULL:
spm_bear_ext.pngjpg spm_mask_ext.pngjpg

We can paste the bear over the water, or the water over the bear. The second image is adjusted to match the first, the "under" image, at the boundary and pasted over it.

Poisson-paste the bear over the water:

call %PICTBAT%poissonPaste ^
  %WATER% spm_bear_ext.png spm_mask_ext.png ^
  spm_pp_out1.png
spm_pp_out1.pngjpg

Poisson-paste the water over the bear:

call %PICTBAT%poissonPaste ^
  spm_bear_ext.png %WATER% spm_mask_ext.png ^
  spm_pp_out2.png
spm_pp_out2.pngjpg

In the next two examples, we negate the effect of the mask. As before, the second image is adjusted to match the first and pasted over it. But the adjustment and pasting is where the mask is black, instead of white.

Poisson-paste the bear under the water:

call %PICTBAT%poissonPaste ^
  %WATER% spm_bear_ext.png spm_mask_ext.png ^
  spm_pp_out3.png 1
spm_pp_out3.pngjpg

Poisson-paste the water under the bear:

call %PICTBAT%poissonPaste ^
  spm_bear_ext.png %WATER% spm_mask_ext.png ^
  spm_pp_out4.png 1
spm_pp_out4.pngjpg

Isolating the bear

An improved boundary for the Poisson pasting (relaxFillMs.bat) can be found by isolating the bear from its background.

Theorem: A 3-D object with a constant colour, illuminated by white light, appears in a photograph to have varying lightness, somewhat varying saturation (or chroma), but constant hue.

Corollary: if we want to segment a photograph by objects, the hue channel is all we need.

In real life, objects rarely have exactly constant colour. And they are illuminated not merely by the main light source but also by light reflected by other objects, and this reflected light will be coloured. This is particularly evident when a low-saturation (grayish) object is close to a high-saturation object (eg bright red). The gray object easily picks up red light from the reflected red object.

And objects that have very low saturation often have widely varying hues, eg a gray object may have adjacent pixels that are sRGB(50.001%,50%,50%) and sRGB(50%,50%,50.001%). This happens at the bear's nose, which is nearly black.

And reducing an image to just hues can cause objects that are clearly distinct to merge into each other.

But it is often a useful first approximation.

Here, we set the Lightness of all pixels to 50%. We could set all Chroma to 100%, but this gives false colours to pixels at the nose, so we set very low Chroma to zero, other Chroma to 100%, with a short ramp between them.

%IMG7%magick ^
  %BEAR% ^
  -colorspace HCL ^
  -channel 1 -level 3,5%% ^
  -channel 2 -evaluate set 50%% ^
  +channel ^
  -colorspace sRGB ^
  spm_hue.png
spm_hue.png

We want just two colours. In general, when we want just (N) colours, using "-colors N" doesn't work well because many pixels we would want in in a particular group actually fall into the other. By using "-colors N+x", virtually all the pixels allocated to one of the groups will be in the correct group. If N+x is too large, we fragment the natural groups too much that the next step won't work. N+2 often works well, when N is small.

%IMG7%magick ^
  spm_hue.png ^
  +dither -colors 4 ^
  spm_hue_4.png
spm_hue_4.png

The sea has essentially two colours. Where the water is tilted up by a wave towards the camera, we see the colour of the water, which is somewhat green. Where the water is tilted away, the far side of a wave, we can't see into the water and instead see reflected light from the sky, which is more blue.

The sea also has a heavily distorted reflection and refraction of the bear. When reduced to four colours, a couple of small shapes have the bear's colour.

The script connCompLimArea.bat removes very small components (here, those that are less than 0.1% of the total image area).

call %PICTBAT%connCompLimArea ^
  spm_hue_4.png spm_hue_4a.png 0.1c
spm_hue_4a.png

The script connCompLimNum.bat here limits the image to two components, mapping the other pixels to the colours of those two components. We have successfully lassooed the bear. Starting from the bear surrounded by water, we have found an outline for the bear.

call %PICTBAT%connCompLimNum ^
  spm_hue_4a.png spm_hue_4a2.png 2
spm_hue_4a2.png

(An alternative method for simplifying spm_hue_4.png is: find the unique colours in the four edges, remap the image to these; any pixels that have changed were a different colour to all edge pixels.)

Change the colours so we have a white background and black object.

%IMG7%magick ^
  spm_hue_4a2.png ^
  -fill White ^
  -draw "color 0,0 floodfill" ^
  -fill Black +opaque White ^
  spm_hue_bw.png
spm_hue_bw.png

Combine with the user-supplied mask:

%IMG7%magick ^
  spm_hue_bw.png ^
  %MASK% ^
  -compose Darken -composite ^
  spm_hue_bw2.png
spm_hue_bw2.png

Now we have an outer and inner bound for a minimum error boundary cut (MEBC). The outer bound has been supplied by the user; the inner bound is from the lassoo. We find the optimum boundary within those limits.

set sdpBLUR_SIG=0
set sdpMASK_FILE=spm_dp_mask2.png

call %PICTBAT%shapeDp ^
  spm_back_crp.png %BEAR% spm_hue_bw2.png ^
  spm_ring_mask2.png
spm_dp_mask2.png spm_ring_mask2.png

We can now use Poisson pasting with this mask.

Poisson-paste the bear over the water:

%IMG7%magick ^
  %WATER% ^
  +depth ^
  -fill Black -colorize 100 ^
  spm_dp_mask2.png ^
  %sGEOM% ^
  -compose Over -composite ^
  spm_mask2_ext.png

call %PICTBAT%poissonPaste ^
  %WATER% spm_bear_ext.png spm_mask2_ext.png ^
  spm_pp_out5.png
spm_pp_out5.pngjpg

(Alternative: use Pp with spm_hue_bw.png, possibly dilated.)

A better boundary?

See Drag-and-Drop Pasting Jiaya Jia, Jian Sun, Chi-Keung Tang, Heung-Yeung Shum, 2009.

poissonPaste.bat adjusts one image to match another at their boundary, by adding an adjustment image. The adjustment image is a relaxed ("membranised") version of the difference between the two input images. If the input images were identical at the boundary, the adjustment image would be zero at the boundary, so the relaxation would also be black and there would be no adjustment (nor any need for an adjustment). If the boundary pixels in the adjustment image were identical to each other, the relaxation would be a constant, flat colour, so the adjustment would be a constant colour-shift.

We see this in the above example: the bear is lightened by the process.

Difficulties arise when one of the inputs contains detail (i.e. gradients) at the boundary that the other doesn't. This creates a variation in the boundary pixels of the adjustment image, and this variation is propagated somewhat into the adjustment, which causes localised colour-shifts.

This isn't obvious in the above examples because both images have variation at the boundary, and the difference in the variation is of a similar magnitude. We can construct an artificial example:

%IMG7%magick ^
  %BEAR% ^
  -crop 2x2@ ^
  ( -clone 0 -fill White -colorize 100 ) ^
  -delete 0 ^
  ( -clone 0 -fill Lime -colorize 100 ) ^
  -delete 0 ^
  ( -clone 0 -fill Red -colorize 100 ) ^
  -delete 0 ^
  ( -clone 0 -fill Black -colorize 100 ) ^
  -delete 0 ^
  -layers merge ^
  spm_chq.png
spm_chq.png
call %PICTBAT%poissonPaste ^
  spm_chq.png %BEAR% %MASK% ^
  spm_chq_out.png
spm_chq_out.pngjpg

The four background colours have clearly propagated into the bear. In this example, no alternative boundary is available, so we can't avoid the effect.

When we have a choice of boundaries, represented in examples above by a white ring, we have chosen the path that minimises the total difference between the input images.

Instead, we can choose the path that minimises the difference in the slopes (a.k.a. gradients; see the Slopes page) of the inputs. For both inputs, we calculate the slopes and record x- and y-deltas for each colour channel in two images (because we need six channels). For each channel we then calculate the difference between the slopes, and use the maximum magnitude of these as the input to the darkestpntpnt process module.

(Instead of using six channels, we could instead use the slopes of the grayscaled images, needing just two channels.)

For the slope*.bat scripts, see the Slopes page.

Below, "-sigmoidal-contrast" and "-auto-level" are merely to aid visibility.

Get the slope of the bear image.

call %PICTBAT%slopeXY ^
  %BEAR% spm_bear_slp.miff

%IMG7%magick spm_bear_slp.miff[0] ^
  -evaluate Divide 2 ^
  -evaluate Add 50%% ^
  -sigmoidal-contrast 10,50%% ^
  spm_bear_slp_0.jpg

%IMG7%magick spm_bear_slp.miff[1] ^
  -evaluate Divide 2 ^
  -evaluate Add 50%% ^
  -sigmoidal-contrast 10,50%% ^
  spm_bear_slp_1.jpg
spm_bear_slp_0.jpg spm_bear_slp_1.jpg

Get the slope of the water image.

call %PICTBAT%slopeXY ^
  spm_back_crp.png spm_water_slp.miff

%IMG7%magick spm_water_slp.miff[0] ^
  -evaluate Divide 2 ^
  -evaluate Add 50%% ^
  -sigmoidal-contrast 10,50%% ^
  spm_water_slp_0.jpg

%IMG7%magick spm_water_slp.miff[1] ^
  -evaluate Divide 2 ^
  -evaluate Add 50%% ^
  -sigmoidal-contrast 10,50%% ^
  spm_water_slp_1.jpg
spm_water_slp_0.jpg spm_water_slp_1.jpg

Subtract: slope(water) - slope(bear)

call %PICTBAT%slopeXYminus ^
  spm_bear_slp.miff spm_water_slp.miff ^
  spm_minus_slp.miff

%IMG7%magick spm_minus_slp.miff[0] ^
  -evaluate Divide 2 ^
  -evaluate Add 50%% ^
  -sigmoidal-contrast 10,50%% ^
  spm_minus_slp_0.jpg

%IMG7%magick spm_minus_slp.miff[1] ^
  -evaluate Divide 2 ^
  -evaluate Add 50%% ^
  -sigmoidal-contrast 10,50%% ^
  spm_minus_slp_1.jpg
spm_minus_slp_0.jpg spm_minus_slp_1.jpg

Get the magnitude of the slope difference.

call %PICTBAT%slopeXYmag ^
  spm_minus_slp.miff ^
  spm_mag_slp.miff

%IMG7%magick spm_mag_slp.miff ^
  -auto-level ^
  spm_mag_slp.jpg
spm_mag_slp.jpg

Find the line of minimum total magnitude within the ring,
and make that a white shape on black background.

call %PICTBAT%minDp ^
  spm_mag_slp.miff spm_hue_bw2.png ^
  spm_mag_msk.png
spm_mag_msk.png

Poisson-paste.

call %PICTBAT%poissonPaste ^
  spm_back_crp.png %BEAR% spm_mag_msk.png ^
  spm_mag_out.png
spm_mag_out.pngjpg

In some ways, this result is more accurate than previous results, creating less bleeding of features from the background to the new foreground. However, the boundary in this example is also tighter, closer to the bear. We have less of the bear's reflection in the water.

Guided gradients

Ordinary (non-guided) Poisson Image Editing creates a result that has a gradient very close to g, the required foreground ("source"). The result is made by adding a membrane to the foreground, where the membrane was made by relaxing a boundary of (background minus foreground).

Guided Poisson Image Editing creates a result that has a gradient very close to v, called a "guidance vector field". v is set to some function of g and f* (the original background). The result is made from the membrane itself, not added to anything, and the membrane is made by relaxing a boundary of simply the background. But the method of relaxation is more complicated.

The guidance vector field might come from the gradient of a single image, or from a mixture of gradients of different images (aka mixed gradients).

Below, spm_ws_div.miff is the right hand side of the equation:

f(x-1,y) + f(x+1,y) + f(x,y-1) + f(x,y+1) - 4*f(x,y) = rhs(x,y)

So each iteration is:

f(x,y) = f(x-1,y) + f(x+1,y) + f(x,y-1) + f(x,y+1) - rhs(x,y) 
                             4                          4

In IM terms: convolve once, subtract something, convolve again, subtract the same thing, and keep going.

For example, we take two other images from the Pérez paper:

A scribble image.

set SCRIBBLE=perez_scribble.jpg
perez_scribble.jpg

A wall image, cropped to the same size as the scribble.

%IMG7%magick ^
  %SCRIBBLE% ^
  perez_wall.jpg ^
  -compose Over -composite ^
  spm_wall_crp.png

set WALL=spm_wall_crp.png
spm_wall_crp.pngjpg

Make a mask in Gimp, roughly the same as the Pérez paper.

set WS_MASK=perez_ws_mask.png
perez_ws_mask.png

For the result: where the mask is black, we want the wall. Where the mask is white, we want the gradient of either the wall or the scribble, whichever is more significant at that pixel.

To do that, we find the gradients (on x and y directions) of both images; find the magitudes of each gradient; find which magnitude is greater; for each pixel, use the slope of the image with the greatest magnitude; find the divergence of that.

Slope of the scribble.

call %PICTBAT%slopeXY ^
  %SCRIBBLE% spm_scrb_sxy.miff

%IMG7%magick ^
  spm_scrb_sxy.miff[0] ^
  -evaluate Divide 2 ^
  -evaluate Add 50%% ^
  spm_scrb_sxy_0.png

%IMG7%magick ^
  spm_scrb_sxy.miff[1] ^
  -evaluate Divide 2 ^
  -evaluate Add 50%% ^
  spm_scrb_sxy_1.png
spm_scrb_sxy_0.png spm_scrb_sxy_1.png

Slope of the wall.

call %PICTBAT%slopeXY ^
  %WALL% spm_wall_sxy.miff

%IMG7%magick ^
  spm_wall_sxy.miff[0] ^
  -evaluate Divide 2 ^
  -evaluate Add 50%% ^
  spm_wall_sxy_0.png

%IMG7%magick ^
  spm_wall_sxy.miff[1] ^
  -evaluate Divide 2 ^
  -evaluate Add 50%% ^
  spm_wall_sxy_1.png
spm_wall_sxy_0.png spm_wall_sxy_1.png

Magnitude of the scribble slope.

call %PICTBAT%slopeXYmag ^
  spm_scrb_sxy.miff spm_scrb_mag.miff
spm_scrb_mag.miffjpg

Magnitude of the wall slope.

call %PICTBAT%slopeXYmag ^
  spm_wall_sxy.miff spm_wall_mag.miff
spm_wall_mag.miffjpg

Which magnitude is greatest?

White for scribble, black for wall.

%IMG7%magick ^
  spm_wall_mag.miff spm_scrb_mag.miff ^
  -colorspace Gray ^
  -compose MinusDst -composite ^
  -threshold 0 ^
  spm_ws_mag_diff.png
spm_ws_mag_diff.png

Use the slope that has the greatest magnitude.

rem FIXME: Can't use mpr??

%IMG7%magick ^
  -define compose:clamp=off ^
  ( spm_wall_sxy.miff[0] ^
    spm_scrb_sxy.miff[0] ^
    spm_ws_mag_diff.png +write mpr:MSK ^
    -alpha off ^
    -compose Over -composite ^
  ) ^
  ( spm_wall_sxy.miff[1] ^
    spm_scrb_sxy.miff[1] ^
    spm_ws_mag_diff.png ^
    -alpha off ^
    -compose Over -composite ^
  ) ^
  -define quantum:format=floating-point ^
  spm_ws_mxslp.miff

%IMG7%magick ^
  spm_ws_mxslp.miff[0] ^
  -evaluate Divide 2 ^
  -evaluate Add 50%% ^
  spm_ws_mxslp_0.png

%IMG7%magick ^
  spm_ws_mxslp.miff[1] ^
  -evaluate Divide 2 ^
  -evaluate Add 50%% ^
  spm_ws_mxslp_1.png
spm_ws_mxslp_0.png spm_ws_mxslp_1.png

Find the divergence.

call %PICTBAT%slopeXYdiv ^
  spm_ws_mxslp.miff spm_ws_div.miff

%IMG7%magick ^
  spm_ws_div.miff ^
  -evaluate Divide 2 ^
  -evaluate Add 50%% ^
  spm_ws_div.jpg

This will be subtracted after each blur.

spm_ws_div.jpg

This divergence is the required slope of the slope of the result. The divergence image will contain negative and positive values, with a mean somewhere near zero.

With the mask, make a hole in the wall.

%IMG7%magick ^
  %WALL% ^
  ( %WS_MASK% -negate ) ^
  -alpha off ^
  -set option:compose:outside-overlay false ^
  -compose CopyOpacity -composite ^
  spm_wall_msk.miff
spm_wall_msk.miffjpg

We relax-fill the hole, using the divergence as the guidance field.

call %PICTBAT%relaxFillMS ^
  spm_wall_msk.miff . spm_wall_msk_rfs2.miff ^
  0.001 1000 . spm_ws_div.miff
0 00:01:49
spm_wall_msk_rfs2.miffjpg

This is essentially the same result as in the Pérez paper. The blue scribble has become purple, i.e. it has increased in redness. This is because the wall is redder than the scribble, at the boundary.

We can test the "null" fill by calculating the div of the wall, and using that as the guidance to fill the hole in the wall.

Find the divergence.

call %PICTBAT%slopeXYdiv ^
  spm_wall_sxy.miff spm_wall_div.miff

%IMG7%magick ^
  spm_wall_div.miff ^
  -evaluate Divide 2 ^
  -evaluate Add 50%% ^
  spm_wall_div.jpg

This will be subtracted after each blur.

spm_wall_div.jpg

Relax-fill with this divergence.

call %PICTBAT%relaxFillMS ^
  spm_wall_msk.miff . spm_wall_fill_wall.miff ^
  0.001 1000 . spm_wall_div.miff

We have reconstructed the wall.

spm_wall_fill_wall.miffjpg

The result is very similar to the original wall image, but slightly blurred, as if the wall has been given "-blur 0.46".

Other results:

Make a black image, and holed black image.

%IMG7%magick ^
  %WALL% ^
  -alpha off ^
  -fill Black -colorize 100 ^
  +depth ^
  +write spm_ws_gray.png ^
  ( %WS_MASK% -negate ) ^
  -alpha off -compose CopyOpacity -composite ^
  spm_ws_gray_m.png
spm_ws_gray.png spm_ws_gray_m.png

Relax the holed wall.

call %PICTBAT%relaxFillMS ^
  spm_wall_msk.miff . spm_wall_msk_rf.miff
spm_wall_msk_rf.miffjpg

Relax the holed wall, guided by gray(50%), ie zero.

call %PICTBAT%relaxFillScr ^
  spm_wall_msk.miff . spm_wall_msk_gr_rfs.miff ^
  0.001 100 . spm_ws_gray.png

The result is the same (but slower).

spm_wall_msk_gr_rfs.miffjpg

Relax the holed gray(50%), guided by the divergence.

call %PICTBAT%relaxFillScr ^
  spm_ws_gray_m.png . spm_gray_rfs.miff ^
  0.001 100 . spm_ws_div.miff
spm_gray_rfs.miffjpg

Poisson-paste, with user-defined mask.

call %PICTBAT%poissonPaste ^
  %WALL% %SCRIBBLE% perez_ws_mask.png ^
  spm_ws_out2.png
spm_ws_out2.pngjpg

Poisson-paste, with max-magnitude mask.

call %PICTBAT%poissonPaste ^
  %WALL% %SCRIBBLE% spm_ws_mag_diff.png ^
  spm_ws_out4.png
spm_ws_out4.pngjpg

Guided colour adjustment

Another image from the Pérez paper.

%IMG7%magick ^
  perez_flower.jpg ^
  -crop 350x350+0+0 +repage ^
  spm_flower.png

set FLOWER=spm_flower.png
spm_flower.pngjpg

A loose mask, made in Gimp.

set FLR_MASK=flower_mask.png
flower_mask.png

The mask is very loose, entirely surrounding the flower; we haven't bothered to try and capture just the tendrils.

The goal is to make the background (the non-flower) monochrome, without changing the flower.

Gray version of flower.

%IMG7%magick ^
  %FLOWER% ^
  -colorspace Gray ^
  +depth ^
  spm_flr_gr.png
spm_flr_gr.pngjpg

Gray version, with hole.

%IMG7%magick ^
  spm_flr_gr.png ^
  ( %FLR_MASK% -negate ) ^
  -alpha off ^
  -compose CopyOpacity -composite ^
  spm_flr_gr_h.png
spm_flr_gr_h.pngjpg

Slope of colour flower.

call %PICTBAT%slopeXY ^
  %FLOWER% spm_flr_sxy.miff

%IMG7%magick ^
  spm_flr_sxy.miff[0] ^
  -evaluate Divide 2 ^
  -evaluate Add 50%% ^
  spm_flr_sxy_0.png

%IMG7%magick ^
  spm_flr_sxy.miff[1] ^
  -evaluate Divide 2 ^
  -evaluate Add 50%% ^
  spm_flr_sxy_1.png
spm_flr_sxy_0.pngjpg spm_flr_sxy_1.pngjpg

Divergence of slope of flower.

call %PICTBAT%slopeXYdiv ^
  spm_flr_sxy.miff spm_flr_div.miff

%IMG7%magick ^
  spm_flr_div.miff ^
  -evaluate Divide 2 ^
  -evaluate Add 50%% ^
  spm_flr_div.png

We will use this as the guidance field.

spm_flr_div.pngjpg

Relax-fill the holed grayscale version, with guidance.

call %PICTBAT%relaxFillMS ^
  spm_flr_gr_h.png . spm_flr_bwc.png ^
  0.001 1000 . spm_flr_div.miff

A good result, but the entire flower has become slightly gray.

spm_flr_bwc.pngjpg

This is essentially the same result as shown in the Pérez paper.

If the result seems too good to be true, well, it is.

Where the mask is black, the result is grayscale, of course. Where the mask is white, the result is the colour input with green "subtracted" (speaking approximately). So the flower has become less green: values in the green channel have decreased, while values in the red and blue channels have increased. This has happened because that is the change at the boundary. Similarly, yellow colour has leached out from the flower to the grayscale areas between the tendrils. This is addressed in Inner Dirichlet boundary below.

Modulating texture

The guidance field might be from the background image, but modulated in some way.

Yet another image from the Pérez paper:

set BOY=perez_boy.jpg
perez_boy.jpg

With Gimp, make a mask that includes just the facial features.

set BOY_MASK=boy_mask.png
boy_mask.png

Get the slope of BOY.

call %PICTBAT%slopeXY ^
  %BOY% spm_boy_sxy.miff

Modify the slope.

%IMG7%magick ^
  spm_boy_sxy.miff ^
  -fuzz 2%% -fill Black -opaque Black ^
  -define quantum:format=floating-point ^
  spm_boy_sxy2.miff

Get the divergence of the modified slope.

call %PICTBAT%slopeXYdiv ^
  spm_boy_sxy2.miff spm_boy_div2.miff

%IMG7%magick ^
  spm_boy_div2.miff ^
  -evaluate Divide 2 ^
  -evaluate Add 50%% ^
  spm_boy_div2.png
spm_boy_div2.pngjpg

Poisson-paste.

call %PICTBAT%poissonPaste ^
  %BOY% . %BOY_MASK% ^
  spm_boy_out2.png ^
  . spm_boy_div2.miff
spm_boy_out2.pngjpg

We have adjusted the slope so where it is within ±2% of 50% (i.e. zero), it becomes 50%. This eliminates small changes in slope. It has a side-effect of reducing overall contrast, because slope accumulates to make contrast, [[ so we compensate very approximately with "-level -80%%,80%%", but this is a kludge.]] Instead, we could match the result to the input image, using either imgGainBias.bat or matchHisto.bat. Another option would be to match the adjusted slope to the non-adjusted slope.

In the next example, after eliminating small slopes, we sharpen the remaining slopes. This increases local contrast.

Modify the slope.

%IMG7%magick ^
  spm_boy_sxy.miff ^
  -fuzz 2%% -fill Black -opaque Black ^
  -unsharp 0x3+1+0 ^
  -define quantum:format=floating-point ^
  spm_boy_sxy3.miff

Get the divergence of the modified slope.

call %PICTBAT%slopeXYdiv ^
  spm_boy_sxy3.miff spm_boy_div3.miff

%IMG7%magick ^
  spm_boy_div3.miff ^
  -evaluate Divide 2 ^
  -evaluate Add 50%% ^
  spm_boy_div3.png
spm_boy_div3.pngjpg

Poisson-paste.

call %PICTBAT%poissonPaste ^
  %BOY% . %BOY_MASK% ^
  spm_boy_out3.png ^
  . spm_boy_div3.miff
spm_boy_out3.pngjpg

This seems useful, so we put that in a script, poissonSmooth.bat. The script creates its own mask, which is black at the four edges, and otherwise white. At large values of sigma, it is effective at removing paper texture from scanned documents. This is a form of adaptive blur; see also Adaptive blur and sharpen.

call %PICTBAT%poissonSmooth ^
  toes.png spm_psm1.png 1
spm_psm1.pngjpg
call %PICTBAT%poissonSmooth ^
  toes.png spm_psm2.png 3
spm_psm2.pngjpg
call %PICTBAT%poissonSmooth ^
  toes.png spm_psm3.png 5
spm_psm3.pngjpg
call %PICTBAT%poissonSmooth ^
  toes.png spm_psm4.png 10
spm_psm4.pngjpg

A different effect is obtained by increasing the contrast of divergence, around the 50% level. This changes the slope of the slope of the result (increasing overall contrast), so a small change here goes a long way.

%IMG7%magick ^
  spm_boy_div2.miff ^
  -evaluate Divide 2 -evaluate Add 50%% ^
  -sigmoidal-contrast 2,50%% ^
  -evaluate Subtract 50%% -evaluate Multiply 2 ^
  -define quantum:format=floating-point ^
  spm_boy_div4.miff

%IMG7%magick ^
  spm_boy_div4.miff ^
  -evaluate Divide 2 ^
  -evaluate Add 50%% ^
  spm_boy_div4.png
spm_boy_div4.pngjpg

Poisson-paste.

call %PICTBAT%poissonPaste ^
  %BOY% . %BOY_MASK% ^
  spm_boy_out4.png ^
  . spm_boy_div4.miff
spm_boy_out4.pngjpg

More generally, poissonPaste.bat gives us a method of modifying the pixels in part of an image and blending the modified part into the original; an alternative to alpha blending.

Inner Dirichlet boundary

See Image Editing without Color Inconsistency Using Modified Poisson Equation, Qin Chuan, Wang Shuozhong, Zhang Xinpeng, 2008.

In examples above, the colour difference at the boundary has influenced all result pixels inside the boundary, so the bear has become lightened, the scribble has become purple, and the flower has become less green. This is also known as colour inconsistency or colour contamination. Sometimes we want this effect, but sometimes we don't.

The pixels to be filled already have an outer boundary condition, where the adjustment to be added is "water" minus "bear", so when this is added to "bear" the result at the outer boundary is "water". Chuan et al suggest we add an additional condition at the bear pixels, setting the adjustment to be added to "zero", so when this is added to "bear" the result at the inner boundary is "bear".

The script poissonPasteIDB.bat takes two masks, one for each of the outer and inner boundaries. The inner boundary mask is white where we want the bear unchanged, and black elsewhere.

Chuan et al suggest the user should create the bear mask. But we have created one automatically, so we can use that. We shrink spm_hue_bw.png by a few pixels. Black now has pixels that are in the bear. Add this to MASK, so now we have a white ring that is to be solved by Poisson equation.

The two masks together form what is often called a trimap; a map that labels some pixels as certainly inside the object, some as certainly outside, and others as unknown.

For non-guided Poisson-pasting, poissonPasteIDB does the following:

  1. Make TMP_MASK that has a white ring where we want the transition.
  2. Make TMP1 that is transparent in the transition area, 50% gray where we want the foreground, and (background minus foreground) where we want the background.
  3. Relax-fill TMP1, making TMP2.
  4. Add the foreground to TMP2. This is the required result.

For guided pasting, poissonPasteIDB does the following:

  1. Make TMP_MASK that has a white ring where we want the transition.
  2. Make TMP1 that is transparent in the transition area, foreground where we want the foreground, and background where we want the background.
  3. Relax-fill TMP1 with guidance, making TMP2.
  4. TMP2 is the required result.

As a happy side-effect: the area to be filled is now the ring between the outer and inner masks. This ring shape is smaller than before, so fewer pixels need filling by relaxFillMS.bat, so this is faster than without an inner mask.

Create the inner boundary mask.

%IMG7%magick ^
  spm_hue_bw.png ^
  -morphology Dilate disk:3 ^
  -negate ^
  spm_bear_sm.png
spm_bear_sm.png
call %PICTBAT%poissonPasteIDB ^
  spm_back_crp.png %BEAR% ^
  %MASK% spm_bear_sm.png ^
  spm_idb_out.png
spm_idb_out.pngjpg

Another example:
Bleed colours into the water but not the bear.

call %PICTBAT%poissonPasteIDB ^
  spm_chq.png %BEAR% ^
  %MASK% spm_bear_sm.png ^
  spm_idb_out2.png
spm_idb_out2.pngjpg

The result has pixels at the outer boundary the same colour as the background, and pixels at the inner boundary the same colour as the bear. Hence we now have colour consistency.

We can do the same trick with the flower.

Create the inner boundary mask with Gimp.
Ensure it has no alpha channel.

set FLR_IN_MASK=flower_inner_mask.png

%IMG7%magick ^
  %FLR_IN_MASK% ^
  -alpha off ^
  %FLR_IN_MASK%
flower_inner_mask.png

Check the masks.

%IMG7%magick ^
  %FLR_MASK% ^
  ( %FLR_IN_MASK% -negate ) ^
  -compose Darken -composite ^
  %FLOWER% ^
  +swap ^
  -alpha off ^
  -compose CopyOpacity -composite ^
  -alpha Background ^
  +depth ^
  spm_flr_chk.png
spm_flr_chk.png

Poisson-paste the grayscale version.

call %PICTBAT%poissonPasteIDB ^
  spm_flr_gr.png %FLOWER% ^
  %FLR_MASK% %FLR_IN_MASK% ^
  spm_flr_bwc3.png

The tendrils have lost some colour, become whiter.

spm_flr_bwc3.pngjpg

Poisson-paste the grayscale version, with guidance.

call %PICTBAT%poissonPasteIDB ^
  spm_flr_gr.png %FLOWER% ^
  %FLR_MASK% %FLR_IN_MASK% ^
  spm_flr_bwc4.png ^
  . spm_flr_div.miff

The tendrils are a good colour, but have slightly blurred.

spm_flr_bwc4.pngjpg

However, the tendrils have become somewhat gray and the monochrome areas between tendrils have become somewhat yellow. Chuan et al suggest a solution. In the guidance field, where the pixels represent either the yellow flower or the grayscale background, the values are low (i.e. close to zero). At the border between the flower and background, the gradient field has relatively high values (i.e. more distant from zero). If we make the high values even higher, we will get a steeper gradient between the two, so less colour bleed. Where the gradient field is above a certain threshold, they increase the values by a certain factor. For the threshold, they use 80% of the maximum value M. For the factor, they use K. They calculate:

ForeIn: mean of foreground at the inner boundary
ForeOut: mean of foreground at the outer boundary
BackOut: mean of background at the outer boundary

K = ForeIn - BackOut 
    ForeIn - ForeOut

M = max(guidance_field)

I suppose the same multiplier is applied to all three channels. But what if K is less than one? Less than zero?

More simply, we can sharpen the guidance field. Before August 2023, we didn't need to add and subtract 50% around the -unsharp. But some recent update to IM causes problems when we have negative values: they become far too negative.

Sharpen the guidance.

%IMG7%magick ^
  spm_flr_div.miff ^
  -evaluate Add 50%% ^
  -unsharp 0x0.5+1+0.05 ^
  -evaluate Subtract 50%% ^
  -depth 32 ^
  -define quantum:format=floating-point ^
  spm_flr_div_u.miff

%IMG7%magick ^
  spm_flr_div_u.miff ^
  -evaluate Divide 2 ^
  -evaluate Add 50%% ^
  spm_flr_div_u.png
spm_flr_div_u.pngjpg

Poisson-paste the grayscale version, with sharpened guidance.

call %PICTBAT%poissonPasteIDB ^
  spm_flr_gr.png %FLOWER% ^
  %FLR_MASK% %FLR_IN_MASK% ^
  spm_flr_bwc5.png ^
  . spm_flr_div_u.miff

This has reduced the bleed of yellow into the monochrome,
but introduced purple to the base of the flower.

spm_flr_bwc5.pngjpg

As an alternative to the previous, we can restore the sharpness with a small "-unsharp" applied just in the transition area, between the two masks:

%IMG7%magick ^
  spm_flr_bwc4.png ^
  ( +clone -unsharp 0x0.5+1+0 ) ^
  ( %FLR_IN_MASK% -negate ^
    %FLR_MASK% ^
    -compose Darken -composite ^
  ) ^
  -compose Over -composite ^
  spm_flr_bwc4_u.png
spm_flr_bwc4_u.pngjpg

I regard this as the best result for the yellow flower on grayscale background.

Colour-correction for panoramas

Above, we have seen that colour-change at the boundary is propagated inwards, creating a colour cast. We have seen this can be prevented by an inner Dirichlet boundary. On the other hand, we can use this to our advantage.

Poisson pasting can be used to correct colours when photographs are to be joined with overlaps ("stitched"), such as panoramas or aerial photography. Suppose we have these three input images:

#1:

%IMG7%magick ^
  -size 150x100 xc: ^
  -sparse-color bilinear ^
    0,0,#67f,0,%%[fx:h-1],#6f7,%%[fx:w-1],0,#45d ^
  -gravity Center -pointsize 30 ^
  -fill Black -annotate +0+0 "#1" ^
  -fill White -annotate -2-2 "#1" ^
  spm_pan_in1.png
spm_pan_in1.pngjpg

#2:

%IMG7%magick ^
  -size 150x100 xc: ^
  -sparse-color bilinear ^
    0,0,#f88,0,%%[fx:h-1],#7f6,%%[fx:w-1],0,#faa ^
  -fill White -gravity Center -pointsize 30 ^
  -fill Black -annotate +0+0 "#2" ^
  -fill White -annotate -2-2 "#2" ^
  spm_pan_in2.png
spm_pan_in2.pngjpg

#3:

%IMG7%magick ^
  -size 150x100 xc: ^
  -sparse-color bilinear ^
    0,0,#a6f,0,%%[fx:h-1],#8d7,%%[fx:w-1],0,#c7f ^
  -fill White -gravity Center -pointsize 30 ^
  -fill Black -annotate +0+0 "#3" ^
  -fill White -annotate -2-2 "#3" ^
  spm_pan_in3.png
spm_pan_in3.pngjpg

Suppose they should overlap like this:

set OFFS_2=+130+10
set OFFS_3=+250-10

%IMG7%magick ^
  spm_pan_in1.png ^
  ( spm_pan_in2.png -repage %OFFS_2% ) ^
  ( spm_pan_in3.png -repage %OFFS_3% ) ^
  -background None ^
  -layers merge +repage ^
  spm_pan_out1.png

set CROP_2=150x100%OFFS_2%
spm_pan_out1.png

There are many possible ways to blend these images, including:

  1. Leave #1 unchanged; adjust #2 to match #1; adjust #3 to match the adjusted #2; paste #2 over #1 and #3 over #2.
  2. Leave #1 and #3 unchanged; adjust #2 across full width to match #1 and #2; paste #2 over #1 and #3.
  3. Leave #1 and #3 unchanged; adjust #2 from left edge to where #3 overlaps it; paste #2 over #1 and under #3.
  4. Adjust #1 so left side is unchanged but right side has half the adjustment to match #2; adjust #3 so right side is unchanged but left side has half the adjustment to match #2; adjust #2 to match the adjusted #1 and #3; paste them.

We will use the third way.

The boundary condition, where we require the images to match, are at most of the left edge of #2 and part of the top (an inverted "L"), and an "L" shape near the right edge of #2.

We make a mask of the "L" shapes. The following is messy but depends only on the image sizes, their order, and the "-repage" arguments. "-fuzz" is used in case twice gray(50%) isn't exactly white. "-layers merge" puts offsets in the output, in this case +0-10, so we can later use #2 offsets directly for cropping.

"-set colorspace sRGB" is needed for IM v7, to get three channels. In IM v7, "-border" uses the current "-compose" method, so we need to explicitly use "plus".

%IMG7%magick ^
  -size 150x100 ^
  xc:gray(50%%) -set colorspace sRGB ^
  -fuzz 0.1%% ^
  ( xc:Red ^
    -shave 1x1 -bordercolor gray(50%%) -border 1 ^
    -repage %OFFS_2% ^
  ) ^
  -background Black ^
  -compose Plus -layers merge +repage ^
  -fill gray(50%%) -opaque Red ^
  ( xc:Red ^
    -shave 1x1 ^
      -compose Over ^
      -bordercolor gray(50%%) -border 1 ^
    -repage %OFFS_3% ^
  ) ^
  -background Black ^
  -compose Plus -layers merge ^
  -fill Black +opaque White ^
  spm_L-both.png

Note: the output has offsets.

spm_L-both.png

Crop to #2:

%IMG7%magick ^
  spm_L-both.png ^
  -crop %CROP_2% +repage ^
  spm_L_cr.png
spm_L_cr.png

We use this to blend image #2 with the other two images.

Subtract #2 from the others:

%IMG7%magick ^
  spm_pan_in1.png ^
  ( spm_pan_in3.png -repage %OFFS_3% ) ^
  -background None ^
  -layers merge ^
  ( spm_pan_in2.png -repage %OFFS_2% ) ^
  -compose Mathematics ^
    -define compose:args=0,-0.5,0.5,0.5 ^
  -layers merge ^
  spm_pan_sub.png

Note: output has offsets.

spm_pan_sub.png

Crop to #2:

%IMG7%magick ^
  spm_pan_sub.png ^
  -crop %CROP_2% +repage ^
  spm_pan_sub_cr.png
spm_pan_sub_cr.png

Mask by the L-shapes:

%IMG7%magick ^
  spm_pan_sub_cr.png ^
  spm_L_cr.png ^
  -compose CopyOpacity -composite ^
  spm_pan_sub_lcr.png
spm_pan_sub_lcr.png

Relax-fill to get an adjustment image:

call %PICTBAT%relaxFillMS ^
  spm_pan_sub_lcr.png . spm_pan_rf.png ^
  1e-6 1000
spm_pan_rf.png

Add the adjustment image to #2:

%IMG7%magick ^
  spm_pan_in2.png ^
  spm_pan_rf.png ^
  -compose Mathematics ^
    -define compose:args=0,2,1,-1 -composite ^
  spm_pan_2adj.png
spm_pan_2adj.png

Merge the modified #2 with the original #1 and #3:

%IMG7%magick ^
  spm_pan_in1.png ^
  ( spm_pan_2adj.png -repage %OFFS_2% ) ^
  ( spm_pan_in3.png  -repage %OFFS_3% ) ^
  -background None -compose Over ^
  -layers merge +repage ^
  spm_pan_123.png
spm_pan_123.png

As required, the images blend seamlessly. Image #2 has been adjusted to exactly match two adjacent images. If image #2 had been entirely surrounded by other images, similar processing could be done.

The script colCorr2.bat composites one image over another, at a given offset, with colour-correction to the top image. We can use the script to composite successive images over previous results. For example:

Composite #2 over #1:

call %PICTBAT%colCorr2 ^
  spm_pan_in1.png spm_pan_in2.png %OFFS_2% ^
  spm_cc_12.png
spm_cc_12.png

Composite #3 over previous result:

call %PICTBAT%colCorr2 ^
  spm_cc_12.png spm_pan_in3.png %OFFS_3% ^
  spm_cc_123.png
spm_cc_123.png

This result is different to the previous one. Instead of adjusting #2 to match both #1 and #3, we adjust #2 to match just #1, then adjust #3 to match the adjusted #2.

We can readily add further conditions. For example, we might want the central column of image #2 to be unchanged. This needs gray(50%) in the adjustment image.

Add a gray line:

%IMG7%magick ^
  spm_pan_sub_lcr.png ^
  -stroke gray(50%%) ^
  -draw "line 75,0,75,99" ^
  spm_pan_sub_lcr2.png
spm_pan_sub_lcr2.png

Relax-fill to get an adjustment image:

call %PICTBAT%relaxFillMS ^
  spm_pan_sub_lcr2.png . spm_pan_rf2.png ^
  1e-6 1000
spm_pan_rf2.png

Add the adjustment image to #2:

%IMG7%magick ^
  spm_pan_in2.png ^
  spm_pan_rf2.png ^
  -compose Mathematics ^
    -define compose:args=0,2,1,-1 -composite ^
  spm_pan_2adj2.png
spm_pan_2adj2.png

Merge:

%IMG7%magick ^
  spm_pan_in1.png ^
  ( spm_pan_2adj2.png -repage %OFFS_2% ) ^
  ( spm_pan_in3.png   -repage %OFFS_3% ) ^
  -background None -compose Over ^
  -layers merge +repage ^
  spm_pan_1232.png
spm_pan_1232.png

We can do a similar trick in the script, by adding a small gray circle in the centre of the adjustment image:

Composite #2 over #1:

set cc2DRAW=circle 75,50 80,50

call %PICTBAT%colCorr2 ^
  spm_pan_in1.png spm_pan_in2.png %OFFS_2% ^
  spm_cc_12b.png
spm_cc_12b.png

Composite #3 over previous result:

call %PICTBAT%colCorr2 ^
  spm_cc_12b.png spm_pan_in3.png %OFFS_3% ^
  spm_cc_123b.png

set cc2DRAW=
spm_cc_123b.png

How does it work?

The technical papers in the references above (Pérez etc) describe the mathematics of Poisson image editing. Here is a less-mathematical explanation. The goal is to create an image where some pixels have desired values (the boundary constraint) while others are such that the slope either changes as slowly as possible, or the rate of change follows a guidance.

I visualise this as the pixel values at the boundary points representing heights, and we stretch a thin rubber membrane, and glue it to all the boundary points. Then the height of the rubber membrane at points inside the boundary are the required pixels values at those points. At the initial guess, which might be some constant value, the membrane is over-stretched adjacent to the boundary points, and not at all stretched between other points. We need to "relax" the membrane so it is equally stretched at all points. At each infinitely small point of the membrane, the forces on that point caused by the stretches in all directions must sum to zero. If the sum wasn't zero, there would be a net force on the point, so it would move.

We simulate the forces and movement in software. Suppose a pixel has a value 0.2, and it has two adjacent pixels with values 1.0 and 0.1, then there is a net force pulling that pixel upwards, to a new value that is the average of 1.0 and 0.1, which is 0.55.

This new value will change forces on any adjacent pixels, so they need to be re-calculated. The process repeats until no values change more than a specified threshold, or we hit a limit on the number of iterations.

In pratice, we set each pixel to the average of four neighbours, the pixels to the north, south, east and west. Instead, we could use the eight adjacent neighbours. This would give the same final result, with virtually no impact on performance.

We can think of this as an infill process. When fully-transparent pixels have an unknown colour, surrounded by boundary pixels of known colours, we "relax-fill" the missing pixels.

More mathematically (sorry), we can think of this as millions of unknowns (the pixel values), with the same number of simultaneous equations.

relaxFill.bat

The script relaxFill.bat aims at a zero rate of change of gradient.

At each iteration, for every pixel (and every channel), we set the new pixel value to the average value of the four neighbours to the north, south, east and west. This is a blurring operation. If we repeat this often enough, every pixel will be almost equal to the average of the four neighbours. This means the slope (gradient) of the image will be almost the same everywhere. It doesn't change quickly. Put it another way, the slope of the slope (the "second derivative") will be almost zero.

Generally, pixels won't be exactly equal to the average of the neighbours. That happens only when the result is a bilinear gradient, which only happens when the boundary values when considered as heights are co-planar. But the repeated iteration gets the overall image as close to this result as it can get.

A bit of maths (sorry): if we say that each pixel value is a function of x and y, the pixel is f(x,y), then the following is approximately true for every pixel in the relaxed result:

f(x-1,y) + f(x+1,y) + f(x,y-1) + f(x,y+1) - 4*f(x,y) = 0

The left hand side of that equation approximates the slope of the slope of the image.

To get to that state, each iteration sets a new value for each pixel f'(x,y):

f'(x,y) = f(x-1,y) + f(x+1,y) + f(x,y-1) + f(x,y+1) 
                              4

In ImageMagick code, this is:

-morphology convolve:n 3x3:0,0.25,0,0.25,0,0.25,0,0.25,0

... where n is however many iterations we want to make between each test for completion.

The iterative process has to start somewhere. We need an initial guess at the result, and each iteration takes us closer to the actual result. With luck there is no local minimum so the initial guess doesn't matter; any initial guess will eventually give us the same result. But the closer the initial guess is to the actual result, the quicker we will get there.

relaxFillScr.bat

The relaxFillScr.bat is a little more complex. Instead of aiming to set the slope of the slope to zero for every pixel, it aims at some other particular slope of the slope for each pixel. The values to be aimed at are given in the guidance image. So in the relaxed result, the following is approximately true:

f(x-1,y) + f(x+1,y) + f(x,y-1) + f(x,y+1) - 4*f(x,y) = guide(x,y)

So each iteration is:

f'(x,y) = f(x-1,y) + f(x+1,y) + f(x,y-1) + f(x,y+1) - guide(x,y) 
                              4                           4

IM can't do this in a single operation. (For example, morphology can't use more than one image as input.) So each iteration has two phases: first we convolve as above, then we subtract another image. These phases fight each other: we smooth then roughen the image, and repeat this smoothing and roughening. This makes the process slow to converge.

In preparation for the script, we set the guidance image to the divergence of the required gradient, so it is the sum of the slope of the slope in the x and y directions.

relaxFillMS.bat

This calls either relaxFill.bat or relaxFillScr.bat at multiple scales. For relaxFill.bat, this always improves performance. However, for relaxFillScr.bat the improvement may be small or even negative.

A recursive mechanism is used, dividing the dimensions by two while the minimum dimension is greater than 50. At each level in the recursive stack, the script resizes images and calls either relaxFill.bat or relaxFillScr.bat, using the previous result as the initial guess. In the initial levels, striving for high accuracy is a waste of effort so the target score is relaxed by a factor that is the ratio of the areas (which is roughly four) at each level.

Here is a simple example using relaxFillMS.bat to relax-fill with no explicit guidance:

Example image, with transparent centre:

%IMG7%magick ^
  -size 400x300 xc:None ^
  -size 20x300 ^
    xc:#f80 -gravity West -composite ^
    xc:#ff0 -gravity East -composite ^
  -size 400x20 ^
    xc:#08f -gravity North -composite ^
    xc:#080 -gravity South -composite ^
  spm_bnd.png
spm_bnd.png

Relax-filled:

call %PICTBAT%relaxFillMS ^
  spm_bnd.png . spm_bnd_rf.png
spm_bnd_rf.png

When the image to be filled is very sparse, the result may quickly converge to a poor solution.

We can aim for a low difference between successive approximations. Or we can specify a large number of iterations between tests. Or we can attempt a better approximation for the first guess. A -sparse-color with Shepards method, using a power of 1.0, works well.

Make a sample sparse image, and a same-size Shepard's sparse colour image.

%IMG7%magick ^
  -size 600x400 xc:None ^
  -fill #d22 -draw "point 100,100" ^
  -fill #3f5 -draw "point 200,75" ^
  -fill #fff -draw "point 250,275" ^
  -fill #23e -draw "point 400,300" ^
  -fill #000 -draw "point 480,80" ^
  +write spm_sparse.png ^
  +write sparse-color:spm_shep.lis ^
  -define shepards:power=1 ^
  -sparse-color shepards @spm_shep.lis ^
  spm_shep.png
spm_sparse.png spm_shep.png

Relax-fill the image.

call %PICTBAT%relaxFillMS ^
  spm_sparse.png ^
  spm_shep.png ^
  spm_sparse_result.png ^
  0.0001 1000
spm_sparse_result.png

An alternative remedy is to use an initial guess that is pure noise. This is, of course, a very poor initial guess, and it ensures convergence does not occur quickly.

Make a same-size noisy image.

%IMG7%magick ^
  spm_sparse.png ^
  +noise Random ^
  spm_noise.png
spm_noise.png

Relax-fill the image.

call %PICTBAT%relaxFillMS ^
  spm_sparse.png ^
  spm_noise.png ^
  spm_sparse_result2.png ^
  0.001 1000
spm_sparse_result2.png

If the process did not fully stabilize, the result may be partially transparent, so we remove transparency.

%IMG7%magick ^
  spm_sparse_result2.png ^
  -alpha off ^
  spm_sparse_opaq2.png
spm_sparse_opaq2.png

poissonPaste.bat

This is a higher-level script. For inputs it uses a background, foreground and mask. The mask should be white where we want the adjusted foreground, and hence defines the area that will be relaxed.

The script optionally takes a guidance image.

poissonPasteIDB.bat

This is like poissonPaste.bat, but takes an extra mask, that is white where we want the foreground to be unchanged. Hence the area to be relaxed is between the two masks.

If it is called with this inner mask blank or set to dot, ".", it instead calls poissonPaste.bat.

Cross sectional example

To further illustrate the relaxation process, we make a simple example, and show the intensity graphs at a cross-section through the images, at the middle row.

Create a background.

%IMG7%magick ^
  -size 200x200 gradient: ^
  -rotate 90 ^
  spm_xb.png

call %PICTBAT%xSection spm_xb.png
spm_xb.png spm_xb_xs.png

Create a foreground. The square is 40% lighter than its surround.

%IMG7%magick ^
  -size 200x200 xc:gray(30%%) ^
  -fill gray(70%%) -draw "rectangle 80,80 120,120" ^
  spm_xf.png

call %PICTBAT%xSection spm_xf.png
spm_xf.png spm_xf_xs.png

Create a mask.

%IMG7%magick ^
  -size 200x200 xc:Black ^
  -fill White -draw "circle 100,100 150,100" ^
  spm_xm.png
spm_xm.png

The goal is to blend the background and foreground, where the mask is white.

We show the elapsed time for the process, and the number of iterations.

Poisson paste with minimal iterations.

set rfCOMP_METRIC=PAE
set ppTGT_SCORE=999
set ppNUM_STEPS=1

call %PICTBAT%poissonPaste ^
  spm_xb.png spm_xf.png spm_xm.png ^
  spm_xout1.png

call %PICTBAT%xSection spm_xout1.png
0 00:00:01
rfmsITER=2 
spm_xout1.png spm_xout1_xs.png

Poisson paste with a few iterations.

set rfCOMP_METRIC=PAE
set ppTGT_SCORE=0.1
set ppNUM_STEPS=10

call %PICTBAT%poissonPaste ^
  spm_xb.png spm_xf.png spm_xm.png ^
  spm_xout2.png

call %PICTBAT%xSection spm_xout2.png
0 00:00:01
rfmsITER=20 
spm_xout2.png spm_xout2_xs.png

Poisson paste to approximate completion.

set rfCOMP_METRIC=PAE
set ppTGT_SCORE=0.001
set ppNUM_STEPS=1000

call %PICTBAT%poissonPaste ^
  spm_xb.png spm_xf.png spm_xm.png ^
  spm_xout3.png

call %PICTBAT%xSection spm_xout3.png
0 00:00:09
rfmsITER=6000 

In the finished result, the square is 40% lighter than its surround.

spm_xout3.png spm_xout3_xs.png

We will repeat the above, but with an explicit guidance field. First, we make a guidance field: the divergence of the slope of the foreground.

call %PICTBAT%slopeXY spm_xf.png spm_xf_sxy.miff

call %PICTBAT%slopeXYdiv spm_xf_sxy.miff spm_xf_div.miff

%IMG7%magick ^
  spm_xf_div.miff ^
  -evaluate Divide 2 ^
  -evaluate Add 50%% ^
  spm_xf_div.png

call %PICTBAT%xSection ^
  spm_xf_div.miff ^
  spm_xf_div_xs.png
spm_xf_div.png spm_xf_div_xs.png

Now we use this divergence as the guidance:

Poisson paste with minimal iterations.

set rfCOMP_METRIC=PAE
set ppTGT_SCORE=999
set ppNUM_STEPS=1

call %PICTBAT%poissonPaste ^
  spm_xb.png spm_xf.png spm_xm.png ^
  spm_xout1g.png . spm_xf_div.miff

call %PICTBAT%xSection spm_xout1g.png
0 00:00:01
rfmsITER=2 
spm_xout1g.png spm_xout1g_xs.png

Poisson paste.

set rfCOMP_METRIC=PAE
set ppTGT_SCORE=0.1
set ppNUM_STEPS=10

call %PICTBAT%poissonPaste ^
  spm_xb.png spm_xf.png spm_xm.png ^
  spm_xout2g.png . spm_xf_div.miff

call %PICTBAT%xSection spm_xout2g.png
0 00:00:01
rfmsITER=30 
spm_xout2g.png spm_xout2g_xs.png

Poisson paste to approximate completion.

set rfCOMP_METRIC=PAE
set ppTGT_SCORE=0.001
set ppNUM_STEPS=1000

call %PICTBAT%poissonPaste ^
  spm_xb.png spm_xf.png spm_xm.png ^
  spm_xout3g.png . spm_xf_div.miff

call %PICTBAT%xSection spm_xout3g.png
0 00:00:40
rfmsITER=12000 
spm_xout3g.png spm_xout3g_xs.png

Using an explicit guidance field of the divergence of the slope of the foreground eventually gives almost the same result as using no explicit guidance field. But it takes eight times as long.

We can use a sharper divergence:

call %PICTBAT%shpDiv spm_xf.png spm_xf_div2.miff

%IMG7%magick ^
  spm_xf_div2.miff ^
  -evaluate Divide 2 ^
  -evaluate Add 50%% ^
  spm_xf_div2.png

call %PICTBAT%xSection spm_xf_div2.png
spm_xf_div2.png spm_xf_div2_xs.png

Poisson paste.

set rfCOMP_METRIC=PAE
set ppTGT_SCORE=0.001
set ppNUM_STEPS=1000

call %PICTBAT%poissonPaste ^
  spm_xb.png spm_xf.png spm_xm.png ^
  spm_xout3g2.png . spm_xf_div2.miff

call %PICTBAT%xSection spm_xout3g2.png
0 00:00:31
rfmsITER=9000 
spm_xout3g2.png spm_xout3g2_xs.png

Successive over-relaxation

I have experimented with successive over-relaxation in an attempt at increasing performance. See Wikikpedia: Successive over-relaxation. This calculates each new value as a blended average of the old value and the calculated new mean:

v' = (1-w)*v + w*f(v)

Where w is the relaxation factor, usually 0 < w <= 2.

However, this only improves performance by up to 16%, and only gives that when w is carefully (manually) chosen, and the optimum w depends on the input. When w is wrong for the image, it can halve the performance, or fail to converge.

Large mosaics

There might be a large mosaic of photographs in an irregular grid, where a small number are known to be "correct", with a requirement to adjust all the others to create a smooth final result. There could be millions of images, each with millions of pixels. Applications include aerial, satellite and drone photography, and panoramas.

A possible approach is:

  1. Composite the input images together. For practical reasons, this might be a scaled-down version.
  2. Make transparent all the pixels corresponding to images that are to be adjusted.
  3. Relax this image, making an adjustment image.
  4. Add this adjustment to (1), where the pixels in (2) are transparent.
  5. If we didn't scale-down, the job is finished. Otherwise...
  6. For each input image, crop out and scale up the adjustment, and add it.
  7. Due to the scaling, the matching won't be perfect, so apply a process similar to Colour-correction for panoramas above to each individual image.
  8. Repeat the previous step until stability.
  9. The result is all the images, colour-corrected to seamlessly match each other.

Seamless-blend and Saliency-blend

After I wrote this page in 2017, IM has implemented operations "-compose seamless-blend -composite" and "-compose saliency-blend -composite". These are documented in Forum: Seamless Blending and Saliency Blending.

For both methods, three numbers can be given in "-define compose:arg=". These are:

  1. the maximum number of iterations to compute before stopping;
  2. the RMSE convergence stopping value;
  3. the number of iterations between printout to the terminal.

The last is relevant only when "-verbose" is in effect.

%IMG7%magick ^
  -verbose ^
  spm_back_crp.jpg ^
  %BEAR% ^
  %MASK% ^
  -alpha off ^
  -define compose:args=5000x0.00001+500 ^
  -compose seamless-blend -composite ^
  -depth 16 ^
  spm_biw.png

This is very close to the result spm_bear_adj.png above.

spm_biw.pngjpg

We do a Poisson-paste with an Inner Dirichlet Boundary, using spm_bear_sm.png. We need a black shape on a white background, so we negate the image.

set IDB=spm_bear_sm.png

%IMG7%magick ^
  -verbose ^
  %IDB% -negate +write mpr:IDB +delete ^
  spm_back_crp.jpg ^
  ( %BEAR% -read-mask mpr:IDB ) ^
  %MASK% ^
  -alpha off ^
  -define compose:args=5000x0.00001+500 ^
  -compose seamless-blend -composite ^
  -depth 16 ^
  spm_biw2.png
spm_biw2.pngjpg

The second example, with the Inner Dirichlet Boundary, has slightly darkened the bear (towards its original colour), but not as much as in the method shown above in the Inner Dirichlet boundary section.

Above, we showed how to make Guided gradients using low-level operations. We can do this directly with the new "-compose saliency-blend -composite" operation.

%IMG7%magick ^
  %WALL% %SCRIBBLE% %WS_MASK% ^
  -compose saliency-blend -composite ^
  spm_salbl.png
spm_salbl.pngjpg

Another use for seamless-blend is "filling holes", aka "inpainting", where we expect pixels to be the average of the 4-adjacent pixels. This is true of gradient images, and cartoons between the edges, but not true of ordinary photos.

For example, sparseMap.tiff is a distortion map from Direct polar distortion: distorting from a shape to ellipse. After inverting this map, 69% of the pixels are transparent. They have "missing" colours.

Here is sparseMap.tiff:

sparseMap.png

Invert it:

%IM7DEV%magick ^
  sparseMap.tiff ^
  -process 'invdispmap' ^
  +channel ^
  -define quantum:format=floating-point ^
  -depth 32 ^
  spm_inv_map.miff
spm_inv_map.miffpng

Flatten against blue to clearly show which pixels are transparent:

%IMG7%magick ^
  spm_inv_map.miff ^
  +channel ^
  -background Blue ^
  -layers Flatten ^
  spm_sm_blue.jpg
spm_sm_blue.jpg

Fill the holes with seamless-blend:

%IMG7%magick ^
  spm_inv_map.miff ^
  +channel ^
  ( +clone -alpha extract -negate +write mpr:ALP +delete ) ^
  ( -clone 0 -alpha off -fill Black -colorize 100 ) ^
  mpr:ALP ^
  -define compose:args=10000x1e-7+100 -compose seamless_blend -composite ^
  -alpha off ^
  spm_filled_map.miff

The filled map is:

spm_filled_map.png:

spm_filled_map.miffpng

The filled map can be used like this:

%IMG7%magick ^
  toes.png spm_filled_map.miff ^
  -virtual-pixel None ^
  -compose distort -composite ^
  spm_fm_toes.png
spm_fm_toes.png

We can test that a round trip of the forwards displacement followed by the inverse displacement results in the original image.

%IMG7%magick ^
  toes.png ^
  sparseMap.tiff ^
  -compose distort -composite ^
  spm_filled_map.miff ^
  -compose distort -composite ^
  spm_fm_rndtrip.png
spm_fm_rndtrip.png

The result looks okay. The displacements shrink and then enlarge, so we can't expect a perfect result. How close is it?

%IMG7%magick compare -metric RMSE toes.png spm_fm_rndtrip.png NULL: 
996.271 (0.0152021)

The distortion is quite low.

These built-in methods are faster than the script-based methods shown above, partly because they don't need to read and write an image at each iteration. However, they can't be supplied with a "first guess", so we can't make a first guess from downsized inputs.

Scripts

For convenience, .bat scripts are also available in a single zip file. See Zipped BAT files.

poissonPaste.bat

rem Seamless boundary by Poisson pasting.
rem %1 is input background (without holes)
rem %2 is input foreground, same size (relevant only when no guidance)
rem %3 is input mask, same size
rem   mask is white where we want adjusted foreground, or black for background
rem %4 is output.
rem %5 0 or 1: whether to negate the mask.
rem %6 optional guidance image
rem %7 radius of final unsharp. 0=no unsharp. Default 0.
@rem
@rem Also uses:
@rem   ppTGT_SCORE target RMSE or PAE score. [default 0.001]
@rem   ppNUM_STEPS number of iteration steps between tests. [default 100]
@rem   ppPOST processing to apply to transition area at end, eg "-unsharp 0x0.5+1+0".
@rem
@rem   rfRLX_FUNC is a relaxation function. [default: average of 4 neighbours]
@rem   rfCOMP_METRIC default RMSE. PAE is more sensitive.
@rem
@rem Updated:
@rem   24-July-2022 for IM v7.
@rem

@if "%3"=="" findstr /B "rem @rem" %~f0 & exit /B 1

@setlocal enabledelayedexpansion

@call echoOffSave

call %PICTBAT%setInOut %1 pp

set IN_A=%INFILE%
set IN_B=%2
set MASK=%3

if not "%4"=="" if not "%4"=="." set OUTFILE=%4

set NEG_MASK=%5
if "%NEG_MASK%"=="." set NEG_MASK=
if "%NEG_MASK%"=="" set NEG_MASK=0

if not %NEG_MASK%==0 if not %NEG_MASK%==1 exit /B 1

if %NEG_MASK%==0 (
  set NEG1=-negate
  set NEG2=
) else (
  set NEG1=
  set NEG2=-negate
)

set GUID=%6
if "%GUID%"=="" set GUID=.
set HAS_GUID=1
if "%GUID%"=="." set HAS_GUID=0

set UNSHP=%7
if "%UNSHP%"=="." set UNSHP=
if "%UNSHP%"=="" set UNSHP=0

if "%ppPOST%"=="" (
  set sUNSHP=
) else (
  set sUNSHP=^^^( +clone %ppPOST% ^^^) %MASK% -compose Over -composite
)

if "%ppTGT_SCORE%"=="" set ppTGT_SCORE=0.001
if "%ppNUM_STEPS%"=="" set ppNUM_STEPS=100

set TMPDIR=\temp\
set TMP_APPRX=%TMPDIR%pp_tmp_apprx.miff
set TMP1=%TMPDIR%pp_tmp1.miff
set TMP2=%TMPDIR%pp_tmp2.miff

:: Provide a first approximation: the background over the foreground.
:: However, this seems to make performance worse.
:: %IMG7%magick ^
::   %IN_B% ^
::   %IN_A% ^
::   -compose Over -composite ^
::   %TMP_APPRX%
set TMP_APPRX=.

if %HAS_GUID%==1 (
  %IMG7%magick ^
    %IN_A% -colorspace sRGB ^
    ^( %MASK% -alpha off %NEG1% ^) ^
    -alpha off -compose CopyOpacity -composite ^
    +depth ^
    %TMP1%

  if ERRORLEVEL 1 exit /B 1
) else (
  %IMG7%magick ^
    -define compose:clamp=off ^
    %IN_A% -colorspace sRGB ^
    %IN_B% ^
    -compose Mathematics ^
      -define "compose:args=0,-0.5,0.5,0.5" -composite ^
    ^( %MASK% -alpha off %NEG1% ^) ^
    -alpha off -compose CopyOpacity -composite ^
    +depth ^
    %TMP1%

  if ERRORLEVEL 1 exit /B 1
)

call %PICTBAT%relaxFillMS ^
  %TMP1% %TMP_APPRX% %TMP2% ^
  %ppTGT_SCORE% %ppNUM_STEPS% ^
  . %GUID%

if ERRORLEVEL 1 exit /B 1

if %HAS_GUID%==1 (
  %IMG7%magick ^
    %TMP2% ^
    %sUNSHP% ^
    +depth ^
    %OUTFILE%

  if ERRORLEVEL 1 exit /B 1
) else (
  %IMG7%magick ^
    -define compose:clamp=off ^
    %IN_B% ^
    %TMP2% ^
    -compose Mathematics ^
      -define "compose:args=0,2,1,-1" -composite ^
    +depth ^
    ^( %MASK% -alpha off %NEG2% ^) ^
    -alpha off ^
    -compose CopyOpacity -composite ^
    %IN_A% ^
    +swap ^
    -compose Over -composite ^
    %sUNSHP% ^
    +depth ^
    %OUTFILE%

  if ERRORLEVEL 1 exit /B 1
)

call echoRestore

@endlocal & set ppOUTFILE=%OUTFILE%& set rfmsITER=%rfmsITER%

poissonPasteIDB.bat

rem Seamless boundary by Poisson pasting, with inner Dirichlet boundary.
rem %1 is input background (without holes)
rem %2 is input foreground, same size
rem %3 is input mask, same size
rem   mask is white where we want adjusted foreground, or black for background
rem %4 inner mask, white where we want exact foreground
rem %5 is output.
rem %6 0 or 1: whether to negate the mask.
rem %7 optional guidance image
@rem
@rem Also uses:
@rem   ppTGT_SCORE target RMSE or PAE score. [default 0.001]
@rem   ppNUM_STEPS number of iteration steps between tests. [default 100]
@rem
@rem   rfRLX_FUNC is a relaxation function. [default average of 4 neighbours]
@rem   rfCOMP_METRIC default RMSE. PAE is more sensitive.
@rem
@rem Updated:
@rem   24-July-2022 for IM v7.
@rem

@if "%4"=="" findstr /B "rem @rem" %~f0 & exit /B 1

@setlocal enabledelayedexpansion

@call echoOffSave

call %PICTBAT%setInOut %1 ppi

set IN_A=%INFILE%
set IN_B=%2
set MASK=%3
set MASK_IDB=%4

if "%MASK_IDB%"=="." set MASK_IDB=

if not "%5"=="" if not "%5"=="." set OUTFILE=%5

set NEG_MASK=%6
if "%NEG_MASK%"=="." set NEG_MASK=
if "%NEG_MASK%"=="" set NEG_MASK=0

set GUID=%7
if "%GUID%"=="" set GUID=.
set HAS_GUID=1
if "%GUID%"=="." set HAS_GUID=0

if %NEG_MASK%==0 (
  set NEG1=-negate
  set NEG2=
) else (
  set NEG1=
  set NEG2=-negate
)

if "%MASK_IDB%"=="" (
  call %PICTBAT%poissonPaste ^
    %IN_A% %IN_B% %MASK%
    %OUTFILE% %NEG_MASK% %GUID%

  goto finished
)

if "%ppTGT_SCORE%"=="" set ppTGT_SCORE=0.001
if "%ppNUM_STEPS%"=="" set ppNUM_STEPS=100

set TMPDIR=\temp\
set TMP1=%TMPDIR%ppi_tmp1.miff
set TMP2=%TMPDIR%ppi_tmp2.miff

set TMP_MASK=%TMPDIR%ppi_msk.png

%IMG7%magick ^
  %MASK% ^
  ( %MASK_IDB% -alpha off -negate ) ^
  -alpha off ^
  -compose Darken -composite ^
  %TMP_MASK%

if %HAS_GUID%==1 (
  %IMG7%magick ^
    -define compose:clamp=off ^
    -compose CopyOpacity ^
    ^( %IN_A% -colorspace sRGB ^
       ^( %MASK%     -alpha off %NEG1% ^) ^
       -alpha off -composite ^
    ^) ^
    ^( %IN_B% ^
       ^( %MASK_IDB% -alpha off %NEG2% ^) ^
       -alpha off -composite ^
    ^) ^
    -compose Over -composite ^
    %TMP1%

  if ERRORLEVEL 1 exit /B 1
) else (
  %IMG7%magick ^
    -define compose:clamp=off ^
    %IN_A% -colorspace sRGB ^
    %IN_B% ^
    -compose Mathematics ^
      -define "compose:args=0,-0.5,0.5,0.5" -composite ^
    ^( +clone -fill gray^(50%%^) -colorize 100 ^) ^
    ^( %MASK_IDB% ^) ^
    -compose Over -composite ^
    +depth ^
    ^( %TMP_MASK% -alpha off %NEG1% ^) ^
    -alpha off -compose CopyOpacity -composite ^
    %TMP1%

  if ERRORLEVEL 1 exit /B 1
)

call %PICTBAT%relaxFillMS ^
  %TMP1% . %TMP2% ^
  %ppTGT_SCORE% %ppNUM_STEPS% ^
  . %GUID%

if ERRORLEVEL 1 exit /B 1

if %HAS_GUID%==1 (
  %IMG7%magick ^
    %TMP2% ^
    %OUTFILE%

  if ERRORLEVEL 1 exit /B 1
) else (
  %IMG7%magick ^
    %IN_B% ^
    %TMP2% ^
    -compose Mathematics ^
      -define "compose:args=0,2,1,-1" -composite ^
    +depth ^
    ^( %MASK% -alpha off %NEG2% ^) ^
    -alpha off ^
    -compose CopyOpacity -composite ^
    %IN_A% ^
    -colorspace sRGB ^
    +swap ^
    -compose Over -composite ^
    +depth ^
    %OUTFILE%

  if ERRORLEVEL 1 exit /B 1
)

:finished
call echoRestore

@endlocal & set ppiOUTFILE=%OUTFILE%
	

poissonSmooth.bat

rem From image %1,
rem make output %2
rem that is smoothed by Poisson-pasting with guidance field
rem that is the div of the slope
rem set to zero where the slope is within %3 percent of zero.
@rem
@rem Updated:
@rem   24-July-2022 for IM v7.
@rem

@if "%1"=="" findstr /B "rem @rem" %~f0 & exit /B 1

@setlocal

@call echoOffSave

call %PICTBAT%setInOut %1 psm

if not "%2"=="." if not "%2"=="" set OUTFILE=%2

set SLP_LIM_PC=%3
if "%SLP_LIM_PC%"=="." set SLP_LIM_PC=
if "%SLP_LIM_PC%"=="" set SLP_LIM_PC=10

set TMPDIR=\temp\
set SXY=%TMPDIR%psm_sxy.miff
set SXY2=%TMPDIR%psm_sxy2.miff
set DIV=%TMPDIR%psm_div.miff
set MASK=%TMPDIR%psm_msk.miff


call %PICTBAT%slopeXY %INFILE% %SXY%

if ERRORLEVEL 1 exit /B 1

%IMG7%magick ^
  %SXY% ^
  -fuzz %SLP_LIM_PC%%% -fill Black -opaque Black ^
  -define quantum:format=floating-point ^
  -depth 32 ^
  %SXY2%

call %PICTBAT%slopeXYdiv %SXY2% %DIV%

if ERRORLEVEL 1 exit /B 1

%IMG7%magick ^
  %INFILE% ^
  -fill White -colorize 100 ^
  -gravity Center ^
  -shave 1x1 ^
  -bordercolor Black -border 1 ^
  %MASK%

call %PICTBAT%poissonPaste ^
  %INFILE% . %MASK% ^
  %OUTFILE% ^
  . %DIV%

if ERRORLEVEL 1 exit /B 1


call echoRestore

endlocal & set psmOUTFILE=%OUTFILE%

connCompLimArea.bat

rem Given image %1,
rem makes %2
rem from connected components that are at least %3 pixels.
@rem   %3 may have suffix 'c' or '%' for percentage or 'p' for proportion of image w*h.
@rem   Default 0.
@rem
@rem Each output connected component is the mean of its input pixels.
@rem
@rem Updated:
@rem   24-July-2022 for IM v7.
@rem

@if "%1"=="" findstr /B "rem @rem" %~f0 & exit /B 1

@setlocal enabledelayedexpansion

@call echoOffSave

call %PICTBAT%setInOut %1 ccla

if not "%2"=="" if not "%2"=="." set OUTFILE=%2


set THRESH=%3
if "%THRESH%"=="." set THRESH=
if "%THRESH%"=="" set THRESH=0


set TH_LAST=%THRESH:~-1%

if "%TH_LAST%"=="^%" set TH_LAST=c

if /I "%TH_LAST%"=="c" (

  for /F "usebackq" %%L in (`%IMG7%magick identify ^
    -format "nTHRESH=%%[fx:w*h*%THRESH:~0,-1%/100]" ^
    %INFILE%`) do set %%L

) else if /I "%TH_LAST%"=="p" (

  for /F "usebackq" %%L in (`%IMG7%magick identify ^
    -format "nTHRESH=%%[fx:w*h*%THRESH:~0,-1%]" ^
    %INFILE%`) do set %%L

) else (
  set nTHRESH=%THRESH%
)

if %nTHRESH%==0 (
  set sTHRESH=
) else (
  set "sTHRESH=-define connected-components:area-threshold^=%nTHRESH%"
)


%IMG7%magick ^
  %INFILE% ^
  -define connected-components:verbose^=true ^
  -define connected-components:mean-color=true ^
  %sTHRESH% ^
  -connected-components 4 ^
  %OUTFILE%


call echoRestore

@endlocal & set cclaOUTFILE=%OUTFILE%

connCompLimNum.bat

rem Given image %1,
rem makes %2
rem from the largest %3 connected components.
@rem
@rem For performance, use the output from connCompLimitArea.bat
@rem to reduce the number of components this script considers.
@rem (or add area parameter to this script?)
@rem
@rem When an eliminated component is adjacent to more than one other component,
@rem   it may merge with the "wrong" one.
@rem
@rem Each output connected component is the mean of its input pixels.
@rem
@rem Updated:
@rem   24-July-2022 for IM v7.
@rem   29-July-2022 Bug in "-connected-components" hinders merging. "-virtual-pixel None" may help.
@rem     Also repeating "-connected-components" with rotated image.
@rem     (See https://github.com/ImageMagick/ImageMagick/issues/5368 )
@rem


@if "%1"=="" findstr /B "rem @rem" %~f0 & exit /B 1

@setlocal enabledelayedexpansion

rem @call echoOffSave

call %PICTBAT%setInOut %1 ccln

if not "%2"=="" if not "%2"=="." set OUTFILE=%2

set NUM=%3
if "%NUM%"=="." set NUM=
if "%NUM%"=="" set NUM=1

set N_CONNECT=8
set N_CONNECT=4

goto skip1

%IMG7%magick ^
  %INFILE% ^
  -define connected-components:verbose^=true ^
  -connected-components %N_CONNECT% ^
  NULL:

:skip1

set sKEEP=
set CNT=1
set LIM_AREA=0
set nTHRESH=
for /F "usebackq skip=1 tokens=4" %%A in (`%IMG7%magick ^
  %INFILE% ^
  -define connected-components:verbose^=true ^
  -connected-components %N_CONNECT% ^
  NULL:`) do (
  rem echo %%A
  if !CNT!==%NUM% set nTHRESH=%%A
  set /A CNT+=1
)

echo %0: nTHRESH=%nTHRESH%

if "%nTHRESH%"=="" (
  echo %0: Found CNT=%CNT%, not NUM=%NUM%
  exit /B 1
)

%IMG7%magick ^
  %INFILE% ^
  -virtual-pixel None ^
  -define connected-components:verbose=true ^
  -define connected-components:mean-color=true ^
  -define connected-components:area-threshold=%nTHRESH% ^
  -connected-components %N_CONNECT% ^
  -rotate 180 ^
  -connected-components %N_CONNECT% ^
  -rotate -180 ^
  %OUTFILE%

goto end


set sKEEP=
set CNT=1
set LIM_AREA=0
for /F "usebackq skip=1 tokens=1 delims=: " %%A in (`%IMG7%magick ^
  %INFILE% ^
  -define connected-components:verbose^=true ^
  -connected-components %N_CONNECT% ^
  NULL:`) do (
  rem echo %%A
  if !CNT! LEQ %NUM% set sKEEP=!sKEEP!,%%A
  set /A CNT+=1
)

echo sKEEP=%sKEEP%

rem Remove initial comma
set sKEEP=%sKEEP:~1%

echo sKEEP=%sKEEP%


:: FIXME: No, this makes the others transparent!!
:: We need to find the limiting area.

set UNIQ=\temp\ccln_uniq.png

%IMG7%magick ^
  %INFILE% ^
  -define connected-components:verbose^=true ^
  -define connected-components:keep=%sKEEP% ^
  -define connected-components:mean-color=true ^
  -connected-components %N_CONNECT% ^
  -unique-colors ^
  %UNIQ%

%IMG7%magick identify %UNIQ%

:end

call echoRestore

@endlocal & set cclnOUTFILE=%OUTFILE%

colCorr2.bat

rem Given images %1 and %2
rem %3 is an offset for #2 with respect to #1
rem makes output %4
rem that is #2 composited over #1, and colour-corrected.
@rem
@rem Also uses:
@rem   cc2DRAW if provided, draws this in gray before relax-fill.
@rem
@rem
@rem Updated:
@rem   24-July-2022 for IM v7.
@rem


@if "%3"=="" findstr /B "rem @rem" %~f0 & exit /B 1

@setlocal enabledelayedexpansion

@call echoOffSave

set INFILE1=%1
set INFILE2=%2
set OFFS_2=%3
set OUTFILE=%4

set TMP_FILE=\temp\cc2.miff

if "%cc2DRAW%"=="" (
  set sDRAW=
) else (
  set sDRAW=-fill gray^(50%%^) -stroke gray^(50%%^) -draw "%cc2DRAW%"
)

%IMG7%magick ^
  ( %INFILE1% +write mpr:IN1 ) ^
  -fill gray(50%%) -colorize 100 ^
  -fuzz 0.1%% ^
  ( %INFILE2% +write mpr:IN2 ^
    -fill Red -colorize 100 ^
    -shave 1x1 -bordercolor gray(50%%) -border 1 ^
    -repage %OFFS_2% ^
  ) ^
  -background Black ^
  -compose Plus -layers merge +repage ^
  -fill Black +opaque White ^
  -crop %OFFS_2% +repage ^
  +write mpr:LMASK ^
  +delete ^
  mpr:IN1 ^
  ( mpr:IN2 -repage %OFFS_2% ) ^
  -compose Mathematics ^
    -define compose:args=0,-0.5,0.5,0.5 ^
  -background None ^
  -layers merge ^
  -crop %OFFS_2% +repage ^
  mpr:LMASK ^
  -alpha off ^
  -compose CopyOpacity -composite ^
  %sDRAW% ^
  %TMP_FILE%

if ERRORLEVEL 1 exit /B 1

rem Relax-fill to get an adjustment image:

call %PICTBAT%relaxFillMS ^
  %TMP_FILE% . %TMP_FILE% ^
  1e-6 1000

if ERRORLEVEL 1 exit /B 1


:skip

rem Add the adjustment image to #2:

%IMG7%magick ^
  %INFILE2% ^
  %TMP_FILE% ^
  -compose Mathematics ^
    -define compose:args=0,2,1,-1 -composite ^
  -repage %OFFS_2% ^
  %INFILE1% ^
  +swap ^
  -background None -compose Over ^
  -layers merge +repage ^
  %OUTFILE%

if ERRORLEVEL 1 exit /B 1

endlocal& set cc2OUTFILE=%OUTFILE%

xSection.bat

rem From image %1,
rem make horizontal cross-section.
@rem
@rem Updated:
@rem   24-July-2022 for IM v7.
@rem

@if "%1"=="" findstr /B "rem @rem" %~f0 & exit /B 1

@setlocal

@call echoOffSave

call %PICTBAT%setInOut %1 xs

if not "%2"=="" if not "%2"=="." set OUTFILE=%2

set WW=
for /F "usebackq" %%L in (`%IMG7%magick identify ^
  -format "WW=%%w\nHH=%%h\nW_2=%%[fx:int(w/2)]\nH_2=%%[fx:int(h/2)]" ^
  %INFILE%`) do set %%L
if "%WW%"=="" exit /B 1

set TMP=%TEMP%\xs.miff

%IMG7%magick ^
  %INFILE% ^
  -crop %WW%x1+0+%H_2% +repage ^
  %TMP%

call %PICTBAT%graphLineCol %TMP% . . . %OUTFILE%

call echoRestore

endlocal & set xsOUTFILE=%OUTFILE%

All images on this page were created by the commands shown, using:

%IMG7%magick identify -version
Version: ImageMagick 7.1.1-15 Q16-HDRI x64 a0a5f3d:20230730 https://imagemagick.org
Copyright: (C) 1999 ImageMagick Studio LLC
License: https://imagemagick.org/script/license.php
Features: Cipher DPC HDRI OpenCL OpenMP(2.0) 
Delegates (built-in): bzlib cairo freetype gslib heic jng jp2 jpeg jxl lcms lqr lzma openexr pangocairo png ps raqm raw rsvg tiff webp xml zip zlib
Compiler: Visual Studio 2022 (193532217)

To improve internet download speeds, some images may have been automatically converted (by ImageMagick, of course) from PNG or TIFF or MIFF to JPG.

Source file for this web page is seamlpm.h1. To re-create this web page, run "procH1 seamlpm".


This page, including the images except where shown otherwise, is my copyright. Anyone is permitted to use or adapt any of the code, scripts or images for any purpose, including commercial use.

Anyone is permitted to re-publish this page, but only for non-commercial use.

Anyone is permitted to link to this page, including for commercial use.


Page version v2.0 25-June-2017.

Page created 23-Oct-2023 04:27:03.

Copyright © 2023 Alan Gibson.