snibgo's ImageMagick pages

New FX tests

Tests for a proposed faster "-fx".

• Performance
• Compatibility and accuracy
• Error checking and reporting
• Many small images
• Scripts

This page shows black-box tests performed on a new version of "-fx", comparing results to an old version of "-fx".

The tests were performed on my Windows 10 laptop. The CPU (Intel Core i7-4700MQ @2.40GHz) has 4 cores, and can run 8 threads. The IM version was v7.1.0-20, Q32, with HDRI and OpenMP. Two builds were created, for the old and new "-fx", and installed in the directories "%FXOLD%" and "%FXNEW%".

These tests are not exhaustive. I welcome any additions.

This is a companion page to New FX.

Performance

The script testPerf.bat runs a magick command for the old fx, and repeats it for the new fx. The command runs a "-fx" operation for each of the input strings in testPerf.txt. There are about 35 input strings.

Compatibility and accuracy

To test compatiblity, we run the same "-fx" operation using the old and new fx code, and find the RMSE distortion between the two results. The distortion is on a nominal scale of zero to one. If it is very close to zero, we consider the test has passed.

For accuracy, the results were compared to hand-calculations.

On my computer, the tests are run with the new fx installed at %FXNEW%:

set FXNEW=%IM7DEVFX2%
set FXNEW=%IM7DEV2%

set FXNEW 

FXNEW=C:\cygwin64\home\Alan\imdevins711_10\bin\

As of 31-March-2023, I patch the file fx.c that is in directory %IM7DEV%.

set FXNEW=%IM7DEV%

set FXNEW 

FXNEW=C:\cygwin64\home\Alan\imdevins7114\bin\

call %IM7SRCFX%\snibgo\goodFxSyntax.bat

Tests with a single input image:

Tests with two input images:

Tests that fail (they show significant differences):

"2300m" has an SI prefix meaning "multiplied by 10^-3", and the negative power doesn't work in the old "-fx".

"23 ;" shows different processing of the blank statement after the semi-colon. The old "-fx" returns 0 from blank statements. In the new "-fx", blank statements have no effect, so the return is whatever the previous statement returned.

Expressions with "rand()" give different results each time.

The old "-fx" has a problem with multiple unary prefixes:

%FXOLD%magick xc: -fx "23+(--2.5)" txt: 
%FXNEW%magick xc: -fx "23+(--2.5)" txt: 

# ImageMagick pixel enumeration: 1,1,0,4294967295,srgb
0,0: (109521666023,109521666023,109521666023)  #FFFFFFFFFFFFFFFFFFFFFFFF  srgb(2550%,2550%,2550%)
# ImageMagick pixel enumeration: 1,1,0,4294967295,srgb
0,0: (109521666023,109521666023,109521666023)  #FFFFFFFFFFFFFFFFFFFFFFFF  srgb(2550%,2550%,2550%)

I regard the new result to be correct.

For "xx=3; yy=xx++; yy", the old "-fx" returns zero. This is because it doesn't return a value from "xx++", but doesn't report a syntax problem and instead returns zero, which is used to set yy.

Sample formats

The script fmtTst.bat tests "-format %[string]" for the strings shown, for the old and new "-fx". There are two images, so two values are shown for each of the old and the new. It shows "same" when the outputs are identical, or "DIFFERENT" when they are different.

call %IM7SRCFX%\snibgo\fmtTst.bat

"-format %[fx:string]" using pixel operands without channel qualifiers are different because, as explained at New FX, the old "%[fx:u]" etc give an intensity, and the new versions give the red channel.

In cases where we don't have pixel operands without channel qualifiers, "-format %[fx:string]" give identical results.

"-format %[hex:string]" are entirely the same.

"-format %[pixel:string]" are entirely the same.

The script goodpc.bat tests examples of "%[...]" that were kindly supplied by GeeMack. (Many thanks, GeeMack!) For these tests, occurrences of "rand()" were changed to "0.23". This exposed two bugs in the new fx, now fixed, so all examples create identical outputs for the old and new fx code.

call %WebDisk%\web\im\geemack\goodpc.bat

Error checking and reporting

Here are some invalid input strings, with each resulting error message. This error checking is done at translation, and has no impact on run-time performance.

call %IM7SRCFX%\snibgo\badFxSyntax fxnt_err.lis

A message that includes "ValStack underflow" suggests the translator has a bug. The count of "underflow" messages in the above is: 0 .

A small selection of Fred's ImageMagick scripts were executed in sequence for the old "-fx" and the new "-fx2". For convenience on my computer, they were first modified to use the "magick" command. The elapsed times were measured, and the outputs from the old and new "-fx" were compared with "-metric RMSE".

The scripts were: adaptivegamma, defisheye, disperse, pano2rect, sharpedge, statsfilt, tunnelize, warplog and xpand.

The "disperse" script uses randomness, creating a different result each time, so the RMSE difference from that script was ignored.

The table shows the worst RMSE of the scripts (excluding "disperse"), on a nominal scale of 0.0 to 1.0. An RMSE of less than 1e-02 will generally not be noticable.

Many small images

The new fx code was developed to increase performance of large images, by decreasing the processing required at each pixel, at the expense of increasing the processing required at each image, which includes more extensive syntax checking and translation to a run-time RPN language.

A consequence is that the total processing required for small images has increased. We want to ensure the reduced performance is not excessive.

We consider two cases: without image statistics, then with image statistics.

Without image statistics

%FXOLD%magick xc:white -duplicate 999 +noise random -brightness-contrast %%[fx:50-(0.34*100)] null:
%FXNEW%magick xc:white -duplicate 999 +noise random -brightness-contrast %%[fx:50-(0.34*100)] null:

0 00:00:07
0 00:00:06

The times are shown in the format "D HH:MM:SS".

%FXOLD%magick xc:white -duplicate 999 +noise random -fx (50-(0.34*100))/100 null:
%FXNEW%magick xc:white -duplicate 999 +noise random -fx (50-(0.34*100))/100 null:

0 00:00:01
0 00:00:00

Conclusion: when no image statistics are involved, there is very little difference in performance between the old and new code.

With image statistics

%FXOLD%magick xc:white -duplicate 999 +noise random -brightness-contrast %%[fx:50-(mean*100)] null:
%FXNEW%magick xc:white -duplicate 999 +noise random -brightness-contrast %%[fx:50-(mean*100)] null:

0 00:00:09
0 00:00:10

In the above "%[fx:...]" example, the mean of each image is required. The old and the new calculate the mean of each image (and all the other statistics of each image).

%FXOLD%magick xc:white -duplicate 999 +noise random -fx (50-(mean*100))/100 null:
%FXNEW%magick xc:white -duplicate 999 +noise random -fx (50-(mean*100))/100 null:

0 00:00:03
0 00:00:03

In the above "-fx" example, the mean of only the first image is required. The old fx calculates the mean of just that image. The new fx calculates the mean of every image in the list, so it takes longer.

Conclusion: the new fx code does decrease performance when we have many small images, but this is not excessive.

As is often the case, rearranging the command can increase performance. For example, repeating the above example but using an embedded "%[fx:mean]":

%FXNEW%magick xc:white -duplicate 999 +noise random -fx "(50-(%%[fx:mean]*100))/100" null:

0 00:00:01

Scripts

For convenience, .bat scripts are also available in a single zip file. See Zipped BAT files.

The scripts use some utility programs such as StopWatchcGrep and chStrs. I do not supply the source or binaries of these.

testPerf.bat

set IMG=%~1
if "%IMG%"=="" set IMG=-size 1000x1000 xc:#abd

set SCRFILE=tp.scr

echo off

(
  for /F "tokens=*" %%L in (%~dp0testPerf.txt) do (
    echo ^( -clone 0 -fx "%%L" +delete ^)
  )

  echo NULL:
) >%SCRFILE%

echo on

type %SCRFILE%


call StopWatch
%FXOLD%magick %IMG% -script %SCRFILE%
call StopWatch
%FXNEW%magick %IMG% -script %SCRFILE%
call StopWatch

goodFxSyntax.bat

set xc2=xc:#abd xc:#432

%FXOLD%magick %xc2% -append +repage gfs.png

set VPIX=-virtual-pixel Black
set PREC=-precision 6
set OUTSPEC=-define quantum:format=floating-point -depth 32


echo off
(
for /F "tokens=*" %%L in (%~dp0goodfxsyntax.txt) do (
  ( %FXNEW%magick %PREC% gfs.png %VPIX% -fx "%%L" %OUTSPEC% gfs_new.miff 2^>^&1 ) |tail -5

  %FXOLD%magick %PREC% gfs.png %VPIX% -fx "%%L" %OUTSPEC% gfs_old.miff

  set cleaned=%%L
  set "cleaned=!cleaned:<=&lt;!"
  set "cleaned=!cleaned:>=&gt;!"
  set "cleaned=!cleaned:&=++amp++!"
  set "cleaned=!cleaned:%%=__!"
  set "cleaned=!cleaned:++amp++=&!"

  %IMG7%magick %PREC% gfs_old.miff gfs_new.miff -metric RMSE -format "=start= '!cleaned!' distortion=%%[distortion] =end=\n" -compare info: 
)
) >fxn_gfs.lis 

echo on

cgrep /ifxn_gfs.lis /ofxn_gfs1a.lis /sdistortion= /f\s\r\n

chStrs /ifxn_gfs1a.lis /ofxn_gfs1a.lis /f"\(" /t^&lt;
chStrs /ifxn_gfs1a.lis /ofxn_gfs1a.lis /f"\)" /t^&gt;
chStrs /ifxn_gfs1a.lis /ofxn_gfs1a.lis /f"__" /t%%

chStrs /ifxn_gfs1a.lis /ofxn_gfs1b.lis /f"=start=" /t\(tr\)\(td\)
chStrs /ifxn_gfs1b.lis /ofxn_gfs1b.lis /f"distortion=" /t\(/td\)\(td\)
chStrs /ifxn_gfs1b.lis /ofxn_gfs1b.lis /f"=end=" /t\(/td\)\(/tr\)

cPrefix /ifxn_gfs1b.lis /X /t"\(tr\)\(th\)-fx input string\(/th\)\(th\)distortion\(/th\)\(/tr\)"
cPrefix /ifxn_gfs1b.lis /X /t"\(table\)" /b"\(/table\)"


echo off
(
for /F "tokens=*" %%L in (%~dp0goodfxsyntax2.txt) do (
  ( %FXNEW%magick %PREC% -size 4x4 %xc2% %VPIX% -fx "%%L" %OUTSPEC% gfs2_new.miff 2^>^&1 ) |tail -5

  %FXOLD%magick %PREC% -size 4x4 %xc2% %VPIX% -fx "%%L" %OUTSPEC% gfs2_old.miff

  set cleaned=%%L

  set cleaned=!cleaned:%%=__!

  %IMG7%magick %PREC% gfs2_old.miff gfs2_new.miff -metric RMSE -format "=start= '!cleaned!' distortion=%%[distortion] =end=\n" -compare info:
)
) >fxn_gfs2.lis

echo on

cgrep /ifxn_gfs2.lis /ofxn_gfs2a.lis /sdistortion= /f\s\r\n

chStrs /ifxn_gfs2a.lis /ofxn_gfs2b.lis /f"=start=" /t\(tr\)\(td\)
chStrs /ifxn_gfs2b.lis /ofxn_gfs2b.lis /f"distortion=" /t\(/td\)\(td\)
chStrs /ifxn_gfs2b.lis /ofxn_gfs2b.lis /f"=end=" /t\(/td\)\(/tr\)

cPrefix /ifxn_gfs2b.lis /X /t"\(tr\)\(th\)-fx input string\(/th\)\(th\)distortion\(/th\)\(/tr\)"
cPrefix /ifxn_gfs2b.lis /X /t"\(table\)" /b"\(/table\)"


echo off
(
for /F "tokens=*" %%L in (%~dp0goodBadFxSyntax.txt) do (
  ( %FXNEW%magick %PREC% -size 4x4 %xc2% %VPIX% -fx "%%L" %OUTSPEC% gfs3_new.miff 2^>^&1 ) |tail -5

  %FXOLD%magick %PREC% -size 4x4 %xc2% %VPIX% -fx "%%L" %OUTSPEC% gfs3_old.miff

  set cleaned=%%L

  set cleaned=!cleaned:%%=__!

  %IMG7%magick %PREC% gfs3_old.miff gfs3_new.miff -metric RMSE -format "=start= '!cleaned!' distortion=%%[distortion] =end=\n" -compare info:
)
) >fxn_gfs3.lis

echo on

cgrep /ifxn_gfs3.lis /ofxn_gfs3a.lis /sdistortion= /f\s\r\n

chStrs /ifxn_gfs3a.lis /ofxn_gfs3b.lis /f"=start=" /t\(tr\)\(td\)
chStrs /ifxn_gfs3b.lis /ofxn_gfs3b.lis /f"distortion=" /t\(/td\)\(td\)
chStrs /ifxn_gfs3b.lis /ofxn_gfs3b.lis /f"=end=" /t\(/td\)\(/tr\)

cPrefix /ifxn_gfs3b.lis /X /t"\(tr\)\(th\)-fx input string\(/th\)\(th\)distortion\(/th\)\(/tr\)"
cPrefix /ifxn_gfs3b.lis /X /t"\(table\)" /b"\(/table\)"

goodpc.bat

set IMGS=xc:#abd xc:#654

set SLICES=5

set PREC=-precision 6

chStrs /i%~dp0goodpc.txt /ogoodpc_norand.txt /frand() /t0.23

%FXNEW%magick %IMGS% -define fx:debug=true -format "%%[fx:%~1]\n" info:

echo off
(
  for /F "tokens=*" %%L in (goodpc_norand.txt) do (
    for /F "usebackq tokens=*" %%B in (`%FXOLD%magick %PREC% %IMGS% -format "<td>%%L</td>" info:`) do set SOLD=%%B

    for /F "usebackq tokens=*" %%B in (`%FXNEW%magick %PREC% %IMGS% -format "<td>%%L</td>" info:`) do set SNEW=%%B

    set goodBad=same

    if !SOLD! NEQ !SNEW! set goodBad=DIFFERENT

    set NoLtGt=%%L
    set "NoLtGt=!NoLtGt:<=&lt;!"
    set "NoLtGt=!NoLtGt:>=&gt;!"

    echo =start= "!NoLtGt!" =mid= !SOLD! !SNEW! =mid2= !goodBad! =end=
  )
) >fxn_ft.lis

cGrep /ifxn_ft.lis /ofxn_ftb.lis /srand() /x

chStrs /ifxn_ftb.lis /ofxn_ftb.lis /f"=start=" /t"\(tr\)\(td\)"
chStrs /ifxn_ftb.lis /ofxn_ftb.lis /f"=mid=" /t"\(/td\)"
chStrs /ifxn_ftb.lis /ofxn_ftb.lis /f"=mid2=" /t"\(td\)"
chStrs /ifxn_ftb.lis /ofxn_ftb.lis /f"=end=" /t"\(/td\)\(/tr\)"

cPrefix /ifxn_ftb.lis /X /t"\(tr\)\(th\)format string\(/th\)\(th cspan\)old result\(/th\)\(th cspan\)new result\(/th\)\(th\)same/different\(/th\)\(/tr\)"
cPrefix /ifxn_ftb.lis /X /t"\(table\)" /b"\(/table\)"

chStrs /ifxn_ftb.lis /ofxn_ft_gm.lis /fcspan /tcolspan=\q2\q

echo on

badFxSyntax.bat

set LIS_FILE=%1

if "%LIS_FILE%"=="" set LIS_FILE=fxn_err.lis

echo off

(
echo ^<table^>
echo ^<tr^>^<th^>-fx input string^</th^>^<th^>Error message^</th^>^</tr^>

for /F "tokens=*" %%A in (%~dp0badfxsyntax.txt) do (
  for /F "usebackq tokens=*" %%B in (`%FXNEW%magick xc:#8bd -fx "%%A" NULL: 2^>^&1`) do set msg=%%B

  set cleaned=!msg!
  set "cleaned=!cleaned:<=&lt;!"
  set "cleaned=!cleaned:>=&gt;!"
  set "cleaned=!cleaned:&=++amp++!"
  set "cleaned=!cleaned:%%=__!"
  set "cleaned=!cleaned:++amp++=&!"

  set msg = !cleaned!

  set msg2=!msg:magick: =!

  if !msg2! NEQ !msg! (
    set msg3=!msg2:fatal/fx2.inc/=!
    echo ^<tr^>^<td^>%%A^</td^>^<td^>!msg3!^</td^>^</tr^>
  )
)
echo ^</table^>
) >%LIS_FILE%

echo on

echo created %LIS_FILE%

fmtTst.bat

set IMGS=xc:#abd xc:#654

set PREC=-precision 6

%FXNEW%magick %IMGS% -define fx:debug=true -format "%%[fx:%~1]\n" info:

%FXOLD%magick %IMGS% -format "%%[fx:%~1]\n" info:


for %%A in (fx) do (

  echo %%A

  echo off
  (
    for /F "tokens=*" %%L in (%~dp0fmtTstDiff.txt) do (
      for /F "usebackq tokens=*" %%B in (`%FXOLD%magick %PREC% %IMGS% -format "<td>%%[%%A:%%L]</td>" info:`) do set SOLD=%%B

      for /F "usebackq tokens=*" %%B in (`%FXNEW%magick %PREC% %IMGS% -format "<td>%%[%%A:%%L]</td>" info:`) do set SNEW=%%B

      set goodBad=same

      if !SOLD! NEQ !SNEW! set goodBad=DIFFERENT

      echo =start= "%%A:%%L" =mid= !SOLD! !SNEW! =mid2= !goodBad! =end=
    )
  ) >fxn_ft.lis
  echo on

  chStrs /ifxn_ft.lis /ofxn_ftb.lis /f"=start=" /t"\(tr\)\(td\)"
  chStrs /ifxn_ftb.lis /ofxn_ftb.lis /f"=mid=" /t"\(/td\)"
  chStrs /ifxn_ftb.lis /ofxn_ftb.lis /f"=mid2=" /t"\(td\)"
  chStrs /ifxn_ftb.lis /ofxn_ftb.lis /f"=end=" /t"\(/td\)\(/tr\)"

  cPrefix /ifxn_ftb.lis /X /t"\(tr\)\(th\)format string\(/th\)\(th cspan\)old result\(/th\)\(th cspan\)new result\(/th\)\(th\)same/different\(/th\)\(/tr\)"
  cPrefix /ifxn_ftb.lis /X /t"\(table\)" /b"\(/table\)"

  chStrs /ifxn_ftb.lis /ofxn_ft_diff_%%A.lis /fcspan /tcolspan=\q2\q
)

for %%A in (fx pixel hex) do (

  echo %%A

  echo off
  (
    for /F "tokens=*" %%L in (%~dp0fmtTst.txt) do (
      for /F "usebackq tokens=*" %%B in (`%FXOLD%magick %PREC% %IMGS% -format "<td>%%[%%A:%%L]</td>" info:`) do set SOLD=%%B

      for /F "usebackq tokens=*" %%B in (`%FXNEW%magick %PREC% %IMGS% -format "<td>%%[%%A:%%L]</td>" info:`) do set SNEW=%%B

      set goodBad=same

      if !SOLD! NEQ !SNEW! set goodBad=DIFFERENT

      echo =start= "%%A:%%L" =mid= !SOLD! !SNEW! =mid2= !goodBad! =end=
    )
  ) >fxn_ft.lis
  echo on

  chStrs /ifxn_ft.lis /ofxn_ftb.lis /f"=start=" /t"\(tr\)\(td\)"
  chStrs /ifxn_ftb.lis /ofxn_ftb.lis /f"=mid=" /t"\(/td\)"
  chStrs /ifxn_ftb.lis /ofxn_ftb.lis /f"=mid2=" /t"\(td\)"
  chStrs /ifxn_ftb.lis /ofxn_ftb.lis /f"=end=" /t"\(/td\)\(/tr\)"

  cPrefix /ifxn_ftb.lis /X /t"\(tr\)\(th\)format string\(/th\)\(th cspan\)old result\(/th\)\(th cspan\)new result\(/th\)\(th\)same/different\(/th\)\(/tr\)"
  cPrefix /ifxn_ftb.lis /X /t"\(table\)" /b"\(/table\)"

  chStrs /ifxn_ftb.lis /ofxn_ft_%%A.lis /fcspan /tcolspan=\q2\q
)

%FXOLD%magick identify -version

Version: ImageMagick 7.1.1-5 (Beta) Q32-HDRI x86_64 852a723f1:20230319 https://imagemagick.org
Copyright: (C) 1999 ImageMagick Studio LLC
License: https://imagemagick.org/script/license.php
Features: Cipher DPC HDRI Modules OpenMP(4.5) 
Delegates (built-in): bzlib cairo fftw fontconfig freetype heic jbig jng jpeg lcms ltdl lzma pangocairo png raqm raw rsvg tiff webp wmf x xml zip zlib
Compiler: gcc (11.3)

%FXNEW%magick identify -version

Version: ImageMagick 7.1.1-5 (Beta) Q32-HDRI x86_64 852a723f1:20230319 https://imagemagick.org
Copyright: (C) 1999 ImageMagick Studio LLC
License: https://imagemagick.org/script/license.php
Features: Cipher DPC HDRI Modules OpenMP(4.5) 
Delegates (built-in): bzlib cairo fftw fontconfig freetype heic jbig jng jpeg lcms ltdl lzma pangocairo png raqm raw rsvg tiff webp wmf x xml zip zlib
Compiler: gcc (11.3)

Page version v1.0 8-January-2022.

Page created 04-Jul-2023 19:39:48.

Copyright © 2023 Alan Gibson.