snibgo's ImageMagick pages

Client-server

A simple program makes IM a non-stop background resource, a server for clients.

• References
• Non-stop IM
• Named pipes
• A simple client-server
• Performance
• Conclusion
• C code and Scripts

This page describes a program that runs continuously, accepting IM commands from stdin. This can be used interactively, or (via named pipes) as a server to clients in separate processes.

This page was inspired by a post on the ImageMagick forum. The program does not implement time-outs or other features that might be desired. It is a proof-of-concept only.

The methods shown might be useful in embedded video systems such as Raspberry Pi or similar. It might also be useful as a server to web-based clients, though more security and robust error-checking would be needed.

References

• ImageMagick forum post: "Batch Mode", 2022. The inspiration behind this page.
• Microsoft Learn: Pipes (Interprocess Communications), 2021. Explains Windows architecture and system calls. This page does not directly use these.

Non-stop IM

The program imserv.c is fairly simple: wait for a line of text from stdin, then execute it as a "magick" command, and repeat. It loops until the "command" it receives is "EOF" (three capital letters, without the quotes).

It is an ordinary console program. I build it with the GNU toolchain, and it runs either under Windows (CMD or BAT), or Cygwin bash. It uses the GNU extension function getline. Aside from that, I expect it can be built on any platform.

The program reads lines from stdin, stripping trailing \n or \r\n. If the final character on a line is backslash "\" the program interprets this as "line-continuation", meaning: remove the backslash, then read another line and append that to the current line, and keep going until the line doesn't end with backslash.

After any necessary appending, the program splits the line into tokens and sends them to the ImageMagick library via the MagickImageCommand() function.

The program sends strings directly to ImageMagick, not via any shell, so strings seen by the program should not have escapes appropriate to any shell.

The commands are independent of each other, as if they had been executed within seperate "magick" CLI commands. For example, we may "-write mpr:MYIMG" in one command, and read it later in that command, but we can't access it in other commands. Similarly, all settings are initialised for each command.

We can run the program interactively by typing commands, and finally typing "EOF" when we have finished.

We can run the program by piping commands from a program such as "echo" or "cat", or redirecting input from a file.

And we can run the program by reading commands from a named pipe (aka "FIFO" for "First In, First Out"), where a different process writes to that same named pipe.

Named pipes

The Windows CMD interface does not include facilities to manage named pipes. We could write C programs to do this, but instead I use Cygwin bash commands. For this page, I run those commands from the Windows "bash" command. This means the commands are first interpreted by Windows CMD, then CMD passes the required string to Cygwin bash. This has implications for what characters need escaping, and how to escape them, and how to express environment variables, and whether CMD or bash expands them.

We make a named pipe in the current directory:

bash -c "mkfifo mypipe"

A pipe is a file. From bash, it looks like this:

bash -c "ls -l mypipe" 

prw-rw-rw- 1 Alan None 0 Jan 19 01:03 mypipe

Note that the first letter in the flags is "p" for "pipe".

From Windows, it looks like this:

dir mypipe* 

19/01/2023  01:03               130 mypipe.lnk

Note that to Windows, it has the extension ".lnk".

We can test for the existence of a named pipe like this:

bash -c "if [ -p mypipe ]; then echo Yes; else echo No; fi" 

Yes

bash -c "if [ -p nosuch ]; then echo Yes; else echo No; fi" 

No

We can delete a pipe like this:

bash -c "rm mypipe"

Those are the basic commands for named pipes. Now we will use a named pipe for the stdin of imserv.exe.

A simple client-server

This web page is built within a single window, by executing the commands shown with a green background. So this window does the work of managing the server process, which occurs in a different window. A diagram illustrates these acivities:

For convenience, we ensure the directory containing imserv.exe is on the system path:

set PATH=%IM7DEV%;%PATH%

We make a named pipe in the current directory:

bash -c "mkfifo impipe"

Test that this has worked:

bash -c "if [ -p impipe ]; then echo Yes; else echo No; fi" 

Yes

We use Windows "start" to create a new window named "imCmdServ", and execute a bash command to start imserv.exe in that window, immediately returning control to this window without waiting for imserv.exe or bash to terminate. We redirect stdin to read from the named pipe and we redirect stderr to a conventional file.

start "imCmdServ" bash -c "imserv.exe --terminate EOF --verbose <impipe 2>clis_pipe.log"

imserv.exe will terminate when it receives "EOF", and then the window will close.

For testing, we first ensure clis_rose.png doesn't exist. Then we send a request to the server to create that file:

del clis_rose.png 2>nul

bash -c "echo ""rose: -resize 400x400 clis_rose.jpg"" >impipe"

Terminate the server:

bash -c "echo EOF >impipe"

Remove the pipe:

bash -c "rm impipe"

Here is clis_pipe.log, the log from imserv.exe:

IM command succeeded
Terminate received

The log shows that the IM command succeeded, and that the program received the command to terminate.

Here is clis_rose.png, the file created by the server:

How do we know when IM has finished processing? We can easily test for the existence of clis_rose.png, but that isn't enough: perhaps IM has created the file but is still writing it. In our simple situation, we can send another dummy request such as "xc: dummy.png" and wait for dummy.png to exist. Our server can process only one command at a time, so we know that if dummy.png exists, the previous command must have completed (or failed).

This is a simple client-server, a toy system, too simple for many purposes. In particular:

1. In a production system, a client would not manage the server processes, and certainly would not have the ability to terminate the server.
2. A production system would have proper error-checking, for example, to prevent a client from writing to a pipe that has no readers so that the client becomes blocked.
3. A production system would need security features, especially if there are multiple clients or multiple servers, and if the clients and servers have different levels of privilege, and if random end-users (eg hostile hackers) can be clients that issue commands to the server.
4. A more versatile system would have a queue mechanism, so a client could send multiple requests with no (or very little) blocking, and the server would process requests from the queue in its own time.
5. A production system needs feedback, so a client can know when requested work is finished, and whether there were errors.
6. We might have multiple servers on a multiprocessor responding to the same pipe. Multiple clients send tens or thousands of requests down a pipe; for each request, the first server that is free processes that request. This complicates the issue of feedback.

Performance

The script cliServPerf.bat compares the performance of the client-server with other methods. For each method, we do a few thousand iterations of "xc: NULL:". This command is about as simple as it can be, so the time represents the overhead of starting and stopping ImageMagick.

The methods tested are:

1. Windows BAT: a BAT script that repeats the command "magick xc: NULL:".
2. Cygwin bash: a bash script that repeats the command "magick xc: NULL:".
3. Redirection: run imserv.exe, redirecting from a file with multiple lines of "xc: NULL:".
4. IM script: run "magick -script csp_mag.scr" where the script repeats the command "xc: NULL:".
5. Client-server: create a named pipe; run imserv.exe in another window, to read from the pipe; cat a file of multiple line of "xc: NULL:" to the pipe; terminate imserv.exe; remove the pipe.

Perhaps we are being a bit unfair to the client-server test, as we include the overhead of creating the pipe, etc. In this method, we also remove and recreate clis_cliserv_proof.png, as proof that the server has worked.

We run the script:

call %PICTBAT%cliServPerf 4000 clis_perf.lis

The timings are shown in my usual format: "d hh:mm:ss", which is "days hours:minutes:seconds".

Prove the server worked (the image is a single red pixel):

In the worst case, calling "magick" a few thousand times in a Windows BAT script, we need about 13 milliseconds per call, about 1/75 of a second. For some applications, this time would be dwarfed by time required for the actual image processing. For my purposes, this overhead is not significant, but for real-time processing of video, where we might have a budget of 30ms per frame, the saving would be worthwhile.

Using an IM script is simple, and much faster than invoking the magick program multiple times. But we need to know all the processing before starting any processing, so this isn't suitable for real-time video or some other applications.

Redirection into imserv.exe, and using imserv.exe in a client-server configuration, have similar speeds. The client-server carries a small overhead, which is not surprising.

Conclusion

The imserv.exe program can be driven from a redirected file or in a client-server cofiguration. This is faster than issuing individual magick commands, either in Windows CMD or in bash.

However, an IM script is even faster. This is simpler than using imserv.exe, and should generally be preferred, if circumstances permit.

This page has shown how imserv.exe can be used as a continuously running service to clients. However, this is proof-of-concept only, and more development would be needed to make a robust production system.

C code and Scripts

For convenience, .bat scripts are also available in a single zip file. See Zipped BAT files.

imserv.c

#include <stdio.h>
#include <stdlib.h>
#include <string.h>

#include <MagickWand/MagickWand.h>

// Inspired from a forum post: https://github.com/ImageMagick/ImageMagick/issues/5898
//
// c:\cygwin64\home\Alan\imagemagick-7.1.0-19\ImageMagick\im-win-forfx\ImageMagick
// bash snibgo\buildwand.sh snibgo\imserv
// %IM7DEV%imserv

typedef struct {
  ExceptionInfo
    *exception;

  ImageInfo
    *image_info;

  MagickBooleanType
    exitOnFileEof,
    exitOnImErr,
    exitNow;

  int
    verbose;

  char
    *sTerminate;

  FILE *
    fh_data;
} imservT;

#define VERSION "imserv v1.0  Copyright (c) 2023 Alan Gibson"

static void usage (void)
{
  printf ("Usage: imserv {OPTION}...'\n");
  printf ("Reads IM commands from stdin and executes them.\n");
  printf ("\n");
  printf ("  -feof,  --exitOnFileEof string  whether a file eof forces program exit\n");
  printf ("  -imerr, --exitOnImErr string    whether an IM error forces program exit\n");
  printf ("  -t,     --terminate string      when program receives this string, it will terminate\n");
  printf ("  -f,     --file string           write verbose text to stderr or stdout\n");
  printf ("  -v,     --verbose               write text information to stderr\n");
  printf ("  -v2,    --verbose2              write more text information to stderr\n");
  printf ("          --version               write version information to stdout\n");
  printf ("  -h,     --help                  write this help to stdout\n");
  printf ("\n");
}


static MagickBooleanType IsArg (char * pa, char * ShtOpt, char * LongOpt)
{
  // LocaleCompare is not case-sensitive,
  // so we use strcmp for the short option.

  if ((strcmp(pa, ShtOpt)==0) || (LocaleCompare(pa, LongOpt)==0))
    return MagickTrue;

  return MagickFalse;
}


#define NEXTARG \
  if (i < argc-1) i++; \
  else { \
    fprintf (stderr, "imserv: ERROR: [%s] needs argument\n", pa); \
    status = MagickFalse; \
  }


static MagickBooleanType menu (
  const int argc,
  const char **argv,
  imservT * pims
)
// Returns MagickTrue if okay.
{
  int
    i;

  MagickBooleanType
    status = MagickTrue;

  pims->exitOnFileEof = MagickTrue;
  pims->exitOnImErr = MagickFalse;
  pims->verbose = 0;
  pims->fh_data = stderr;
  pims->sTerminate = "EOF";
  pims->exitNow = MagickFalse;

  for (i=1; i < argc; i++) {
    char * pa = (char *)argv[i];

    if (IsArg (pa, "-feof", "--exitOnFileEof")==MagickTrue) {
      NEXTARG;
      if (LocaleCompare(argv[i], "true")==0) pims->exitOnFileEof = MagickTrue;
      else if (LocaleCompare(argv[i], "false")==0) pims->exitOnFileEof = MagickFalse;
      else {
        fprintf (stderr, "imserv: ERROR: option %s needs true or false\n", pa);
        status = MagickFalse;
      }
    } else if (IsArg (pa, "-imerr", "--exitOnImErr")==MagickTrue) {
      NEXTARG;
      if (LocaleCompare(argv[i], "true")==0) pims->exitOnImErr = MagickTrue;
      else if (LocaleCompare(argv[i], "false")==0) pims->exitOnImErr = MagickFalse;
      else {
        fprintf (stderr, "imserv: ERROR: option %s needs true or false\n", pa);
        status = MagickFalse;
      }
    } else if (IsArg (pa, "-t", "--terminate")==MagickTrue) {
      NEXTARG;
      pims->sTerminate = (char *)argv[i];
    } else if (IsArg (pa, "-f", "--file")==MagickTrue) {
      NEXTARG;
      if (LocaleCompare (argv[i], "stdout")==0) pims->fh_data = stdout;
      else if (LocaleCompare (argv[i], "stderr")==0) pims->fh_data = stderr;
      else {
        fprintf (stderr, "imserv: ERROR: option %s needs stdout or stderr\n", pa);
        status = MagickFalse;
      }
    } else if (IsArg (pa, "-v", "--verbose")==MagickTrue) {
      pims->verbose = 1;
    } else if (IsArg (pa, "-v2", "--verbose2")==MagickTrue) {
      pims->verbose = 2;
    } else if (IsArg (pa, "-version", "--version")==MagickTrue) {
      fprintf (stdout, "%s\n", VERSION);
      pims->exitNow = MagickTrue;
    } else if (IsArg (pa, "-h", "--help")==MagickTrue) {
      usage ();
      pims->exitNow = MagickTrue;
    } else {
      fprintf (stderr, "imserv: ERROR: unknown option [%s]\n", pa);
      status = MagickFalse;
    }
  }

  if (status == MagickFalse) {
    usage ();
    pims->exitNow = MagickTrue;
  }

  return (status);
}


static MagickBooleanType ExecIm (imservT * pims, int argc, char **argv)
{
  MagickBooleanType
    status;

  status = MagickImageCommand (pims->image_info, argc, argv, NULL, pims->exception);

  if (status == MagickFalse) {
    fprintf (pims->fh_data, "MagickImageCommand failed\n");
    MagickError (pims->exception->severity, pims->exception->reason, pims->exception->description);
    status = MagickFalse;
  }

  if (pims->exception->severity != UndefinedException) {
    fprintf (pims->fh_data, "MagickImageCommand exception\n");
    CatchException (pims->exception);
    status = MagickFalse;
  }

  if (pims->verbose) fprintf (pims->fh_data, "IM command %s\n", status ? "succeeded" : "failed");

  return status;
}


static inline void StripTrailNewline (char ** Cmd, ssize_t *CmdLen)
// Strip trailing \n, then \r
{
  if (*(*Cmd + *CmdLen-1) == '\n') {
    *(*Cmd + *CmdLen - 1) = '\0';
    (*CmdLen)--;
  }
  if (*(*Cmd + *CmdLen-1) == '\r') {
    *(*Cmd + *CmdLen - 1) = '\0';
    (*CmdLen)--;
  }
}


static MagickBooleanType GetCommand (imservT * pims, char ** Cmd, ssize_t *CmdLen)
// Allocates Cmd; needs to be freed somewhere.
// Returns whether okay.
// Return MagickFalse if either oom, or file eof.
{
  MagickBooleanType okay = MagickTrue;
  *Cmd = NULL;
  *CmdLen = 0;
  size_t max_line_len = 0;
  ssize_t inlen = getline (Cmd, &max_line_len, stdin);

  if (inlen < 0) {
    fprintf (pims->fh_data, "getline failed; perhaps eof\n");
    if (pims->exitOnFileEof) okay = MagickFalse;
  }

  StripTrailNewline (Cmd, &inlen);

  // While inbuf ends with line-continuation character,
  // get another line and append them.

  #define cContinueLine '\\'

  char * p = *Cmd + inlen - 1;
  while (*p == cContinueLine) {
    char * NextLine = NULL;
    size_t NextLineLen = 0;
    ssize_t NewLen = getline (&NextLine, &NextLineLen, stdin);

    if (NewLen < 0) {
      fprintf (pims->fh_data, "next getline failed; perhaps EOF\n");
      if (pims->exitOnFileEof) okay = MagickFalse;
      break;
    }

    StripTrailNewline (&NextLine, &NewLen);

    inlen += NewLen - 1;
    *Cmd = realloc (*Cmd, (size_t)(inlen + 2));
    if (! *Cmd) {
      fprintf (pims->fh_data, "realloc failed\n");
      okay = MagickFalse;
      break;
    }
    memcpy (p, NextLine, (size_t)(NewLen+1));
    free (NextLine);
    p = *Cmd + inlen - 1;
  }

  *CmdLen = (okay) ? inlen : 0;

  return okay;
}


static MagickBooleanType ProcessCommands (imservT * pims)
// Returns only when file eof, or EOF received, or error.
{
  MagickBooleanType okay = MagickTrue;

  char ** argv;
  int argc;

  char * inbuf = NULL;
  ssize_t inlen;


  while (okay) {
    okay = GetCommand (pims, &inbuf, &inlen);
    if (!okay) break;

    if (*pims->sTerminate && (strcmp (inbuf, pims->sTerminate)==0)) {
      if (pims->verbose) fprintf (pims->fh_data, "Terminate received\n");
      okay = MagickFalse;
      break;
    }

    if (pims->verbose > 1) fprintf (pims->fh_data, "inlen=%li [%s]\n", inlen, inbuf);

    if (inlen) {
      argv = StringToArgv (inbuf, &argc);
      if (!argv) {
        fprintf (pims->fh_data, "StringToArgv failed\n");
        okay = MagickFalse;
        break;
      }

      MagickBooleanType ImOkay = ExecIm (pims, argc, argv);

      if (pims->exitOnImErr && !ImOkay) okay = MagickFalse;

      ssize_t j;
      for (j=0; j < (ssize_t) argc; j++) {
        if (pims->verbose > 1) fprintf (pims->fh_data, "  [%s]", argv[j]);
        argv[j] = DestroyString(argv[j]);
      }
      if (pims->verbose > 1) fprintf (pims->fh_data, "\n");

      argv=(char **) RelinquishMagickMemory (argv);
    }

    if (inbuf) free (inbuf);
    inbuf = NULL;
  }

  if (inbuf) free (inbuf);

  return okay;
}


int main (int argc, const char **argv)
{
  imservT ims;

  MagickBooleanType status = menu (argc, argv, &ims);
  if (status == MagickFalse) return (-1);
  if (ims.exitNow) return 0;

  MagickWandGenesis ();

  ims.exception = AcquireExceptionInfo ();
  ims.image_info = AcquireImageInfo ();

  MagickBooleanType okay = ProcessCommands (&ims);

  ims.image_info = DestroyImageInfo (ims.image_info);
  ims.exception = DestroyExceptionInfo (ims.exception);

  MagickWandTerminus ();

  return (okay ? 0 : 1);
}

cliServPerf.bat

rem Explore performance of client/server methods.
rem %1 number of iterations
rem %2 output text file for table

setlocal enabledelayedexpansion

set PATH=%IMG7%;%IM7DEV%;%PATH%

set nIter=%1

echo off

(
  echo echo off
  for /L %%N in (1,1,%nIter%) do echo magick xc: NULL:
  echo echo on
) >clis_mag.bat

(
  for /L %%N in (1,1,%nIter%) do echo magick xc: NULL:
) >clis_mag.sh

(
  for /L %%N in (1,1,%nIter%) do echo xc: NULL:
) >clis_mag.lis

(
  for /L %%N in (1,1,%nIter%) do echo xc: -write NULL: +delete
) >clis_mag.scr


copy clis_mag.lis clis_mag2.lis
echo xc:red clis_cliserv_proof.png >>clis_mag2.lis

del clis_cliserv_proof.png 2>nul

call StopWatch
call clis_mag.bat
call StopWatchEnv tBat
bash clis_mag.sh
call StopWatchEnv tBash
%IM7DEV%imserv <clis_mag.lis >nul 2>&1
call StopWatchEnv tRedir
magick -script clis_mag.scr
call StopWatchEnv tImScr
bash -c "mkfifo impipe_cs"
start "imCmdServ" bash -c "imserv.exe --terminate EOF --exitOnFileEof false <impipe_cs"
bash -c "cat clis_mag2.lis >impipe_cs"
bash -c "echo EOF >impipe_cs"
bash -c "until [ -f clis_cliserv_proof.png ]; do echo Waiting; sleep 0.25; done"
bash -c "rm impipe_cs"
call StopWatchEnv tCliServ

(
  echo ^<table^>
  echo ^<tr^>
  echo ^<th^>Method^</th^>^<th^>Elapsed^<br /^>time^</th^>
  echo ^</tr^>^<tr^>
  echo ^<th^>Windows BAT^</th^>^<td^>%tBat%^</td^>
  echo ^</tr^>^<tr^>
  echo ^<th^>Cygwin bash^</th^>^<td^>%tBash%^</td^>
  echo ^</tr^>^<tr^>
  echo ^<th^>Redirection^</th^>^<td^>%tRedir%^</td^>
  echo ^</tr^>^<tr^>
  echo ^<th^>IM script^</th^>^<td^>%tImScr%^</td^>
  echo ^</tr^>^<tr^>
  echo ^<th^>Client-server^</th^>^<td^>%tCliServ%^</td^>
  echo ^</tr^>
  echo ^</table^>
) >%2

echo on

endlocal

%IMG7%magick -version

Version: ImageMagick 7.1.0-49 Q16-HDRI x64 7a3f3f1:20220924 https://imagemagick.org
Copyright: (C) 1999 ImageMagick Studio LLC
License: https://imagemagick.org/script/license.php
Features: Cipher DPC HDRI OpenCL 
Delegates (built-in): bzlib cairo freetype gslib heic jng jp2 jpeg jxl lcms lqr lzma openexr pangocairo png ps raqm raw rsvg tiff webp xml zip zlib
Compiler: Visual Studio 2022 (193331630)

bash --version

GNU bash, version 4.4.12(3)-release (x86_64-unknown-cygwin)
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>

This is free software; you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

%IM7DEV%imserv --version

imserv v1.0  Copyright (c) 2023 Alan Gibson

Page version v1.0 19-January-2023.

Page created 19-Jan-2023 01:05:29.

Copyright © 2023 Alan Gibson.