Copyright Notice

This text is copyright by CMP Media, LLC, and is used with their permission. Further distribution or use is not permitted.

This text has appeared in an edited form in WebTechniques magazine. However, the version you are reading here is as the author originally submitted the article for publication, not after their editors applied their creativity.

Please read all the information in the table of contents before using this article.
Download this listing!

Web Techniques Column 17 (September 1997)

A few months ago, I taught a Perl class at Digital Ink, the producers of the www.washingtonpost.com web site. As I was walking through their web site, I stumbled across some pages that had a neat feature for their images: upon loading the page, I'd immediately get a rather large, but fast-loading, black-and-white gif of the image. Then slowly, the color would magically appear!

I thought this rather neat, and wondered how I would do that. I first decided not to look at the source of the page, but just try to figure it out on my own. The basic mechanism that I came up with is the topic of this month's column. My guess was that the image wasn't really a single image, but two images, fed into the browser by something called ``server push'' or ``server animation push''. More properly described as the use of a special MIME type to cause the browser (either Netscape family products or Microsoft products) to keep reading multiple versions for the same URL.

However, when I finished my prototype for the program described later, I noticed that it wouldn't cache -- that is, if I went back to the same site later, I saw the black-and-white preview GIF again, before loading the real image (again!). This was an unsatisfactory result for long-term or heavily used sites. Upon further investigation, I discovered that the www.washingtonpost.com site was using the Netscape-proprietary LOWSRC option to the IMG tag!

As far as server load and user bandwidth, the Digital Ink method is thus clearly superior, because the images can be cached. However, having constructed this slightly-more-portable script, I suspect that you'll still get something out of the basic techniques anyway. So, on we go with the program in [listing one, below].

Lines 1 through 4 begin nearly every program I write, especially for CGI-style scripts. Here, I'm turning on taint-checking, warnings, all the best compiler restrictions, and disabling the buffering on STDOUT.

Line 6 pulls in the File::Copy module, needed for the copy routine that I use later. (This module is distributed with all recent Perl versions.)

Similarly, line 7 pulls in the HTTP::Date module, but extracts just the time2str routine. This module is part of the LWP library, an essential piece of every Webmaster's Perl toolkit. If you don't have yours installed yet, get it from the CPAN (with a nice index at http://www.perl.com/CPAN/CPAN.html). The time2str routine will give us a nice HTTP-compatible formatted date from a UNIX-internal timestamp.

Lines 9 through 24 gather together everything that might need to be changed to have this script work in different locations. Lines 11 through 13 define $REAL and $PREVIEW, using their common parent directory, defined in $IMAGE_TOP. ($IMAGE_TOP is not used further in the program, so if your $REAL and $PREVIEW don't have any common part, you can skip the definition.)

Line 15 sets up the PATH variable (for child processes). This PATH must contain the directories of the NETPBM tools, which on my ISP's system have been installed in /usr/local/netpbm/bin.

Lines 26 through 30 define program-wide constants. In this case, just the boundary string that will be included as part of the multipart MIME type. The string is rather arbitrary, but should be something that will be unlikely to occur in the output text.

Lines 32 through 36 grab the input provided by the CGI interface. Here, we're not really going into any great effort to parse the entire CGI input. (For that, I would have used the very nice CGI.pm library.) Instead, we just need to capture the part of the URL after the CGI script name. Because of the interface, this parameter will either be undef, empty, or start with a forward-slash.

Lines 38 through 54 construct the $file, $real, and $preview variables, based on the $PATH_INFO variable. The main part of the logic is placed inside an eval() block, which is being used here to trap any user-induced errors and generate the appropriate message.

Line 42 parses the $PATH_INFO variable, looking for reasonable filenames for either a GIF or a JPEG. The leading slash is automatically removed, because I've placed it outside the memory reference parentheses for memory #1. Also, scary names like .. are automatically rejected here, because the name must end in something like ``.gif'' or ``.jpg''. Also, the forward-slash cannot be part of the name, restricting all images to the files contained directly within $REAL. This is a good thing from a security standpoint.

Lines 44 through 46 capture the matched filename, and construct a full path for it, also verifying that the file exists.

If either of the two die() operators in that eval() block are executed, Perl doesn't die, but instead exits the block immediately, making the text available in the $@ variable. Line 48 checks this variable -- if it was empty, everything worked OK, but if not, we print the right HTTP headers to trigger an error condition, including dumping the text of $@ in the body of the message.

Line 54 constructs the preview file name, based on the original filename with ``.gif'' appended, and a directory path of thNexte preview directory.

Next, it's time to construct the preview if necessary. Line 56 tests first for the existence of the preview file, and ensures that its modification time is more recent than the modification time of the original file. This lets us install new versions of the full-sized image without having to adjust the preview directory.

If it's time to create the preview, line 57 creates a temp file name in the same directory as the preview file. This is important because we are going to rename the file into place after it has completed, and the rename() operator must be able to work. If you want to build the new preview into a directory like /tmp, you'll need to make sure that $PREVIEW and /tmp are on the same filesystem.

Lines 59 through 61 are cribnotes to me about the commands that need to be constructed to turn either a GIF or a JPEG into the right preview GIF. Lines 63 and 64 construct the appropriate command in $command, based on those formulas. Line 65 executes the command, capturing the noise generated on standard output and standard error into $messages. I don't refer to this variable later, but while I was debugging, I dropped this variable into a log file so that I could make sure that the right PBM tools were being invoked.

Line 66 checks for the existence of a non-empty $preview_temp after all that. If something didn't work, usually $preview_temp ended up empty. A more robust program would have checked the exit status of each and every command invoked in the pipeline, but that would have only muddied this example, so I left that out.

If the file exists, line 67 renames the file into place. It's quite possible that multiple simultaneous processes have all generated their own version of this preview. Hopefully, those will all be identical, so it won't matter who actually succeeds in renaming the file in that case. No locking is thus necessary, unless you are being really conservative on CPU cycles. If the rename fails, we don't care, because the unlink in line 69 will be blowing that temporary file away anyway.

By the time we make it to line 72, we have a valid $real filename, a potentially valid $preview file, and now it's time to spit out the result for the client program. Line 72 grabs the last-modified time of the image to include it into the response headers. (I thought this would prevent Netscape from asking for the image repeatedly, but apparently not. However, it will be good for proxy caches anyway.) Line 73 prints that header.

Line 74 prints the header that makes multiple contents possible -- the multipart/x-mixed-replace MIME type. Both of the two major browser families (from Netscape and Microsoft) support this type in recent releases -- as does my favorite alternate browser, w3-mode inside GNU Emacs. The important part is getting ``--'' followed by the boundary marker (defined with $BOUNDARY) between the various pieces.

For more information on this MIME type, just search for ``Server Push'' in your favorite web search engine. I quickly found a dozen or so tutorials and sample pieces of code. (Ironically, when this column is placed online on my website at http://www.stonehenge.com/merlyn/WebTechniques/, those searches will also locate this column!)

Speaking of pieces, line 75 defines the beginning of the first piece. If the $preview file exists, that'll be the first piece, and we send its contents, followed by another boundary, using the send_this() subroutine defined later.

Whether or not we sent the preview file, lines 80 and 81 send the real full-strength image, followed by the terminating boundary (indicated by a trailing --).

The only thing left to define is the sending routine, in lines 83 through 91. The parameter is a filename, stored into $name in line 84.

Line 86 determines if the image is a GIF or a JPEG by the filename. In a more robust program, we'd peek into the content, but this'll be close enough for most situations. (And if the file was inappropriately named, the web server would probably have sent the wrong content-type anyway!)

Lines 88 and 89 send the right HTTP headers. The Content-length header is very important, because the browsers need it to keep from reading the second image as part of the first.

Finally, line 90 copies the file defined in $name to STDOUT, using the copy() routine defined in File::Copy.

To use this program, put your images into the directory pointed to by $REAL, and create a $PREVIEW directory with a mode that is writeable by the user that runs CGI scripts (usually nobody). A safe bet here would be to make the directory mode 0777. Then, presuming you've installed the script as /cgi/two-pass, change your IMG tags appropriately to:

        <IMG SRC="/cgi/two-pass/somefile.jpg" ALT="some file">

where somefile.jpg`` is the name of the real image you copied to $REAL. That's it. Again, using this program as-is would not necessarily be wise, but hopefully the Server Push technique and the caching technique will be useful in programs that you write. Enjoy!

Listing One

        =1=     #!/home/merlyn/bin/perl -Tw
        =2=     use strict;
        =3=     
        =4=     $|++;
        =5=     
        =6=     use File::Copy;
        =7=     use HTTP::Date qw(time2str);
        =8=     
        =9=     ### Configuration
        =10=    
        =11=    my $IMAGE_TOP = "/home/merlyn/Web/Images";
        =12=    my $REAL = "$IMAGE_TOP/Real";
        =13=    my $PREVIEW = "$IMAGE_TOP/Preview";
        =14=    
        =15=    $ENV{PATH} = join ":",
        =16=      qw(
        =17=         /usr/local/netpbm/bin
        =18=         /usr/local/bin
        =19=         /usr/ucb
        =20=         /usr/bin
        =21=         /bin
        =22=         );
        =23=    
        =24=    ### end Configuration
        =25=    
        =26=    ### constants
        =27=    
        =28=    my $BOUNDARY = "ThisRandomString";
        =29=    
        =30=    ### end constants
        =31=    
        =32=    ### CGI input
        =33=    
        =34=    my $PATH_INFO = $ENV{PATH_INFO} || "";
        =35=    
        =36=    ### end CGI input
        =37=    
        =38=    my $file;
        =39=    my $real;
        =40=    
        =41=    eval {
        =42=      $PATH_INFO =~ /^\/([a-zA-Z0-9._-]+\.(jpe?g|gif))$/i
        =43=        or die "missing filename in PATH_INFO: $PATH_INFO\n";
        =44=      $file = $1;
        =45=      $real = "$REAL/$file";
        =46=      -r $real or die "$file not found\n";
        =47=    };
        =48=    if ($@) {
        =49=      print "Status: 404 Not Found\n";
        =50=      print "Content-type: text/plain\n\n";
        =51=      print "error: $@\n";
        =52=      exit 0;
        =53=    }
        =54=    my $preview = "$PREVIEW/$file.gif";
        =55=    
        =56=    unless (-e $preview and -M $preview < -M $real) {
        =57=      my $preview_temp = "$preview.$$";
        =58=    
        =59=      ## JPEG: djpeg * | ...
        =60=      ## GIF: giftopnm * | ...
        =61=      ## ... | ppmtopgm | pgmtopbm | ppmtogif >*.gif
        =62=    
        =63=      my $command = ($real =~ /\.gif$/i) ? "giftopnm" : "djpeg";
        =64=      $command .= " $real | ppmtopgm | pgmtopbm | ppmtogif >$preview_temp";
        =65=      my $messages = `($command) 2>&1`;
        =66=      if (-s $preview_temp) {       # presuming non-empty file means all is OK
        =67=        rename $preview_temp, $preview;
        =68=      }
        =69=      unlink $preview_temp;
        =70=    }
        =71=    
        =72=    my $last_modified = time2str((stat $real)[9]);
        =73=    print "Last-modified: $last_modified\n";
        =74=    print "Content-type:multipart/x-mixed-replace;boundary=$BOUNDARY\n\n";
        =75=    print "--$BOUNDARY\n";
        =76=    if (-e $preview) {
        =77=      send_this($preview);
        =78=      print "\n--$BOUNDARY\n";
        =79=    }
        =80=    send_this($real);
        =81=    print "\n--$BOUNDARY--\n";
        =82=    
        =83=    sub send_this {
        =84=      my $name = shift;
        =85=    
        =86=      my $type = ($name =~ /\.gif$/i) ? "gif" : "jpeg";
        =87=    
        =88=      print "Content-type: image/$type\n";
        =89=      print "Content-length: ", (-s $name), "\n\n";
        =90=      copy $name, \*STDOUT;
        =91=    }

Randal L. Schwartz is a renowned expert on the Perl programming language (the lifeblood of the Internet), having contributed to a dozen top-selling books on the subject, and over 200 magazine articles. Schwartz runs a Perl training and consulting company (Stonehenge Consulting Services, Inc of Portland, Oregon), and is a highly sought-after speaker for his masterful stage combination of technical skill, comedic timing, and crowd rapport. And he's a pretty good Karaoke singer, winning contests regularly.

Schwartz can be reached for comment at merlyn@stonehenge.com or +1 503 777-0095, and welcomes questions on Perl and other related topics.