Copyright Notice

This text is copyright by CMP Media, LLC, and is used with their permission. Further distribution or use is not permitted.

This text has appeared in an edited form in WebTechniques magazine. However, the version you are reading here is as the author originally submitted the article for publication, not after their editors applied their creativity.

Please read all the information in the table of contents before using this article.
Download this listing!

Web Techniques Column 63 (Jul 2001)

[suggested title: Calculating download time]

It's a simple problem, really. You click on a link to a site. It starts loading. But you're on a slow modem dialup. So it keeps loading, and loading, and loading, the little browser icon in the upper right continuing its animation as if that will distract you enough to not notice that the page still hasn't finished. But still it keeps loading. Until finally, just when you're about to hit ``stop'', it's done.

Why are these pages so big? Why are most pages so unfriendly to slow links? I suspect it's because most pages are being designed on intranets these days, and nobody ever bothers to go home to test it out from their ISP connection. And that's unfortunate.

As a simple test, we could at least write a little program to download the entire page with all of its links, and see how many bytes would be needed to satisfy the browser. Of course, we'd have to go through all of the HTML, looking for embedded images, sounds, objects (like flash) and frame links. Hmm, sounds like a lot of work. Unless you have the right tools, like Perl's wonderful LWP library. So I wrote such a program, and I'm presenting it in [listing one, below].

Line 1 turns on warnings: a good thing while developing. Line 2 enables the common compiler restrictions, forcing us to declare our variables, avoid the use of barewords, and stay away from symbolic references. Line 3 enables autoflushing on STDOUT, so each line of output will be immediately visible as it is printed.

Lines 5 through 8 pull in 4 modules from the LWP library. If you don't have LWP installed, you can use the CPAN module to do this without a lot of hassle as follows:

  $ perl -MCPAN -eshell
  cpan> install Bundle::LWP

You may have to answer some questions if this is your first time installing with

The LWP::UserAgent module provides the web client object. HTTP::Cookies handles cookie-based interactions. HTTP::Request::Common creates common web requests. And HTML::LinkExtor parses HTML to find external URL links.

If you didn't know that these were all part of the LWP CPAN bundle, you could have just asked for them explicitly, as in:

  $ perl -MCPAN -eshell
  cpan> install LWP::UserAgent
  cpan> install HTTP::Cookies
  cpan> install HTTP::Request::Common
  cpan> install HTML::LinkExtor

Lines 10 through 28 define the %LINKS global hash, selecting the types of tag attributes that we believe will be loaded by the browser directly. I got this list easily; I grabbed the HTML::Tagset module and copied the value of linkElements out of there into here. Then I decided which of the items was a browser load. Please don't consider this list authoritative; it's just a best guess (and I'd appreciate feedback if I erred).

Note that one of the items is the href attribute of the link tag, which often contains the CSS file, loaded by the browser. However, it also often contains other URLs that are not, and this is distinguishable only by looking at other attributes in the tag, which are unfortunately not provided by HTML::LinkExtor. Maybe a future version of HTML::LinkExtor will address this need.

Also, I'm not doing anything to look within the JavaScript of a page to get dynamic rollovers. That's a hard problem. I'd probably need a full JavaScript interpreter to do that. Ugh.

Lines 30 through 33 create the ``virtual browser'' as the global $ua variable, by first creating a generic object (line 30), giving it the right proxy information from the environment (line 31), setting a distinct user agent (line 32), and then setting up an in-memory ``cookie jar'' (line 33). The cookie jar allows us to visit pages that depend on having visited the referring URL, since any cookies will be properly handled by virtual browser.

Line 35 calls the main routine report for each URL mentioned on the command line. Line 37 ends the program when we're done (not strictly necessary, since it's just subroutines from here down, but I like to keep my programs maintainable).

Now for the good stuff. The report routine beginning in line 39 is given a single URL, extracted in line 40. This routine pulls down the page, and examines it for all URLs that would also have been loaded in a typical frame- and image- grokking browser. To do this, we'll maintain and populate two top-level data structures: the @todo array in line 42, and the %done hash in line 43.

The @todo array is a list of items that must still be processed, in the form of two-element references to arrays. The first element is the source URL (used to properly set the referer header), and the second element is the URL to fetch. We'll initially load the URL of interest into @todo, with an empty referer string. The %done hash serves double-duty, both as a way of tracking which URLs we've already done (the keys), but also the number of bytes for that URL (the value) for later analysis.

Line 45 begins the ``while there's still more to-do'' loop. Line 46 pulls off the first arrayref from the list, and explodes it into the referer string and the URL to be fetched. Line 47 skips over the ones we've already done.

Lines 49 and 50 fetch the page, by creating a GET request with the proper referer header, and fetching it with the virtual browser. The result is always an HTTP::Response object.

If the fetch is successful, line 52 detects that. We'll take the content into $content in line 53, and put its length into the value for %done as well, keyed by the URL we just fetched.

If the content was HTML, this means the browser would then have displayed the HTML, and will then crawl through looking for images, sounds, subframes, and other things, so we must do likewise. Line 55 detects the presence of such HTML by noting the MIME type of the content.

Line 57 pulls out the ``base'' URL from the HTTP::Response object. Normally, this is the same as the original URL. However, if the HTML header contains a ``base'' URL, then a browser would have calculated all relative URLs from that base URL, not the fetching URL, so we must do likewise. Luckily, the base method just does the right thing and gives us what we need, regardless of whether it was specified or not.

Line 58 sets up the HTML::LinkExtor object, giving it no callback function, but specifying the base URL for relative links. Without a callback function, the data is merely gathered into an internal object variable, which we'll access with a links method later. Lines 59 and 60 hand the HTML content to the object, triggering a complete parse pass.

Lines 61 through 71 pull out the link information, item by item. Each link (in $link) is an arrayref. The arrayref is dereferenced in line 62 to reveal an initial tagname (ending up in $tag) followed by key/value pairs of attributes, which are rolled up into the %attr hash.

Line 63 detects the tagname being an item of interest. The HTML::LinkExtor module finds all possible outbound URLs, but we're interested only in the ones that would be loaded by the browser right away. If the tag is in %LINKS, then we check each of the attributes in the list to see if it's something we saw in the HTML content (line 65) and it's a non-empty value (line 66).

If we have a URL of interest, we push that onto the ``to-do'' list, as a two-element item, with the $base URL for a referer. I was puzzled if this should be $base or $url, and settled on $base for no solid reason. You could probably write me to convince me I'm wrong on this and I wouldn't take it personally.

Well, that handles the typical HTML page. But we also have some other kinds of responses from a fetch. The other typical response is a ``redirect'', where the remote server indicated that a different URL should be fetched. That's handled in lines 73 to 76. First, we count the length of the content (because the browser would still be fetching all of the response content), then fetch the location header (the new URL), and then queue up a fetch of this new URL. Again, I wasn't sure what referer string should be given to this fetch, so I settled on $url. Again, I could be easily argued out of it (and I bet it's inconsistent amongst browser implementations).

And if it's not a good fetch or a redirect, it's probably an error, detected in line 77. Line 78 merely dumps this information to the invoker, on the odds that this is something that won't make much sense to try to correct from, and simple information is all that's required.

When all the to-do list has been completed, we drop out of the big loop, down to the reporting phase, beginning in line 83. Line 84 creates a local temporary variable, initializing it to 0. This will be the total number of bytes fetched by the browser for this URL. Line 86 shows the URL name for labelling purposes.

Lines 87 to 90 go through all the entries in the %done hash. Elements of the hash are sorted by descending numeric order on the values, so that we get a quick idea of the piggiest part of the page. As each URL and bytesize is pulled up, the total is updated in line 88, and formatted nicely in line 89 using printf.

Lines 91 and 92 do what we came here for. The total downloaded bytes are shown in line 91, and line 92 computes the download time at a conservative 2000 bytes per second on a 28.8 dialup line. (I'm painfully aware of how slow this is as I spend much of my time bouncing around from one hotel to another, very much missing my cable modem at home.)

Let's look at a couple of typical reports, for both and for my own archive of columns (and listings) at

    $ dltime =>
          121133 TOTAL
              61 seconds at 28.8 =>
           29735 TOTAL
              15 seconds at 28.8

Ahh... look at that. That's 61 seconds of download time for the WebTechniques site, but only 15 seconds for my site. How nice. The big bottleneck on the WebTechniques homepage seems to be the cover JPG. Maybe they can reduce the JPEG Q factor a bit to save some time. Or maybe this is an acceptable tradeoff. But at least now we know how long it'll take me to visit the WebTechniques homepage from a hotel room. Time enough to hum the ``Final Jeopardy'' theme twice while I'm waiting. So until next time, enjoy!


        =1=     #!/usr/bin/perl -w
        =2=     use strict;
        =3=     $|=1;
        =5=     use LWP::UserAgent;
        =6=     use HTTP::Cookies;
        =7=     use HTTP::Request::Common;
        =8=     use HTML::LinkExtor;
        =10=    my %LINKS =                     # subset of %HTML::Tagset::linkElements
        =11=    (
        =12=     'applet'  => ['archive', 'codebase', 'code'],
        =13=     'bgsound' => ['src'],
        =14=     'body'    => ['background'],
        =15=     'embed'   => ['src'],
        =16=     'frame'   => ['src'],
        =17=     'iframe'  => ['src'],
        =18=     'ilayer'  => ['background'],
        =19=     'img'     => ['src', 'lowsrc'],
        =20=     'input'   => ['src'],
        =21=     'layer'   => ['background', 'src'],
        =22=     ## 'link'    => ['href'], ## durn, some of these are stylesheets
        =23=     'script'  => ['src'],
        =24=     'table'   => ['background'],
        =25=     'td'      => ['background'],
        =26=     'th'      => ['background'],
        =27=     'tr'      => ['background'],
        =28=    );
        =30=    my $ua = LWP::UserAgent->new;
        =31=    $ua->env_proxy;
        =32=    $ua->agent("dltime/1.00 ".$ua->agent); # identify ourselves
        =33=    $ua->cookie_jar(HTTP::Cookies->new); # capture cookies if needed
        =35=    report($_) for @ARGV;
        =37=    exit 0;
        =39=    sub report {
        =40=      my $start = shift;
        =42=      my @todo = ["", $start];
        =43=      my %done;
        =45=      while (@todo) {
        =46=        my ($refer, $url) = @{shift @todo};
        =47=        next if exists $done{$url};
        =49=        my $request = GET $url, [referer => $refer];
        =50=        my $response = $ua->simple_request($request);
        =52=        if ($response->is_success) {
        =53=          $done{$url} = length (my $content = $response->content);
        =55=          next if $response->content_type ne "text/html";
        =57=          my $base = $response->base; # relative URLs measured relative to here
        =58=          my $p = HTML::LinkExtor->new(undef, $base) or die;
        =59=          $p->parse($content);
        =60=          $p->eof;
        =61=          for my $link ($p->links) {
        =62=            my ($tag, %attr) = @$link;
        =63=            if ($LINKS{$tag}) {
        =64=              for (@{$LINKS{$tag}}) {
        =65=                next unless exists $attr{$_};
        =66=                next unless length (my $a = $attr{$_});
        =67=                ## print "$base $tag $_ => $a\n"; ## debug
        =68=                push @todo, [$base, $a];
        =69=              }
        =70=            }
        =71=          }
        =73=        } elsif ($response->is_redirect) {
        =74=          $done{$url} = length $response->content; # this counts
        =75=          my $location = $response->header('location') or next;
        =76=          push @todo, [$url, $location]; # but get this too
        =77=        } elsif ($response->is_error) {
        =78=          print "$url ERROR: ", $response->status_line, "\n";
        =79=        }
        =81=      }                             # end of outer loop
        =83=      {
        =84=        my $total = 0;
        =86=        print "$start =>\n";
        =87=        for my $url (sort { $done{$b} <=> $done{$a} } keys %done) {
        =88=          $total += $done{$url};
        =89=          printf "  %10d  %s\n", $done{$url}, $url;
        =90=        }
        =91=        printf "  %10d TOTAL\n", $total;
        =92=        printf "  %10.0f seconds at 28.8\n\n", $total/2000;
        =93=      }
        =95=    }

Randal L. Schwartz is a renowned expert on the Perl programming language (the lifeblood of the Internet), having contributed to a dozen top-selling books on the subject, and over 200 magazine articles. Schwartz runs a Perl training and consulting company (Stonehenge Consulting Services, Inc of Portland, Oregon), and is a highly sought-after speaker for his masterful stage combination of technical skill, comedic timing, and crowd rapport. And he's a pretty good Karaoke singer, winning contests regularly.

Schwartz can be reached for comment at or +1 503 777-0095, and welcomes questions on Perl and other related topics.