Copyright Notice

This text is copyright by CMP Media, LLC, and is used with their permission. Further distribution or use is not permitted.

This text has appeared in an edited form in SysAdmin/PerformanceComputing/UnixReview magazine. However, the version you are reading here is as the author originally submitted the article for publication, not after their editors applied their creativity.

Please read all the information in the table of contents before using this article.
Download this listing!

Unix Review Column 50 (Jan 2004)

[suggested title: Sharing Open Source code through the CPAN]

The Comprehensive Perl Archive Network (CPAN) is a wonderful place, full of contributed items for you to use, such as scripts and modules. Modules are the core of the CPAN: little building blocks for you to include into your applications.

Modules end up in the CPAN by being wrapped in a distribution. A distribution is typically a compressed tar archive, identified with a particular version number, and containing one or more modules separated out into packages. A distribution also contains installation instructions and often contains tests to validate proper build and installation.

Let's look at a sample distribution that I entered into the CPAN this past summer to see the essential parts of any distribution. As a joke, I created the Acme::Current module which gives the current year, month, and day... as constants! The module is out-dated and updated every 24 hours, using an automated script that I'll describe shortly.

The module itself is in Current.pm, shown in [listing one, below]. Line 1 defines the module name by switching to its package. Line 2 enables strict, a good thing for larger program development.

Lines 4 and 6 declare the package variables $VERSION, $YEAR, $MONTH, and $DAY. While I could have done this with our rather than use vars, that would have restricted my module to installations with Perl 5.6 or later. When contributing to the CPAN, it's important to think about portability.

Line 8 provides the version number of the module. For my joke module, it also happens to be the entire active code of the module as well. The values for the three constants are established using hardwired values, and then they are wrestled into a single integer by sprintf. The name $VERSION is special to Perl's require operator, and also to the indexing software used to track changes to CPAN modules. Because the indexing software is simplistic, I had to keep this line within the guidelines established in the ExtUtils::MakeMaker manpage.

The resulting version number for these constants is 20030720. Any version number greater than this supercedes this release, so it's important that the version number monotonically increase with time.

Line 10 provides the mandatory ``true value'' required by the require operator, on which the use operator is built. Line 12 defines the end of the executable portion of the file, as we are now beginning the POD for the module. While this line is essentially optional, it does save a bit of time in the parser, because the parser doesn't need to scan beyond the __END__ marker to find non-POD code after the POD.

The POD starts in line 14. Again, the format of POD for a CPAN distribution is a bit restricted because the automated tools are extracting key information. The NAME heading is very specific, for example. The synopsis here shows a typical use of this module. I left out the remainder of the POD for space... you can see it in the CPAN if you wish.

To create a distribution from this module, we need to create a distribution directory, and put this file beneath that directory. I called mine Acme-Current, and placed this module into lib/Acme/Current.pm below that directory.

At the top of the directory, I include the next essential component: the Makefile.PL, shown in [listing two below]. This file is a Perl program that creates a Makefile that can be used to build, test, install, and create distribution archives for this particular distribution. Line 1 pulls in the ExtUtils::MakeMaker module, which defines the meaning of the rest of the file.

Lines 2 through 8 create a Makefile when executed. The parameters declare the name of my distribution (Acme::Current), which file I'm getting the distribution version number from (the only module here), and also that to properly install this module, the user has to have any version of Test::More also installed. It's important to list every non-core module in this list if I've used them in my module, because the automated installation tools (like CPAN.pm) can then install any necessary dependencies before continuing on to my module.

But what is Test::More? I've used it in my one test file for this distribution, which I've placed in t/01-core.t below Acme-Current, the contents of which are shown in [listing three, below]. Once again, it's a Perl program, using the Test::More testing harness. To be recognized as a test, the file should match the fileglob of t/*.t. If there's more than one test, they're run in alphabetical order.

Line 1 pulls in the Test::More module, declaring that I'm not prepared to define how many tests I'm running. If this had been a real module and not a joke module, I should change this to:

  use Test::More tests => 4;

because at the moment, there are four tests. Of course, I'd have to change this every time I added or deleted a test, so the no_plan option was good while I was monkeying around.

The first test (in line 2) is a use_ok test, which verifies the syntax of the Acme::Current module. At this point, we should now have the three constants defined in the Acme::Current package namespace.

To test the values, I compute the current time vector using gmtime. Then, using a series of is tests, I compare the value as given by the module with the value as it should be. If the numbers are the same, the test passes. Otherwise, the test fails, and the discrepancy is noted. A failed test normally prevents the module from being installed as well, unless the user is fairly insistent (and a bit crazy).

These tests succeed as long as ``today'' (from a GMT perspective) is the same as the values that I set when making up the version number. If someone tries to install a stale version, the tests will fail. Obviously, for this to work, a new version must be uploaded every day, and the process to do that will be described shortly.

Besides the module itself, the Makefile.PL, and one or more test files, I also need a MANIFEST file to denote the contents of the distribution archive. My MANIFEST looks like [listing four, below]. It's just a list of files. Note that a comment is permitted following some delimiting whitespace. One additional file is listed there: the README file, which should be a simple information file. The README file is extracted separately and archived in the CPAN so that users can download and examine just the README without having to unpack the entire distribution. My README is given in [listing five, below].

So, now I've got the essentials of a distribution. To make a distribution archive, I issue the following commands:

        perl Makefile.PL
        make all test tardist

The first command creates a Makefile, which the second command uses to build the module (copying it into a staging area), running the tests on the distribution (everything matching t/*.t), and then if that all works, creating a file whose name looks like Acme-Current-20030720.tar.gz. Whew!

Well, once we have the distribution archive, it's time to get that into the CPAN. I had already applied and received my CPAN PAUSE ID, by visiting http://pause.cpan.org. Using that same website, I can also insert items into my CPAN author directory, by file upload, URL fetching, or FTP transfer.

I decided that since this joke would need to be updated nightly, I would build a Perl program to update the constants in the .pm file and then submit the file using a URL fetch. By experimenting with WWW::Mechanize, I found the right combination of steps, and put that into a cron-trigger job that I call Maintain in the same Acme-Current directory, presented in [listing six, below].

Lines 1 through 3 start nearly every long Perl program that I write, turning on the compiler restrictions. Lines 5 and 6 force the current directory to be the same directory in which the script is located.

Lines 8 and 9 pull in the WWW::Mechanize (found in the CPAN) and File::Copy modules. Lines 11 through 14 define the necessary paths and authentications for the CPAN upload. I use a private directory on my webserver to provide the source for the upload. My CPAN PAUSE ID is merlyn, but my CPAN PAUSE password is not shown, because it is provided on STDIN from cron for security.

Lines 16 through 26 edit the .pm file to create the proper version number and date constants. I use an in-place edit here, looking for the $VERSION assignment line by recognizing the specific pattern. Once that's complete, I can now go about the task of testing and creating a new distribution.

Line 28 performs the steps shown earlier, throwing away any output text. Because this is a joke module, I fail silently, erring on the side of just not uploading a new module. Had this been a more important module, I'd be parsing through the output text to verify that it is as expected. At this point, we've either rebuilt the same distribution archive as before, or we now have a new archive.

Line 30 cleans up some cruft that had been left over when the same version number was used twice (which is true 23 times a day). Line 31 does the actual business of attempting an upload for any new distributions now made. Each name that fits the fileglob pattern is submitted to the CPAN, using the subroutine beginning in line 33.

Line 34 saves the local file name parameter. Line 36 computes this same name in the webserver directory, and if it already exists, then we've already uploaded this file and return immediately (in line 37).

Line 39 copies this new file to the webserver directory. Line 40 creates a WWW::Mechanize object to let us talk to the PAUSE site and schedule a new upload.

Line 42 establishes my PAUSE ID and password using HTTP ``basicauth''. Line 44 gets the initial form that I see to log in. Line 45 follows the login link, which will look at the ``basicauth'' credentials, and present me with a menu of actions to take regarding my CPAN author directory.

Line 46 follows the link to upload a file. The resulting form has some very specific form elements, into which I stuff with the URL that maps to my new distribution (line 47). Line 49 submits that form, and line 50 reports that a new distribution was submitted. Because this is run from cron, I get one email every day with this single text line in it.

The next step is to set up this script to run every hour. Why every hour? Because I don't want to figure out when GMT rolls over, so I let the machine just do the work every hour, and upload only when there's a new distribution. Simple, but effective. The cron entry looks like this:

  ## Acme::Current joke module
  11 * * * * /home/merlyn/Perl/Acme-Current/Maintain%MYPASSWORD

I do this at 11 minutes past the hour every hour of every day. Why 11? Because I have different things firing from my crontab, and I try to stagger them so that they don't all start at the same time, and ``11 minutes past the hour'' was the next slot available.

Thus, 24 times a day, the pm file gets edited, and the Makefile.PL is transformed into a Makefile, and the distribution gets tested and an archive gets created. But only once a day is this version number different from the previous number, and then the PAUSE machinery is contacted to trigger an upload.

OK, so it's a lot of work for a joke that I eventually disabled anyway, but hey, that's what Acme is for. I hope I've shown how simple it is to create a distribution though, and hopefully inspired you to get your open source code pieces into the CPAN. Until next time, enjoy!

Listings

        =0=     #### LISTING ONE ####
        =1=     package Acme::Current;
        =2=     use strict;
        =3=     
        =4=     use vars qw($VERSION);
        =5=     
        =6=     use vars qw($YEAR $MONTH $DAY);
        =7=     
        =8=     $VERSION = sprintf "%04d%02d%02d", $YEAR = 2003, $MONTH = 7, $DAY = 20;
        =9=     
        =10=    1;
        =11=    
        =12=    __END__
        =13=    
        =14=    =head1 NAME
        =15=    
        =16=    Acme::Current - Determine current year, month, day (GMT)
        =17=    
        =18=    =head1 SYNOPSIS
        =19=    
        =20=      use Acme::Current;
        =21=      printf "It's now %04d/%02d/%02d.\n",
        =22=        $Acme::Current::YEAR,
        =23=        $Acme::Current::MONTH,
        =24=        $Acme::Current::DAY;
        =25=      if ($Acme::Current::MONTH == 12 and $Acme::Current::DAY == 25) {
        =26=        print "Merry Christmas!\n";
        =27=      }
        =28=    ..... REST OMITTED .....
        =0=     #### LISTING TWO ####
        =1=     use ExtUtils::MakeMaker;
        =2=     WriteMakefile(
        =3=       NAME => 'Acme::Current',
        =4=       VERSION_FROM => 'lib/Acme/Current.pm',
        =5=       PREREQ_PM => {
        =6=                     'Test::More' => 0,
        =7=                    },
        =8=     );
        =0=     #### LISTING THREE ####
        =1=     use Test::More qw(no_plan);
        =2=     BEGIN { use_ok('Acme::Current'); }
        =3=     my @now = gmtime;
        =4=     is($Acme::Current::YEAR, $now[5]+1900, 'year');
        =5=     is($Acme::Current::MONTH, $now[4]+1, 'month');
        =6=     is($Acme::Current::DAY, $now[3], 'day');
        =0=     #### LISTING FOUR ####
        =1=     README
        =2=     Makefile.PL
        =3=     MANIFEST                        This list of files
        =4=     lib/Acme/Current.pm
        =5=     t/01-core.t
        =0=     #### LISTING FIVE ####
        =1=     Acme::Current gives you all the power of those myriad of date/time
        =2=     modules without all that complexity, as long as all you want is the
        =3=     current date (GMT-based), and you keep the module up to date.
        =0=     #### LISTING SIX ####
        =1=     #!/usr/bin/perl
        =2=     use strict;
        =3=     $|++;
        =4=     
        =5=     use FindBin qw($Bin);
        =6=     BEGIN { chdir $Bin or die "Cannot chdir to $Bin: $!" }
        =7=     
        =8=     use WWW::Mechanize;
        =9=     use File::Copy;
        =10=    
        =11=    my $CPAN_DIR = "/web/private/CPAN";
        =12=    my $CPAN_URI = "http://www.stonehenge.comm/private/CPAN";;
        =13=    my $CPAN_USER = "merlyn";
        =14=    chomp(my $CPAN_PASSWORD = <STDIN>);
        =15=    
        =16=    my @now = gmtime;
        =17=    my ($Y, $M, $D) = ($now[5]+1900, $now[4]+1, $now[3]);
        =18=    
        =19=    {
        =20=      local @ARGV = "lib/Acme/Current.pm";
        =21=      local $^I = "~";
        =22=      while (<>) {
        =23=        s/\$YEAR = \d+, \$MONTH = \d+, \$DAY = \d+;/\$YEAR = $Y, \$MONTH = $M, \$DAY = $D;/;
        =24=        print;
        =25=      }
        =26=    }
        =27=    
        =28=    system "perl Makefile.PL >/dev/null && make all test tardist </dev/null >/dev/null 2>&1";
        =29=    
        =30=    unlink glob "*.tar";            # in case it blocked from a .gz
        =31=    submit_to_cpan($_) for glob "Acme-Current-*.tar.gz";
        =32=    
        =33=    sub submit_to_cpan {
        =34=      my $local_file = shift;
        =35=    
        =36=      my $cpan_dir_file = "$CPAN_DIR/$local_file";
        =37=      return if -f $cpan_dir_file;  # already submitted;
        =38=    
        =39=      copy $local_file, $cpan_dir_file;
        =40=      my $agent = WWW::Mechanize->new();
        =41=    
        =42=      $agent->credentials('pause.perl.org:443', 'PAUSE',
        =43=                          $CPAN_USER, $CPAN_PASSWORD);
        =44=      $agent->get('https://pause.perl.org');
        =45=      $agent->follow(qr/Login/);
        =46=      $agent->follow(qr/Upload a file/);
        =47=      $agent->current_form->value('pause99_add_uri_uri',
        =48=                                  "$CPAN_URI/$local_file");
        =49=      $agent->click('SUBMIT_pause99_add_uri_uri');
        =50=      print "$local_file submitted\n";
        =51=      ### print $agent->current_form->dump;
        =52=    }

Randal L. Schwartz is a renowned expert on the Perl programming language (the lifeblood of the Internet), having contributed to a dozen top-selling books on the subject, and over 200 magazine articles. Schwartz runs a Perl training and consulting company (Stonehenge Consulting Services, Inc of Portland, Oregon), and is a highly sought-after speaker for his masterful stage combination of technical skill, comedic timing, and crowd rapport. And he's a pretty good Karaoke singer, winning contests regularly.

Schwartz can be reached for comment at merlyn@stonehenge.com or +1 503 777-0095, and welcomes questions on Perl and other related topics.