Copyright Notice

This text is copyright by CMP Media, LLC, and is used with their permission. Further distribution or use is not permitted.

This text has appeared in an edited form in SysAdmin/PerformanceComputing/UnixReview magazine. However, the version you are reading here is as the author originally submitted the article for publication, not after their editors applied their creativity.

Please read all the information in the table of contents before using this article.

Unix Review Column 4 (September 1995)

Perl has a substantial collection of data recognizing and capturing operators and features. In this column, I'm going to build a parser for a small somewhat free-form database, like a mailing list.

First, let's take a look at some sample data:

        Name: Randal L. Schwartz
        Company: Stonehenge Consulting Services
        Street: 4470 SW Hall Suite 107
        City: Beaverton
        State: Oregon
        Zip: 97005
        Phone: 503-777-0095
        
        Name: John Big-booty
        City: San Angeles
        State: California
        Zip: 93021
        Phone: 291-555-2213
        
        Company: Lips, Inc.
        Street: 4221 Wayback Lane
        City: Springfield
        State: Kansas
        Zip: 65554

Each entry is delimited from the next by a blank line. Note that not all fields are present in each entry, so we'll have to consider that when we're reading it in to a database. (Please note that these entries are for illustrative purposes only -- any similarity to actual persons, living or dead, is merely a coincidence.)

I'm going to put the data into a list of associative arrays. (This is not possible in earlier versions of Perl, so if these examples don't work, make sure you have Perl version 5.000 or later.) Each associative array represents one entry from the database. The keys of the associative array are the names of the fields (like ``Name'' or ``City'') with the values being the corresponding data.

First, we'll need to break the data up into entries. The simplest way is to take advantage of ``paragraph mode'' while reading the file. In paragraph mode, each ``line'' read from the file is actually any number of lines up to the next blank line or end of file. (Isn't it convenient that there's a Perl mode to read this kind of data? Some would probably accuse me of selecting this structure to match the Perl mode, but I'm not telling. :-) Paragraph mode is selected by setting $/ to an empty string:

        #!/usr/bin/perl
        $/ = "";
        while (<>) {
                ## one entry per $_ here
        }

Wow, we're almost done (not!). The text in $_ will now look like a number of lines that are separated by \n. We need to parse this data, which is most easily done using multi-line match mode. Multi-line match mode (indicated by a trailing m on the regular expression) allows the caret (^) to match not only the beginning of the string, but also just after any embedded newline in the string. Now, to grab the Name, Street, and City fields, it'd look like this:

        #!/usr/bin/perl
        $/ = "";
        while (<>) {
                %entry = (); # initialize empty entry
                if (/^Name: (.*)/m) {
                        $entry{"Name"} = $1;
                }
                if (/^Street: (.*)/m) {
                        $entry{"Street"} = $1;
                }
                if (/^City: (.*)/m) {
                        $entry{"City"} = $1;
                }
                ## save %entry here
        }

As you can see, the code for parsing the entry is looking rather repetitive, and I haven't even done all of the fieldnames yet. This was my first approach, so let me throw it away and try something more general.

It's easier to think of the data as ``field: value'', and grab both at the same time. Let's try that direction:

        #!/usr/bin/perl
        $/ = "";
        while (<>) {
                # for each entry:
                %entry = (); # initialize
                foreach (split /\n/) {
                        # for each line in entry:
                        $entry{$1} = $2
                                if /^(.*): (.*)$/;
                }
                ## save %entry here
        }

Ahh! This is getting close. However, there's a bug here. Can you tell what it is before reading ahead? Nope? Well, suppose the value contains a colon, as in:

        Company: White Elephants: A division of Trunks-R-Us

The first .* will match all the way up to the second colon, because regular expression quantifiers are greedy by default. To fix that, just put a ? after that first colon, which says to match as few characters as possible (be ``stingy'') rather than as many as possible (``greedy''). (To save space, I won't repeat the program part -- just look for the change in later revisions.)

Note that the order of data, or omission of some of the fields, is irrelevant. I could put the city before the name, or whatever.

Now I have a valid %entry for each entry. All I have to do is save it. I can't save the actual associative array into a list, but I can save a pointer to the associative array using a reference. At first thought, you might think to do this:

        #!/usr/bin/perl
        $/ = "";
        @entries = (); # master list
        while (<>) {
                # for each entry:
                %entry = (); # initialize
                foreach (split /\n/) {
                        # for each line in entry:
                        $entry{$1} = $2
                                if /^(.*): (.*)$/;
                }
                # save %entry to @entries:
                push @entries, \%entry;
        }

And this will indeed make a list of references to associative arrays. However, it's a list of references to the same associative array, %entry! What we need instead is to make a new anonymous associative array for each of the entries. There's a couple of ways we can do that. One is to use the data from %entry inside the initialization of a brand new associative array:

        #!/usr/bin/perl
        $/ = "";
        @entries = (); # master list
        while (<>) {
                # for each entry:
                %entry = (); # initialize
                foreach (split /\n/) {
                        # for each line in entry:
                        $entry{$1} = $2
                                if /^(.*): (.*)$/;
                }
                # save %entry to @entries:
                push @entries, {%entry};
        }

The other, probably spiffier way is to create a new anonymous associative array to begin with, and get rid of %entry altogether:

        #!/usr/bin/perl
        $/ = "";
        @entries = (); # master list
        while (<>) {
                # for each entry:
                $ref_entry = {}; # anon hash
                foreach (split /\n/) {
                        # for each line in entry:
                        $$ref_entry{$1} = $2
                                if /^(.*): (.*)$/;
                }
                # save this entry to @entries:
                push @entries, $ref_entry;
        }

I like this one better, even though the syntax of de-referencing the reference to the anonymous associative array gets slightly uglier in the middle of the loop. The syntax $$ref_entry{$1} says to use $ref_entry as the ``name'' of an associative array, and look at the key for $1 in that associative array.

Whichever you prefer, we now have what we started out to get -- a list of references to (anonymous) associative arrays representing the original data entries.

For my first trick, I'll print the data in a mailing list form. To do this, I'll have to walk through the data, looking for things with complete addresses. Let's give it a whirl:

        #!/usr/bin/perl
        [parsing code from above goes here]
        foreach $ref (@entries) {
                $name = $$ref{"Name"};
                $company = $$ref{"Company"};
                $street = $$ref{"Street"};
                $city = $$ref{"City"};
                $state = $$ref{"State"};
                $zip = $$ref{"Zip"};
                next unless defined $address;
                print "$name\n$street\n";
                print "$city, $state $zip\n";
        }

Here, I'm looking for entries that have an address, because there are entries in my database for which I have only a phone number. Those funky $$ref{"Something"} things are once again using $ref as the ``name'' of an associative array. Recall that each element of the @entries list is a reference to an associative array.

The entries here will come out in the order that I've defined them. What if I want them in zip code order? No problem (well, no major problem) -- just sort the entries list before using it:

        #!/usr/bin/perl
        [parsing code from above]
        @entries = sort {
                $$a{"Zip"} <=> $$b{"Zip"}
        } @entries;
        [printing code from above]

Here, I'm using a user-defined (that'd be me) sort to rearrange the contents of the @entries list. A user-defined sort routine is handed two elements of the list to be sorted (here, @entries) as $a and $b. Since each of the elements are references to an associative array, I can dereference them to get the ``Zip'' field from each one.

If I wanted something more complicated, like state first, and then city within state, it's just a small matter of typing a little more:

        #!/usr/bin/perl
        [parsing code from above]
        @entries = sort {
                $$a{"State"} cmp $$b{"State"} or
                $$a{"City"} cmp $$b{"City"}
        } @entries;
        [printing code from above]

Here, the left part of the or operator will be evaluated first. If the cmp returns -1 or +1, then the right part of the or operator can be skipped. This happens when the states differ. If the states are the same, then the right part of the or operator must be evaluated (because the left part is 0, which is false), yielding the comparison between cities (a difficult task to do in real life).

For my last trick, I'll give you the entire program that prints only the phone numbers for persons or companies for which we have phone numbers, sorted by phone number:

        #!/usr/bin/perl
        $/ = "";
        @entries = (); # master list
        while (<>) {
                # for each entry:
                $ref_entry = {}; # anon hash
                foreach (split /\n/) {
                        # for each line in entry:
                        $$ref_entry{$1} = $2
                                if /^(.*): (.*)$/;
                }
                # save this entry to @entries:
                push @entries, $ref_entry;
        }
        @entries = sort {
                $$a{"Phone"} cmp $$b{"Phone"}
        } @entries;
        foreach $ref (@entries) {
                $phone = $$ref{"Phone"};
                next unless defined $phone;
                $name = $$ref{"Name"};
                $company = $$ref{"Company"};
                print "$name ($company) $phone\n";
        }

This is just built up from pieces from the previous snippets, hacked just slightly to meet the new goal.

Perl's richness of language features, while possibly initially intimidating, can yield tremendous flexibility in the long run if you take the time to explore them. Hopefully, you've seen a few new cool tricks in this column. Keep reading for future cool tricks and basic techniques.


Randal L. Schwartz is a renowned expert on the Perl programming language (the lifeblood of the Internet), having contributed to a dozen top-selling books on the subject, and over 200 magazine articles. Schwartz runs a Perl training and consulting company (Stonehenge Consulting Services, Inc of Portland, Oregon), and is a highly sought-after speaker for his masterful stage combination of technical skill, comedic timing, and crowd rapport. And he's a pretty good Karaoke singer, winning contests regularly.

Schwartz can be reached for comment at merlyn@stonehenge.com or +1 503 777-0095, and welcomes questions on Perl and other related topics.