Copyright Notice

This text is copyright by InfoStrada Communications, Inc., and is used with their permission. Further distribution or use is not permitted.

This text has appeared in an edited form in Linux Magazine magazine. However, the version you are reading here is as the author originally submitted the article for publication, not after their editors applied their creativity.

Please read all the information in the table of contents before using this article.

Linux Magazine Column 09 (Feb 2000)

[suggested title: Subroutine references]

In last month's column, I looked at using ``references'' in Perl, and showed the basic syntax for creating references to arrays, hashes, and even scalars. I also described the canonical form of converting a non-reference expression into a reference, and also how to use the shortcut rules to make this simpler.

A reference can also be taken to a subroutine (often called a ``coderef''), like so:

  sub hello { print "hello, $_[0]!" }
  my $hello_ref = \&hello;

Here, the referencing operator \ is used in front of the subroutine name. The name must begin with an ampersand &, even though we frequently don't need the ampersand on invocation.

Accessing this subroutine involves ``dereferencing'', much as accessing a variable required dereferencing as well. In fact, the syntax rule is the same as before. Take a normal use of the named item, remove the name, and replace it with a block returning a reference:

  &hello("Randal") # normal form
  &{$hello_ref}("Randal") # using the reference

Either of these invoke &hello, passing it the single parameter of "Randal".

Of course, the rules for simplification apply here as well. Because the only thing inside the curly braces is a simple scalar variable, we can drop the braces:

  &$hello_ref("Randal")

And there's even an arrow form (which I managed to get introduced into Perl 5.004 on a bet):

  $hello_ref->("Randal")

Now, what's the point of a subroutine reference? Indirection, like everything else! I can pass a subroutine reference into a subroutine to create ``pluggable'' behavior. For example, suppose I wanted to write a filter that removed all words that started with ``z'', moving them to the end. If the subroutine is a filter, it will need to know how to get the next item, and then how to hand back the result. Let's pass those as subroutine refs:

  sub get_input { shift @my_array }
  sub send_output { print $_[0] }
  filter(\&get_input, \&send_output);
  sub filter {
    my $input = shift;
    my $output = shift;
    my @elided = ();
    while (defined(my $item = $input->())) {
      if ($item =~ /^z/) {
        push @elided, $item;
      } else {
        $output->($item);
      }
    }
    $output->($_) for @elided; # 5.005 syntax
  }

Now, there's a bunch of stuff going on here, so let's look at it closely. First, I defined &get_input and &send_output, my input and output routines. Then I passed references to those routines as the first two (and only) parameters to my &filter subroutine. The expectation is that the filter routine will call my input routine each time it needs the next item, passing no parameters, and expecting a return string, or undef if nothing further remains to be processed. Similarly, the output routine will be called with a single parameter for each item to be passed along.

The guts of &filter are mostly normal, although I've got a few indirect subroutine invocations mixed in. Note that each input line goes into a local $item variable, and then tested to see if it begins with z.

We could just as easily call this same filter routine to go from STDIN into a variable:

  sub get_from_stdin { <STDIN> }
  sub put_into_array { push @array, shift }
  filter(\&get_from_stdin, \&put_into_array);

And there you have it. Same subroutine, but now magically reading STDIN and writing to an array. As long as the subroutine being passed has the right ``signature'' (combination of inputs and outputs), we can plug anything we want into there.

The subroutines created for this kind of pluggable behavior are likely to be used in only one place. Can we eliminate the creation of a program-wide subroutine name? Sure, by using anonymous subroutines:

  my $get = sub { <STDIN> };

This is similar to:

  sub Name_that_I_Did_Not_Have_To_Create { <STDIN> }
  my $get = \&Name_that_I_Did_Not_Have_To_Create;

Except I didn't have to create that name. So, we can simplify invocations of &filter:

  my $getter = sub { <STDIN> };
  my $putter = sub { print $_[0] };
  filter($getter, $putter);

But did we even need the variables? No! So, let's go one step further:

  filter(sub { <STDIN> }, sub { print $_[0] });

If you can follow this, you can see that we now have a powerful tool, being able to pass behaviors into a subroutine instead of just data.

I don't like that we didn't label what was what, so let's also introduce named parameters:

  filter(
    Get => sub { <STDIN> },
    Put => sub { print $_[0] },
  );
  sub filter {
    my %args = @_;
    my $input = $args{Get};
    my $output = $args{Put};
    my @elided = ();
    while (defined(my $item = $input->())) {
      if ($item =~ /^z/) {
        push @elided, $item;
      } else {
        $output->($item);
      }
    }
    $output->($_) for @elided;
  }

which works because we're turning the input parameters into a hash, then extracting the appropriate getter or putter.

But what if we didn't specify a getter or putter? Unlike other references, coderefs do not ``autovivify''. That'd be a nice way to get a ``do-what-I-mean'' operator, but that probably won't ever get introduced into Perl 5. Instead, we can provide a default:

  sub filter {
    my %args = @_;
    my $input = $args{Get} || sub { <STDIN> };
    my $output = $args{Put} || sub { print $_[0] };
    # rest identical...
  }
  filter(Put => sub { push @output, shift });

Now we have a default STDIN-to-STDOUT filter, or we can override the input and output steps to be whatever we want. We can even add error checking:

  sub filter {
    my %args = @_;
    my $input = delete $args{Get} || sub { <STDIN> };
    my $output = delete $args{Put} || sub { print $_[0] };
    die "too many args to filter" if %args;
    # rest identical...
  }

Yes, I know that delete is no longer guaranteed to deliver the value as it once was in very old days, but this is likely only to be a mess if we use a tied hash. Not so here.

Now, one thing that's been bugging me since the first instance of &filter is that pesky /^z/ hardwired into the subroutine. Can we make that more general? Sure! Let's make it a pluggable behavior as well:

  sub filter {
    my %args = @_;
    my $input = delete $args{Get} || sub { <STDIN> };
    my $output = delete $args{Put} || sub { print $_[0] };
    my $defer = delete $args{Defer} || sub { shift =~ /^z/ };
    die "too many args to filter" if %args;
    while (defined(my $item = $input->())) {
      if ($defer->($item)) {
        push @elided, $item;
      } else {
        $output->($item);
      }
    }
    $output->($_) for @elided;
  }
  filter( Get => sub { shift @array },
          Defer => sub { shift > 10 });

The new interface is that the Defer coderef must return a true/false value for each item passed to it. Items that return true will be moved to the end of the list. The default value is the one from before: items beginning with z are pushed back. Mostly nonsense, but just to show you that you can have weird defaults. In that last invocation, I'm pushing back all items that are numerically larger than 10.

Now, let's look at variable scope. Suppose I call filter passing a coderef, and the subroutine referenced by that coderef accesses a lexical variable outside the scope of the subroutine, like so:

  sub pushback {
    my @input = @_;
    my @output;
    filter(
      Get => sub { shift @input },
      Put => sub { push @output, $_ });
    return @output;
  }

Can the getter routine access @input in this manner? Yes. Perl binds the subroutine to its defining environment via the ``closure'' mechanism, which is a fancy way to say that all lexical variables visible to the definition are also available at the invocation. Note the word ``lexical'' here; that'll be important later.

This &pushback routine is nice, but it still uses that silly ``begins with z'' selection. Let's make that more general:

  sub pushback2 {
    my $coderef = shift;
    my @input = @_;
    my @output;
    filter(
      Get => sub { shift @input },
      Put => sub { push @output, shift },
      Defer => sub { local $_ = shift; $coderef->() });
    return @output;
  }

Here, I'm setting up a local $_ so that the coderef called from subroutine can access $_ similar to the grep and map operators. That means we can do things like this:

  @out = pushback2 sub { /^z/ }, @in;
  @numbers = pushback2 sub { $_ > 10 }, @source;

And with a little bit of hackery (in the form of a prototype declaration), we can get even closer:

  sub pushback3 (&;@) {
    my $coderef = shift;
    my @input = @_;
    my @output;
    filter(
      Get => sub { shift @input },
      Put => sub { push @output, shift },
      Defer => sub { local $_ = shift; $coderef->() });
    return @output;
  }
  @in = qw(alpha zeta orange zero mostel);
  @out = pushback3 { /^z/ } @in;

A prototyped entry of & in the first position accepts a block in lieu of a coderef expression, and the word sub is then optional.

Enough on syntax: let's look a bit more at closures. A closure can access the enviroment of all of its external lexical variables (and here's the fun part) regardless of whether or not that enclosing scope is still valid or has been exited. More smoke and mirrors:

  {
    my $a;
    $getter = sub { $a };
    $setter = sub { $a = shift };
  }
  $setter->(3); # set $a to 3
  print $getter->(); # prints 3

Even though $a has gone out of scope, there's still a private reference to $a associated with both $getter and $setter. What's more, if this code is entered again, a new $a is created, with a new set of closure references created as well. That's best illustrated with a subroutine that spits out coderefs, like so:

  sub make_getter_setter {
    my $a;
    return sub { $a },
           sub { $a = shift };
  }
  ($get_a1, $set_a1) = make_getter_setter();
  ($get_a2, $set_a2) = make_getter_setter();

The $a that $get_a1 and $set_a1 share is different from the $a that $get_a2 and $set_a2 share. And those individual $a variables will stay alive as long as the coderefs also stay alive. Think of the possibilities!

Another use of closures is with named subroutines, just to come back to where we started:

  BEGIN {
    my @pushback;
    sub get_next { @pushback ? shift @pushback : <STDIN> }
    sub unget { unshift @pushback, shift }
  }

Here, we have a couple of named subroutines, but both accessing a common lexical variable @pushback. Since this block executes at compile time, we'd expect the value of @pushback to come and go before any code got executed. However, since the subroutines have global scope (not scoped to the block), they live throughout the duration of the program, and will hold a closure reference to the effectively private-but-shared anonymous array created from @pushback. That gives us ``local'' variables with ``global'' lifetimes.

I hope you've enjoyed this little excursion into subroutine references, and find them part of your bag of Perl tricks. For further information, check the documentation that comes with Perl, especially perlref, as well as chapter 4 of my book Programming Perl, Second Edition from O'Reilly and Associates (co-authored by Larry Wall and Tom Christiansen). Until next time, enjoy!


Randal L. Schwartz is a renowned expert on the Perl programming language (the lifeblood of the Internet), having contributed to a dozen top-selling books on the subject, and over 200 magazine articles. Schwartz runs a Perl training and consulting company (Stonehenge Consulting Services, Inc of Portland, Oregon), and is a highly sought-after speaker for his masterful stage combination of technical skill, comedic timing, and crowd rapport. And he's a pretty good Karaoke singer, winning contests regularly.

Schwartz can be reached for comment at merlyn@stonehenge.com or +1 503 777-0095, and welcomes questions on Perl and other related topics.