in reply to Re^2: A mini-language for sequences (part 1)
in thread A mini-language for sequences (part 1)

The following are some (lightly edited :) initial reactions to your follow post.

Note: Initial.

There are some positive reactions lower down, but I thought that I would get these out of the way first in the hope that the later reactions would erase any bitter taste these might leave.

  1. I never said "this crazy FP stuff".
  2. I get the FP -v- FP-inspired difference.
  3. Phrases like "spectrum of problems", "shape of the problems", "shape-flexing capability", "code at a higher level" & "think at a higher level" all leave me...um...suspicious.

    What is he trying to conceal with flowery language?

    Put that down to having suffered the hyperbole of too many "new programming paradyms". (I realise FP isn't new.)

  4. "The reason you perceive the fp-inspired code as hard to follow is because your brain hasn't "internalized" the meaning of products and zips and the rest of our new vocabulary. To you, these words probably seem like gratuitous functional jargon."

    My question is: "Do I need this new vocabulary?".

  5. "But the words have meaning. When I saw the example problem as first posted by the Seeker of Perl Wisdom, I immediately thought, "Ah, he wants the count of each value from the zipped products." And the code immediately followed."

    Fair enough, but then you had to construct a fairly complicated library of intrastructure in order to be able to code the solution in terms of your vocabulary.

    When I saw the example problem, I thought: "He want's to iterate over the two arrays of strings and count the unique pairs of aligned characters".

    1. "...iterate over the two arrays..."

      Nested for loops.

    2. "...count the unique pairs...".

      Increment a hash.

    3. "...pairs of aligned characters."

      Two substrs and another for loop.

    The code I posted fell straight out of that. No need for any auxillary vocabulary. No need to code a bunch of extra infrastructure. Perl already has all the vocabulary and in-built infrastructure required.

  6. "That's the reward. To have the right words to describe the problem. To have more ways of stringing together ideas. To reduce the gulfs between problem and understanding, understanding and code." & "If this sounds like worthless babble, that's OK."

    {nods} Yes. It does a bit :)

  7. I didn't appreciate functional programming until I had used it to solve real-world sized problems for over a year. Only then could I "feel" fp in my bones and appreciate its potential.

    I've been trying to get into FP since I first starting reading articles on Lisp in Byte magazine circa. 1979/80. I currently have Common Lisp, Scheme, and most recently Haskell.

  8. Have you read Why Functional Programming Matters? If not, read it. Now."

    I pulled the PDF and tried to read it. After a little while it began to seem familiar. Maybe it appeared on Byte, or CoACM or something else I've read a long time ago? Maybe it was referenced from such an article and I tracked it down and tried to read it? Maybe it's just the code examples that seem familiar from some other FP tutorial I've read down the years? Whatever, the PDF sits open on my desktop and I will endevour to try and read it to the end, but I doubt that I made it that far last time either.

    The problem is that in common with many other FP (and, notoriously OO) tutorials, it selects very specifically limited and condusive (to their aims) examples. It then spends an inordinate amount of time concentrating on how to construct the solution--in it's own, very esoteric, terms. Your real world examples are infinitely better (IMO).

    But the single line that sums up my feelings of what I've seen of FP tutorials and advocacy comes from the Abstract on the page you linked to above.

    Since modularity is the key to successful programming, functional languages are vitally important to the real world.

    The problem with that statement is that it implies that the only way to achieve modularity, is through FP--which is clearly not the case.

  9. I love Perl Monks, but don't look at the fp examples on Perl Monks as your window into the fp world. Take a look at the stuff going on in the Haskell Communities Report. Read some of the papers from the ICFP proceedings. That's where you'll find the good stuff.

    As I mentioned, my FP exploration goes back a long way, and my research is definitly not confined to what I see on PM. My exploration of Haskell is very new (and did come about because of articles mentioning it here at PM), so thankyou for those links--they will give me many hours of good reading.

I hope that doesn't all seem negative? It isn't meant to be. I'm continuing to enjoy reading your article and pursue various angles that lead from it. Thankyou for writing it and I look forward to Part 2.

For me, the most significant thing that came out of your response to my initial post is:

Instead of passing in \(@site1,@site2) you can pass in any number of input arrays. And the same code works, as is.

Now that is a benefit that I can understand and internalise :).

I did try to download your code and try this out, but there seem to be some bits missing to allow the examples to work? Perhaps in part 2. This is what I extracted from your OP: {**Code moved to bottom of post**} but I couldn't work out how to fix it up so that I could try out the example with 3 or more lists?

Anyway, the concept of writing a generic solution to the nested loops problem is one that I've been trying to get to grips with for some time. The upshot of your article (for me) is that I've finally gotten around to putting some concerted effort into it, and I came up with a solution. It doesn't look much like FP in it's construction, but it does rely heavily of constructing anonymous subroutine iterators, and constructing lists of these. And combining these iterator functions to contruct a higher order iterator. So, having (probably) butchered the FP vocabulary to death :), this is what my solution looks like: 

#! perl -slw
use strict;

sub loops {
    my @iters = map{
        my @list = ( @$_, undef );
        sub { $list[ @list ] = shift @list || () };
    } @_;
    my @rv = map{ $iters[ $_ ]() } 0 .. $#iters;
    return sub {
        my $rv = [ @rv ];
        for my $i ( reverse 0 .. $#iters ) {
            $rv[ $i ] = $iters[ $i ]() and return $rv;
            $rv[ $i ] = $iters[ $i ]();
        }
        return;
    };
}

my $iter = loops [ 'a' .. 'd' ], [ 1 .. 4 ], [ 'me', 'you' ];

print "@$_" while $_ = $iter->();
__END__
[13:51:21.14] P:\test>loops
a 1 me
a 1 you
a 2 me
a 2 you
a 3 me
a 3 you
b 1 me
b 1 you
b 2 me
b 2 you
b 3 me
b 3 you
c 1 me
c 1 you
c 2 me
c 2 you
c 3 me

[download]

I'm pretty pleased with that. Your article has been invaluable to me because it has prompted me to look at this problem in a completely different way to that in which I have been looking at it in the past. And it has triggered a lot of thoughts about using similar techniques to generalise several other pieces of code I have kicking around.

So thankyou again for posting your very thought provoking article. I really look forward to part 2.

I think the conclusion I'm drawing is that it's possible to learn and adopt techniques from other languages and styles of coding, without having to adopt the vocabularies and working practices wholesale. Once you understand (if I have?), the underlying techniques, it becomes possible to utilise them whilst retaining the flavour of the language in which your writing.

The code I extracted from your OP. Moved to the bottom to avoid interupting the main post. (Wouldn;t it be nice if readmore tags only expanded when asked? Even if viewing the containing post directly). 

#! perl -w
use strict;
use List::Util qw( reduce );
use Data::Dumper;

#package Sequence;

#sub new {
#    my ($proto, $seq) = @_;
#    bless $seq, $proto;
#}

#sub seqsub(&) {
#    Sequences->new(@_);
#}
    
sub seq {
    my ($i, $elems) = (0, \@_);
    seqsub {
        $i < @$elems
            ? ( $elems->[ $i++ ] )
            : do { $i = 0; () };
    }
}

sub enumerate {
    local $Data::Dumper::Terse = 1;
    local $Data::Dumper::Indent = 0;
    my ($i, $seq) = (0, $_[0]);
    while (my @val = $seq->()) {
        @val = map { ref ($_) ? Dumper($_) : $_ } @val;
        printf "%2d => %s\n", $i++, "@val";
    }
    $seq;
}
    
sub seq_prod2 {
    my ($s, $t) = @_;
    my @sval;
    seqsub {
        @sval = $s->() unless @sval;
        my @tval = $t->();
        @tval ? ( @sval, @tval )
              : do { @sval = $s->();
                     @sval ? ( @sval, $t->() ) : () };
    }
};
    
sub seq_prod {
    reduce { seq_prod2($a,$b) } @_ ;
}


sub seqs {
    map seq(@$_), @_;
}

sub seq_from_spec {
    seq_prod( seqs(@_) );
}

sub seq_foreach {
    my ($seq, $fn) = @_;
    while (my @val = $seq->()) {
        $fn->(@val);
    }
    $seq;
}

sub seq_foreach_from_spec {
    my ($spec, $fn) = @_;
    seq_foreach( seq_from_spec( @$spec ), $fn );
}

sub seq_filter {
    my ($seq, $filter_fn) = @_;
    seqsub {
        my @val;
        1 while @val = $seq->() and !$filter_fn->(@val);
        return @val;
    }
}
    
sub seq_map {
    my ($seq, $fn) = @_;
    seqsub {
        my @val = $seq->();
        @val ? $fn->(@val) : ();
    }
}
    
sub seq_reset {
    my $seq = shift;
    if ($seq) {
        1 while $seq->();
    }
    $seq;
}

sub seq_zip {
    my $seqs = seq( @_ ); # seq of seqs (!)
    my $seq_count = @_;
    seqsub {
        my @outvals;
        while (my $seq = $seqs->()) {
            if (my @val = $seq->()) {
                push @outvals, @val;
            }
            else {
                seq_reset( $seqs->() ) for 1 .. $seq_count;
                seq_reset( $seqs );
                return ();
            }
        }
        return @outvals;
    }
}

my @site1 = qw( AATKKM aatkkm );
my @site2 = qw( GGGGGG gggggg );

my %counts;

seq_foreach_from_spec( [ \(@site1, @site2) ],
    sub {
        seq_foreach(
            seq_zip( ( map seq(split//), @_ ) ),
            sub { $counts{"@_"}++ }
        )
    }
);

print Dumper(\%counts), "\n";

[download]
Examine what is said, not who speaks.
"Efficiency is intelligent laziness." -David Dunham
"Think for yourself!" - Abigail
"Memory, processor, disk in that order on the hardware side. Algorithm, algorithm, algorithm on the code side." - tachyon

Replies are listed 'Best First'.
Re^4: A mini-language for sequences (part 1)
by tmoertel (Chaplain) on Nov 07, 2004 at 23:50 UTC
    First, thank you for your most-recent response. There are many interesting ideas within it, and I want to get to them. But before we do that, let's put our discussion within proper context.

    I must make clear that we have wandered onto dangerous ground. We are using my meditation as the testing ground for your conclusions about the merits functional programming, and it was never indented to be anything other than a meditation.

    To be blunt: If you're looking at my mediation as anything other than a meditation, you're making a mistake. My meditation is not meant to be practical. It is not meant to be advocacy for functional programming. It is a meditation: a discourse intended to express my focusing on one idea – sequences – and going with it as deep as I can.

    The point of the meditation is not to say that "this way is better." The point is to ignore earthly concerns that normally hold us back (such as practicality) and to keep moving forward with an idea, just to see where it takes us. Maybe we'll end up at a really cool place. Maybe not. Regardless, we hope to learn something from the journey, if not from the destination.

    At certain points along our journey you have asked, "Why did we walk this strange path to get to this spot? I know a road that most people would think is better."

    But we're not trying to find the best path just now. We're trying to walk a different path to see if we can learn anything. Instead of comparing our path to the road and dismissing it as inferior after the first few miles, can we not keep on walking, just to walk, and to see if we encounter anything new or valuable?

    In the end, maybe we will still prefer the road, but then again, maybe knowing the path will come in handy. Maybe some destinations will best be reached by traveling both road and path. Maybe the path will be a valuable addition to our options for future travel.

    To come at it practically, don't judge my examples. Judge the ideas the examples represent. But only later, after due consideration.

    Now, on to specific items.

    My question is: "Do I need this new vocabulary?"
    Whether you need any new vocabulary depends on the vocabulary and on the problems you are trying to solve. If the cost of learning the vocabulary and then using the vocabulary to solve your problems is less than the cost of solving your problems otherwise, then you probably ought to use the vocabulary.

    Do you need my meditation's sequence vocabulary? Most likely, no. It exists only for this meditation. If I were to create a CPAN module from it, as some have suggested, the resulting library would look and be organized much differently than presented here. The goals of meditation and of practical programming are far apart.

    I would, however, argue that much of the vocabulary of ideas and techniques within the meditation are useful and could earn its place in most personal programming toolboxes.

    But the single line that sums up my feelings of what I've seen of FP tutorials and advocacy comes from the Abstract on the page you linked to above.
    Since modularity is the key to successful programming, functional languages are vitally important to the real world.
    The problem with that statement is that it implies that the only way to achieve modularity, is through FP--which is clearly not the case.
    I don't think that Hughes implies that FP is the only way to achieve modularity. Rather, he argues that FP (especially modern FP) reduces the cost of modularity to the point where it can be used, and its benefits reaped, almost ubiquitously – not just at object or function boundaries but at much finer granularities.

    In most languages, the smallest unit of modularity is the object or function. But in Haskell, for example, I can go much finer. I can slice function calls in half, turn operators into functions and functions into operators, flip arguments around, and do a whole lot more to reshape code until it fits my needs. The potential for code reuse goes up dramatically because I can modularize on a much finer granularity than before; the artificial distinctions that other languages impose between language and library cease to exist. Libraries aren't merely collections of functions but real vocabularies that extend the language.

    I've been trying to get into FP since I first starting reading articles on Lisp in Byte magazine circa. 1979/80. I currently have Common Lisp, Scheme, and most recently Haskell.
    To help me understand and quantify where you're coming from, how many hours would you conservatively estimate that you have spent writing functional code in the last year?
    The problem is that in common with many other FP (and, notoriously OO) tutorials, ("Why Functional Programming Matters") selects very specifically limited and condusive (to their aims) examples. It then spends an inordinate amount of time concentrating on how to construct the solution--in it's own, very esoteric, terms. Your real world examples are infinitely better (IMO).
    The problem for people who write about FP to non-FP audiences is that the introductory slant of their writing prevents the use of complex example problems, where FP's merits are readily visible. Anybody can solve simple problems in any language, and so readers understandably wonder why FP is so great when their "normal" solutions to the same example problems would arguably be better.

    For this reason, some FP authors will pick simple example problems that just happen to be hard to solve in non-FP languages. While the intent is to keep the problems simple enough that readers can follow them and yet "hard" enough to show the merits of FP, many readers conclude that the examples are contrived or even rigged to put FP in an artificially glowing light.

    Sadly, this is a case when the limitations of the medium undo the message.

    FP languages let you manipulate pieces of programming logic as easily as most languages let you manipulate data. As a result, it is easy to create FP programs that morph their shape to match the structure of the problems they solve. Thus the straightforward FP solution is often general enough to scale from the simplest to the most complex problems within a wide spectrum of related problems.

    This is deeply cool magic, but you won't appreciate it until you experience it. And you won't experience it in introduction-to-FP papers. You can't put examples like that into a paper. The best way, then, to read the papers is to study the examples carefully, understand the thinking behind them, and then meditate on how far that thinking will take you. Often, it will take you surprisingly far.

    I hope that doesn't all seem negative? It isn't meant to be. I'm continuing to enjoy reading your article and pursue various angles that lead from it.
    No, I don't take it as negative. But I do see a slight danger of your drawing conclusions about FP without due consideration. Be deliberate about forcing your mind to remain wide open while you invest serious time in studying and coding functional programs.

    The best advice I can give you is to force yourself to write functional code, the more the better. If you're already looking at Haskell, that's your best bet. (There is no finer modern FP language, IMHO.) Get a copy of Paul Hudak's The Haskell School of Expression: Learning Functional Programming through Multimedia and work through it. Solve the problems. Don't skip the proofs.

    For me, the most significant thing that came out of your response to my initial post is:
    Instead of passing in \(@site1,@site2) you can pass in any number of input arrays. And the same code works, as is.
    Now that is a benefit that I can understand and internalise :). I did try to download your code and try this out, but there seem to be some bits missing to allow the examples to work?
    I think the problem is that you commented out the OO portion of my code but forgot to recode seqsub accordingly. Add the following to your code (or uncomment the OO portion that you commented out), and you should be good to go: 
        sub seqsub(&) { $_[0] }

    [download]
    Here's our example solution put into a subroutine so that we can call it repeatedly with differently "shaped" inputs: 
        sub count_zipped_combo_vals {
        my %counts;
        seq_foreach_from_spec( \@_, sub {
            seq_foreach(
                seq_zip( ( map seq(split//), @_ ) ),
                sub { $counts{"@_"}++ }
            )
        });
        \%counts;
    }

    [download]
    Now we can see how will it handles problems of different shapes. We'll try 1-, 2-, and 3-array cases, but the same code should work for any number of arrays greater than zero. 
        # a few problem cases to try

    my @site1 = qw( AATKKM aatkkm        );
    my @site2 = qw( GGGGGG gggggg        );     
    my @site3 = qw( XXXXXX ++++++ yyyyyy );
    # ... and so on ...

    my @sites = \( 
        @site1, @site2, @site3, # ... and so on ...
    ); 

    # try 1-, 2-, and 3- array cases
    print Dumper( count_zipped_combo_vals( @sites[0..$_] ) )
        for 0 .. $#sites;


    # $VAR1 = {
    #           'A' => 2,
    #           'k' => 2,
    #           'a' => 2,
    #           'M' => 1,
    #           'T' => 1,
    #           'K' => 2,
    #           'm' => 1,
    #           't' => 1
    #         };
    # $VAR1 = {
    #           'K G' => 2,
    #           'A G' => 2,
    #           'm g' => 1,
    #           'a g' => 2,
    #           'A g' => 2,
    #           'M G' => 1,
    #           'k g' => 2,
    #           'k G' => 2,
    #           'T G' => 1,
    #           'a G' => 2,
    #           'm G' => 1,
    #           't G' => 1,
    #           'K g' => 2,
    #           'M g' => 1,
    #           't g' => 1,
    #           'T g' => 1
    #         };
    # $VAR1 = {
    #           't G y' => 1,
    #           'T G +' => 1,
    #           'K G X' => 2,
    #           'a G X' => 2,
    #           'm G +' => 1,
    #           'a G +' => 2,
    #           'T g y' => 1,
    #           'm g +' => 1,
    #           'a g +' => 2,
    #           'k G X' => 2,
    #           'a g X' => 2,
    #           'm g X' => 1,
    #           'T g X' => 1,
    #           't G +' => 1,
    #           'M G y' => 1,
    #           'k g y' => 2,
    #           't g y' => 1,
    #           'm G X' => 1,
    #           't G X' => 1,
    #           'K G +' => 2,
    #           't g +' => 1,
    #           'a g y' => 2,
    #           'T G X' => 1,
    #           'm g y' => 1,
    #           'm G y' => 1,
    #           'A G X' => 2,
    #           'M g +' => 1,
    #           'k g +' => 2,
    #           'k g X' => 2,
    #           'A g X' => 2,
    #           'A g +' => 2,
    #           'M g y' => 1,
    #           'k G +' => 2,
    #           'a G y' => 2,
    #           'M G +' => 1,
    #           'A g y' => 2,
    #           'K g X' => 2,
    #           'K G y' => 2,
    #           'K g y' => 2,
    #           'K g +' => 2,
    #           'k G y' => 2,
    #           'A G y' => 2,
    #           'T g +' => 1,
    #           't g X' => 1,
    #           'M G X' => 1,
    #           'M g X' => 1,
    #           'T G y' => 1,
    #           'A G +' => 2
    #         };

    [download]
    Thanks again for taking the time to consider our discussion. I hope that in the end your will find the journey to be the reward.

    Cheers,
    Tom

    Tom Moertel : Blog / Talks / CPAN / LectroTest / PXSL / Coffee / Movie Rating Decoder

[reply]
[d/l]
[select]
      I hope you can appreciate my skepticism when I respond to:
      FP languages let you manipulate pieces of programming logic as easily as most languages let you manipulate data. As a result, it is easy to create FP programs that morph their shape to match the structure of the problems they solve. Thus the straightforward FP solution is often general enough to scale from the simplest to the most complex problems within a wide spectrum of related problems.
      with the simple phrase "you have drank from the FP Koolaide".

      Fortran solved everything. Then Pascal solved everything. Then Smalltalk solved everything. Then Prolog solved everything. Now you're saying FP solves everything.

      Sure.

      Having been around the block a few times, lemme say that I can certainly see FP being useful as yet another approach to a problem. But your unrelentless praise for the latest new thing should be taken in context of the history of discovering just another interesting programming technique.

      FP will be good for some problems, horrible for others. Just like every other style discovered before it. Understand that, and you'll understand why everyone's not jumping on your bandwagon.

      -- Randal L. Schwartz, Perl hacker
      Be sure to read my standard disclaimer if this is a reply.

[reply]
        (Updated Mon Nov 8 14:36 EST 2004: Clarified wording.)

        Randal,

        Thanks for jumping into the conversation. Your viewpoint is always welcome.

        I am concerned, however, that you have quoted my most flowery praise for FP and used it to dismiss much of my views as mere Koolaide visions without giving the whole of my views due consideration.

        To be clear, I never claimed that FP was the solution for "everything" or that it was the one, true way. If you genuinely believe that this was my claim, I am sorry for not having presented my beliefs more clearly.

        Let me take the opportunity to do that now:

        1. FP is not a panacea.
        2. FP is another tool for the toolbox (but a very powerful one).
        3. FP, like all tools, reduces the cost of some things; increases the cost of others.
        4. Modern FP languages (like Haskell) are not your grandfather's FP language. If you haven't looked at FP in the last 5 years, look again.
        5. Modern FP languages have reduced the cost of function manipulation to the point where small-scale reuse becomes practical.
        6. Small-scale reuse provides significant cost reductions.
        7. Much like Perl, modern FP languages require a serious investment of time before they become understood (and appreciated).
        8. Most people who dismiss FP haven't spent much time writing code in modern FP languages.

        Regarding the following:

        Having been around the block a few times, lemme say that I can certainly see FP being useful as yet another approach to a problem. But your unrelentless praise for the latest new thing should be taken in context of the history of discovering just another interesting programming technique.
        Having also been around the block a few times, let me say that my appreciation of FP comes not from having been smitten by the latest new thing but rather from the investment of much time and effort. Since the late 1980s I have been coding in and evaluating FP languages alongside the more common languages such as C, C++, and Perl that I use to earn a living in industry. However, it is only recently that I have watched FP flower and come into its own.

        While most of industry has turned its back on FP, having long ago made up its mind, the academic world has been working. Steady, slow, and ceaseless, they have been working. And they have made much progress. Please don't dismiss that progress without due consideration. If you haven't spent serious time coding in a modern FP language lately, please consider the possibility that you're drawing conclusions from outdated first-hand knowledge.

        FP will be good for some problems, horrible for others. Just like every other style discovered before it. Understand that, and you'll understand why everyone's not jumping on your bandwagon.

        To help me put this comment in perspective, how much coding have you done in modern FP languages lately?

        The reason I ask is because if a man came up to me and summed up Perl by saying that Perl was good for some problems, horrible for others, just like every other language discovered before it, and that's why not everybody is jumping on the Perl bandwagon, I would want to know how much he had used Perl. It's not that I disagree with his statement, which is clearly true for just about everything, but that I don't find the statement as useful as knowing why some people are on the bandwagon. I would want to know whether the man could tell me. I would want to know whether the man understood Perl's strengths and weaknesses and could provide useful information that would help me make sensible choices about when to use Perl.

        Cheers,
        Tom

        Tom Moertel : Blog / Talks / CPAN / LectroTest / PXSL / Coffee / Movie Rating Decoder

[reply]
      To help me understand and quantify where you're coming from, how many hours would you conservatively estimate that you have spent writing functional code in the last year?

      Methinks you're asking, where do you sit in The Evolution of a Haskell Programmer.

      To wit I reply, currently indeterminate, but I'm aiming for "Tenured professor" :)

      Examine what is said, not who speaks.
      "Efficiency is intelligent laziness." -David Dunham
      "Think for yourself!" - Abigail
      "Memory, processor, disk in that order on the hardware side. Algorithm, algorithm, algorithm on the code side." - tachyon
[reply]

In Section Meditations