I've added Hadoop::Streaming to cpan. This module aims to simplify the Streaming interface to Hadoop, by automating the input/output mapping. I found the initial module on github, then extended it with new features and tests and pushed it to CPAN.

Please take a look and let me know how to make it more useful, especially to the new-to-hadoop and new-to-hadoop-streaming user! Specifically I've added new documentation to the root level Hadoop::Streaming, and I'd love feedback.

Thank you,
--spazm 

    package My::Hadoop::Example::Wordcount;
    use Moose::Role;

    sub map 
    { 
        my ($self,$line) = @_;
        my @words = split( /\W+/, $line);
        $self->emit( $_ => 1 ) for @words;
    }
    sub reduce 
    { 
        my ( $self, $key, $value_iterator) = @_;
        my $sum = 0;
        while( $value_iterator->has_next() ) 
        { 
            my $value = $value_iterator->next();
            $sum += $value;
        }
        $self->emit( $key, $sum );
    } 
    sub combine 
    { 
        my ( $self, $key, $value_iterator) = @_;
        my $sum = 0;
        while( $value_iterator->has_next() ) 
        { 
            my $value = $value_iterator->next();
            $sum += $value;
        }
        $self->emit( $key, $sum );
    } 

    package My::Hadoop::Example::Wordcount::Mapper;
    use Moose;
    with Hadoop::Streaming::Mapper,   My::Hadoop::Example::Wordcount;

    package My::Hadoop::Example::Combiner::Wordcount::Mapper;
    use Moose;
    with Hadoop::Streaming::Combiner, My::Hadoop::Example::Wordcount;

    package My::Hadoop::Example::Wordcount::Reducer;
    use Moose;
    with Hadoop::Streaming::Reducer,  My::Hadoop::Example::Wordcount;

    1;

[download]
Driver files:

my_mapper: 

  #!/usr/bin/perl
  use My::Hadoop::Example;
  My::Hadoop::Example::Mapper->run();

[download]
my_combiner: 
  #!/usr/bin/perl
  use My::Hadoop::Example;
  My::Hadoop::Example::Combiner->run();

[download]
my_reducer: 
  #!/usr/bin/perl
  use My::Hadoop::Example;
  My::Hadoop::Example::Reducer->run();

[download]
Hadoop streaming jar command: 
  hadoop                                     \
      jar $streaming_jar_name                \
      -D mapred.job.name="my hadoop example" \
      -input    my_input_file                \
      -output   my_output_hdfs_path          \
      -mapper   my_mapper                    \
      -combiner my_combiner                  \
      -reducer  my_reducer

[download]

In reply to Re^3: Hadoop and perl by spazm
in thread Hadoop and perl by InfiniteLoop

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.