And on fetching the various chunks, what do you need to do with them? How do you want to organize, manipulate or summarize them? Is each "candidate" uniquely identified by the "candidate number"? If so, do you want a simple listing that shows, e.g.

candidate-number  R-score  [protein-name ...]
[download]

my @candidates;  # should use a hash instead, if 
my $c_id;        #   the numbering has lots of big gaps

while (<>) {
    if ( /CANDIDATE\s+(\d+)/ ) {
        $c_id = $1;
    }
    elsif ( /DNA\s+.Note.\s*(\S*.*)/ ) {
        my $proteins = $1;
        my @p_set = ();
        while ( $proteins =~ /\[(NP\s+\d+)\]/g ) {
            push @p_set, $1;
        }
        $candidates[$c_id]{proteins} = [ @pset ];
    }
    elsif ( /R-score\s*=\s*([.\d]+)/ ) {
        $candidates[$c_id]{rscore} = $1;
    }
}
[download]

BioGeek, I don't mean to complain, but can simplify your question, a bit, get it down to the basics (I don't want to try to figure out all that HTML). Also, consider <readmore> tags if posting long questions. Thanks, and I look forward to seeing a distilled version.

First of all, does bork.embl-heidelberg.de offer any sort of batch download, or text based save that could be used?

I'm guessing your looking to do this in a (semi) automated manner, and starting from saving as text from your web browser wont help in the long run, since your scripting the fetch? I also might suggest that you may want to consider contacting the web site maintainers. They may not be thrilled about getting hammered by your script, and may be very open to helping find an easier way to provide this information to you instead.

That being said, I used wget to download the page and wrote this code which seems to capture what you want (this is a quick translation from a command line test, and in no way should be considered production ready or tested).

while (<>) {
  chomp;
  if(/CANDIDATE (\d+)/){
    $cand = $1;
    @acc = ()
  } # if candidate line
  elsif (m{http://www.ncbi.nlm.nih.gov:80/entrez/query.fcgi}
      && m{(\w+)</A>}) {
    push(@acc,$1);
  } # else if ncbi accession #
  elsif (m{R-score</A>\s*=\s*(\S+);}) {
    print "$cand\t", join(",",@acc), "\t$1\n";
  } # else if R-score, print this candidates info
} # while reading web page
[download]

• Collapse the three lines of declaration into one line:

  my ( $R_score, @protein, @R_score_protein );
  [download]

  Arrays are by default initialized as empty arrays, so you don't need to set them to (). <em$R_score is a scalar, not an array, so by default it is undef, which acts like the empty string if you try to concatenate to it; but you should initialize it to "" if it might be read before it is assigned.

• In any case, you don't need $R_score, since you only use it to transfer a value from $1 into @R_score_protein. Delete the variable and assign $1</em, into the array.

  my ( @protein, @R_score_protein );
  ...
  push( @R_score_protein, $1 );
  [download]

• Does 'strict' not complain about that unquoted string? In any case, get that URL out of the code and into a configuration variable at the beggining of the file. Better yet, provide for the user to override the URl at run time:

  Read more... (602 Bytes)

• if you can process proteins and R scores separately:

  while ( $content_all =~ m{R\-score<\/A>\s=\s(\d\.\d+)\;}g ) {
      push( @R_score_protein, $1 );
  }
  while ( $content_all =~ m{\[(NP_\d+)\]}g ) {
      push @protein, $1;
  }
  [download]

  Better yet, you could move the regular expressions into 'local' variables or 'configuration' variables:

  my $protein_re = qr|\[(NP_\d+)\]|;
  my $score_re = qr|R-score</A>\s=\s(\d\.\d+)|;
  while ( $content_all =~ m{$protein_re} ) .....
  while ( $content_all =~ m{$score_re} ) ....
  [download]

  Tidier, more readable, more maintainable.

• I suspect you want to associate the protein codes and the R scores, which the version above doesn't provide. You need to break the input into distinct lines, which I'll arrange to have happen magically in the break_into_lines() routine:

  Read more... (2 kB)