my $lproc = 621557;
my @sids  = get_sids($url, $pid, $lproc);
[download]

sub get_sids{
  
  my ($url, $pid, $lproc) = @_;
  my $page = 1;
  my $uri = URI->new($url);
  my ($i, @sids);

  while ($page){
    
    # build the uri
    $uri->query_form(page => $page, pid => $pid);
    my $uri_string = $uri->as_string;

    # get the content, check for success
    my $content = get $uri->as_string;
    die qq{LWP get failed: $!\n} unless $content;
    
    # build the tree
    my $t = HTML::TreeBuilder->new_from_content($content)
      or die qq{new from content failed: $!\n};
    
    # get a list of all anchor tags
    my @anchors = $t->look_down(_tag => q{a})
      or die qq{no tables found in : $!\n};
    
    # look at each anchor
    my $more = 1; # flag
    for my $anchor (@anchors){
      # get the href
      my $href = $anchor->attr(q{href});
      
      if ($href){

        # test for a sid in the query fragment
        my $uri = URI->new($href);
        my %q = $uri->query_form;
        my $sid = $q{sid};
        next unless $sid;
        
        # exit the while loop if it
        # is the last processed sid
        $more--, last if $sid == $lproc;
        
        # otherwise save it
        push @sids, $sid;
        
      }
    }
    
    last unless $more;
    
    # see if there is another page
    $page = get_next_page($t);
    
    # avoid accidental indefinite loops
    # hammering the server, adjust to suit
    die if $i++ > 5;
    
  }
  
  # send 'em back
  return @sids;
  
}
[download]

Update: corrected the sub

OK, have looked it over and think I understands most of it fairly well now. Adapted it to do the whole list as well as import PID/Lproc for processing

There's a bug somewhere though making it abort if a PID have 0 SIDs. Instead of moving on to next PID for processing it simply ends.

Here's what I have so far (I added some print stuff so I can see it progressing):

Read more... (3 kB)

In the main loop

        die qq{no sids found\n} unless @sids;
[download]

does what is says on the tin. :-)

Perhaps replace that line with something like

if (not @sids){
  print qq{no sids found in $cpid\n};
  next;
}
[download]

That way it will print a message and simply move on if none are found.

On a side note you could declare

    my $url = q{http://csr.wwiionline.com......etc.};
[download]

before the main loop as it is the same everytime.

Also,

my $pidlist = <$settings>;

# Get list of pids and process all
while ( $pidlist = <$settings> ) {
[download]

is reading a line from the file and then throwing it away. It is starting on the second line of the file. Remove the first declaration and change the while to

while (my $pidlist = <$settings>){
  #...
[download]

I reckon it's shaping up nicely. :-)