That looks as though you are off to a good start.

I have a couple of observations and some more questions. 

my $pid = <PIDLIST>;

[download]
will read the first line from the file. To look at each line in turn loop over the file handle with a while loop.

The docs for LWP::Simple show how to check if the get is successful. Always best to do that.

First question, what does a line in the pidlist.txt file actually look like? Is there any processing to be done to extract any data you need?

Secondly, could you show a (cut down) sample of what the downloaded content looks like. It may be that you can extract the data you need on the fly and write the whole lot out in one go rather than save each page to disk and parse it later.

Lastly, it looks as though the site needs a login and uses cookies. I'm curious to know how you managed to download any pages.

Looking good though!

This is some code illustrating the points above. 

#! /usr/bin/perl

use strict;
use warnings;
use LWP::Simple qw(get);
use File::Slurp ;

# pid = Persona ID, one of a players 3 identities.
# sid = Sortie ID, identifier for a mission taken by the persona.
# We want to crawl all sortie list pages and collect new sid's we 
# have't seen before and then move on to next persona.

my $pbase = q{http://csr.wwiionline.com/scripts/services/persona/sorti
+es.jsp};
my $pcnt = 1;
my $pidfile = q{c:/scr/pidlist.txt};

# Open list of pid's
open my $pidlist, q{<}, $pidfile or die qq{Could not open $pidfile: $!
+};

# loop over list of pids one at a time

while (my $pid = <$pidlist>){
  print $pid;
  chomp $pid;
  
  # what does a line from the pidlist look like?
  # do need to do any work on it?
  
  while (1) {
    my $url = qq{$pbase?page=$pcnt&pid=$pid};
    my $content = get($url);
    die qq{get failed for $url: $!} unless $content;
    
    # parse $content
    # extract data we need
    # test if there is more work to do
    
    
    # Update page number and grab next.
    $pcnt += 1; 
  };
}
# Close files
close $pidlist or die $!;
print qq{\nDone!\n};

[download]

In reply to Re^3: collect data from web pages and insert into mysql by wfsp
in thread collect data from web pages and insert into mysql by SteinerKD

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.