http://qs1969.pair.com?node_id=625293

myuserid7 has asked for the wisdom of the Perl Monks concerning the following question:

Hello, everyone. I'm writing a Perl script that downloads and saves to disk all of the MP3 files linked on a given Web page. For Web sites that do not require the use of cookies, I'm able to use LWP::Simple::getstore() without any problem. And I already know how to download HTML Web pages from sites that use cookies, using LWP::UserAgent->new(), HTTP::Cookies::Netscape->new(), HTTP::Request->new(), etc.

But I do not see how to do both at the same time -- in other words, download and store an MP3 file from a site that requires one to be logged in with a cookie. I tried using the HTTP::Request->new(), $ua->request(), but it does not work because the content type is not 'text/html', but instead 'audio/mpeg'.

Any ideas?
  • Comment on How download MP3 file from Web site that requires cookies

Replies are listed 'Best First'.
Re: How download MP3 file from Web site that requires cookies
by jettero (Monsignor) on Jul 06, 2007 at 16:38 UTC
    my $cookie_jar = new HTTP::Cookies(file => "cookies.txt"); $cookie_jar->load; END { $cookie_jar->save }; my $mech = new WWW::Mechanize( user_agent => "mp3slurpy/1.0", cookie_j +ar => $cookie_jar ); $mech->get( $login_page ); $mech->submit_form( form_number => 3, fields => { username => 'mungo', password => 'lost-and-alone', } ); $mech->get( $mp3_file, ":content_file" => $output_file );

    PS, there's no real reason to choose WWW::Mechanize over LWP::UserAgent except for the ease of using the API... Mech uses LWP though, so there's no disadvantage for picking it either.

    -Paul

      @fenLisesi
      I should have explained that the reason for the script is to process Web pages that have dozens of links to MP3 files, so I don't have to do it manually.

      @zer
      I'm afraid I don't exactly understand what you mean by "if you are using an external website you can use a proxy to copy the cookie used." The Web site with MP3 files is an external Web site. If the files were local, I wouldn't be wanting to download them. What you mean by using a proxy to copy cookies?

      @Codon
      Yes, but too manual for what I want to do.

      @Paul
      Thank you for the input! Unfortunately, I spent a bunch of time trying to install WWW::Mechanize, but was unsuccessful. I'm using ActiveState Perl on a Windows XP machine. So I used their PPM (Programmer's Package Manager), and did "install WWW::Mechanize" and got error messages. Every variation I tried, failed. I don't believe I've ever downloaded a CPAN module before, but after this experience, I'm not sure I want to try again. Anyway, I gave up in frustration, and switched over to LWP::UserAgent, which seems to be installed by default, and managed to get the code working, using $ua->get(). Thanks!
Re: How download MP3 file from Web site that requires cookies
by fenLisesi (Priest) on Jul 06, 2007 at 16:31 UTC
    I am probably misinterpreting your question, but I am inclined to ask why you don't login first and then get your MP3. That is, you don't have to do it all in one step. The user agent is just like a browser; you can keep filling forms and clicking buttons. Cheers.
Re: How download MP3 file from Web site that requires cookies
by zer (Deacon) on Jul 06, 2007 at 17:29 UTC
    if you are using an external website you can use a proxy to copy the cookie used.

    Otherwise you may log in using your browser and HTTP::Cookie::Find to use the cookies off of other browsers.

    there are other more detailed packages that can help you with specific browsers such as mozilla and IE.

    Best of luck

Re: How download MP3 file from Web site that requires cookies
by Codon (Friar) on Jul 06, 2007 at 20:17 UTC
    Have you tried DownThemAll!? It won't help you with your Perl solution, but it may make it superfluous.

    Ivan Heffner
    Sr. Software Engineer
    WhitePages.com, Inc.
Re: How download MP3 file from Web site that requires cookies
by ForgotPasswordAgain (Priest) on Jul 06, 2007 at 16:43 UTC
    The best way to learn web programming is by downloading porn.