while ($CONTENT =~ <div class=\"usertext-body may-blank-within md-cont
+ainer \"><div class=\"md\">(.+?)<\/div>
<\/div><\/form><ul class=\"flat-list buttons\"> //gs )
[download]

This line gives an error. The thing to the right side of the =~ should be a match (a regular expression) but it is not. Perhaps move one of the terminating slashes to the beginning of the expression instead of having 2 at the end.

Addendum: This line:

open(OUT, "">C:/Users/user/perl_tests/reddittest.txt");

is also badly formed. See open for examples of how to call this function correctly.

Thank you very much!

The following is a malformed regular expression:

while ($CONTENT =~ <div class=\"usertext-body may-blank-within md-cont
+ainer \"><div class=\"md\">(.+?)<\/div><\/div><\/form><ul class=\"fla
+t-list buttons\"> //gs )
[download]

It is at least missing the s/ start.

Personally, I suggest that you do the content extraction by using HTML::TreeBuilder and XPath or CSS selectors (via HTML::TreeBuilder::XPath and HTML::Selector::CSS).

Also note that Reddit has an API available, so you maybe don't need to scrape at all but can get the comments in a machine readable format directly.

Also note that on CPAN, there are many Reddit modules available, and it seems that Reddit::Client is using the Reddit API.

I don't recommend to use HTML::TreeBuilder::XPath because it's really slow (on files > 1M). HTML::TreeBuilder::LibXML is much faster and have almost the same syntax.

Thank you very much! I will look into these alternatives. Thanks again for your suggestions.

Hello Lisa1993, and welcome to the Monastery!

As Corion says, you’d be better off using a dedicated module to extract the HTML you want. But in the meantime...

hippo has identified the syntax errors in your code. But, even when these are fixed, the regular expression won’t match any of the content on the web page in question. To get it to match, I had to tweak it in two places:

use strict;
use warnings;
use LWP::Simple;

my $URL     = 'https://www.reddit.com/r/unitedkingdom/comments/58m2hs/
+' .
              'i_daniel_blake_is_released_today/';
my $CONTENT =  get($URL);
my $regex   = '<div class="usertext-body may-blank-within md-container
+ ">' .
              '<div class="md">(.+?)</div>\s*</div>' .
              '</form><ul class="flat-list buttons">';
my $x       = '';
my $count   =  0;

while ($CONTENT =~ m{$regex}gs)
{
    $x .= $1;
    ++$count;
}

print $x;
print "Count: $count\n";
[download]

First, you need to allow for (optional) whitespace between the two closing </div> tags. Second, you need to remove the space at the end of the regex. Also note that it isn’t necessary to escape the quotation character, and you can avoid escaping forward slashes by changing the regex delimiter (as shown above).

With these changes, I get 80 matches.

Also note the inclusion of use strict and use warnings, and the declaration of variables using my. This is basic good practice in Perl.

Hope that helps,

That's brilliant! You've made my day! Thank you very much.

Can I just ask too, is there any way to run the script for multiple URL's at once? Or would I need a more complicated programme for that?

Thanks again!

It's trivial by wrapping part of your code in a for() loop, and turning the single scalar $URL link into an array @URLS, that contains a list of urls instead. The for() loop iterates over this list. Note that this assumes the regex is the same for all urls. Untested:

use strict;
use warnings;
use LWP::Simple;

my @URLS = qw(
    http://one.example.com
    http://two.example.com
    http://three.example.com
);

my $regex   = '<div class="usertext-body may-blank-within md-container
+ ">' .
              '<div class="md">(.+?)</div>\s*</div>' .
              '</form><ul class="flat-list buttons">';

for my $URL (@URLS){
    my $CONTENT =  get($URL);
    my $x       = '';
    my $count   =  0;
    
    while ($CONTENT =~ m{$regex}gs){
        $x .= $1;
        ++$count;
    }

    print "---$URL---\n";
    print $x; 
    print "Count: $count\n";
}
[download]

An alternative would be to request a JSON object rather than the rendered web page, do this by appending .json to the end of your URL like so:

https://www.reddit.com/r/unitedkingdom/comments/58m2hs/i_danie+l_blake
+_is_released_today/.json
[download]

I've no idea what tool you're using further down the line for analysis, but HTML seems like a odd format to store such data. Here is a short example, simply printing the name of the poster and the comment:

#!/usr/bin/perl

use strict;
use warnings;
use Mojo::UserAgent;

my $url ='https://www.reddit.com/r/unitedkingdom/comments/58m2hs/i_dan
+ie+l_blake_is_released_today/.json';

my $ua = Mojo::UserAgent->new;
my $data = $ua->get( $url )->res->json;

foreach my $comment ( @{$data} ) {
    foreach my $child ( @{ $comment->{'data'}->{'children'} } ) {
        print $child->{'data'}->{'author'} . " posted:" .$/;
        print $child->{'data'}->{'body'} . "\n" if( $child->{'data'}->
+{'body'} );
    }
}
[download]

You'll need the Mojo::UserAgent module:

#install via cpan
cpan Mojo::UserAgent
#or cpanm
cpanm Mojo::UserAgent
[download]

From the brief example above you can see how to get just what you want, or add some other bells and whistles. The example isn't particulary pretty in it's output, I'll leave that an an exercise for you. You can examine the JSON in browser (some plugins exist to prettify the content) or you can use something like json_pp to print it from the command line.

Update: So I read some other comments you made, if you're trying to do this for various sub-reddits you can easily adapt the above example to:

• For each sub reddit url (append .json)
• Get each thread
• Follow the existing code to print comments (or save to a file)
• sleep for a few seconds...