joyfedl has asked for the wisdom of the Perl Monks concerning the following question:

I have a problem with my script. i cant output the response very well

this is the error i get

away_teamsARRAY(0x55fbf77894a0)timesARRAY(0x55fbf743be78)home_teamsARRAY(0x55fbf71ab7b0)scoresARRAY(0x55fbf71aba98)

this is the output i expect

times home_teams scores away_teams 12.05. 17:00 Brighton 1 - 4 Manchester City

Full code

#!/usr/bin/perl - wT use strict; use warnings; use LWP::UserAgent; use HTTP::Request; use HTML::TreeBuilder; use Data::Dumper; use CGI::Carp qw(fatalsToBrowser warningsToBrowser); # show errors in +browser print "Content-type: text/html\n\n"; # print output to browser my $url = 'https://www.flashscore.com/football/england/premier-league- +2018-2019/results/'; my $ua = LWP::UserAgent->new; sub get_scores { my $response = $ua->get($url); if ($response->is_success) { my $tree = HTML::TreeBuilder->new; $tree->parse($response->decoded_content); return $tree; } else { die $response->status_line; } } my $results = get_scores(); my @times = map { $_->as_text } $results->look_down(_tag => 'div', cla +ss => 'event__time'); my @home_teams = map { $_->as_text } $results->look_down(_tag => 'div' +, class => 'event__participant event__participant--home'); my @scores = map { $_->as_text } $results->look_down(_tag => 'div', cl +ass => 'event__scores fontBold'); my @away_teams = map { $_->as_text } $results->look_down(_tag => 'div' +, class => 'event__participant event__participant--away'); my %dict_res; for my $ind ($results) { push @{$dict_res{'times'}}, @times[$ind]; push @{$dict_res{'home_teams'}}, @home_teams[$ind]; push @{$dict_res{'scores'}}, @scores[$ind]; push @{$dict_res{'away_teams'}}, @away_teams[$ind]; print %dict_res; }

Replies are listed 'Best First'.
Re: Web scrapping
by 1nickt (Canon) on Jul 11, 2025 at 09:44 UTC

    I think you should find another source for your football results, unless you have obtained special permission for what you are doing.

    From the Terms and Conditions : "... you are not permitted to use the content of the website by embedding, aggregating, scraping or recreating it without our express consent ..."

    E.g. https://www.api-football.com/, https://www.football-data.org/ (free). Using an API is much easier than scraping a web page anyway.


    The way forward always starts with a minimal test.
Re: Web scrapping
by ysth (Canon) on Jul 10, 2025 at 18:03 UTC
    Please do show what you want the response to look like; it is not at all clear from your code.

    You have warnings enabled, and this code does generate some; did you see and understand them?
      I only noticed the space between dash and "w" because of hippo's reply. Perhaps Perlmonk's input field validation garbled an em-dash here? 🤔
        Perhaps Perlmonk's input field validation garbled an em-dash here

        (a) No. If there's any "garbling", blame html, not perlmonks
        (b) I hope not; em-dash won't work in place of hyphen on the command line.

Re: Web scraping
by hippo (Archbishop) on Jul 10, 2025 at 22:21 UTC
    #!/usr/bin/perl - wT

    Well, that line is not going to do what you think it is. Can you see what you have done wrong here?


    🦛

      At least i have tried something here is the try

      But still cant get the output and no response

      HTML Part to Scrap

      <div id="g_1_UFOgEYGu" class="event__match event__match--withRowLink e +vent__match--static event__match--twoLine" data-event-row="true"> <a href="https://www.flashscore.com/match/football/UFOgEYGu/#/matc +h-summary" target="_self" rel="nofollow" class="eventRowLink" aria-de +scribedby="g_1_UFOgEYGu" title="Click for match detail!"></a> <span class="event__check--hidden"></span> <div class="event__time">12.05. 17:00</div> <div class="wcl-participant_7lPCX event__homeParticipant" data-tes +tid="wcl-matchRow-participant"> <img data-testid="wcl-participantLogo" data-size="XXS" class=" +wcl-assetContainer_ZFzzG wcl-logo_EkYgo wcl-sizeXXS_-SmAO" alt="Brigh +ton" loading="lazy" src="https://static.flashscore.com/res/image/data +/G0q9xjRq-b92lfEJC.png" /> <span class="wcl-simpleText_Asp-0 wcl-scores-simpleText-01_pV2 +Wk wcl-name_3y6f5" data-testid="wcl-scores-simpleText-01">Brighton</s +pan> </div> <div class="wcl-participant_7lPCX event__awayParticipant" data-tes +tid="wcl-matchRow-participant"> <img data-testid="wcl-participantLogo" data-size="XXS" class="wcl-assetContainer_ZFzzG wcl-logo_EkYgo wcl-sizeXXS +_-SmAO" alt="Manchester City" loading="lazy" src="https://static.flashscore.com/res/image/data/0vgscFU0 +-lQuhqN8N.png" /> <strong class="wcl-simpleText_Asp-0 wcl-scores-simpleText-01_p +V2Wk wcl-bold_roH-0 wcl-name_3y6f5" data-testid="wcl-scores-simpleTex +t-01">Manchester City</strong> </div> <span class="wcl-matchRowScore_jcvjd wcl-isFinal_Am7cC event__scor +e event__score--home" data-testid="wcl-matchRowScore" data-state="fin +al" data-highlighted="false" data-side="1">1</span> <span class="wcl-matchRowScore_jcvjd wcl-isFinal_Am7cC event__scor +e event__score--away" data-testid="wcl-matchRowScore" data-state="fin +al" data-highlighted="false" data-side="2">4</span> </div>

      Full code

      #!/usr/bin/perl use Mojo::UserAgent; use Mojo::DOM; my $ua = Mojo::UserAgent->new; my $res = $ua->get('https://www.flashscore.com/football/england/premie +r-league-2018-2019/results/')->result; if ($res->is_success) { my $dom = Mojo::DOM->new($res->body); my @matches = $dom->find('div.event__match event__match--withRowLi +nk event__match--static event__match--twoLine')->each; # Iterate through the event__match event__match--withRowLink even +t__match--static event__match--twoLine and extract text foreach my $results (@matches) { my $Time = $results->find('div.event_ _time')->map('text')->jo +in; my $teamA = $results->find('div.wcl-participant_7lPCX event__h +omeParticipant')->map('text')->join; my $teamB = $results->find('div.wcl-participant_7lPCX event__a +wayParticipant')->map('text')->join; my $teamA_Scores = $results->find('span.wcl-matchRowScore_jcvj +d wcl-isFinal_Am7cC event__score event__score--home')->map('text')->j +oin; my $teamB_Scores = $results->find('span.wcl-matchRowScore_jcvj +d wcl-isFinal_Am7cC event__score event__score--away')->map('text')->j +oin; print "$teamA : $teamA_Scores: $Time"; print "$teamB : $teamB_Scores: $Time"; } } else { print "Cannot parse the result. " . $res->message . "n"; }

        Is this "AI" generated code? It looks like something it would come up with since it doesn't understand what the problem is, or have any domain knowledge. The content of the page is constructed after it's loaded. The table you're trying to parse simply doesn't exist at the point your code runs. The data 'is' in the page, see cjs.initialFeeds['results'], however this is very fragile and subject to change, even automatically. As usual 1nickt has given a sensible response, you could also use a headless browser such as WWW::Mechanize::Chrome to automate an actual browser to visit the page, waiting for everything to actually happen and acquiring the data via the query mechanisms it provides, or passing the complete HTML to your Mojo::DOM code. The sensible option is the free APIs already provided, rather than playing whack a mole with this other site.

Re: Web scrapping
by LanX (Saint) on Jul 10, 2025 at 17:53 UTC
    There are multiple issues

    >

    040: push @{$dict_res{'away_teams'}}, @away_teams[$ind]; 041: print %dict_res;

    • you are pushing into an array reference and then printing it, that's why you get the ARRAY(0x55fbf77894a0) output
    • @away_teams[$ind] is a one element array-slice, you might prefer $away_teams[$ind]
    • it's unclear why you even try to push into an array with multiple values, if you want to print a singular one right away in line 41
    • hashes have random order, you cant rely on print %dict_res having your desired order. prefer a hash-slice here, like print @dict_res{@order}

    Cheers Rolf
    (addicted to the Perl Programming Language :)
    see Wikisyntax for the Monastery

      i dont want to print single results, I want to print all

        > i dont want to print single results, I want to print all

        you are printing inside a loop iterating over $results

        And now I notice another issue,

        • $result is a scalar

        I don't think you know what you are doing and should try debugging step by step to learn the essentials of Perl programming.

        If this is AI generated code, good luck finding someone who's going to fix all these issues.

        Cheers Rolf
        (addicted to the Perl Programming Language :)
        see Wikisyntax for the Monastery

A reply falls below the community's threshold of quality. You may see it by logging in.