Dear Perl Monks, please share some of your great wisdom!:)

I use Selenium::Remote::Driver and need to set up a "normal" user-agent. I've tried to use Selenium::UserAgent, but it has only some UserAgents for mobiles and tables, and none for usual desktop PC's. Maybe there's a way to expand the list of devices in Selenium::UserAgent, or how to set up a correct User-Agent (I'd like would be Firefox) with Selenium::Remote:Driver + Firefox manually?

I'm trying to parse a website which is protected from bots (solve the puzzle if you are a human). When I try to parse it with the default Selenium+Firefox UserAgent - the protection appears.

I've tried to use Selenium::UserAgent and it worked - the protection has disappeared, but I wasn't able to scrape the needed data, because the target site promotes its mobile application instead of showing the needed data this way.

So, after that I've checked the UserAgent of my home computer's browser and set it up using LWP::UserAgent:

my $ua = LWP::UserAgent->new( "Mozilla/5.0 (X11; Linux x86_64; rv:105.
+0) Gecko/20100101 Firefox/105.0" );

my $driver = Selenium::Remote::Driver->new( browser_name => 'firefox',
+ ua => $ua );

[download]

But this way the protection arrived again.

After that, I've connected to my server through the VNC viewer, opened the same Firefox I've been using with Selenium, and there was no anti-bot protection this way. So, that's why I'm sure that I need to use a correct UserAgent and/or some other settings.

Please, help:)

Update:

Looks like I need to set up the Accept header, like:

$req->header('Accept' => '*/*');

[download]

But how to do it with Selenium::Remote::Driver ?

Update 2:

I've set up the Accept header with the code below, but nothing's changed.

$ua->default_header('Accept' => "*/*");

[download]

In reply to How to set up a correct custom User-Agent when using Perl's Selenium::Remote::Driver and Firefox by Jaharmy

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.