kixsnap has asked for the wisdom of the Perl Monks concerning the following question:

I have a program i'm writing in perl using Selenium::Remote::Driver that is also using threads.

Is S:R:D not thread friendly? When I start a thread and try to open a selenium driver instance, I would get a random error(dont have on hand, I got past this). For now, I set a shared variable that makes it so only one selenium instance is executing commands at a time, but i'm still running into errors, right now is with setting cookies in the selenium instance.

Is there a way around this?
This is how my program goes for instance:
setup driver

$driver = Selenium::Remote::Driver->new( 'auto_close' => 0, 'browser_name' => 'chrome', extra_capabilities' => { 'chromeOptions' => { 'args' => ['window-size=1920,1080', 'blink-settings=imagesEnabled=false']}});


Thread 1 Starts
Thread 2 Starts
Thread 2 Randomly won the race of getting selenium started first, does stuff
Thread 1 Waits til thread 2 is done doing selenium commands.
....Fast forward both threads are done setting up....
Thread 1 Wins the race now and starts setting a cookie in theirenter code here selenium instance on a webpage we're on, www.mywebsite.com

$driver->add_cookie("anything", "1", "/", 'www.mywebsite.com','1','1');

this thread ends up terminating because of the error

Thread 1 terminated abnormally: Error while executing command: invalid argument: invalid argument: invalid 'secure' (Session info: chrome=81.0.4044.129) at /Library/Perl/5.18/Selenium/Remote/Driver.pm line 402 thread 1. at /Library/Perl/5.18/Selenium/Remote/Driver.pm line 353 thread 1.

No matter what we change it to, this will still error out, anyone have any ideas? is this really just not thread friendly and i have to try something else?

Replies are listed 'Best First'.
Re: Selenium::Remote::Driver and threads
by 1nickt (Canon) on May 02, 2020 at 15:16 UTC

    Greetings kixsnap and welcome to the Monastery. I bring you glad tidings of great joy for all mankind.

    Yes, you can use Selenium::Remote::Driver in a parallel application. One of the more convoluted small helper apps I have built was a program that ran continuously offline looping through records from a DB and making a series of calls for each using S::R::D and Firefox. The program had been running for several months when I left that assignment.

    Things that might be different between my environment then and yours:

    • I was not using threads (and never have), although the solution I implemented would work under a threading environment as far as I know.
    • I was running on a Unix server with no GUI, so I used a set of X window emulators as large as the number of worker processes for the Firefox instances to operate in. I ran a daemon to make sure the emulators were always available.
    • Firefox may have changed since then (~ 4-5 years ago); I haven't done any web client automation work since then so I don't know.
    • I was using the magnificent and mighty MCE (MCE::Loop back then; now I would most likely use MCE::Flow...) and MCE::Shared. The code presented below is mostly a redacted version of what I built back then; please bear in mind that there may be some naive MCE idioms there; do your own reading to find out the state of the MCE and MCE::Shared art nowadays.

    =pod # $ DISPLAY=:99 perl script.pl You need to set the C<DISPLAY> env variable because this script uses h +eadless Firefox instances run by Selenium. =cut use strict; use warnings; use MCE::Shared; use MCE::Loop; use Tie::Cycle; my $pid = $$; END { report() if $$ == $pid } # only parent process reports results # Overall results hash, shared between processes my $results = MCE::Shared->hash(); # Process the data my %data = get_data(); # in this case, was keyed by HTTP method for my $method ( keys %data ) { MCE::Loop->init( chunk_size => 100, max_workers => 8 ); tie my $screen, 'MCE::Shared', { module => 'Tie::Cycle' }, [ 0 .. +7 ]; # We have a fixed number of screens in our virtual X buffer; # this range should match the number of MCE worker processes mce_loop { my ( $mce, $chunk_ref, $chunk_id ) = @_; my $driver = get_selenium_driver(); # or maybe an obj that already knows what SRD commands to run # Set the screen ID for a Firefox instance (for Selenium) $ENV{'DISPLAY'} = ':99.' . $screen; for ( @{ $chunk_ref } ) { my %this_job_data = %{ $_ }; if ( $this_job_data{skip} ) { $results->incr('SKIP'); next; } $driver->do_the_thing( $this_job_data ); $results->incr('OK'); $results->incr($this_job_data->{something}); } } @{ $data{ $method } }; MCE::Loop->finish; } ## sub report { # do something with $results } __END__

    Hope this helps!


    The way forward always starts with a minimal test.