Script needs major optimizing.

Jesse Smith has asked for the wisdom of the Perl Monks concerning the following question:

This little script is able to get my server load to say 69.62. Then once it stops running after like five minutes, it goes back down to about 1. All this script is supposed to do is check if the user that views the page is the Facebook bot or some one else. When it's some one else it simply redirects you to the video on youtube. When it's Facebook, it gives the Facebook bot the Youtube information needed to post the video on Facebook in the correct embedded format so you get embedded views credited on youtube.

#!/usr/bin/perl

use CGI ':standard';

$video = param('video');

use LWP::Simple qw(!head);
$url = get ("http://www.domain.org/facebookvideos/$video/");

$url =~ s*<meta property="og:description" content="Text of the talk:  
+<a href='http://domain.org/general-conference/([^&]+)/([^&]+)/([^&]+)
+ '>http://domain.org/general-confere...</a> ">*<meta property="og:des
+cription" content="Text of the talk: http://domain.org/general-confer
+ence/$1/$2/$3" />*g;
#$url =~ s*tion\" content\=\"*tion\" content\=\"Want more\?\? http://w
+ww.domain2.org/ *g;
$url =~ s*og:url" content="http://www.domain2.org/videos/*og:url" cont
+ent="http://www.domain2.org/facebook/*g;
#$url =~ s*Want more\?\? http://www.domain2.org/ Text*Text*g;

#facebookexternalhit   or   Mozilla
if ($ENV{'HTTP_USER_AGENT'} =~ /facebookexternalhit/) 
{
   print "Content-Type: text/html\n\n";
print <<EOM;
$url
EOM
}

else {
print "Location: http://www.youtube.com/watch?v=$video\n\n";
EOM
}
[download]

generates

top - 23:51:16 up 23:18,  1 user,  load average: 69.62, 36.45, 15.11
Tasks: 134 total,  45 running,  86 sleeping,   0 stopped,   3 zombie
Cpu(s): 99.7%us,  0.3%sy,  0.0%ni,  0.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:    786432k total,   786432k used,        0k free,        0k buffers
Swap:     4264k total,     4264k used,        0k free,        0k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                     
 1512 site100   19   0 55248  11m 1876 R  3.3  1.5   0:01.39 facebook.cgi                 
 1943 site100   19   0 55248  11m 1876 R  3.3  1.5   0:01.09 facebook.cgi                 
 4018 site100   19   0 55248  11m 1876 R  3.3  1.5   0:01.09 facebook.cgi                 
 4024 site100   19   0 55280  11m 1876 R  3.3  1.5   0:02.78 facebook.cgi                 
 4029 site100   19   0 55248  11m 1876 R  3.3  1.5   0:02.81 facebook.cgi                 
 4034 site100   19   0 55280  11m 1876 R  3.3  1.5   0:02.79 facebook.cgi                 
 4044 site100   19   0 55248  11m 1876 R  3.3  1.5   0:01.09 facebook.cgi                 
 4057 site100   19   0 55280  11m 1876 R  3.3  1.5   0:01.04 facebook.cgi                 
 4060 site100   19   0 55280  11m 1876 R  3.3  1.5   0:01.11 facebook.cgi

How do you make the script do what it should do in a second or two instead of it spending five minutes killing the server???

Comment on Script needs major optimizing. Download Code

Replies are listed 'Best First'.
Re: Script needs major optimizing. by Corion (Patriarch) on Dec 30, 2011 at 08:28 UTC
Have you tried caching the video information from `http://www.domain.org/facebookvideos/` instead of re-fetching the information every time?	[reply] [d/l]
Re: Script needs major optimizing. by i5513 (Pilgrim) on Dec 30, 2011 at 12:23 UTC
That redirect could be done by modrewrite in apache, and run the script only if facebook user agent is used So no every http request is generating a new process Regards,	[reply]
Re^2: Script needs major optimizing. by Jesse Smith (Acolyte) on Dec 31, 2011 at 08:21 UTC
Thanks. I didn't think of using mod_rewrite. Though it's not from getting a ton of real people clicks that generates the high server load and crashing it. I just posted a video link to the page, and then removed the link. It's as if the Facebook robot is doing a DOS attack on my server, then stops about five minutes later!!!!! The page has 180,242 likes. Looking at the logs, from the post and delete, in five minutes, Facebook hit the URL 153 times then finally stopped, with a special guest appearance from the Google bot!!! It's like there's an infinite loop some where in there!!! Then when I re-posted the video link and keep it, it looks like when it shows up in SSH --> top, it's just actual Facebook users clicking it that makes the script show up, and that does not crash the server or make the load go very high (at 2.81 right now)! It looks like it's the first post of a video that causes Facebook to crash the server.	[reply]
Re: Script needs major optimizing. by Anonymous Monk on Dec 30, 2011 at 10:46 UTC
`# you initialise/calculate $url here if ($ENV{'HTTP_USER_AGENT'} =~ /facebookexternalhit/) { # you only need $url here print "Content-Type: text/html\n\n"; print <<EOM; $url EOM } else { # you don't need $url here print "Location: http://www.youtube.com/watch?v=$video\n\n"; EOM }` [download] You are effectively doing a HTTP GET for every request you get, facebook bot or not.	[reply] [d/l]