jkeenan1 has asked for the wisdom of the Perl Monks concerning the following question:

Short version:

Is there a Perl technique or a CPAN distribution which would enable me to capture the DOM of some executed HTML in the way that Safari Inspector or Firefox Web Inspector does?

Full version:

We need to be able to identify the file type for every file we receive from upstream servers and push to a downstream server. Simply looking at the file's extension works in the overwhelming majority of cases: .gif, .jpg, .swf, .png, etc. However recently we have a case where the HTML we are supplied from upstream contains URLs which, rather than directly pointing to "img" with extensions, point to a site where JavaScript is executed and where, ultimately, a Flash file is displayed. Our current code can't see what that JavaScript does and, consequently, fails to correctly identify a file as an ".swf" to our downstream recipients.

Pseudo-code: 

<html><head></head>
<body>
<script>
Big hunk of Javascript with URLs not ending in a file extension
</script>
<noscript>
<img src="http://path/to/image.gif></img>
</noscript>
</body>
</html>

[download]
Should this HTML ultimately be displayed on a browser that does not support scripting, the .gif will be displayed. But in all other cases the JS inside the <script> tag is executed and, ultimately, a Flash file is displayed.

I have tried using LWP::UserAgent to make the call. The call succeeds, but the HTTP::Response object simply contains the URL in the original call. However, when I execute this in a browser with developer tools, I get better results. In either Safari's Inspector or Firefox's Web Inspector, I can see what in Safari, at least, is labelled the DOM view of the result. More pseudo-code: 

<html><head></head>
<body>
<script>
Big hunk of Javascript with URLs not ending in a file extension
<div>
<object>
Hunk of code including URL ending in ".swf" and "type" clearly indicat
+ing shockwave flash file.
</object>
</div>
</script>
<noscript>
<img src="http://path/to/image.gif></img>
</noscript>
</body>
</html>

[download]

I want to capture this DOM view and parse it so that I can accurately identify the Flash file to the downstream server. If Safari and Firefox (and probably other browsers as well) can do this, my hunch is that Perl can do it as well. But I'm not primarily a web developer, so I turn to you.

Thank you very much.

Jim Keenan

Replies are listed 'Best First'.
Re: Seek Perl equivalent to DOM Inspector (js pipe dream)
by Anonymous Monk on Sep 28, 2013 at 22:58 UTC
[reply]
[reply]
Re: Seek Perl equivalent to DOM Inspector
by jkeenan1 (Deacon) on Oct 01, 2013 at 16:54 UTC

    I have investigated WWW::Scripter and been in communication with that library's author, Father Chrysostomos. But this is proving a tough nut to crack. Let me give another example:

    Slightly obfuscated and simplified source code: 
    $ cat source.html
<html>
   <head></head>
   <body>
<script
src="http://bs.example.com/SmashingPumpkin/adServer.bs?cn=rsb&c=28&pli
+=8064559&PluID=0&w=336&h=280&ord=[RANDOM_NUMBER]&ucm=true"></script>
<noscript>
<a
href="http://bs.example.com/SmashingPumpkin/adServer.bs?cn=brd&FlightI
+D=8064559&Page=&PluID=0&Pos=1510952713"
target="_blank"><img
src="http://bs.example.com/SmashingPumpkin/adServer.bs?cn=bsr&FlightID
+=8064559&Page=&PluID=0&Pos=1510952713"
border=0 width=336 height=280></a>
</noscript>
   </body>
</html>

    [download]

    What I see in Safari Web Inspector (trimmed, obfuscated) 

    $ cat formatted.html
<html><head></head>
   <body>
     <script
src="http://bs.example.com/SmashingPumpkin/adServer.bs?cn=rsb&amp;c=28
+&amp;pli=8064559&amp;PluID=0&amp;w=336&amp;h=280&amp;ord=[RANDOM_NUMB
+ER]&amp;ucm=true"></script>
     <div id="ebDiv850954202003777" dir="ltr" style="display: inline;
width: 336px; height: 280px; margin: 0px auto; ">
       <div id="ebStdBannerDiv_16672157_850954202003777" dir="ltr"
style="border: 0px; padding: 0px; margin: 0px; top: 0px; left: 0px;
overflow: visible; font-size: 0px; line-height: 0px; text-align: left;
width: 336px; height: 280px; display: inline; ">
         <object
           id="ebStdBannerFlash_16672157_850954202003777"
           dir="ltr"
           name="ebStdBannerFlash_16672157_850954202003777"
           style="border:0px; padding:0px; margin:0px; text-align:left
+;
width:336px; height:280px; "
codebase="http://download.macromedia.com/pub/shockwave/cabs/flash/swfl
+ash.cab#"
           version="8,0,0,0"
           type="application/x-shockwave-flash"
data="http://ds.example.com/BurstingRes//Site-23100/Type-2/9bcc2f07-af
+05-4f07-a0f0-ca803d331bd4.swf">
...

    [download]

    My object is to be able to pluck out either of the last two lines -- 'type' or 'data' -- you see above.

    Using WWW::Scripter, I called: 

    $scripter->use_plugin("JavaScript");
my $doc = $scripter->document();
say STDERR Dumper $doc;

    [download]
    The latter call resulted in 23,500+ lines in Data::Dumper output -- yet nowhere in there could I find the strings 'swf', 'shockwave' or 'flash'. So I couldn't be sure if the code was executing the script at all.

    Suggestions?

    Thank you very much.

    Jim Keenan

[reply]
[d/l]
[select]

      Suggestions?

      :) Get serious? 

      $scripter->use_plugin("JavaScript");
my $doc = $scripter->document();
say STDERR Dumper $doc;
__END__
Can't call method "use_plugin" on an undefined value at - line 1.

      [download]

      Even if I were to fill in the missing blanks from that code snippet, all the html/javascript/website stuff is bogus too

      Maybe all you need is WWW::Scripter::Plugin::Ajax but I wouldn't be surprised if that doesn't fix anything :) writing a DOM+JS is tricky

[reply]
[d/l]

Back to Seekers of Perl Wisdom