While creating a Web content management system, I created some sample plugins to test out what I had built. One of them would get the HTML from another site and stick everything between the body tags into the current document. It's kind of like having frames without the headache of frames. This wasn't intended to be anything fancy and I knew it would break many things such as JavaScript, images with relative links, etc. The owner of the company, however, was so excited and wants me to develop this into something truly useful. Here's what I have so far (somewhat simplified).

package Onsite::Template::Plugin::Mirror;

use strict;
use warnings;

use HTML::TokeParser::Simple;

BEGIN {
    require LWP::Simple;
}

use base qw/Onsite::Template::Plugin::Base/;

sub run {
    my $self = shift;
    my $site = $self->get_site;
    my $html = LWP::Simple::get( $site );

    my $output = '';
    if ( ! $html ) {
        $output = "<strong>Could not retrieve Web page for '$site'</st
+rong>";
    }
    else {
        my $p = HTML::TokeParser::Simple->new( \$html );
        my $token;
        do { $token = $p->get_token } until $token->is_start_tag('body
+');
        do {
            $token = $p->get_token;
            last if ! $token; # just in case someone forgets a closing
+ body tag
            $output .= $token->return_text;
        } until $token->is_end_tag('body');
    }
    return $output;
}
[download]

Note: this runs through Template Toolkit as a plugin.

For the end user, this is as simple as using a URL similar to the following.

http://somesite.com/mod_perl/index.cgi?page=3&site=http://some_target.
+com
[download]

Since I need to try actually turn this into something serious and emulate frames without actually using them, here are the issues that have crossed my mind.

• Convert relative URLs to absolute URLs.
• Hyperlinks need http://somesite.com/mod_perl/index.cgi?page=3&site= prepended to them.
• CSS links in head section can possibly be parsed and added to div tags, thus preserving look and feel.
• Every hyperlink needs something like an MD5 hash to prevent spoofing (otherwise, pranksters could link our site directly to a pr0n site.

I'm not terribly worried about Javascript or bad HTML as the only sites that we would target would be sites that we create. However, I'm concerned about what else I might be missing. If there are many hyperlinks, creating an MD5 has for every one could be time consuming, even under mod_perl. Since I've not tried to do something like this before, I'd love to see some suggestions on things I might trip up on, or anything that I may have overlooked.

Cheers,
Ovid

Join the Perlmonks Setiathome Group or just click on the the link and check out our stats.

---

---

Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!

Titles consisting of a single word are discouraged, and in most cases are disallowed outright.

Read Where should I post X? if you're not absolutely sure you're posting in the right place.

Please read these before you post! —

• How do I compose an effective node title?
• How do I post a question effectively?
• Markup in the Monastery

Posts may use any of the Perl Monks Approved HTML tags:

a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr

You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)

	For:		Use:
	&		&
	<		&lt;
	>		&gt;
	[		&#91;
	]		&#93;

Link using PerlMonks shortcuts! What shortcuts can I use for linking?

See Writeup Formatting Tips and other pages linked from there for more info.