in reply to Re^2: Testing with Test::Mock::Tiny::HTTP
in thread Testing with Test::Mock::HTTP::Tiny

it would seem the simpler solution is to just serialize ...

yep, good point haukex++ as this is the easiest and most secure by far. Since I have more time:

use strict;
use warnings;

use HTTP::Tiny;
use Test::Mock::HTTP::Tiny;
use JSON;

my $http = HTTP::Tiny->new;
my $resp = $http->get('http://www.way-finder.uk/');

# EDIT: it's captured_data() not mocked_data() see below
#my $json_str = eval { JSON::encode_json(Test::Mock::HTTP::Tiny->mocke
+d_data()) };
my $json_str = eval { JSON::encode_json(Test::Mock::HTTP::Tiny->captur
+ed_data()) };
die "failed to encode json" if $@;

open my $fh, '>', 'mock_html.dat';
print $fh $json_str;
close $fh;

[download]

and

use strict;
use warnings;
use Test::More;
use Test::Mock::HTTP::Tiny;

use WWW::Crawl;
use JSON;
plan tests => 1;

$/ = undef;
open my $fh, '<', 't/mock_html.dat' or die "Can't open datafile";
my $replay = <$fh>;
close $fh;

$replay = eval { JSON::decode_json($replay) };
ok(!$@, "parsed JSON replay data") or BAIL_OUT($@);
is(ref($replay), 'ARRAY', "parsed JSON replay data is an ARRAY");

die "Nothing to replay" unless $replay;

Test::Mock::HTTP::Tiny->set_mocked_data($replay);

my $crawl = WWW::Crawl->new(
    'timestamp' => 'a',
);

my @links = $crawl->crawl('https://www.testing.crawl', \&link);

cmp_ok ( scalar @links, '==', 8, 'Correct link count');

sub link {
    diag ($_[0]);
}

[download]

note: eval() around JSON subs is because it dies on error last time I checked.

bw, bliako

Replies are listed 'Best First'.
Re^4: Testing with Test::Mock::Tiny::HTTP
by Bod (Parson) on Sep 27, 2023 at 14:39 UTC
    Since I have more time

    Thank you...I was also thinking that JSON might be the sensible solution here.

    eval() around JSON subs is because it dies on error last time I checked

    It took a moment, but I worked that out before reading your confirmation...

    But why fully qualify the methods that are exported by default? 

    $replay = eval { JSON::decode_json($replay) };

    [download]
    and not 
    $replay = eval { decode_json($replay) };

    [download]

[reply]
[d/l]
[select]
      But why fully qualify the methods that are exported by default?

      as a mental note to myself that this sub is from that package

[reply]
[d/l]
        as a mental note to myself that this sub is from that package

        That makes sense!

        I don't think I'll adopt this for obvious method names like encode_json but it seems a very sensible approach for methods that are rather ambiguous such as get in LWP::Simple

[reply]
[d/l]
[select]
Re^4: Testing with Test::Mock::Tiny::HTTP
by Bod (Parson) on Sep 27, 2023 at 20:33 UTC

    Thanks for this bliako
    For anyone stumbling this way in future, the method to create the file is captured_data, not mocked_data

    Encoding to JSON gives a nice JSON file so that bit seems to be working :)

    However, I am still getting an HTTP 599 error when I try to decode it :(

    $/ = undef;
open my $fh, '<', 't/mock_html.dat' or die "Can't open datafile";
my $replay = <$fh>;
close $fh;

$replay = eval { decode_json($replay) };

ok ( !$@, 'Parsed JSON' ) or BAIL_OUT($@);

is ( ref($replay), 'ARRAY', '$replay is an ARRAY ' );

die "Nothing to replay" unless $replay;

Test::Mock::HTTP::Tiny->set_mocked_data( $replay );

diag ( Dumper (Test::Mock::HTTP::Tiny->mocked_data) );

my $crawl = WWW::Crawl->new(
    'timestamp' => 'a',
);

    [download]

    The diag gives the full JSON object as expected but Test::Mock::HTTP::Tiny is not feeding into the HTTP::Tiny->get call.

[reply]
[d/l]
[select]
      However, I am still getting an HTTP 599 error when I try to decode it :

      It works for me. To be sure, edit Test/Mock/HTTP/Tiny.pm (perldoc -l Test::Mock::HTTP::Tiny will tell you where this file is located) and put debugging messages after ## no critic. Just in case you haven't noticed, the return from my $resp = $http->get('http://www.way-finder.uk/'); is not an HTTP::response (which offers $resp->decoded_content) but a hash with keys headers, success, content, .... So you are looking for:

      my $content = $resp->{success} && length $resp->{content} ? $resp->{co
+ntent} : undef;
ok(defined $content, "got content");

      [download]
[reply]
[d/l]
[select]
        ...put debugging messages...

        Putting debugging messages in HTTP::Tiny has identified the problem 😀

        The mock HTTP data is working fine...but WWW::Crawl pulls links out of the page and then goes and looks at those. And those linked pages are not in the mocked data. The code is designed to deal with HTTP 404 errors for missing pages but Test::Mock::HTTP::Tiny gives HTTP 599 errors instead.

        Thanks for your help with this. I now need to find a solution where it ignores expected 599 errors and not unexpected ones...

[reply]
[d/l]
        Just in case you haven't noticed, the return from my $resp = $http->get('http://www.way-finder.uk/'); is not an HTTP::response

        Yes thanks - I had noticed that. At the point where we mock the data, we don't use $resp. But within the module we do this: 

        my $resp = $self->{'http'}->get($url);
if (!$resp->{'success'}) {
    croak "WWW::Crawl: HTTP Response " . $resp->{'status'} . " - " . $
+resp->{'reason'};
}

$page = $resp->{'content'};

        [download]
        It's here that the module is croaking when I try to use mocked data.

        perldoc -l Test::Mock::HTTP::Tiny

        That's helpful thanks...
        Definitely easier than searching manually!

        I'll add some debugging code a little later when more time is available. I'll try and get the module uploaded to GitHub later.

[reply]
[d/l]
[select]

In Section Seekers of Perl Wisdom