Ok, I've been thinking about a daemon, but I thought maybe it is possible to cache the thing somehow and do without establishing another daemon. I considered daemon approach a little desperate, but I'm beginning to think maybe it wouldn't be that bad.

I understand that hash with all possible answers wouldn't be that big, but it's questions that are making the whole thing impossible, and storing IPs in some memory efficient OR cachable way that provides lightning fast lookup speed is what I seek. I want to supply a single IP and get an answer containing it's AS number. I just can't construct a hash of all ips, it's ~4GB. So I need prefixes, thus I need a method for a really quick lookup of an individual IP in prefix list.

My data right now is 120000 prefixes. My first (simple) approach was to construct a hash like this: 

%iphash = (
    192 => { 192.168.0.0/24 => 23244, 
             192.168.1.0/24 => 4324,
            },
    193 => { 193.1.0.0/16 => 2452 },
);

[download]
and so on, these are of course just some stupid values. Then I would look up the first octet of an ip address, taking the keys of a subhash (which are all the prefixes that start from this octet), iterate through them using Net:IP overlaps and most probably - get an answer finally. But it was more than 3 seconds, and it's way too much. I guess it was pretty much memory consuming too, but since it was too slow I didn't even get to checking memory.

So I found a Net::Patricia module, fed it with the prefix list and assigned "user_data" which were the AS numbers. Now, using match_string method I get what I want in less than a second. Perfect! However the memory problem occurred and I don't know how to cache it, share it, and so on. 

my $pt = new Net::Patricia;
(fill it)
my $memd = new Cache::Memcached {(usual stuff)};
$memd->set( 'pt', $pt );
(...)
my $ptm = $memd->get('pt');
my $asn = $ptm->match_string($given_ip);

[download]

And what I get is: 

perl in free(): warning: chunk is already free
perl in free(): warning: page is already free
Assertion failed: (prefix->ref_count > 0), function Deref_Prefix, file
+ patricia.c, line 362.
zsh: abort (core dumped)  perl ipmemcached_00.pl

[download]

I'm not saying it "doesn't work", I know I might have just gotten it wrong, although I tried various approaches and all I got where core dumps (which surprised me a lot, I must say) or undefs, hence this thread.

In reply to Re^2: (mem)caching of ip prefixes? by Fangorn
in thread (mem)caching of ip prefixes? by Fangorn

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.