Jay

actually, I'm not "stuck" with a flat text file, but I cant find a really good way to split things up... for instance, if the user searches for "boy", making a file called "boy.resutls" with that info, would not work becuase some of the results from a "boy" search also come in a "girl" search, and a "child" search. But not all of them. If I did that I would have to open and search all the files anyway, defeateing the whoe purpose of splitting them up.

If, as the results came in, I put them in seperate files based on the first digit, like a "1" file, a "2" file a "3" file, and then opened each file as it is needed in future searhces, would this be faster do you think?
Is is faster to open one file with all entries and search it, or open various files with less entries and search each one individually as needed?

Would it be faster to put them in Mysql and then for each one do a search like --"SELECT * FROM okphotos WHERE id LIKE '$id';-- for each result? I am trying all these as we speak, but your suggestions will save me from trying something in vain.. Thanks.

kbeen.

mySQL could/would cache that data in memory if the table is used often, however you lose speed if the MySQL server is on a different machine than the script. If you have a select for each ID you gain speed again though, specially when the cache becames larger. I think a SQL server is a good way to go.

Don't use LIKE though, use =, although LIKE without % and ? might be optimized that way by the server... dunno.

use numeric fields if possible, if not, then create a index on a substring of the id. for example,

id char(10), Update: don't use varchar ...
index index_id id(3)


Tiago

Warning... untested code.

#!/usr/bin/perl -wT
use strict;
use DB_File;

my $dbmfile = '/tmp/lookuptable';

tie my %lookuptable, "DB_File", $dbmfile, O_RDWR|O_CREAT, 0640, $DB_HA
+SH
      or die "Cannot open file '$dbmfile': $!\n";

# do whatever it is to get the ids...
my $id = '123456789';

# check lookup cache:
my $cachevalue = $lookuptable{$id};

if ($cachevalue eq 'Y') {
   # link is ok
} elsif ($cachevalue eq 'N') {
   # link is bad
} else {
   # determine if it is good or bad.
   my $isok = 'Y';

   # add the calculated value to the cache
   $lookuptable{$id} = $isok
}
[download]

-Blake

A good suggestion; but I'd be careful about which DBM module actually gets used, because most of them have pretty limiting restrictions. SDBM f.ex, the only DBM module that ships with Perl/Win32, can only store a very small number of keys, making it next to useless for anything beyond persistent configuration information or such.

Good point. I've always found BerkeleyDB to be more than sufficient for anything like this. I'd recommend it over most of the other dbmish inplementations.

-Blake

Everyone has given good advice above me, but I thought I'd add my personal experience as well. I've also worked on projects where speed was critical, and for the most speed-critical parts we used custom persistant caches to keep things speedy.

The implementation of such a cache is fairly straightforward, pretty much a hash with a daemon running around it. During quiet moments the cache is cleaned with old nodes removed and/or flushed to disk/database. Connection is usually via a unix domain socket.

If you have your front-end scripts populating the cache on misses, then you can have a very simple and very fast single-threaded cache. It can be used with any of the other strategies (databases, disk files, DBM files, etc) mentioned above. It can pre-populate itself with information if desired.

I'd suggest you think twice before having your cache make HTTP connections to the remote server. While this seems very elegant from a design standpoint (ask cache, get back answer, don't care about hit or miss), it means you run the risk of your cache being blocked on HTTP, or seriously increasing the complexity of the code. A misbehaving cache can kill your overall performance when everything depends upon it.

Hope that your project gets well and that your customers are blinded by its speed. ;)

Cheers,
Paul

I will only touch on Q1 since I have no experience with what you want to do in Q2.. however it might well turn out that you don't need to go to such superb lengths of twisting your arms as your Q2 suggests, if the following notes work well enough for you, and I believe you'll see a huge difference.

First of all, a few bad mistakes in your code. :-( The biggest one being that you try to emulate a hash's functionality by using a string you run a regex over, instead of just using a hash. Rely on Perl to do the work for you; hash lookup is very fast and efficient and will do the job better than any crutch you may be able to think of.

• as has been mentioned, use substr() instead of a regex to extract the Y/N part
• It is un-Perlish and 99.9% of the time a bad idea to use the three-expression for(;;){}
• If you say if($confirmed_thumb){} if(!$confirmed_thumb){};, why not use an else?

local $_; # should always do this unless you know why you don't want i
+t

my %id_cache;
open (IMAGE_LIST, $image_list); # you didn't need quotes here..
while(<IMAGE_LIST>) {
    $id_cache{substr($_,1)} = substr($_,0,1) eq 'Y';
}
close (IMAGE_LIST);

open (IMAGE_LIST, ">>$image_list"); # although you do here
for (@thumb_id_list) {
    unless(exists $id_cache[$_]) {
        # IDs that didn't appear in the file won't exist in the hash
    # therefor we have to connect to partner website for them
        $id_cache[$_] = &check_image($session,$_);
        print IMAGE_LIST $id_cache[$_] ? "Y$_\n" : "N$_\n";
    }

    # $id_cache[$_] is now defined,
    # regardless of whether it was in the file or not
    push (@display_thumbs,$_) if $id_cache[$_];
}
close (IMAGE_LIST);
[download]

Now that's much shorter and clearer, no? It will also run a lot faster - even if from an algorithmic point of view it is still anything but satisfactory. (Note: you must take proper care of file locking, or you'll end up corrupting your image list file.) Since your IDs appear to be of fixed length, I could go into all sorts of optimizations possible with that knowledge, here, first and foremost being to store the keys in a binary file to get rid of the overhead of reading and parsing a textfile for every invocation.

However, I won't. Because I think you're stepping down the wrong road. You will have a problem with that approach very soon because you're slurping the entire list into memory from scratch every time. That will become slow very soon, but the killer argument is that your memory usage will go through the roof - something you should avoid at all cost for medium or higher traffic CGIs. A partial solution to the speed problem would be to use Storable to put the %id_cache into a binary file; this will be at least an order of magnitude faster than anything you are likely to be able to code yourself.

However, there's still the problem of memory usage. It would be better if you can check your keys from disk without reading the entire list into memory. Unfortunately that requires very clever storage in order to be fast, and coming up with a good way to do this is what the people at IBM and Oracle get paid a lot of money for - it is an extraordinarily tricky task. I would advise against the method that has been mentioned by others to abuse the file system as a database, because while that works, on just about every Unix filesystem the number of files you can create is limited; subdividing the key into a path with several small bits increases the problem a lot, while on the other hand keeping it as one single filename makes for very large directories that some filesystems are pretty slow to search.

Really, you should use a database.

Also, accumulate the unknown permission images rather than calling &check_image() on each, and use LWP::Parallel to check them at once. This should save your script a couple seconds of waiting for each reply in turn; just make sure you don't totally cripple the target server with a flood of requests (if memory serves, the module lets you define how many of the requests to fire off at once).

Hm. If you've got the authority to do it, you might well want to change from running a CGI on your client's site to somthing more persistant (like a mod_perl handler thingy), so you don't have to parse some out-of-core version of your permissionlist on every search.

Failing that (or even not failing that), you could also...

Go the multiple-files idea one further, on the theory that your OS's FS layer is probably smarter about caching then you are: since apparently all you really need to do is lookup each 48-bit (or so) result ID and get an answer of Allowed, Not Allowed, or Donno. (In other words, a one-trigit result).

Therefore, create a file like allowed-to-show-cache/12/39/89/83/98/12/93 with a length of 0 or 1 if you cannot or can show the file, and then use a -f test to see if you already know, and if that works, do a -s _ to see if you can forward the result's picture or not. This would be really good with somthing like reiserfs (IIRC), which is designed to be really efficent on directory-tree lookups like this by having some sort of natural tree structure. You never have to acatualy open the files at all. While this is probably larger then the other solution in terms of disk space, it involves less parsing.

There's probably something else none of us are seeing... but I don't know what.

Hmm...

Don't write a file-hashing thing like that yourself; use Cache::Cache which does it for you.

Just be carefull with not pounding the other servers with more requests than what it can handle.

I would say a Dual Pentium III 1000MHz can probably handle 50 hits per second as a top limit before active connections may start to accumulate, and eventually refuse new connections. This of course varies according to several factors, but it's probably a good gauge for a server running Apache on BSD or Linux.

i know i may be stating the obvious, but sometimes i have to do that to myself. (grin)