Beefy Boxes and Bandwidth Generously Provided by pair Networks
Don't ask to ask, just ask
 
PerlMonks  

Simple data-store with Perl

by bliako (Monsignor)
on Nov 10, 2021 at 12:38 UTC ( #11138687=CUFP: print w/replies, xml ) Need Help??

I was in need of a data-store for clients to read and write short blobs (Edit: i.e. share data between them, that data can be simple strings, including json, or binary data. If the clients are in Perl, serialised nested perl data structure is also a possibility). A shared-memory hashtable seems ideal in my case. A DB seems far-fetched, especially because I need to keep these operations ultra-fast and do not need persistency, ACIDity, and frankly, neither I want to think how to do SQL which invariably, in my case, is googled out and ends up a hit-and-miss and inefficient. OTOH, building a C shared-memory framework with IPC from scratch will take long time. I am leaning towards using Redis especially because it has a C api as my app is in C. It also has a Perl api.

But here is a Pure Perl solution (yet another example of programming sitting on the hunched shoulders of Giants). It seems very fast to me with 100_000 inserts in 0.35 secs within same machine:

# simple-perl-server.pl package MyPackage; # base reapped from (with thanks): # https://metacpan.org/pod/Net::Server use base qw(Net::Server); my %Store; my $port = shift or die "No port\n"; sub process_request { my $self = shift; while (<STDIN>) { s/[\r\n]+$//; if( /^(.+?)=(.+?)$/ ){ my $x = $1; my $y = $2; #print "Added '$x' => '$y'\015\012"; print STDERR "Added '$x' => '$y'\n"; $Store{$x} = $y; } elsif( /^(.+?)=$/ ){ my $x = $1; if( exists $Store{$x} ){ print "'$x' => '".$Store{$x}."'\015\012"; } else { print "'$x' => <undef>\015\012"; } } elsif( /quit/i ){ print STDERR "$0 : quitting ...\n"; exit(0); } } # while <STDIN> } MyPackage->run(port => $port, ipv => '*');
# simple-perl-client.pl # EDIT: this has been edited an hour after posting to set N=100_000 # time reported is correct though # mostly from here: # https://codereview.stackexchange.com/questions/106421/network-chat +-in-perl # and here: # https://gist.github.com/chankeypathak/1b1b9b3a27799eb5e277 use strict; use warnings; use IO::Socket::INET; use Time::HiRes qw/gettimeofday/; my $N = 100_000; # number of inserts my $start_time = Time::HiRes::gettimeofday(); my $server = shift or die "No server\n"; my $port = shift or die "No port\n"; my $client_socket = IO::Socket::INET->new( PeerPort => $port, PeerAddr => $server, Proto => 'tcp' ) or die "Can't create send socket: $!!\n"; print "Connected to $server:$port!\n"; my $child; if($child = fork) { while( 1 ){ my $received = <$client_socket>; exit unless defined $received; print $received; } } die "fork: $!\n" unless defined $child; # set print "$0 : doing $N sets ...\n"; for my $i (1..$N){ $client_socket->send("x$i=$i\n"); } print "$0 : done $N sets.\n"; print "$0 : doing $N gets ...\n"; # get for my $i (1..$N){ $client_socket->send("x$i=\n"); } print "$0 : done $N gets.\n"; my $end_time = Time::HiRes::gettimeofday(); print "Closing connection ...\n"; $client_socket->send("quit\n\n"); close $client_socket; sleep(1); print "$0 : done $N sets/gets in ".($end_time-$start_time)." seconds.\ +n";

Run the server as: perl simple-perl-server.pl 16001

Run the client as: perl simple-perl-client.pl 127.0.0.1 16001

It's quite fast: 0.345 secs for 100_000 inserts, (out-of-the-box redis needed 10x that)

bw, bliako

Replies are listed 'Best First'.
Re: Simple data-store with Perl
by NERDVANA (Pilgrim) on Nov 12, 2021 at 20:17 UTC
    You might also be interested in LMDB_File, the "Lightning DB" from OpenLDAP project. It takes more code to use it, and it isn't pure-perl, but it supports transactions and uses memory-mapping to be able to load very large scalars without read/write loops. It uses filesystem locks to support concurrent transactions.

      For modules that map a hash to disk, LMDB_File is not the fastest though.

      Size op perl GDBM NDBM ODBM SDBM DB +_File CDB_File LMDB_File ------ -- ---------- ---------- ---------- ---------- ---------- ----- +----- ---------- ---------- 20 rd 1666666.7 4419.9 8025.7 8650.5 6004.2 4 +342.2 4396.6 6968.6 20 wr 1818181.8 5383.6 9058.0 11160.7 9049.8 4 +965.2 6583.3 7479.4 200 rd 1754386.0 30599.8 53092.6 57388.8 63938.6 28 +835.1 55157.2 37828.6 200 wr 2409638.6 34764.5 67181.7 60024.0 49627.8 36 +832.4 124223.6 57028.8 600 rd 1785714.3 92081.0 100654.3 110436.2 130378.1 91 +185.4 150564.6 63151.2 600 wr 2857142.9 83114.0 116369.3 128287.4 76036.0 92 +692.7 307219.7 108303.2 2000 rd 1806684.7 221877.1 186968.3 189969.6 200541.5 157 +257.4 315059.9 74881.1 2000 wr 2656042.5 217793.7 188076.0 206015.7 96459.9 219 +442.6 623830.3 146166.8 20000 rd 1225565.3 217185.9 174087.1 162620.1 164974.3 132 +289.1 418550.1 74129.5 20000 wr 1623640.2 176447.8 160959.3 156257.3 107578.9 134 +935.5 800768.7 139251.5 200000 rd 528887.9 162823.0 147817.9 132551.1 131489.6 107 +172.2 373562.3 67801.9 200000 wr 656525.5 83695.0 82843.5 84043.8 101899.1 79 +178.1 587721.9 131520.2

      Higher is better/faster.

      Maybe CDB_File is an option when doing a simple tie

      Considering other options:

      Size op perl BerkeleyDB KyotoCab Redis Redis2 Redi +sFast RedisFast2 ------ -- ---------- ---------- ---------- ---------- ---------- ----- +----- ---------- 20 rd 800000.0 4059.3 4412.1 3667.7 4538.2 4 +990.0 5217.8 20 wr 714285.7 4176.2 2523.3 3082.6 4170.1 4 +747.2 4873.3 200 rd 1169590.6 24746.3 26430.6 8324.3 9318.8 12 +487.5 12046.7 200 wr 2409638.6 29726.5 17913.1 8275.1 8682.4 11 +898.4 11308.4 600 rd 2068965.5 76511.1 92307.7 9424.6 8761.3 14 +136.9 12960.9 600 wr 2898550.7 72202.2 57066.8 10471.6 11069.5 14 +948.1 13962.6 2000 rd 1881467.5 144009.2 138956.4 10658.7 9824.3 14 +306.4 13094.4 2000 wr 2638522.4 178269.0 154547.6 10715.2 10597.8 15 +203.1 14677.6 20000 rd 1225940.9 146482.6 129354.4 9337.7 9300.4 12 +765.2 12690.3 20000 wr 1474600.0 130359.4 242283.3 10723.9 10503.4 14 +652.1 13632.5 200000 rd 530653.2 119285.1 123884.2 9061.5 9004.3 13 +044.8 12572.5 200000 wr 637429.9 74454.9 228455.5 10342.3 10269.5 14 +175.5 14034.8

      Note that KyotoCabinet::DB is not on CPAN.

      Comparing LMDB_File to its direct competitors (adding another option: tie to an DBD::SQLite database with Tie::Hash::DBD):

      Size op perl SDBM CDB_File LMDB_File KyotoCab S +QLite ------ -- ---------- ---------- ---------- ---------- ---------- ----- +----- 20 rd 1666666.7 7423.9 5614.8 6700.2 3478.9 7 +886.4 20 wr 1666666.7 7806.4 13333.3 7902.0 2206.5 9 +775.2 200 rd 1351351.4 62695.9 52952.1 39463.3 33984.7 45 +228.4 200 wr 1709401.7 47494.7 134228.2 50314.5 15725.7 43 +131.3 600 rd 2047781.6 138153.4 144092.2 66386.4 92893.6 66 +217.9 600 wr 2884615.4 80699.4 281558.0 113938.5 70863.4 59 +970.0 2000 rd 1901140.7 216849.2 297176.8 78929.7 119911.3 84 +613.1 2000 wr 2649006.6 96098.4 575539.6 159083.7 143502.9 70 +526.8 20000 rd 1274210.0 228574.0 509100.2 80934.3 109472.1 73 +961.5 20000 wr 1649348.5 114533.8 863036.2 148118.5 198214.1 71 +024.7 200000 rd 534996.8 169792.5 485755.2 73310.5 107320.8 50 +125.8 200000 wr 661559.4 95249.5 673387.1 139133.0 218396.9 50 +760.9

      Enjoy, Have FUN! H.Merijn
        Interesting, I hadn't seen KyotoCab before. But are you sure about these numbers? Most are showing writes faster than reads, and big writes faster than small writes?

        Edit: Oh, and CDB_File doesn't seem like it really compares with the rest because (from the docs, at least) it sounds like you have to create a complete new file to make any changes.

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: CUFP [id://11138687]
Front-paged by Arunbear
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others perusing the Monastery: (6)
As of 2022-01-20 10:42 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?
    In 2022, my preferred method to securely store passwords is:












    Results (56 votes). Check out past polls.

    Notices?