Beefy Boxes and Bandwidth Generously Provided by pair Networks
go ahead... be a heretic
 
PerlMonks  

Re: Simple data-store with Perl

by NERDVANA (Pilgrim)
on Nov 12, 2021 at 20:17 UTC ( #11138764=note: print w/replies, xml ) Need Help??


in reply to Simple data-store with Perl

You might also be interested in LMDB_File, the "Lightning DB" from OpenLDAP project. It takes more code to use it, and it isn't pure-perl, but it supports transactions and uses memory-mapping to be able to load very large scalars without read/write loops. It uses filesystem locks to support concurrent transactions.

Replies are listed 'Best First'.
Re^2: Simple data-store with Perl
by Tux (Canon) on Nov 13, 2021 at 11:07 UTC

    For modules that map a hash to disk, LMDB_File is not the fastest though.

    Size op perl GDBM NDBM ODBM SDBM DB +_File CDB_File LMDB_File ------ -- ---------- ---------- ---------- ---------- ---------- ----- +----- ---------- ---------- 20 rd 1666666.7 4419.9 8025.7 8650.5 6004.2 4 +342.2 4396.6 6968.6 20 wr 1818181.8 5383.6 9058.0 11160.7 9049.8 4 +965.2 6583.3 7479.4 200 rd 1754386.0 30599.8 53092.6 57388.8 63938.6 28 +835.1 55157.2 37828.6 200 wr 2409638.6 34764.5 67181.7 60024.0 49627.8 36 +832.4 124223.6 57028.8 600 rd 1785714.3 92081.0 100654.3 110436.2 130378.1 91 +185.4 150564.6 63151.2 600 wr 2857142.9 83114.0 116369.3 128287.4 76036.0 92 +692.7 307219.7 108303.2 2000 rd 1806684.7 221877.1 186968.3 189969.6 200541.5 157 +257.4 315059.9 74881.1 2000 wr 2656042.5 217793.7 188076.0 206015.7 96459.9 219 +442.6 623830.3 146166.8 20000 rd 1225565.3 217185.9 174087.1 162620.1 164974.3 132 +289.1 418550.1 74129.5 20000 wr 1623640.2 176447.8 160959.3 156257.3 107578.9 134 +935.5 800768.7 139251.5 200000 rd 528887.9 162823.0 147817.9 132551.1 131489.6 107 +172.2 373562.3 67801.9 200000 wr 656525.5 83695.0 82843.5 84043.8 101899.1 79 +178.1 587721.9 131520.2

    Higher is better/faster.

    Maybe CDB_File is an option when doing a simple tie

    Considering other options:

    Size op perl BerkeleyDB KyotoCab Redis Redis2 Redi +sFast RedisFast2 ------ -- ---------- ---------- ---------- ---------- ---------- ----- +----- ---------- 20 rd 800000.0 4059.3 4412.1 3667.7 4538.2 4 +990.0 5217.8 20 wr 714285.7 4176.2 2523.3 3082.6 4170.1 4 +747.2 4873.3 200 rd 1169590.6 24746.3 26430.6 8324.3 9318.8 12 +487.5 12046.7 200 wr 2409638.6 29726.5 17913.1 8275.1 8682.4 11 +898.4 11308.4 600 rd 2068965.5 76511.1 92307.7 9424.6 8761.3 14 +136.9 12960.9 600 wr 2898550.7 72202.2 57066.8 10471.6 11069.5 14 +948.1 13962.6 2000 rd 1881467.5 144009.2 138956.4 10658.7 9824.3 14 +306.4 13094.4 2000 wr 2638522.4 178269.0 154547.6 10715.2 10597.8 15 +203.1 14677.6 20000 rd 1225940.9 146482.6 129354.4 9337.7 9300.4 12 +765.2 12690.3 20000 wr 1474600.0 130359.4 242283.3 10723.9 10503.4 14 +652.1 13632.5 200000 rd 530653.2 119285.1 123884.2 9061.5 9004.3 13 +044.8 12572.5 200000 wr 637429.9 74454.9 228455.5 10342.3 10269.5 14 +175.5 14034.8

    Note that KyotoCabinet::DB is not on CPAN.

    Comparing LMDB_File to its direct competitors (adding another option: tie to an DBD::SQLite database with Tie::Hash::DBD):

    Size op perl SDBM CDB_File LMDB_File KyotoCab S +QLite ------ -- ---------- ---------- ---------- ---------- ---------- ----- +----- 20 rd 1666666.7 7423.9 5614.8 6700.2 3478.9 7 +886.4 20 wr 1666666.7 7806.4 13333.3 7902.0 2206.5 9 +775.2 200 rd 1351351.4 62695.9 52952.1 39463.3 33984.7 45 +228.4 200 wr 1709401.7 47494.7 134228.2 50314.5 15725.7 43 +131.3 600 rd 2047781.6 138153.4 144092.2 66386.4 92893.6 66 +217.9 600 wr 2884615.4 80699.4 281558.0 113938.5 70863.4 59 +970.0 2000 rd 1901140.7 216849.2 297176.8 78929.7 119911.3 84 +613.1 2000 wr 2649006.6 96098.4 575539.6 159083.7 143502.9 70 +526.8 20000 rd 1274210.0 228574.0 509100.2 80934.3 109472.1 73 +961.5 20000 wr 1649348.5 114533.8 863036.2 148118.5 198214.1 71 +024.7 200000 rd 534996.8 169792.5 485755.2 73310.5 107320.8 50 +125.8 200000 wr 661559.4 95249.5 673387.1 139133.0 218396.9 50 +760.9

    Enjoy, Have FUN! H.Merijn
      Interesting, I hadn't seen KyotoCab before. But are you sure about these numbers? Most are showing writes faster than reads, and big writes faster than small writes?

      Edit: Oh, and CDB_File doesn't seem like it really compares with the rest because (from the docs, at least) it sounds like you have to create a complete new file to make any changes.

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: note [id://11138764]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others perusing the Monastery: (3)
As of 2022-01-27 00:30 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?
    In 2022, my preferred method to securely store passwords is:












    Results (70 votes). Check out past polls.

    Notices?