Re^5: is Sqlite db good enough to handle large data.

In selecting SQLite you are giving up the flexibility for simplicity. We don't know your application as well as you do. Your job as a software engineer highly qualified to design a product worth buying, and capable of keeping your clients happy, is to research what the various alternatives can provide, and to make the decision based on the total suitability of the solution you choose. Someone already provided links to SQLite's documentation, which I've read in depth a few times in the past 90 days. I know that the documentation spells out pretty clearly what constitute reasonable use cases and what constitute blockers. Now find discussions on PostgreSQL, MySQL, and other alternatives so that you can make the informed decision.

At this point we've strayed pretty far off of the original question; whether or not SQLite can handle big data. We've identified that's not really the question to be asking. And none of this is specific to Perl. Interesting to Perl developers, sure. But it seems like you really want the answer to be yes. It would be irresponsible of us to green light a decision that should be made with a lot more details than would be appropriate to raise here in this forum.

You have read https://www.sqlite.org/whentouse.html, right? Digital Ocean has a lot of great information in its support documents. Check out this one: SQLite vs MySQL vs PostgreSQL: A comparison of relational database management systems.

Dave

Comment on Re^5: is Sqlite db good enough to handle large data.

Replies are listed 'Best First'.
Re^6: is Sqlite db good enough to handle large data. by bliako (Abbot) on Jul 22, 2019 at 11:12 UTC
Is there a higher level DB API you recommend so that the program keeps the same if/when changing to another DB?	[reply]
Re^7: is Sqlite db good enough to handle large data. by Your Mother (Archbishop) on Jul 22, 2019 at 13:50 UTC
DBIx::Class fits the bill largely. As long as one doesn't embed incompatible raw SQL and understands the limitations in SQLite. SQLite is somewhat easy to mock simplistic functionality for other engines too. Here is a snippet I use to help DBD::SQLite pretend to be MySQL for testing– `if ( $dbh->{Driver}->{Name} eq "SQLite" ) { require Date::Calc; $dbh->sqlite_create_function("NOW", 0, sub { sprintf('%d-%02d-%02d %02d:%02d:%02d', Date::Calc::Today_a +nd_Now()); }); require Math::BigInt::Random; $dbh->sqlite_create_function("RAND", 0, sub { Math::BigInt::Random::random_bigint( min => -9223372036854 +775808, max => 9223372036854775807 ); }); }` [download] Update: I want to mention because it likely seems I am saying the opposite: MySQL is half-garbage and I would never choose it unless constrained to do so by managers, tools, support… PostgreSQL and MariaDB (a MySQL fork that tries to correct a lot of the idiotic defaults and such) are better choices for open source RDBMS.	[reply] [d/l]
Re^7: is Sqlite db good enough to handle large data. by erix (Prior) on Jul 22, 2019 at 20:44 UTC
The DB API is SQL	[reply]
Re^8: is Sqlite db good enough to handle large data. by bliako (Abbot) on Jul 23, 2019 at 10:05 UTC
Agreed. I meant how to avoid using db-specific SQL in our scripts and let a module do that instead. For example the use of `sqlite_create_function()` in YourMother's response.	[reply] [d/l]