in reply to Re: is Sqlite db good enough to handle large data.
in thread is Sqlite db good enough to handle large data.

Basically, I am writing desktop application for retailers which will manage buy and sales Information for them. so wondering if sqlite db is ideal for the same.

  • Comment on Re^2: is Sqlite db good enough to handle large data.

Replies are listed 'Best First'.
Re^3: is Sqlite db good enough to handle large data.
by davido (Cardinal) on Jul 21, 2019 at 18:14 UTC

    SQLite is not ideal if you need multiple consumers of your application. What I mean is, if you have several retail locations trying to share the database, it's more difficult to do right. A serverless database is really just intended for in-process, on-system use, not for multi-process, many-system use. SQLite also is difficult to use in situations where you need master/replica configurations for redundancy or shared load, live backups, row locking, and so on.

    When you talk about whether SQLite is appropriate for big data you are only considering one of the factors that would go in to deciding if it is an appropriate choice. And you've got a vision of how it will fit into your big data plan that differs from what we may be thinking. It can certainly handle large amounts of data. And it may have a place in certain parts of big data system.

    But with the very limited information you're providing, I would suggest that no, it's not ideal. Why? Because one retail location could become two, or five, or twenty five, or five hundred. Why should each location have a local database? It could make sense for each location to cache information, or to keep track of information ephemerally, and in such a case SQLite might be fine. There are thousands of uses of SQLite. But the moment you start thinking in terms of a central repository for multiple applications or multiple instances or installations of an application to access, SQLite runs out of gas.

    On the other hand, perhaps you're standing up a service that multiple locations can access, and that service you're contemplating backing with SQLite for your data. Again, maybe not ideal; how will you handle running multiple instances of your application as traffic from multiple locations grows beyond what a single application worker can handle? How will you deal with live backups in a 24/7 world? How will you handle hot promotion of replicas to master should master ever go down? You wouldn't want mission critical data deployed alongside a single application instance on a single piece of hardware; 1990 is 29 years behind us, and we've learned a lot about scalability and redundancy since then.

    So while SQLite can handle tons of data, it may not provide flexibility and scalability in areas that have nothing to do with the size of the data set.

    I love SQLite, and recently used it in a system that comprises thousands of servers. But in this case the servers each do some job and store in local SQLite databases batches of information that later get posted up to a service that receives from each of those thousands of servers. The SQLite databases are ok, in this case, being single-server instances; there's no need for them to become a shared resource, and there's no competition from other processes to simultaneously write into SQLite. So it was a good choice; minimal, nothing to maintain, almost nothing to install.

    It sounds like the application you are contemplating has no current expectation of growing beyond a single instance. But in a retail world I can't imagine that would be a good plan for any business that may someday grow. Admittedly I know nothing about the application you're considering. But I do have some retail experience: the first half of my career was as a retail buyer for a chain of stores that was at one time a dozen locations, and later 80, and later hundreds. Solutions that would have worked for one store would never have worked for a dozen. And solutions that worked for a dozen would never have been able to scale to 80, let alone hundreds. Small business, mom and pop stores sometimes do grow beyond their single location, and if an application cannot support that, it is targeting the specific narrow niche of businesses that are content not to grow.


    Dave

      Hi Dave,
      Thanks for the detailed information.
      I am targeting small stores to begin with, I mean one store would be having single application per desktop.
      So, can we consider sqlite db in that case.

        In selecting SQLite you are giving up the flexibility for simplicity. We don't know your application as well as you do. Your job as a software engineer highly qualified to design a product worth buying, and capable of keeping your clients happy, is to research what the various alternatives can provide, and to make the decision based on the total suitability of the solution you choose. Someone already provided links to SQLite's documentation, which I've read in depth a few times in the past 90 days. I know that the documentation spells out pretty clearly what constitute reasonable use cases and what constitute blockers. Now find discussions on PostgreSQL, MySQL, and other alternatives so that you can make the informed decision.

        At this point we've strayed pretty far off of the original question; whether or not SQLite can handle big data. We've identified that's not really the question to be asking. And none of this is specific to Perl. Interesting to Perl developers, sure. But it seems like you really want the answer to be yes. It would be irresponsible of us to green light a decision that should be made with a lot more details than would be appropriate to raise here in this forum.

        You have read https://www.sqlite.org/whentouse.html, right? Digital Ocean has a lot of great information in its support documents. Check out this one: SQLite vs MySQL vs PostgreSQL: A comparison of relational database management systems.


        Dave

        taht q was already answered