Beefy Boxes and Bandwidth Generously Provided by pair Networks
P is for Practical
 
PerlMonks  

Storable for user preferences

by Bod (Parson)
on Sep 30, 2023 at 18:17 UTC ( [id://11154774] : perlquestion . print w/replies, xml ) Need Help??

Bod has asked for the wisdom of the Perl Monks concerning the following question:

I need to store user preferences...

Currently, there are only two user preferences and these are stored as fields in the User table of the database. However, the number of preferences is going to increase. I guess there will be around 20 or so eventually. Experience suggests that most users won't change most of their preferences from the default setting. Therefore, I don't really want to add all the preferences to the User table when nearly all of them will be set to the default values.

I could implement the preferences in the database with an EAV model. But I wondered if Perl's core Storable might be a better solution.

I've not used Storable before so I'd appreciate the monastic wisdom about any issues I might face using Storable generally and it's suitability for this specific application...

Replies are listed 'Best First'.
Re: Storable for user preferences
by NERDVANA (Deacon) on Sep 30, 2023 at 23:23 UTC

    The "Relationally Correct" approach is to create one column for each preference. This gives you efficient fetching of all the prefs with a single select query, it lets you write convenient queries to see which users are using which options, it lets you run aggregate queries across the preferences, and so on. It also lets you declare an accurate data type for each preference. You don't need to put all the columns in the users table; you could make a 1:1 relation to a table of user-preferences if that keeps your code cleaner.

    The downside from a programming perspective is that it means you need a schema change every time you need a new preference. I like to argue that the correct solution here is to make schema changes painless so that you can add columns as often as you like. It generally means you spend a lot of effort on tooling, but I think the results are worth it in the end.

    EAV is the second option. The upside is you don't need to change your schema, ever. The downsides are that you have to write *much* more complicated queries to do all the normal relational things with your data. You also have to decide whether to correctly declare the data types (meaning multiple EAV tables or multiple 'value' columns) or just make everything a string, resulting in a lower quality data model. You may run into limits of your DB engine like how many joins are allowed in a single query, or you may have to write code that selects multiple rows to get one user's worth of attributes. The more EAV you use, the more effort you have to spend designing the code that runs the queries, and you can easily end up with performance problems. (though probably not on something as small as user preferences)

    The third option is serialization. I used to recommend against this because it takes power away from SQL (e.g. selecting users where a preference is set to some specific value, comparing the number of users who use preference A vs. preference B, running a query to see if *any* user has selected some invalid option for preference X, performing a schema update where preference Y has been removed, and so on), but now most database engines let you reach into JSON data with special keywords! So, I would still recommend against Storable, but if you are using Postgres, use a "json" or "jsonb" column. If you decide you want to run fancy SQL queries later, you can learn the functions that let you access fields of the json and run the query in the database instead of needing to fetch *every* user to perform those kinds of queries.

    In short, I recommend either continuing with one column per setting, and improve your schema-update scripts and workflows, or the jsonb option if you are using Postgres.

    The specific job you're doing now is sort of low-importance for which method you choose, but if you choose one of the better options it gives you experience for the next time this problem comes up on some more critical/large-data scenario.

      In short, I recommend either continuing with one column per setting, and improve your schema-update scripts and workflows, or the jsonb option if you are using Postgres.

      Thanks for your detailed answer which confirms a lot.

      I'm using MariaDB but that has pretty decent JSON handling.

Re: Storable for user preferences
by afoken (Chancellor) on Oct 01, 2023 at 10:20 UTC

    I would avoid Storable simply because it depends on the Perl version and the perl configuration. That would be acceptable for short-term, temporary storage (say up to a few days), but not for long-term storage. Sooner or later, you will update perl, and that may get you in trouble with Storable. Another problem is that Storabe uses a binary format, which makes it harder to debug.

    The common text-based formats JSON, XML, INI, and YAML have specs that do not depend on Perl at all. Upgrading perl won't be a problem. And of course, those formats are more or less human readable.

    Some relational databases have added support for data in JSON and/or XML format, so accessing JSON/XML data from within SQL is possible and should be quite efficient. I assume that you need the set of user preferencs only in Perl, not in SQL. So simply serializing your user preferences to a JSON or XML string and deserializing them back should work even without JSON/XML support from the database. In general, you would load the user preferences on login and keep them for the entire session. You write them only if the user changes some user preferences. Parsing a little bit of JSON or XML at login should not hurt much. And as a nice extra, the JSON / XML containing the user preferences is just a bit of text for the database. Storing and retrieving text works with every relational database, it keeps your code independant from the actual database.

    Alexander

    --
    Today I will gladly share my knowledge and experience, for there are no sweeter words than "I told you so". ;-)
      I would avoid Storable simply because it depends on the Perl version and the perl configuration

      That seals the decision...thank you.

      As I said in the question, I've not used Storable. I didn't know it was version-dependent. As such, it's not suitable for my needs here.

      ADDED:
      The database is MariaDB, which has a perfectly capable set of JSON functions.

Re: Storable for user preferences
by swl (Parson) on Sep 30, 2023 at 21:39 UTC

    Storable is very good but might be more than is needed for this use case.

    If the preferences are all numeric and text values then it might be simpler to use a text based config format. That way users can see and edit their config values using a text editor. There is always the risk they will make a mess of the formatting but you could always add big warnings to the top of the file so it is caveat emptor if they do. (Edit: And fall back to system defaults with a warning when that happens.)

    See for example Config::Any for a list of options, e.g. YAML, JSON or INI. Best to avoid the Perl format since it is loaded using do {} so will run any code in the file and is thus a potential security issue.

      Was going to reply with something similar so a strong "seconded" to the above. If you’ve got a large chunk mod;//Storable may win out performance wise, but for a user config YAML or JSON are going to be better choices (human editable plain text FTW).

      The cake is a lie.
      The cake is a lie.
      The cake is a lie.

      That way users can see and edit their config values using a text editor

      Our users would struggle to use a test editor! The only way the preferences will be updated is from a Perl script under my control prompted by user input on a webpage.

      I did consider using JSON or similar, but that means reading in the entire set of preferences every time I want to access just one of them. Not a deal breaker, but it would probably be better if I didn't have to. Yes, te preferences will be all integers and perhaps strings, mostly just binary options so I shall have a look at Config::Any as a solution. thanks.

        I shall have a look at Config::Any as a solution

        If you are using some kind of web environment (e.g. CGI), be VERY careful with write access to the configuration. You need to ensure that no process reads the config file while it is rewritten, or else some settings may get lost. You also need to ensure that at most one process writes the config file at any time. Both can be handled with file locking and atomic renames, but it is easy to implement wrong. It is WAY easier and safer to put the configuration in the database that you already use.

        Alexander

        --
        Today I will gladly share my knowledge and experience, for there are no sweeter words than "I told you so". ;-)

        Not a problem. Reading the other posts on the thread it seems the prefs will be stored centrally anyway, in which case the database approaches would seem the better fit.

Re: Storable for user preferences
by hippo (Bishop) on Sep 30, 2023 at 22:45 UTC

    Storable is fine but it isn't clear to me what the benefits would be here. I would go with EAV: stick with the database but have a preferences table which consists of a user ID, a preference name and a preference value. When you query the preferences table for a combination of user and preference name and there is no matching row then you can use the system-wide default value. Why would this not be simple/efficient?


    🦛

Re: Storable for user preferences
by kcott (Archbishop) on Oct 01, 2023 at 05:22 UTC

    G'day Bod,

    There's three basic types you might consider (possibly others that didn't come to mind when I wrote this):

    For scalability and efficiency, a database solution is probably the best. That's based on the 2 to 20 or so preferences that you mentioned; assumes a number of users, now or in the future, totalling in the 100s, 1000s, or more; and also assumes preferences are stored server-side, not client-side. I'd have different advice if those assumptions are incorrect — let us know.

    You will need an admin application to create, add, delete and modify the default preferences. It should also be capable of doing the same for user preferences.

    In your application which uses the preferences, you'll need to access the default preferences; you only need to access user preferences if they exist. The method of access will, of course, vary; however, you'd probably end up with something like:

    my %prefs = (%$default_prefs, %{$user_prefs // {}});

    With database storage, you can SELECT just the data you want; for instance, give me "default" and "userX" data.

    With Storable, you'd need something like:

    my $all_prefs = retrieve('preferences.sto'); my $default_prefs = $all_prefs->{default}; my $user_prefs = $all_prefs->{userX};

    With other types of storage, you similarly need to read/load/retrieve a file, then extract the data you want from that.

    When changing a preference, you just UPDATE a datum in the database; with Storable, you change a datum in a hash, then rewrite an entire file.

    It's possible that there are other, more sophisticated methods with which I'm unfamiliar. See what others have written; there may be better solutions.

    — Ken

      For scalability and efficiency, a database solution is probably the best. That's based on the 2 to 20 or so preferences that you mentioned; assumes a number of users, now or in the future, totalling in the 100s, 1000s, or more; and also assumes preferences are stored server-side, not client-side. I'd have different advice if those assumptions are incorrect — let us know.

      G'day Ken,

      You are spot on with your assumptions 😀

      Yes, you are right. The database solution is going to be right for this purpose. I had come across Storable mentioned a few times recently both here and in other places and thought it might be a good solution as it keeps the preferences in Perl. But I am not keeping any of the other data within Perl that is held in a database so why shouldn't preferences?

      Perhaps it was shiny penny syndrome...

Re: Storable for user preferences
by ikegami (Patriarch) on Oct 04, 2023 at 00:29 UTC

    Avoid Storable. JSON and YAML are much better options for structured data of this sort.

Re: Storable for user preferences
by stevieb (Canon) on Oct 02, 2023 at 07:15 UTC

    At this time, I don't have time to read what are probably very valuable responses, so I must respond to you in the blind based solely on my own experiences.

    Storable is an exceptionally useful and pretty darned efficient and very effective method of serialization of Perl data. In fact, it's the primary storage mechanism for the IPC::Shareable distribution that allows one to share extremely complex Perl structures in memory between processes that I've managed for several years now. In fact, in years of testing, I haven't found anything comparable. Have I found faster? Yes. More capable? Yes. More resilient? Yes. More able for my very specific usage case? No.

    And that's what I leave you with. No. Don't use Storable for your use case. There are far better alternatives. Many of them. Even (by my own testing) a JSON string to a DB is likely preferable than using a proprietary serialization mechanism.

    Storable is a format that you'll have to load from disk (or memory) each time you need read access, and you'll have to write that back each time. There are significant proc cycles used to do this, far more than DB writes. This is compounded exponentially if you store all settings in one Storable file, and if you do, contention issues are a bloody nightmare. If you keep each user in their own file, well, as you can imagine, that has I/O headaches all on its own, and can itself make things unscalable quickly for the CPU and your poor disk array.

    Then you have to deal with compatibility. Not all Storable formats are compatible with one another. The next version of perl could be included with an update that renders all previous saved data as unavailable or inaccessible. This may not be relevant now, but what if you need to restore data from tape later?

    Use a common format to store your data. Sounds like a DB is the correct approach. If it needs to be serialized, use a common, cross-language standard. Storable is a fantastic resource, but not for this case.

    Besides... it doesn't seem like you need to store data *structures* anyways, just data, so in reality there's no need for such complex overhead such as Storable.

    The "what ifs" are too many, and there's no complexity to the data. Just store it in a DB (or JSON or equivalent manner) and be done with it. Don't compound things by introducing an unneeded technology that may confound you, or your successors in the future. If you can't use a DB, I'd advise a JSON-file-per-user approach long before I'd suggest a proprietary serialization mechanism such as Storable without hesitation for your needs.

    My opinion, based on extensive experience,

    -stevieb