in reply to Simple JSON based data storage - what would you recommend?

I only need two things from the API - to be able to store a JSON string, and to be able to retrieve the most recent JSON string stored.

Presumably the views are accessed using some name or number.

It'd be pretty hard to beat your file system for performance for something as simple as this. Each view is simply an appropriately named file within a directory.

Have the background process write the json file to a temp directory, and then, when complete, delete the 'live' file and rename the new one into the live directory.

The foreground process simply slurps the named view from its file in the 'live' directory, and uses a (say) 1/10th second sleep-before-retry, if the file doesn't exist at the moment it tries to slurp it.

Assuming a relatively small (low hundreds or less) number of views; and relatively small files (say a few kb); any half decent file system will do a pretty good job of keeping the 'hot' views in cache and will take care of the 'refresh on write' situation automatically.

For such a simple use-case, anything else is pretty much overkill.

With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
"Science is about questioning the status quo. Questioning authority".
In the absence of evidence, opinion is indistinguishable from prejudice.
  • Comment on Re: Simple JSON based data storage - what would you recommend?

Replies are listed 'Best First'.
Re^2: Simple JSON based data storage - what would you recommend?
by blindluke (Hermit) on Oct 14, 2014 at 16:16 UTC

    Thank you. You're right - I was overthinking the whole thing. I'll update the files using File::Temp's tempfile and rename, and I'll store them under the same dir, with file name corresponding to the view name.

    Then, on the frontend, I'll just have a handler associated with the path '/data/viewname' that returns the viewname file contents (JSON).

    There is a blurt_atomic sub in Sysadm::Install that handles the tempfile/rename routine, I'll either roll my own in a similar way, or just use the module as an added dependency.

    Again, thank you, both for the time, and the good advice.

    - Luke

[reply]

      As long as the back-end is a single machine and not a load-balanced farm *and* the number of saved files is relatively small, this will work fine. A shared filesystems across a farm presents it's own set of challenges as does a large number of files in one directory. For load-balanced farms, I would pin the service to one machine and if the number of files is expected to be greater than 10K, I would use some type of directory partitioning scheme to keep the number of files in a directory low (for those times when you need to manually read the dir).

      -derby
[reply]

In Section Seekers of Perl Wisdom