YordanGeorgiev has asked for the wisdom of the Perl Monks concerning the following question:

I want to implement "hot reload" of the tables and columns meta-data in the following project:

qto - github
https://qto.fi - click just login to get to the ui ...

The page life cycle / control flow goes as follows: 

 - user requests a page
 - the Mojoicious framework builds the "empty templates" and servers t
+hem to the browser
 - once loaded to the browser, client side axios http requests fetch t
+he data from the back-end
 - vue does the data-binding on the client side by using BOTH the data
+ for the page and the global tables and columns meta-data

[download]

by not-breaking however the horizontal scalability provided the world's best framework Mojolicious

the variant would be to have the following flow of events: 

 - 1) alter table in postgres 
 - 2) reload a metatable from UI ( to tell to the Application Layer to
+ reload the meta-data)
 - 3) the meta data hash ref of hash refs is accessible to all the per
+l processes

[download]

The current implementation does not implement this "hot reload" to all the hypnotoad processes ... which breaks the listings in the UI, because they are meta-data "aware" ...

Could it be wise to complicate the current setup by adding Redis with publish subscribe mechanism for the meta-data only ?

Could it be implemented by some kind of interprocess signalling / communication - one proc reloads the meta-data and somehow signlas to the others ... as at the end those DDL changes on the db site will probably not occur so often ...

Could some kind of quick hack not requiring a lot of work be implemented ?

Replies are listed 'Best First'.
Re: multiprocess run-time data structure sharing in Mojolicious web app ...
by 1nickt (Canon) on Jan 29, 2020 at 20:18 UTC

    Hi,

    I don't understand "reload a metatable from UI ( to tell to the Application Layer to reload the meta-data)" ... does that mean that the update is triggered by a user client action?

    Anyway it doesn't matter: yes, I think you should use Redis caching, and I don't think you have to make it as complicated as pub/sub. Just update the cached values in Redis when they change in the DB. If your Mojo workers persist for many requests, you can combine with a hynotoad -s restart, which would gracefully kill any serving requests and spawn new ones reading from the updated cached values.

    Both Redis and the Perl Redis interface are very stable. We use them extensively in our busy Mojo apps at $work.

    Edit: For multi-process data sharing in general, see MCE and MCE::Shared.

    Hope this helps!


    The way forward always starts with a minimal test.
[reply]
[d/l]

      Thank you !!!

      *** yes, I think you should use Redis caching, and I don't think you have to make it as complicated as pub/sub. Just update the cached values in Redis when they change in the DB

      I guess this will be the first iteration of the implementation, which will be triggered once the list/meta_tables is requested only ... via a simple Redis wrapper with setMeta and getMeta subs, which will first serialize the hash ref of hash refs ( with Data::Dumper or Storable ) and set it to redis , than desirialize it and serve to clients ... when requested ...

      *** Both Redis and the Perl Redis interface are very stable

      Glad to hear that - the most important aspect, I forgot to ask even ...

      *** For multi-process data sharing in general, see MCE and MCE::Shared ...

      I guess multi-process sharing will not make sense architecturally-wise ... if one would like to use later one multiple application layer boxes ...

      *** Hope this helps!

      Yes, indeed. Many Thanks ! I will post a git commit-hash link in this discussion later on ... once implemented ...

[reply]

        Once again Thank you for the professional advice!

        I ended up I guess with less or more the same implementation you suggested:

        Basically the Application Layer code fetches the meta data during start-up or some meta-tables and stores into into Redis, after which all calls fetch it from there ...

        here is the fetch

        here is the store

        and the unit test: and the unit test

        The lessons learned from this release was the huge effort one has to undertake when dealing with overall architectural logic change related to the meta data handling of the application.

        I hope I achieved a good foundation for the horizontal scalability of the app ...

        Once again, thank you for the Wisdom !

[reply]

Back to Seekers of Perl Wisdom