in reply to Daemonizing (or otherwise speeding up) a high-overhead script?

OP here. Many thanks for all the responses. I was going to give more detail for what's actually going on, but in fact the question that "hippo" asked led me straight to the obvious solution to this.

As they pointed out, the Catalyst app is already running, and that really is the daemon, and all the workers we need. What's happening now is that we implement our own job queue, and the cron script simply gathers data from the job queue and fires requests into the Catalyst app. Currently, the cron script loads the entire app (because it uses some utility functions that are, um, useful), but it doesn't need to--with a small amount of rewriting, the cron script can get the info it needs, without the zillion database connections, and send everything into the main app.

When I was thinking we needed a separate daemon, I didn't stop to consider why this script itself needed to be daemonized, and in fact, there's nothing in that script that takes much overhead. I was misunderstanding my own app. Wasn't intentionally an XY question, but it seems to have ended up that way.... Thank you!

  • Comment on Re: Daemonizing (or otherwise speeding up) a high-overhead script?

Replies are listed 'Best First'.
Re^2: Daemonizing (or otherwise speeding up) a high-overhead script?
by NERDVANA (Priest) on Aug 25, 2023 at 22:01 UTC
    I'm not a huge fan of running periodic tasks through the public-facing app, but it works. One of the problems would be if the public hit the app with so many requests that there weren't available workers to serve the cron jobs. Another problem is the potential for hackers who found your source code to issue bogus cron tasks from the public.

    An easy solution is to run another copy of the Catalyst app (or even a second Catalyst app entirely) for handling cron tasks, and not expose it to the public. This also lets you restart them independently, knowing whether you're disrupting the users or crashing a long-running cron task.

      I appreciate your concerns. But it's not a public-facing app; it's purely internal, and it only responds to requests that are from the internal network, and have some appropriate API key for the relevant task. We monitor this pretty closely. It can, of course, still happen that we get more requests than we expect, but we have a fair amount of cushioning built into the system.
Re^2: Daemonizing (or otherwise speeding up) a high-overhead script?
by Anonymous Monk on Aug 24, 2023 at 19:35 UTC
    And in fact, it took about 5 lines of changes, and I cut the startup time on our dev box from 19s to .9s. This is awesome!