Re: Persistent timed events

I'd consider using the file system.

Each record becomes a file, named for the time of its due time + a number^* to distinguish time clashes, stored in subdirectories named by date to avoid the "huge directory" problem.

^*Say, an MD5 of the date/time/processid/record contents.

The actioning server would use atomic rename to remove the next entry from the due queue before reading and actioning it, then deleting it.

If your action times need not be precise, say 1 minute granularity, then approx.1440 files per directory should make for a reasonable glob-sort-rename-the-top-entry action, and minimise the need to retry with the next-to-top entry should another actioning server have grabbed the top one first (assuming that there might be a need for multiple servers).

Another method of avoiding server collisions I've used before, is to have each server only glob the file system on a given 5 or 10 second boundary of each minute. Ie. Have up to 12 servers with the first waiting until 5 seconds past each minute before scanning the appropriate directory, the second 10 seconds past the minute and so on.

Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.

Lingua non convalesco, consenesco et abolesco. -- Rule 1 has a caveat! -- Who broke the cabal?

"Science is about questioning the status quo. Questioning authority".

In the absence of evidence, opinion is indistinguishable from prejudice.

Comment on Re: Persistent timed events

Replies are listed 'Best First'.
Re^2: Persistent timed events by tirwhan (Abbot) on Nov 03, 2005 at 14:45 UTC
Thanks, I think I'm being dense here, do you mean create 1440 directories per date and then have the actioning servers pick files out of the due directory until it's empty? Otherwise I'd have tens of thousands of files per date-directory. Or do you mean several records per file? Great idea about the time-sharing servers, I'll definitely use that. Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it. -- Brian W. Kernighan	[reply]
Re^3: Persistent timed events by BrowserUk (Patriarch) on Nov 03, 2005 at 15:57 UTC
No. One directory per date, one file per message named for the time (to the minute; hence the 1440) of the action + disambiguator. This was based upon the assumption that your "several hundred thousand" figure would be spread across several days. `20051103 - 0502.xxx1 0502.xxx2 0503.xxx1 0503.xxx2 0505.xxx1 0506.xxx1 20051104 - ...` [download] If I was misreading you, and your mean several hundred thousand per day, and daily directories would be too big, then I'd sub-divide them into hours `20051103 - 00 - 01 .... - 23 - 5901.xxx1 - 5901.xxx2 - 5902.xxx1` [download] I guess you could go to minutes, if required to keep the final directory sizes small for searching. Having 1440 subdirs in a directory isn't a problem as you are never searching that space, just going through it with a hard-coded path. I'd stick to one record per file, to avoid the need for using (cooperative) locking, as it only takes for someone to use a non-cooperating utility at an inappropriate moment to create a dealock situation. Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error. Lingua non convalesco, consenesco et abolesco. -- Rule 1 has a caveat! -- Who broke the cabal? "Science is about questioning the status quo. Questioning authority". In the absence of evidence, opinion is indistinguishable from prejudice.	[reply] [d/l] [select]