Re^2: Persistent File Queue recommendation requested

Thank you for the input. You state that they don't scale well for 1000s of jobs. Unfortunately, this is the case. The enqueue thread received hundreds of thousands of jobs in a day. So, this probably isn't going to work for my needs. I appreciate the feedback and will probably play with IPC::DirQueue for other things. Thank you.

Comment on Re^2: Persistent File Queue recommendation requested

Replies are listed 'Best First'.
Re^3: Persistent File Queue recommendation requested by Corion (Patriarch) on Jan 27, 2010 at 14:36 UTC
Depending on the nature of the jobs, you can follow the usual approach and use the first two letters or some other (evenly distributed!) criteria to distribute the jobs among subdirectories. This makes scanning for fresh jobs harder though. Alternatively, move jobs that are "in processing" into a separate directory which is not scanned. That will reduce the load that idle jobs produce while scanning for work to do. If you have to have a high throughput and can't batch your 100k requests into jobs of (say) 100 items or so, I'd look at premade solutions or maybe just at dedicating a database machine which serves as the central job directory.	[reply]

Replies are listed 'Best First'.

Re^3: Persistent File Queue recommendation requested
by Corion (Patriarch) on Jan 27, 2010 at 14:36 UTC

Depending on the nature of the jobs, you can follow the usual approach and use the first two letters or some other (evenly distributed!) criteria to distribute the jobs among subdirectories. This makes scanning for fresh jobs harder though. Alternatively, move jobs that are "in processing" into a separate directory which is not scanned. That will reduce the load that idle jobs produce while scanning for work to do.

If you have to have a high throughput and can't batch your 100k requests into jobs of (say) 100 items or so, I'd look at premade solutions or maybe just at dedicating a database machine which serves as the central job directory.

[reply]