Re^2: Gracefully exiting and restarting at a later time.

Okay

This would also work, except that there is a needlessly large redundant file involved. I say redundant, because a directory is also a file, and would contain the same information. In order for this to work, the file would also have to be updated after each node was processed.

So far, keeping the information in the database seems like the best idea (although the updated row in the database would have to be written to a file as well)

Thanks for your thoughts.

Largins

Comment on Re^2: Gracefully exiting and restarting at a later time.

Replies are listed 'Best First'.
Re^3: Gracefully exiting and restarting at a later time. by BrowserUk (Patriarch) on Dec 21, 2011 at 15:24 UTC
except that there is a needlessly large redundant file involved. Hm. If you are going to keep the same list in a DB, it will also end up in a file within the filesystem. And depending upon which DB & schema you use, it will occupy anywhere from a little more to perhaps double as much space as the file. In order for this to work, the file would also have to be updated after each node was processed. You'd have to update the DB after every file to indicate the file had been processed. And that 'indication', whatever form you chose to use, is still going to end up modifiying a file on disk. In the end, whether you use a flat file or a "DB", the same steps have to occur -- build a list; grab them one at a time; process; check them off the list -- and the same essential disk activity must occur. The difference is, with a DB, you'll also get a whole raft of additional IO going on for its internal logging and journalling activity. All of which is required for its ACID compliance and/or transactional safety, but which is unnecessary overkill for such a simple -- build a list and discard each item when you've processed it -- application. Not to mention all the additional complexity involved in setting up, maintaining and using the DB. I like simple, but, each to their own :) With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday' Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error. "Science is about questioning the status quo. Questioning authority". In the absence of evidence, opinion is indistinguishable from prejudice. The start of some sanity?	[reply]

Replies are listed 'Best First'.

Re^3: Gracefully exiting and restarting at a later time.
by BrowserUk (Patriarch) on Dec 21, 2011 at 15:24 UTC

except that there is a needlessly large redundant file involved.

Hm. If you are going to keep the same list in a DB, it will also end up in a file within the filesystem. And depending upon which DB & schema you use, it will occupy anywhere from a little more to perhaps double as much space as the file.

In order for this to work, the file would also have to be updated after each node was processed.

You'd have to update the DB after every file to indicate the file had been processed. And that 'indication', whatever form you chose to use, is still going to end up modifiying a file on disk.

In the end, whether you use a flat file or a "DB", the same steps have to occur -- build a list; grab them one at a time; process; check them off the list -- and the same essential disk activity must occur.

The difference is, with a DB, you'll also get a whole raft of additional IO going on for its internal logging and journalling activity. All of which is required for its ACID compliance and/or transactional safety, but which is unnecessary overkill for such a simple -- build a list and discard each item when you've processed it -- application.

Not to mention all the additional complexity involved in setting up, maintaining and using the DB.

I like simple, but, each to their own :)

With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'

Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.

"Science is about questioning the status quo. Questioning authority".

In the absence of evidence, opinion is indistinguishable from prejudice.

The start of some sanity?

[reply]