except that there is a needlessly large redundant file involved.

Hm. If you are going to keep the same list in a DB, it will also end up in a file within the filesystem. And depending upon which DB & schema you use, it will occupy anywhere from a little more to perhaps double as much space as the file.

In order for this to work, the file would also have to be updated after each node was processed.

You'd have to update the DB after every file to indicate the file had been processed. And that 'indication', whatever form you chose to use, is still going to end up modifiying a file on disk.

In the end, whether you use a flat file or a "DB", the same steps have to occur -- build a list; grab them one at a time; process; check them off the list -- and the same essential disk activity must occur.

The difference is, with a DB, you'll also get a whole raft of additional IO going on for its internal logging and journalling activity. All of which is required for its ACID compliance and/or transactional safety, but which is unnecessary overkill for such a simple -- build a list and discard each item when you've processed it -- application.

Not to mention all the additional complexity involved in setting up, maintaining and using the DB.

I like simple, but, each to their own :)