Re^2: Sharing DB handler among forked processes

I write web spider which grab info about auction bids. Bids are grouped by sections. So i fork each section, collect information for it and try to insert this info into DB. Maybe i don't need shared DB handler but must create new handler for each forked process?

Comment on Re^2: Sharing DB handler among forked processes

Replies are listed 'Best First'.
Re^3: Sharing DB handler among forked processes by kyle (Abbot) on Aug 12, 2007 at 06:06 UTC
Yes, you don't need a shared DB handle. Each forked process can have its own connection to the database and operate on it through that connection. From your description, they don't even need to talk to each other, so you don't need any other IPC. The database shouldn't have any problem dealing with the data coming from different directions at once.	[reply]
Re^4: Sharing DB handler among forked processes by EvanCarroll (Chaplain) on Aug 12, 2007 at 22:23 UTC
It becomes a performance issue vs simplicity. If persistence isn't a concern, and you aren't being charged by the handle, then this is obviously the way to go. But, if you need most any transaction, or any other fun vooodoo, obviously this route won't work. For most cases, I'm advocating a solution that would be the fringe case -- if you don't have a good reason to use IPC don't. Evan Carroll www.EvanCarroll.com	[reply]
Re^3: Sharing DB handler among forked processes by EvanCarroll (Chaplain) on Aug 12, 2007 at 05:28 UTC
Share the scraped info with a master proc using shmem, or sockets. Then have the parent inject the data into the DB. Evan Carroll www.EvanCarroll.com	[reply]