Beefy Boxes and Bandwidth Generously Provided by pair Networks
P is for Practical

Re^3: Mass downloads. (N+5)

by exussum0 (Vicar)
on May 25, 2005 at 19:46 UTC ( #460434=note: print w/replies, xml ) Need Help??

in reply to Re^2: Mass downloads. (N+5)
in thread Mass downloads.

Can't whatever program you run, download them in batch and keep a marker of what node was last successfully downloaded and saved? Heck, when it can't reach the server, have it sleep for 5 or 10 mintues. When you connect again, it'll just pickup where it left off.

Give me strength for today.. I will not talk it away..
Just for a moment.. It will burn through the clouds.. and shine down on me.

Replies are listed 'Best First'.
Re^4: Mass downloads. (N+5)
by BrowserUk (Patriarch) on May 25, 2005 at 20:39 UTC

    Part of what I was hoping to do do was test what throughput I could get via my limited bandwidth using mutiple threads and overlapped IO.

    My intention was to download high proportion of PM and then produce a fully inverted index to it locally.

    With the concern that there was never a time here when I could run the process without having an undue impact, I got permission to do my throughput testing at the weekend at another site where their peak usage is weekdays.

    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    Lingua non convalesco, consenesco et abolesco. -- Rule 1 has a caveat! -- Who broke the cabal?
    "Science is about questioning the status quo. Questioning authority".
    The "good enough" maybe good enough for the now, and perfection maybe unobtainable, but that should not preclude us from striving for perfection, when time, circumstance or desire allow.

Log In?

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: note [id://460434]
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others surveying the Monastery: (5)
As of 2022-08-17 07:26 GMT
Find Nodes?
    Voting Booth?

    No recent polls found