|
Re: Seeking general directions for a Windows specific app
by BrowserUk (Patriarch) on Aug 13, 2008 at 21:37 UTC
|
Firstly I would say: Don't bother.
The filesystem, system caches and disk DD's have access to far more and far better information, from all concurrent activity--not just those processes using your mechanisms--upon which to decide what order to read and write things and when. The very best you could achieve at user-level code is to make things no slower. The likelyhood is that you will not achieve that best.
However, were I to write something along these lines, my starting point would be Jenda's extremely useful Win32::FileOp, which gives you direct access to the OS's own MoveFile and CopyFile functions for the actual file processing parts, along with the OpenDialog(), SaveAsDialog() and BrowseForFolder() functions which are direct access to the systems own GUIs for those functionality. Along with a whole heap of other useful stuff that works with the OS rather than in spite of it.
Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
"Science is about questioning the status quo. Questioning authority".
In the absence of evidence, opinion is indistinguishable from prejudice.
| [reply] |
|
|
The filesystem, system caches and disk DD's have access to far more and far better information, from all concurrent activity--not just those processes using your mechanisms--upon which to decide what order to read and write things and when.
I personally believe that this seems to be the general consensus as of the answers that popped up in this thread. I may well stand corrected out of trusting the expertise of you all. But the impression of things getting so awfully slow with parallel transfers was so strong that I'd like to at least do some (more) experiment and to this end...
However, were I to write something along these lines, my starting point would be Jenda's extremely useful Win32::FileOp, which gives you direct access to the OS's own MoveFile and CopyFile functions
...I will certainly try Win32::FileOp in the first place to do the experiment in a more controlled way! Thank you for the suggestion!
| [reply]
[d/l] |
|
|
One thing to consider, is the source and quality of your USB device driver. Many moons ago when evaluating a 8-port serial card (under OS/2), we found that every additional concurrent transfer halved the overall combined throughput until by the time we had all 8 ports transferring data concurrently, the combined throughput was almost at a crawl.
As we were potentially going to be purchasing many hundreds of these cards, the manufacturers sent us a support technician to help us resolve the problem. Turned out that the install had selected to use a generic serial port driver rather than the manufacturer supplied one. And as that had been written to deal with just the standard 2-port configuration, it was overly conservative in its design and that was the cause of the abysmal throughput.
Once we installed the serial DD designed to deal with the 8-port device, things improved immensely. I'd strongly suggest that you visit the web site for whoever manufactured your USB card and check the availability of the latest drivers. Also, if you installed anything involved in the USB stick using Plug&Pray, check that it used the manufacturers DDs and not generic ones. Especially if your copy of Windows is relatively old. Early builds shipped with only USB v1.x drivers, but most devices work much better with later v2.x drivers.
The availability and capabilities of DDs always trail the hardware.
Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
"Science is about questioning the status quo. Questioning authority".
In the absence of evidence, opinion is indistinguishable from prejudice.
| [reply] |
Re: Seeking general directions for a Windows specific app
by jethro (Monsignor) on Aug 13, 2008 at 20:56 UTC
|
I would suggest: don't. The usb stick is the time hog here, and since it has no seek time like a hard disk you will not save one second by sequentializing the writes IMHO
| [reply] |
|
|
I personally believe that experimental evidence as of my own observation described in the readmore section at the top of the root node would suggest just the opposite, (which probably I didn't make clear enough to start with) but to be fair I didn't take times with a cronograph nor did I repeat the experiment in a controlled manner, thus ++ on faith: however I'm still fairly convinced that they somehow "interfered" to the point of getting ridicolously slow.
Incidentally, it all started with an usb key, but I didn't say anywhere that the use of the utility should be limited to usb keys...
| [reply]
[d/l] |
|
|
Just to make sure about your experimental evidence: A second copy should slow down the first copy to about half speed, any further copy should further lessen the speed of any single copy to 1/n with n the number of copies. That is to be expected. But you have seen a speed reduction much higher than that?
In that case I stand corrected. Must be some hair brained implementation somewhere.
| [reply] |
|
|
Re: Seeking general directions for a Windows specific app
by Jenda (Abbot) on Aug 13, 2008 at 21:52 UTC
|
I know it's not a Perl answer etc. etc. etc. but what about using something a little... more advanced than Windblows Exploder? xplorer2 maybe? F5, [queue], ...
The free Lite version is enough.
P.S.: explorer.exe crashes at least once every day on my bloody Vista and hangs every time I dare to try to mute using the special mute button off the notebook lately. Plus the Open or SaveAs dialog sometimes crashes applications. Thank you Microsloth!
| [reply] |
|
|
I know it's not a Perl answer etc. etc. etc. but what about using something a little... more advanced than Windblows Exploder? xplorer2 maybe? F5, [queue], ...
I personally believe that I must be a very lucky guy for I don't seem to experience all the inconveniences you're mentioning. To be fair I'm not that much of a file-manager-kinda-guy at all: which is full evident under OSen in which cli's are more common and powerful. But really, exploder doesn't explode too much here, and I'm not dissatisfied with it, for what I use it - and when I don't use it, I'm very comfortable with the file browsing mode of a very old version of a famous image viewer, which I more commonly use as a generic file manager... however strange this "setup" may seem! That's more or less all I need in daily operations. I've been more tempted and inclined to replace Windows' shell recently, and I've been investigating some possibilities. (Of course, this has nothing to do with the problem at hand.)
| [reply]
[d/l] |
Re: Seeking general directions for a Windows specific app
by plobsing (Friar) on Aug 14, 2008 at 05:13 UTC
|
Assuming that enqueue operations would speed them up, things only really get interesting when you have 2 operations that can be performed completely in parallel.
for example:
C: -> F: while also doing D: -> G:
Now given a list of drive pairs (from and to), and costs of operation (file size combined with drive speed), what you get looks like a weighted graph. And then you have to pick the sequence of all edges that optimizes something to the effect of:
# warning: untested!
use List::Util 'sum';
use Math::Combinatorics 'combine';
# @edges is the sequence of operations
# edge_value and has_common_drive are left as an excercise to the impl
+ementer
my %seen = ();
my $fitness =
sum(
# sum up operations until blocking
map { sum(
map { edge_value($_) }
@edges[ $_->[0] .. $_->[1] - 1 ]
) }
# take only the first collision after an operation
grep {not $seen{$_->[0]}++}
# sort collisions by distance between
sort { $a->[1] - $a->[0]
<=>
$b->[1] - $b->[0]
}
# find colliding operations
grep { has_common_drive( @{$edges}[@$_] ) }
# all combinations of operations
combine(2, 0..$#edges)
);
I'm not inclined to prove it, but your problem seems to be at least NP complete.
And then you have to consider that you would have to re-evaluate your moving strategy whenever a new edge is added to your graph (whenever you enqueue a new operation).
Don't forget that operations on the same file *MUST* occur in FIFO order when at least one of them is a write. This is likely uncommon, but it is a critical error to do operations otherwise.
The only reasonable solution is to apply heuristics and get a "close enough" answer. But what is "close enough". And how hard is it going to be to get there? Too hard and not worth it IMHO.
In short, I don't think that writing a queue for this would be easy if you hope to get anything close to optimal. | [reply]
[d/l]
[select] |
|
|
To know whether you can do your C: -> F: and D: -> G: in parallel you'd have to know these are all different PHYSICAL drives (well, I would not worry about two remote mapped drives being the same). If C: and D: reside on the same harddrive, they can't be performed completely in parallel. In general, I would leave this to the user, if he enqueued the operation, he wants it to start after the previous one(s) complete.
| [reply] |
|
|
To be fair, I hadn't even remotely thought of (i) the problem of logical drives being really part of the same physical drives, nor -more importantly- of (ii) plobsing++'s subtleties, assuming naively that a much simpler logic involving (something like) a %is_in_use hash holding arrayrefs of "transfers" (of which the first one to be considered active) would do. But eventually you're right that if I still roll up this enqueing utility/experiment of mine, then I may want to skip all the hassle and just enque everything at the user's request.
| [reply]
[d/l]
[select] |