FILE IO, delay after finish.

exodist has asked for the wisdom of the Perl Monks concerning the following question:

I am working on a perl program that writes data to a cf card. I am using sysopen, sysseek, and syswrite. I am currently doing some speed testing before I decide the best way to approach this. I am having my program write 100mb tot he cf and then quit, it notifies me of progress as it goes. It seems to write the full 100mb, then delay for a long time, the whole operation takes ~38 seconds, but it seems to finish after 10-15 seconds then sit for the rest of it. I tried the same thing w/ a usb drive that had a light indicating usage, the light kept going.

I know that in buffered read/writes the data is actually put in memory then synced to disk when finished or when memory pressure requires.... however I was under the impression from google searches that sysread and syswrite were not buffered, therefore I expect my progress to be progress of the data written to the drive, not the progress of data being queued to enter the drive.

If anyone can enlighten me that would help. Also if anyone can give me a way to truely see the real progress (I am trying to get it to print an MiB per second here.) that would be great.

Here is the relivant code:

        sysopen(DEV, "/dev/sdc1", O_RDWR);
        $Size = sysseek(DEV,0,2);
        print("Size: $Size\n");
        sysseek(DEV,0,0);
        my $KCounter = 0;
        my $MCounter = 0;
        my $Temp;
        while (1)
        {
                syswrite(DEV, 'L' x 1024, 1024);
                $Temp = sysseek(DEV,0,1);
                if ($Temp == $Size)
                {
                        print("\nEND\n");
                        close(DEV);
                        exit(0);
                }
                $Temp = ($Temp / 1024) / 1024;
                printf("\r %f MiB", $Temp);
                if ($Temp >= 100)
                {
                        print("\n");
                        close(DEV);
                        exit(0);
                }
[download]

Comment on FILE IO, delay after finish. Download Code

Replies are listed 'Best First'.
Re: FILE IO, delay after finish. by Corion (Patriarch) on Jun 16, 2007 at 18:48 UTC
Using `syswrite` over `write` only means that Perl won't do any buffering. Your operating system will still do buffering, likely until the file gets closed or the process calls `exit`. I guess you can only show the "real" progress as the OS claims by waiting until your program finishes and then calculating the throughput. As an aside, you might want to unbuffer STDOUT so your progress gets displayed: `$\|++;` [download] (unless you're already doing that somewhere). If you're concerned about writing speed, I'd try to write larger buffers, maybe something in the range of the cluster size of the hard disk or in the range of half/twice the cache size of the hard disk to maximize the amount of data available for the disk to write.	[reply] [d/l] [select]
Re^2: FILE IO, delay after finish. by exodist (Monk) on Jun 16, 2007 at 19:24 UTC
Well, essentially what I am trying to do is compare continuous write and random write on different media types. I also did this to test writing with seeking to different positions first. `# foreach (1 .. 1024) # { # syswrite(DEV, 'L' x 1024, 1024); # } # $MCounter++; # printf("\r %f MiB", $MCounter); # if ($MCounter >= 100) # { # close(DEV); # exit(0); # } # sysseek(DEV,(102410242), 1); # print("\nJumping 2mb\n");` [download] This is giving me frustrating results as my cf card is 100x, my drive is sata, and my usb stick unknown speed. Here are the results: `Continuous write, 100 mb: Image stored on a filesystem: 26.964s 100x CF Card: 36.572s USB Stick: 1m28.938s Disk Partition: 2.599s Random write (write 1mb, skip 2mb, loop) Image: 21.111s CF: 49.917s USB: 1m29.354s Partition: 2.832s` [download] As predicted the random write took longer... but my understanding was that for short random read/write operations flash media was supposed to be faster because of a hard drives slow seeking. Then again I am addressing a continuous 300mb section, just skiping 2mb each time, so I guess I should re-think this experiment to really jump around both directions....	[reply] [d/l] [select]
Re: FILE IO, delay after finish. by betterworld (Curate) on Jun 16, 2007 at 18:40 UTC
You should check the return values and error codes of your system calls. sysopen might fail (and return false) e. g. if there are permission problems. syswrite might not write all of your data for some reason and then return how many data has been written, which would explain your problem. It might even write nothing (maybe because the device is full) and return undef and set $! to an explanation. If your device is full, that would explain what your program is doing during the last 23 to 28 seconds: Unsuccessfully try to write stuff and ignore the errors. See perldoc -f syswrite	[reply]