james28909 has asked for the wisdom of the Perl Monks concerning the following question:

Ok, i have been wondering this for a while, and i prob already know how to do it and i am just not realizing it, but....

if i wanted to open a file , and then byte reverse the file without writing it to disk but instead to a scalar to seek and read from, what would be the best way to do this? and without messing with the original data... or is it even possible?

like i said maybe i already know how to do this. in the end i want to open a file in memory, then byte reverse it to another scalar in memory without writing it to disk and still be able to seek and read the byte reversed file that is in memory.

something like:
use strict; use warnings; use File::Slurp; my $read_file = read_file( "file" ); #read_file my $memory_file; #declare memory file my ( $buf, $data, $n, $bytes ); #declare variables while (( $n = read $read_file, $data, 4096 ) != 0 ) { print $memory_file, pack( "v*", unpack("n*", $data )); #byte rev +erses $bytes+=$n; } open ( my $write_file, ">", "reversed" ); #writes byte reversed memor +y file to syswrite ( $write_file, $memory_file ); #disk AFTER it byte reverse +s in #memory. read ( $memory_file, $stuff, 0x200 ); #should return a value but +doesnt print ( $stuff );

hopefully what i am asking is clear enough, if not, just let me know :)


UPDATE, problem solved with the following code, no file::slurp needed and im pretty sure you wouldnt even need to binmode the files either.And also, just to update, this will take a 16 MB file, in 0x34 0x12 0x78 0x56 order and put it in 0x12 0x34 0x56 0x78 order.
Here is the code updated code that will seek and read as well:
open ( my $in_file, '<', "file" ); binmode( $in_file ); my $mem_file; binmode( $mem_file ); my ( $buf, $data, $n, $bytes ); while (( $n = read $in_file, $data, 4096 ) != 0){ $mem_file .= pack( "v*", unpack("n*", $data )); $bytes+=$n; } open ( my $otherMem_file, '<', \$mem_file ); binmode( $otherMem_file ); seek $otherMem_file, 0x00, 0; read $otherMem_file, my $temp, 0x1200; open my $out_file, '>', "reversed"; syswrite( $out_file, $temp);

Also i want to thank everyone who gave me advice, if it werent for you guys/gals here at perlmonks id be lost for sure, if i had the money id buy you all a beer :cheers:

Replies are listed 'Best First'.
Re: Best way to read a file into memory and use normal operation on memory file?
by hexcoder (Curate) on Jul 16, 2014 at 07:27 UTC
    Something like this?
    use File::Slurp; my $file_content = File::Slurp::read_file('file'); $file_content = reverse $file_content; my $memory_file = \$file_content; open my $memfh, '<', $memory_file or die "$!\n"; #...seek #...read close $memfh;
    For details see these articles from the FAQ:
    read-in-an-entire-file-all-at-once and
    open-a-filehandle-to-a-string
Re: Best way to read a file into memory and use normal operation on memory file?
by Anonymous Monk on Jul 16, 2014 at 01:11 UTC
    my $str = "yo ho ho\n"; open my($fh), '+<', \$str; print <$fh>; print $fh "and a bottle\n"; print $str; __END__ yo ho ho yo ho ho and a bottle
      Naturally this needs use v5.8.0;, for perl v5.6 you need IO::String
Re: Best way to read a file into memory and use normal operation on memory file?
by Laurent_R (Canon) on Jul 16, 2014 at 07:01 UTC
    You could just slurp the file into a scalar variable and then reverse it.

    Or, if it is a text file, use the File::ReadBackwards module to read the file backwards line by line and reverse each line.

Re: Best way to read a file into memory and use normal operation on memory file?
by perlfan (Parson) on Jul 16, 2014 at 17:59 UTC
    Filesys::POSIX may offer you an interesting option. You could also set up a RAM disk.
Re: Best way to read a file into memory and use normal operation on memory file?
by james28909 (Deacon) on Jul 16, 2014 at 15:46 UTC
    thanks all for the input, really i already have the byte reversing done, and i dont think reading the whole file in reverse will give me the output i desired :p , but i already have that dealt with anyway, i was just trying to figure out how i could byte reverse it in memory (without having to write it to disk) and only write to disk when needed. some of the tests i did yesterday was trivial, i will post code later on hopefully, if i can get around to it :)
Re: Best way to read a file into memory and use normal operation on memory file?
by james28909 (Deacon) on Jul 16, 2014 at 16:45 UTC
    Let me give yall a better explanation of what i am trying to do, i already have the byte reversing done so no need to read a file backwards, but i can open both files before the byte reversal like this:
    use File::Slurp; open my $in_file, '<', "file"; binmode($file); open ( my $out_file, '>', "reversed" ); binmode($out_file); my ( $buf, $data, $n, $bytes ); while (( $n = read $in_file, $data, 4096 ) != 0 ) { syswrite $out_file, pack( "v*", unpack("n*", $data )); $bytes+=$n; }

    this above code will successfully read a file then byte reverse it to the other file, but you have to have a file on disk to do it.


    now, what i want to be able to do, is open the $in_file, then byte reverse it in memory, and then write it to file after reversing it, or in other words, byte reverse the file in memory, then do whatever i want with it.


    use File::Slurp; open ( my $in_file, '<', "file" ); binmode( $file ); my $temp; open my $mem_file, '>', \$temp; binmode( $mem_file ); my ( $buf, $data, $n, $bytes ); while (( $n = read $in_file, $data, 4096 ) != 0 ) { syswrite $mem_file, pack( "v*", unpack("n*", $data )); $bytes+=$n; } open ( my $out_file, '>', "reversed" ); binmode( $out_file ); syswrite ( $out_file, $mem_file );


    i have tried a few different things and nothing has worked tho i am able to get the program to actually write the file, but it writes SCALAR(XXXXXXXX) or GLOB(XXXXXXXX) ect to the file and not the actual reversed data.

      A few comments. First of all:

      this above code will successfully read a file then byte reverse it to the other file, but you have to have a file on disk to do it.

      Unfortunately, it won't. The code you posted will take a file, split it into chunks of 4096 byte, attempt to reverse each chunk, and then write the chunks to a new file in the same order. Worse, it fails to even reverse each chunk. Try generating a test file with several chunks, e.g. as follows:

      $ for i in `seq 1 256`; do echo -n '0123456789ABCDEF' >>file; done $ for i in `seq 1 256`; do echo "This won't work" >>file; done $

      Try running your script on the resulting file; you'll end up with something like the following:

      1032547698BADCFE1032547698BADCFE1032547698BADCFE... khTsiw not'w ro khTsiw not'w ro khTsiw not'w ro ...

      Which is not "reversed" in even the loosest sense of the word.

      Now, on to your second question. This

      syswrite ( $out_file, $mem_file );

      simply isn't going to work, as you've found out. syswrite expects to be passed data (in the form of a scalar), but what you're passing is a reference to a filehandle, which stringifies to something along the lines GLOB(0x8006c228).

      Furthermore, I'm a little puzzled as to why you want to use an in-memory file to begin with. Your first script -- if you were to fix it -- would be useful since you could use it to reverse files too large to slurp into memory all at once; with the second one, that's obviously not true anymore.

      You don't need to use an in-memory file: there's a much better, easier-to-use facility for holding data in memory, namely variables. So why not do what was suggested above, using e.g. File::Slurp to read all the file's contents into a variable, and then use reverse to reverse it before writing it out again?

      Frankly, I'm puzzled, but perhaps I'm missing the obvious good reason for why you're doing it this way. If that is the case, please enlighten me; otherwise, please meditate on the advice the monks gave you.

        the pack unpack function (on my system atleast) will take a file in 12 34 56 78 order and put it as 34 12 78 56 which is exactly what i need and works flawless everytime. ive even packed the function into an exe and added it as a right click context menu option because it did work so good :p

        maybe it has to do with your machine maybe? im on windows pc ofcourse.

        also i should add that the file i will always be working with is 16 MB's (6,777,216 bytes)

      Is there any reason this has to be done as a file in memory, instead of a simple scalar? Just replace your

      syswrite $mem_file, pack( "v*", unpack("n*", $data ));

      with

      $mem_file .= pack( "v*", unpack("n*", $data ));

      And of course, simply use my $mem_file; instead of the open my $mem_file...;

        yeah that def worked because i can byte reverse the file, then after the while loop i can write it to disk/hdd. i can also seek and read it ;) which is what i was after lol.

        here is the final working code:
        open ( my $in_file, '<', "file" ); binmode( $in_file ); my $mem_file; binmode( $mem_file ); my ( $buf, $data, $n, $bytes ); while (( $n = read $in_file, $data, 4096 ) != 0){ $mem_file .= pack( "v*", unpack("n*", $data )); $bytes+=$n; } open ( my $otherMem_file, '<', \$mem_file ); binmode( $otherMem_file ); seek $otherMem_file, 0x00, 0; read $otherMem_file, my $temp, 0x1200; open my $out_file, '>', "reversed"; syswrite( $out_file, $temp);


        and it give me all expected results byte for byte. :)
        also i should add that i am parsing the file using its hexadecimal representaion of a binary file and i honestly dont think you would even need to binmode any of these files and it doesnt need file::slurp. completely independant code teehee
Re: Best way to read a file into memory and use normal operation on memory file? (byteswap)
by Anonymous Monk on Jul 16, 2014 at 19:54 UTC

    Byte swap on the output, as in the following:

    $ echo -n 'abcd' | dd conv=swab | hexdump -C
    00000000  62 61 64 63                                       |badc|
    

    It seems OP is requesting a ':bswap' io layer or somesuch. AFAIK there is none.

Re: Best way to read a file into memory and use normal operation on memory file?
by james28909 (Deacon) on Jul 16, 2014 at 22:31 UTC
    What i am wanting to do is byte reverse in the proprietary order in which the function does (12 34 56 78 -> 34 12 78 56 or 34 12 78 56 -> 12 34 56 78) and put that in memory so i can access/do operations like read and seek on that particular data in memory at anytime without having to write anything to disk/hard drive.
Re: Best way to read a file into memory and use normal operation on memory file?
by james28909 (Deacon) on Jul 19, 2014 at 06:53 UTC
    you know what, i am lucky i found out how to do this lol. because i just tried to pack almost 1000 lines of code into an exe and it wasnt working right when opening files. the function i was trying to use, was going to open a file like
    my $file = PerlApp::get_bound_file("patch_file") open ( my $patch, '<', $file );

    but what i needed to do was open it with
    my $file = PerlApp::get_bound_file("patch_file"); open ( my $patch, '<', \$file );

    the key defference is the "\". its a good thing i figured this out tho.