I started to write a module (pure Perl) to implement a virtual disk inside a file. This will be used in other project, much bigger, in the future. But I will put this resource as a Perl module to the others use.

I want to share my ideas, and ask for some, since I saw, in previous postes, that this help a lot, and doesn't hurt.

Is not very simple to make a real virtual disk, that can be changed every time, for many programms in the same time, inside a file. Is not only a database to save files. We need to read, write, delete, move, etc...

Thinking about a good way to do that I first implemented a FAT system inside the file, with clusters. So, all the clusters have the same size, a status, and can be linked to other cluster to make a block. Then I'm creating the block handle, where every block can have some clusters, in any point of the file, and this group of clusters (the block) are used to save data. To save files, or directorys, we can use different types of formats and headers, to than save the data in the block. And finally make the IO::File based on the virtual disk, to can use it directly as a normal perl IO.

The objective of this architecture is make a dynamic and fast way to work with the virtual disk. And the file system will be similar to UNIX, where you can mount a virtual disk in some path, and make the virtual disk use many different files, in different machines.

The objective of this is a full portable way to save files, with security. For example, you can save a file with .exe extension, and don't worry if some one will execute it. Or use files with UNIX path on Mac.

But to make a good format I need to think in all the resources that I need to implement. For example, what stats a file will have, data, size, mode, etc... I can have links inside the disk to another file? If yes, need to think in the format.

Well, this is the list of resources now. If you have some idea, send your comments: 


TYPES OF DATA:
  file, directory, link.

STATUS OF A FILE/DIRECTORY/LINK:

  size      The size used by the file.
  rsize     The real size, in clusters, of the file.
  blocks    The number of blocks (clusters) in use.
  AcTime    When the file was last accessed.
  MdTime    When the file was last modified.
  CrTime    When the file was created.
  mode:     rwx  (will use x? if I can't run a file!)
  uid/gid   Will have user access level? since we can
            read the main file disk... Or use just to tell
            who created it.

CRYPTOGRAPHY

  A file can be encrypted with symmetric and asymmetric
  keys. What algorithm to use? Alrady exist a Perl module
  for it?

[download]
About the name of the module, how about IO::VirtualDisk?

I will be very glad for any reply.

Graciliano M. P.
"The creativity is the expression of the liberty".

Replies are listed 'Best First'.
Re: Virtual Disk for Perl
by Abigail-II (Bishop) on Mar 07, 2003 at 07:03 UTC
    You keep calling this a 'virtual disk', but all you are talking about is creating a file system. disk != file system. There can be many file systems on a disk, and a file system can span many disks. And there are other uses for disks than to put file systems on it. (Swap space, raw disks for databases, etc.)

    Abigail

[reply]
Re: Virtual Disk for Perl
by zentara (Cardinal) on Mar 07, 2003 at 12:59 UTC
    Well to amplify some on what Abigail-II said.... You want to create a virtual filesystem on a loopback device. 
    system ('dd','if=/dev/zero','of=image.raw','bs=1024k',"count=$count");
system ('/sbin/mke2fs','-F','-b 2048','image.raw');
system ('mount','-text2','-oloop','image.raw',"$mntloc");

    [download]
    So you can do whatever you want with it, encrypt it, etc.
[reply]
[d/l]
Re: Virtual Disk for Perl
by steves (Curate) on Mar 07, 2003 at 13:39 UTC

    I'm not seeing how this would be used. You'd have a module used by Perl programs that would presumably have those programs writing to this virtual disk (virtual file system). Question #1: How will you synchronize all this virtual file system activity across those programs? Question #2: What does this data end up looking like on the real disk? Eventually you have to write it out somewhere. Question #3: How will non-Perl programs accessing the same files be managed?

    Maybe some proposed examples of usage would clear some of that up. Me thinks you may be trying to reinvent a much bigger wheel than you realize. There's a reason filesystems are implemented in OS kernels.

    If your idea is really to have a filter for data coming from and going to the real filesystem for some sort of consistent Perl view of the world, I'd look at writing a file handle tie module.

[reply]
Re: Virtual Disk for Perl
by gmax (Abbot) on Mar 07, 2003 at 14:10 UTC

    Before you go too far into your project, check something that has been already done, so that you have an idea of how complex the task might be and which risks you are going to face.

    VMWare sells a virtual machine that emulates a x86 computer architecture and creates virtual disks. In its earlier releases, the workstation package included a perl script to mount such virtual disks and to use them as regular filesystems.

    They don't include that utility in their latest distribution, but you can find it in their archives (it's quite a big file, 6.2 MB).

    Thinking about it, there is maybe a good reason why the mounting utility is not in distribution anymore, i.e. that you shouldn't mess around with filesystems at such low level.

     _  _ _  _  
(_|| | |(_|><
 _|

    [download]
[reply]
[d/l]
      I bet this script uses loopbacks which is quite standart method to mount ordinary files with filesystems on Linux (no idea if similar feature exist on other Unix-like systems).

      --
      Ilya Martynov, ilya@iponweb.net
      CTO IPonWEB (UK) Ltd
      Quality Perl Programming and Unix Support UK managed @ offshore prices - http://www.iponweb.net
      Personal website - http://martynov.org

[reply]

        Loopbacks are implemented in BSD systems. I don't have much experiance with commercial *nix, but I'm guessing it's available pretty much everywhere. You could probably get one on Win32 if you really tried (not easily, mind you).

        ----
        Reinvent a rounder wheel.

        Note: All code is untested, unless otherwise stated

[reply]
Re: Virtual Disk for Perl
by gmpassos (Priest) on Mar 07, 2003 at 17:33 UTC
    Yes, Linux already have something for this. But you are missing the point. A pure Perl module, fully portable!!! The app that will use this will be made to work in any OS!

    The idea is to save files and directorys inside a unique file.

    Graciliano M. P.
    "The creativity is the expression of the liberty".

[reply]

      How would lock a resource on that file system? Every time you need to access a resource inside *one* file, you would deny access to any other concurrent request for another resource...

      I had your idea many times, mostly because I needed the creation time, but always gave up and returned to some sort of db (with paths as primary keys for example). Use a DWIMish DBI module if you need portability.

      Ciao, Valerio

[reply]

         Another idea is to treat a zip file as a filesystem.

         You get compression for free, the file may be used by other processes, and you can work upon temporary files until you need to perform the updating - which simplifys the locking others have mentioned.

        Steve
        ---
        steve.org.uk
[reply]
        Yes, when you are using the file to write or read you block all the virtual disk. But you only have the handle of the file opened when you are sending read/write commands. In other words, between this read/write commands different process can use the file. Wasn't easy to make, but it's working! ;-P

        Graciliano M. P.
        "The creativity is the expression of the liberty".

[reply]
Re: Virtual Disk for Perl
by toma (Vicar) on Mar 09, 2003 at 19:24 UTC
    This project has been done, at least partially, already. See perlfs. It works on linux but is intended to be ported to other operating systems.

    I recall that someone wrote another one a long time ago, perhaps in perl 4. I think that it was used in research about distributed file systems and cryptography.

    It should work perfectly the first time! - toma

[reply]
      Many thanks! But is not exactly what I want. It's not portalbe, and pure Perl:

      "PerlFS is a combination of a Linux kernel module and some Perl scripts which allow to write filesystems in Perl. It currently works with Linux kernel 2.2 or 2.4..."

      But I will take a look, maybe to integrate my module to the system or make a XS for it, since for now it has access only from Perl programs.

      I was thinking that I can make an extension of this module to make some IPC, for shared memory for any OS!

      Graciliano M. P.
      "The creativity is the expression of the liberty".

[reply]

Back to Meditations