My one-big website wiki/blog/cms application was based on SQLite. It worked well... until I upgraded my database version and the corresponding DBD::SQLite version, but my web host's version numbers lagged. Well, at this change, the db formats became incompatible and everything went up in smoke.

After I had cleaned up the debris, I decided to change things. In my quest for the "simplest thing that would work," and inspired by the database wars on http://radar.oreilly.com/ (too many links to mention here), I decided to convert my datastore to flat-files. Here is my experience in the hope that other monks will offer me suggestions and insights to make this better.

First item on the tackle-list was organization. Aware that too many files in a folder slow things down, and inspired by CPAN's apparent organization, I decided to implement an alphabetical hierarchy like so -- 

[datastore]/P/PE/PER/PerlMonks
[datastore]/P/PE/PER/PerlTheLanguage
[datastore]/F/FL/FLA/FlatFiles

.. in other words ..

my $FileName;
my ($DIR, $DI, $D);
if (length($FileName) >= 3) {
  $DIR = uc(substr($FileName, 0, 3));
  $DI  = substr($DIR, 0, 2);
  $D   = substr($DI, 0, 1);
}

# in case the FileName is 2 characters long (less than 2 not possible)
elsif (length($FileName) == 2) {
  $DIR = uc($FileName) . '0';
  $DI  = uc($FileName);
  $D   = substr($DI, 0, 1);
}

[download]

A'ite. That works. Now, on to metadata. A file allows only three possible metadata -- the FileName (the title of the web page), the contents of the File (basically, the contents of the web page), and its mtime. What about all manner of other items I had stored in my database table? After pondering a bit, I decided to create a shadow file prefixed with a period. On my Mac, that file is invisible (although it mucks up the landscape on my Windows box), so it is there but not there. I stored all the remaining metadata in this file in the same location as the main file in a key:value pattern 

[datastore]/P/PE/PER/.PerlMonks
[datastore]/P/PE/PER/PerlMonks
[datastore]/P/PE/PER/.PerlTheLanguage
[datastore]/P/PE/PER/PerlTheLanguage
[datastore]/F/FL/FLA/.FlatFiles
[datastore]/F/FL/FLA/FlatFiles
..
# and inside a metadata file
   page_created_on: 2005-07-18
   page_created_by: punkish
  page_modified_on: 2005-07-18
  page_modified_by: punkish
      num_of_stars: 3
     privacy_level: 1
         prev_page: PrevPage
         next_page: NextPage
         firstname: Pun
        middlename: 
          lastname: Kish
           company: 
             title: 
          address1: 
          address2: 
              city: 
             state: 
               zip: 
           country: 
            phone1: 
            phone2: 
            phone3: 
          birthday: 
             email: 
               url: punkish.eidesis.org
contact_created_on: 2005-08-21 00:20:00
contact_created_by: 1
               lat: 43.065816
               lon: -89.409005

[download]

Well, the above is working quite well. Some other things are not done yet but they will also get resolved with time and ingenuity. My question for now -- what insights can monks offer me with regards to the above?

Update: Fixed typo in a link above.
--

when small people start casting long shadows, it is time to go to bed

In reply to Moving from SQL to flat-files by punkish

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.