We also work on this sort of stuff a lot, here are some additional thoughts.

We dropped the requirement for a footer row in favour of an MD5 checksum since it is a better indication of file corruption than a footer count. Once you have an MD5 there's really no need for the count.

>Data records should be easily identified
> Each record type in the file should be easily identified
> from the others. Using the first field as an indicator > works well."

If you used fixed length records (and therefore fields) save yourself some heartache and make sure that all the record indicators are the same length. Yep, we've seen it done otherwise. When done this way a simple chunk off the front of the record tells you what to expect, gymnastics are required otherwise.

Also leave plenty of space, because at some point there might be variations on record types that are acceptable and a sub version can be handy.

A final anecdote. Dates, how might I **** thee, let me count the ways.

We were involved in rescuing 3 months of data that had been entered where on one of the workstations, and I quote... "Sometimes, on this one, the dates don't work. When that happens we just swap the day and month and it's OK.". Fan*******tastic.

In reply to Re: Thoughts on designing a file format. by adamc00
in thread Thoughts on designing a file format. by demerphq

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.