I am starting on my largest Perl project yet.

I have to create ETL (extract-transform-load) scripts for a bunch of data that come from various sources and go off to different tables in Oracle.

The general scenario is thus --

input --> transformation magic --> Oracle

Inputs vary from text files to SQL Server tables to Oracle tables. Transformation can be a simple mapping of fields and load, or very complicated transformations in between.

Currently all the transformation and loading is being done in SQL Server via DTS. The point of this exercise is to migrate to Oracle, and convert all DTS to Perl/Shell scripts. I will be programming them on a Windows box, but they will eventually run on Unix, and will get crontabbed to run unattended. The whole intent is to avoid human-computer-interaction as much as possible.

I have started studying each of the conversions in detail, and noting down the source data, opening up the DTS packages in SQL Server Enterprise Manager and "decoding" them into plain text.

I am right now in the pre-assessment phase (really, I am trying to figure out how long it will take to do the assessment), after which I will do the detailed assessment, which will then hopefully result in a document that will enable the programmer(s) (myself) to sit down and program the scripts.

This is a chance for me to demonstrate what I know about Perl, and what Perl is capable of, to folks generally fairly low on the knowledge-scale about Perl. I want to do things right. Any words of wisdom would be appreciated.

--

when small people start casting long shadows, it is time to go to bed

In reply to Automating data loading into Oracle by punkish

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.