in reply to Re^2: Flat file handling
in thread Flat file handling

Okay, this is progress in the right direction -- much better than the initial post in this thread, which is too much of a mess (too many things wrong with it) to look for any specific problem. My recommendation is to delete the original script, and start a new one, which opens with comment lines that reflect your 5-point synopsis. Then write code in accordance with the comments.

Of course, the 5-point summary still needs a little work...

1. Read in the flat file
Fine. But it would be helpful to describe (briefly and succinctly) what information is actually present in the file, or at least, what information in it is needed by your script, and how it's formatted. (For example, the OP talks about "components" in the file, but there's no indication about what constitutes one "component".)
2. Take the name of the stored procedure from the flat file.
Good. How do you do that? Is there more than one procedure name in the file? If so, how will you keep track of them all (or do you only need the first or last one that you find)?
3. Prepare the first execute statement by reading the flat file line by line. Each EXEC statement input variable is defined line by line in the flat file.
This is not clear. You've already read the file. (Or did steps 1 and 2 just involve reading the first line or few, rather than reading the entire file? You need to be clear about that.) As with the procedure name thing in step 2, how is the input variable determined from the content of each line? You mention "first execute statement", so how many such statements will there be? (Determined by number of lines in the file? Number of lines of a particular format?) Are you talking about really different statements, or a single statement with a placeholder, to be used with the variable that is read from each line?
4. When either there is a change in the unique identifier (ie. there is a new component), or the END of the file is reached, the EXEC is fired and the next EXEC statement build is begun.
Here we go with "components" again. What the heck is a "component"?? If you've read the file and built up the exec statement and now you execute it, where does the "next EXEC statement" come from? Or have you still not read the entire file yet?
5. The EXEC statement/s are fired in turn and the results returned to a output file.
There needs to be enough information in the commentary about the input file so that the code reader (and the code writer!) can look at the input data file and identify key events that will invoke specific behaviors within the script.

Your description seems to indicate a sort of hierarchical structure in the data: a procedure, a set of values to pass to that procedure, another procedure, another set of values, and so on. If this is so, and if the commentary (i.e. internal documentation) describes this clearly, and if the code is written so that it conforms to the commentary, then things will work better all around. Especially the next time you paste your whole script at the start of another SoPW thread.

And for cryin' out loud, if you're going to ask for advice, you should try using the advice you're given -- and even try a few more things on your own after that -- before asking for more advice. (I usually don't downvote SoPW nodes, but I made an exception in your case because you really have shown a lack of consideration in this regard.)

Oh, and do try to use consistent indentation. It really does help (you and us both). If your text editing tool supports auto-indentation, learn how it works; if not, set yourself a goal to acquire one that does and learn to use that.

Replies are listed 'Best First'.
Re^4: Flat file handling
by Win (Novice) on Nov 19, 2004 at 10:34 UTC
    I solved the problem by changing the format of the flat file slightly. I now use the term END after each component and this allowed me to delete the offending code.

    The code works as it should now. After a good nights sleep.
    #! perl -w scipt use strict; use warnings; use DBI; use DBD::ODBC; use File::Copy; my ($data_source, $database, $user_id, $password) = qw( *********** ** +******** ********** ******* ); my $conn_string = "driver={SQL Server}; Server=$data_source; Database= +$database; Trusted_Connection=yes"; my $dbh = DBI->connect( "DBI:ODBC:$conn_string" ) or die $DBI::errstr; my $Request_id; my $Variable_number; my $Variable; my $Choice; my @VARIABLE; my @CHOICE; my $Command; my $Return_results; my $Component_number_old = 1; my $Component_number = 1; my $SPROC; my $unique; my $elements_in_array; my $flat_file = "Flat_file.txt"; open (FLAT_FILE, "<$flat_file"); while (<FLAT_FILE>){ chomp; $Component_number_old = $Component_number; if ($_ =~ /SPROC\sName\:\s([a-z|A-Z|0-9|_]{1,100})/){ print "Hi_A\n"; $SPROC = $1; } elsif ($_ =~ /^\*(.{36})\;(\d{1,5})\;(\d{1,5})\;([a-z|A-Z|0-9|_]{1,5 +0})\;(.{1,100})/){ print "$_\n"; push (@VARIABLE, $Variable); push (@CHOICE, $Choice); print "Hi_B\n"; $unique = $1; $Component_number = $2; $elements_in_array = $3; $Variable = $4; $Choice = $5; } if ($_ =~ /^\sEND/){ print "Hi_E\n"; push (@VARIABLE, $Variable); push (@CHOICE, $Choice); $Command = join(' ', 'EXEC', $SPROC, join(', ', @CHOICE[1 .. $elements_in_array])) . ';'; my $Request_id = $CHOICE[26]; print "$Command\n"; Got_Command($Command, $Request_id); undef @CHOICE; my @CHOICE; } } close FLAT_FILE; my $source = $flat_file; my $dest = 'Bin'; move($source, $dest) or die "Error moving file: $!\n"; sub Got_Command { my($Command,$Request_id) = @_; my $output_file = "Output/Output_file_".$Request_id.".txt"; open (OUTPUT_FILE, "+>>$output_file"); print "$Command\n"; my $sthB_A = $dbh->prepare($Command) or die "Couldn't prepare query +: ".$dbh->errstr; $sthB_A->execute() or die "Couldn't execute query: ".$sthB_A->errstr +; my $Return_results = "Select * from Result_storage_keep where Unique +_identifier = ".$Request_id."\;"; print "$Return_results\n"; my $sth = $dbh->prepare($Return_results) or die "Couldn't prepare qu +ery: ".$dbh->errstr; $sth->execute() or die "Couldn't execute query: ".$sth->errstr; while (my @row = $sth->fetchrow_array ) { print OUTPUT_FILE join("\t", @row); print OUTPUT_FILE "\n"; } }
      Few things.
      1. Why capture $unique and $componen_number, since you aren't using them anywhere?
      2. You don't need $elements_in_array, $#array will give the last elements number.
      $elements_in_array = scalar(@array); $last_element_index = $#array; $last_element = $array[$#array]; # $#array is one less than the number of elements in array # as first index in array is 0 # this is always true if (scalar(@array) == $#array + 1) {};
      3. Push things to your arrays immeadiately, instead of waiting for the next row. It makes it easier to follow whats happening later (at least I think so :)).