justin423 has asked for the wisdom of the Perl Monks concerning the following question:

I have a file that i need to read into a database in the following format.
rectype data ID (autoincremented)
REC1 DATAX DATA2 1
REC2 DATA3 DATA4 2
REC3 DATA5 DATA6 3
REC4 DATA7 DATA8 4
RECZ 5
REC1 DATAY DATA2 6
REC2 DATA3 DATA4 7
REC3 DATA5 DATA6 8
REC4 DATA7 DATA8 9
RECZ

AND SO ON. So here is my issue. The data stored at DATAX and DATAY in each group is relevant to the succeeding records and needs to be added to each succeeding record(REC2, REC3, until RECZ, then the next record is a new grouping, so the value has to re-initialized, and would look like this:
REC1 DATAX DATA2 1
REC2 DATA3 DATA4 2 DATAX
REC3 DATA5 DATA6 3 DATAX
REC4 DATA7 DATA8 4 DATAX
RECZ 5 DATAX

REC1 DATAY DATA2 6 (NEW VALUE)
REC2 DATA3 DATA4 7 DATAY
REC3 DATA5 DATA6 8 DATAY
REC4 DATA7 DATA8 9 DATAY
RECZ DATAY

is there a quick perl or MySQL query using fetchrow that will roll-over the values from DATAX to succeeding records until it gets to RECZ and then re-initialize the value for the next group? I can do it either by storing it in the same table or inserting each REC2 into a REC2 table, each REC3 into a REC3 table. One other thing. The number of rows to read are in the millions, so it should be efficient. Thanks.

Replies are listed 'Best First'.
Re: Sequential data read in MySQL/Perl
by choroba (Cardinal) on Jul 06, 2016 at 21:34 UTC
    Hi justin423,

    Having read your question several times, I can't make heads or tails from it. What is the schema of the database table? How do you store REC2 DATA3 DATA4 2 DATAX with 5 columns and RECZ 5 DATAX with 3 columns? Also, what do you mean by (NEW VALUE) ? Why is the first RECZ numbered with 5, but there's no number for the second RECZ?

    If you just want to know how to persist a value across several iterations of a loop, the answer is simple: declare a variable before the loop.

    #! /usr/bin/perl use warnings; use strict; my $grouping = q(); while (<DATA>) { my @columns = split; print "@columns $grouping\n"; if (@columns > 2) { $grouping = $columns[1] if q() eq $grouping; } else { $grouping = q(); } } __DATA__ REC1 DATAX DATA2 1 REC2 DATA3 DATA4 2 REC3 DATA5 DATA6 3 REC4 DATA7 DATA8 4 RECZ 5 REC1 DATAY DATA2 6 REC2 DATA3 DATA4 7 REC3 DATA5 DATA6 8 REC4 DATA7 DATA8 9 RECZ

    ($q=q:Sq=~/;[c](.)(.)/;chr(-||-|5+lengthSq)`"S|oS2"`map{chr |+ord }map{substrSq`S_+|`|}3E|-|`7**2-3:)=~y+S|`+$1,++print+eval$q,q,a,
      the table is a temp table that has the fields that are common across all of the rows coming in. (REC1 to RECZ above). the common identifier for REC1 to RECZ only appears on REC1, so i want to update the rec2, rec3, rec4 rows with the value in DATAX, which only appears in REC1. I was also trying to show that the value needs to be reset when the program encounters a RECZ, so that a new value is applied for the next group. How do I read the value from the database into a variable using DBI?
        I am also confused about what you are doing. If these 9 values "go together" somehow, then perhaps a table like what I show below is what you need? Each row is a "group" and each column contains "like data". Without some descriptive story re: post from Grandfather, its hard to say what you really need (which may be different than what you are asking for).
        1 DATAX DATA2 DATA3 DATA4 DATA5 DATA6 DATA7 DATA8 2 DATAY DATA2 DATA3 DATA4 DATA5 DATA6 DATA7 DATA8
Re: Sequential data read in MySQL/Perl
by GrandFather (Saint) on Jul 06, 2016 at 23:57 UTC

    Show us something real. Fake up a simple database (see I know what I mean. Why don't you?) and use it to tell the story. It'd probably help if you tell us something of the bigger picture so we have a context for what you are trying to do - it's just possible there's a better way.

    Premature optimization is the root of all job security