Re: How to reference to array keys?
by jwkrahn (Abbot) on Jul 26, 2011 at 22:57 UTC
|
Your buildArray subroutine won't work correctly because as perlsyn explains:
If any part of LIST is an array, "foreach" will get very confused if you add or remove elements within the loop body, for example with "splice". So don't do that.
| [reply] [d/l] |
Re: How to reference to array keys?
by Cristoforo (Curate) on Jul 27, 2011 at 01:40 UTC
|
The blank lines between records aren't blank - they contain spaces. If the data comes this way, I wouldn't be able to use the parsing method below. I had to delete all the spaces in the blank lines so they really were blank lines and then used the paragraph mode to read in 1 record at a time.
Anyhow, this is how I might parse the data (into the %data hash). Assumes RECORD_ID is unique for every set of owners and waiters.
#!/usr/bin/perl
use strict;
use warnings;
use Data::Dumper;
my %data;
<DATA> for 1 .. 10; # toss out header
{
local $/ = ""; # paragraph mode
while (<DATA>) {
last if /^:/;
my $owner;
if (m{^(/.+)$}mg) {
$owner = $1;
}
my @owner_rec = unpack "A13A13A2A10A8A5A8A9A*", $owner;
$data{ $owner_rec[1] }{ owner } = \@owner_rec;
my @waiter;
while (m{^(/.+)$}mg) {
my $waiter = $1;
push @waiter, [ unpack "A13A13A2A10A8A5A8A9A*", $waiter ];
}
$data{ $owner_rec[1] }{ waiter } = \@waiter;
}
}
print Dumper \%data;
__DATA__
[H[2JUniData Release 7.2 Build: (3786)
(c) Copyright Rocket Software, Inc. 1988-2009.
All rights reserved.
Current UniData home is /usr/udthome/.
Current working directory is /usr/local/rfs/udt.
:TERM ,0
:UDT.OPTIONS 20 ON
:LOGTO /ud/JWP
:LIST.QUEUE
FILENAME RECORD_ID M OWNER UNBR UNO TTY TIME DA
+TE
/prod-data/J 00151120273 X jmorg 3584038 247 ts/49 13:38:12 Ju
+l 20
----------------------------------------------------------------------
+----
FILENAME RECORD_ID M WAITING UNBR UNO TTY TIME DA
+TE
/prod-data/J 00151120273 X jmorg 2015244 134 s/109 13:48:32 Ju
+l 20
/prod-data/J 00151120273 X gdavi 1359996 62 ts/20 13:54:22 Ju
+l 20
FILENAME RECORD_ID M OWNER UNBR UNO TTY TIME DA
+TE
/prod-data/J 001!L!311895 X jmorg 2015244 134 s/109 13:48:32 Ju
+l 20
----------------------------------------------------------------------
+----
FILENAME RECORD_ID M WAITING UNBR UNO TTY TIME DA
+TE
/prod-data/J 001!L!311895 X jmorg 5713932 191 ts/46 14:01:42 Ju
+l 20
FILENAME RECORD_ID M OWNER UNBR UNO TTY TIME DA
+TE
/prod-datahi 001!10274882 X rfuse 3354796 61 ts/43 13:39:02 Ju
+l 20
----------------------------------------------------------------------
+----
FILENAME RECORD_ID M WAITING UNBR UNO TTY TIME DA
+TE
/prod-datahi 001!10274882 X jmorg 3584038 247 ts/49 13:39:22 Ju
+l 20
:
This program print out the following (using Data::Dumper)
C:\Old_Data\perlp>perl t5.pl
$VAR1 = {
'001!L!311895' => {
'owner' => [
'/prod-data/J',
'001!L!311895',
'X',
'jmorg',
'2015244',
'134',
's/109',
'13:48:32',
'Jul 20'
],
'waiter' => [
[
'/prod-data/J',
'001!L!311895',
'X',
'jmorg',
'5713932',
'191',
'ts/46',
'14:01:42',
'Jul 20'
]
]
},
'001!10274882' => {
'owner' => [
'/prod-datahi',
'001!10274882',
'X',
'rfuse',
'3354796',
'61',
'ts/43',
'13:39:02',
'Jul 20'
],
'waiter' => [
[
'/prod-datahi',
'001!10274882',
'X',
'jmorg',
'3584038',
'247',
'ts/49',
'13:39:22',
'Jul 20'
]
]
},
'00151120273' => {
'owner' => [
'/prod-data/J',
'00151120273',
'X',
'jmorg',
'3584038',
'247',
'ts/49',
'13:38:12',
'Jul 20'
],
'waiter' => [
[
'/prod-data/J',
'00151120273',
'X',
'jmorg',
'2015244',
'134',
's/109',
'13:48:32',
'Jul 20'
],
[
'/prod-data/J',
'00151120273',
'X',
'gdavi',
'1359996',
'62',
'ts/20',
'13:54:22',
'Jul 20'
]
]
}
};
C:\Old_Data\perlp>
| [reply] [d/l] [select] |
Re: How to reference to array keys?
by jethro (Monsignor) on Jul 26, 2011 at 22:58 UTC
|
my %hash= @array;
print $hash{owner};
Note that calling your array @array and your hash %hash is a bad idea. Variable names should describe the contents
Dump is probably from YAML and map is a perl builin function
| [reply] [d/l] |
Re: How to reference to array keys?
by Not_a_Number (Prior) on Jul 26, 2011 at 21:17 UTC
|
It would help us to help you, tremendously, if you were to give the expected output from your code :).
| [reply] |
Re: How to reference to array keys?
by Marshall (Canon) on Jul 27, 2011 at 12:03 UTC
|
I tried to run the OP's code but there were some fatal errors. I re-wrote the code and I hope my explanation of it will help the OP.
The original code takes the approach of reading all of the data into a single variable, then splitting it out again to an array, then there are all sorts of subroutines to get rid of this, get rid of that...
I don't understand what the end objective really is, but from reading the OP's code, this whole thing has to do with record_id's. So I decided upon a hash structure keyed to record_id's.
There is a fundamental difference in the parsing approach. Rather than read the whole thing in and then go searching around for stuff to delete, I read each line and decide what to keep. Deciding what to keep is different than deciding what to throw away.
The file format illustrates something to avoid if you are designing a log file format. What a log line means is state dependent upon what a previous line said. There are of course reasons to have complex records in a log file, but very often a de-normalized flat "every line speaks for its self" is the best.
Anyway, the data is read line by line. If it is important, something happens. Everything that is not important is ignored. The only two types of lines that matter are: FILENAME lines and data lines. The lines starting with FILENAME tells us how to interpret the 4th field of subsequent data lines. So if a FILENAME line is seen, this bit of status is saved. A list slice is used to just get the 4th thing on the line.
It appears that all of the data lines start with a '/' so I picked that to check against. If a data line is seen, then I get the 2nd column (the record number) and use that in conjunction with the state variable that tells us whether this is the owner or waiting. Then that line gets saved in the data structure.
So that's it! Two if statement do the whole job! There is no "oh, this is a special case at the beginning so we throw away the first 10 lines, or this funny line with ':' in column 1 is what ends the data. If the data format was not dependent upon the previous FILENAME line, then there would be only one if statement.
I did not parse each line to the nth degree. Often that's not necessary and here it would just over complicate what is already a complex data structure, HoHoA. There is a simple sub to split a line out into a hash. I return the hash as a flattened list for simplicity.
I guess it will become apparent what kind of reports are needed with an update from the OP. I show one report. The __DATA__ segment and Data::Dump output is long so that is in a readme section.
#!/usr/bin/perl
use strict;
use warnings;
use Data::Dump qw(pp); #Data::Dumper is also great!
my %recordState; #recordid=>OWNER|WATING=>line
my $OwnerOrWaiting="";
while (<DATA>)
{
s/\s+$//; # delete trailing whitespace
if (/^FILENAME/)
{
$OwnerOrWaiting = (split)[3];
}
if (m|^/|)
{
my ($record_id) = (split)[1];
push @{$recordState{$record_id}{$OwnerOrWaiting}}, $_;
}
}
sub get_fields
{
my $line = shift;
my %fields;
@fields{qw(FILENAME RECORD_ID M USER UNBR UNO TTY TIME DATE)} =
split(' ', $line, 9);
return %fields;
}
print "The record numbers are:\n";
print "\t$_\n" foreach (keys %recordState);
# I don't know what kind of queries you want, but
# for example print some basic data for any queue's that
# have 2 or more people waiting...
foreach my $recNum ( keys %recordState)
{
if ( @{$recordState{$recNum}{WAITING}} >= 2)
{
print "Hey, there are at least 2 guys in this crowd!\n";
foreach my $line ( @{$recordState{$recNum}{WAITING}} )
{
my %temp = get_fields($line);
#this next is a hash slice...
print "@temp{'RECORD_ID','USER','TIME','DATE'}\n";
}
}
}
print "Dumping data...\n";
foreach my $recHashref ( values %recordState)
{
print "OWNER $_\n", for @{$recHashref->{OWNER}};
print "WAITING $_\n", for @{$recHashref->{WAITING}};
}
print pp(\%recordState);
=OUTPUTS
The record numbers are:
001!L!311895
001!10274882
00151120273
Hey, there are at least 2 guys in this crowd!
00151120273 jmorg 13:48:32 Jul 20
00151120273 gdavi 13:54:22 Jul 20
Dumping data...
OWNER /prod-data/J 001!L!311895 X jmorg 2015244 134 s/109 13:
+48:32 Jul 20
WAITING /prod-data/J 001!L!311895 X jmorg 5713932 191 ts/46 14:
+01:42 Jul 20
OWNER /prod-datahi 001!10274882 X rfuse 3354796 61 ts/43 13:
+39:02 Jul 20
WAITING /prod-datahi 001!10274882 X jmorg 3584038 247 ts/49 13:
+39:22 Jul 20
OWNER /prod-data/J 00151120273 X jmorg 3584038 247 ts/49 13:
+38:12 Jul 20
WAITING /prod-data/J 00151120273 X jmorg 2015244 134 s/109 13:
+48:32 Jul 20
WAITING /prod-data/J 00151120273 X gdavi 1359996 62 ts/20 13:
+54:22 Jul 20
=cut
| [reply] [d/l] [select] |
|
|
Hey Marshall sorry to double post, but you don't have to answer my previous question about that if statement, it does work it's just that everything below that if statement looks like it's quoted (text is pink, means quote in gedit). If I am thinking correctly it just checks for line that start with a "/". Right?
I wasn't able to get it working because I didn't have the DATA::Dump package, but I do now and the ouput looks pretty good. Nice Work!
Sorry for the newbieism but what does it mean when you have the lines with the array then curly braces with it like this line below?
push @{$recordState{$record_id}{$OwnerOrWaiting}}, $_;
Could you explain that to me?
Thanks,
Matt
| [reply] |
|
|
| [reply] |
|
|
|
|
|
|
|
|
Hey Matt, extra post is no problem at all.
If I am thinking correctly it just checks for line that start with a "/". Right? Yes, Right!
I see you have a link to some more info on references, I'll hit a few highlights specific to this code.
%recordState is a hash with basically 2 keys. In a complex data structure, everything is a reference until you get to the very last dimension where the data is. The value of $recordState{$record_id} is a reference to a another hash. The key to that hash is $OwnwerOrWaiting which is either set to "OWNER" or "WAITING" and the value is a reference to an array of input lines. Look at the output of Data::Dump and see the "key => value" pairs.
Also take a look at my dump loop, foreach my $recHashref ( values %recordState). There I didn't even use the $record_id key at all! I just get the values of the first dimension hash, where are references to the 2nd dimension hash. In the line
print "WAITING $_\n", for @{$recHashref->{WAITING}};
$recHashref is dereferenced and the WAITING key is accessed. The values of that key is a reference to an array of lines. I say that that whole thing, $recHashref->{WAITING} is an array reference that I want to dereference by enclosing it in another set of curly braces and putting an @ in front. Each line gets printed out. Note that I used the exact same code pattern for OWNER as for WAITING. There is only one OWNER and that could have been: print "OWNER $recHashref->{OWNER}[0]\n"; which would mean give me the 0th line in the array pointed to by $recHashref->{OWNER}. There is a certain value in uniformity and so I did not handle the case of just one OWNER separately.
Maybe I confused you even more, but it this was remotely understandable, then the answer to you question above is that value of $recordState{$record_id}{$OwnerOrWaiting} is a reference to an array, which is dereferenced with the @ so that the current line can be pushed onto it.
It could very well be that the data structure can be simplified once I read more about what you are needing as an end result.
| [reply] [d/l] [select] |
|
|
|
|
|
|
|
|
|
Hey Marshall,
I copy/pasted in your code but from this line -->(line 20) if (m|^/|)... From that line on down it looks like you are missing a quotation mark or a backslash in the if.
Should it be more like this? Or something like this?
if (m|/^/|)
Thanks,
Matt
| [reply] |
|
|
| [reply] [d/l] |
Re: How to reference to array keys?
by mmartin (Monk) on Jul 27, 2011 at 13:55 UTC
|
Hey everybody thanks for the responses.
Not_a_Number,
Well the expected output I haven't really gotten there yet. That's why I wanted to know how to reference to those elements, so later on I could easily pull out the correct data I may need.
jwkrahn,
Does it matter if I'm not using a foreach statement? Because I didn't use any foreach statements in the code... Though I do understand the whole thing about adding/removing elements inside a loop because if you splice something in one place then later on if you loop through you would get a uninitialized variable, or something like that.
jethro,
Yea your right about naming the variables that. They are only like that because the code I posted started out as only a test script for using the "map" function. I really just wanted to see how it works, but then it just grew, and grew and grew. But I do plan on changing them.
Cristoforo,
As far as I know I believe the data does have those spaces on the blank lines. So I guess what your saying, like jwkrahn is trying to say I think is, to remove stuff, or better term exclude stuff, as you read in the file instead of afterwards? I would like to give a try to what you posted, but do you think you could explain the lines in the code between the curly braces, that you posted, where it reads in the <DATA>? I'm kinda new to this so it would really help. And for the "record_id", that piece of data will be the same for the person who has the lock and for the person waiting for that record/file.
Marshall,
This is what the end objective should be... The final script will end up being probably a cron job or something along those lines. When the script runs it will find any "locked" files and email to us a small "log" type file with the more pertinent data. Like who has the lock, what file is locked, how long has it been locked, who is waiting for the file, etc...
Yes, you are right. It does make more sense to get rid of/ignore that stuff, as I read the file in. That would probably eliminate most of the lines of code.
Also, thanks for your code, I will give that a try. My original thought was to use hashes. But I have very little experience using hashes and didn't even know where to begin, so that's why I went with an array.
I want to thank you all for posting, and giving me your suggestions. But I am curious, and it was in the original post. What is the purpose of the "map" function on an array. Does using that allow me to access an element within an element in the array?
There is alot here to take in from all your posts. So it may take me a while to get through all your suggestions. I will post back as soon as I get a little further with this stuff from your posts.
Thanks again,
Matt
| [reply] |
|
|
RE: What is the purpose of the "map" function on an array?
In this code:
@array = map { my ($file, $recordID, $M, $owner, $pid, $userID, $tty,
+$time, $month, $day, $locker) }= split;
The map does absolutely nothing useful at all! I'm actually not quite sure that it even works at all.
Anyway it is not needed here.
@array = my ($file, $recordID, $M, $owner, $pid, $userID, $tty, $time,
+ $month, $day, $locker) = split;
map{} is most often used for data transformation. Read map.
@output = map{some code}@input; is a typical usage. Each element of @input enters the map as $_. Some processing is done. The return value which passes to the left into @output is whatever the last line of the map code evaluates to. There is no need for a "return" statement.
| [reply] [d/l] [select] |
Re: How to reference to array keys?
by mmartin (Monk) on Jul 27, 2011 at 14:05 UTC
|
Sorry forgot to show what my output looks like at the time of the post, for whomever it was who had asked (output below). The data doesn't look too bad. I mean it does match the data with a correct "key" for each field, or "identifier" if you want to call it that. It prints the array first with just a plain print statement, then it does a Dump of the array. This is not at all what the output should look like in the end, but for now it will do just fine. Well anyway here it is...
=========================== Array ============================
/prod-data/J 00151120273 X jmorg 3584038 247 ts/49 13:38:12 Jul 20 Loc
+ked
/prod-data/J 00151120273 X jmorg 2015244 134 s/109 13:48:32 Jul 20 Wai
+ting
/prod-data/J 00151120273 X gdavi 1359996 62 ts/20 13:54:22 Jul 20 Wait
+ing
/prod-data/J 001!L!311895 X jmorg 2015244 134 s/109 13:48:32 Jul 20 Lo
+cked
/prod-data/J 001!L!311895 X jmorg 5713932 191 ts/46 14:01:42 Jul 20 Wa
+iting
/prod-datahi 001!10274882 X rfuse 3354796 61 ts/43 13:39:02 Jul 20 Loc
+ked
/prod-datahi 001!10274882 X jmorg 3584038 247 ts/49 13:39:22 Jul 20 Wa
+iting
==============================================================
---
- M: X
day: 20
file: /prod-data/J
locker: Locked
month: Jul
owner: jmorg
pid: 3584038
recordID: 00151120273
time: 13:38:12
tty: ts/49
usedID: 247
- M: X
day: 20
file: /prod-data/J
locker: Waiting
month: Jul
owner: jmorg
pid: 2015244
recordID: 00151120273
time: 13:48:32
tty: s/109
usedID: 134
- M: X
day: 20
file: /prod-data/J
locker: Waiting
month: Jul
owner: gdavi
pid: 1359996
recordID: 00151120273
time: 13:54:22
tty: ts/20
usedID: 62
- M: X
day: 20
file: /prod-data/J
locker: Locked
month: Jul
owner: jmorg
pid: 2015244
recordID: 001!L!311895
time: 13:48:32
tty: s/109
usedID: 134
- M: X
day: 20
file: /prod-data/J
locker: Waiting
month: Jul
owner: jmorg
pid: 5713932
recordID: 001!L!311895
time: 14:01:42
tty: ts/46
usedID: 191
- M: X
day: 20
file: /prod-datahi
locker: Locked
month: Jul
owner: rfuse
pid: 3354796
recordID: 001!10274882
time: 13:39:02
tty: ts/43
usedID: 61
- M: X
day: 20
file: /prod-datahi
locker: Waiting
month: Jul
owner: jmorg
pid: 3584038
recordID: 001!10274882
time: 13:39:22
tty: ts/49
usedID: 247
Thanks,
Matt
| [reply] [d/l] |
Re: How to reference to array keys?
by mmartin (Monk) on Aug 03, 2011 at 14:08 UTC
|
Hello Monks,
In case your wondering about the 2 blank posts above. I had a question about something but I figured it out so I just deleted their contents. It seems that I might be done with the Perl portion of my scripts. Now onto a Shell script to execute and mail the output of my Perl script.
I wanna thank everyone who posted, and I feel much more comfortable with Perl now.
If you want to see my final code just send me a post and I'll put it up on here.
Thanks again for all your help,
Matt
| [reply] |
Re: How to reference to array keys?
by mmartin (Monk) on Aug 02, 2011 at 20:39 UTC
|
| [reply] |
Re: How to reference to array keys?
by mmartin (Monk) on Aug 03, 2011 at 14:01 UTC
|
| [reply] |