Reading Data from a file

Anonymous Monk has asked for the wisdom of the Perl Monks concerning the following question:

Hello,

I have already posted on this forum about this problem. I did some changes and printed out all the output I get from the program and I still do not understand why I am getting this problem.

The main goal is to read data from files. In here I read data from the group and passwd file on a linux box. I want to isolate some data, such as groups and specific users to generate new group and passwd file on a different system.

The problem happening in the CleanFiles function. When I call it twice, the second time it returns data of the previous call as well as the data from this call. I am trying to understand why the data passed in a previous call stay and are return with the data of the next call.

I have tried to call the function 3 times and the third time I get the data of the two previous calls as well as the data from the last call.

Any suggestion? I am relatively new to Perl.


#!/usr/bin/perl

###################
#Packages required#
###################

use strict;
use List::MoreUtils qw(uniq);
use Data::Dumper;
use warnings;

############
#Parameters#
############

#First we define the path to the file we use
#do not put any /
my $dac02_location = "/Users/tuxy94/Scripts/tmp/dac03_conf";
my $dac03_location = "/Users/tuxy94/Scripts/tmp/dac03_conf/etc";

#name of the groups
my @group = ('jai','smbuser','domadm');
#number of max user on the system
my $i = 8;
#name of additional machines that are in passwd but not in group
my @add_machines=('dacsrv01', 'dacwks01', 'dacwks02', 'dacwks03', 'dac
+wks04', 'dacwks05', 'daceufo201', 'dacluxe02', 'dacluxe03');
#special users to also update the password for
my @special_users=('root','nagios');

###########
#Functions#
###########

#To parse the data and store it in a hash
sub DataParsing{
    my ($file, @fields) = @_;

    my %data = ();
    my $item;

    open(DATA_PARSING, $dac02_location."/".$file) or die "Cannot open 
+the file!";

    while(<DATA_PARSING>){
        chomp;

        foreach $item (@fields){
            if($_ =~ /$item/){
                $data{$item}=[split('[:,]',$_)];
                $data{$item.'_raw'} = $_;
            }
        }
    }

    close(DATA_PARSING);

    return %data;
}

#Clean the file group and return an array with clean data to construct
+ the new one
sub CleanFiles{
    my ($file, @lines) = @_;

    my @data_file = ();

print "Dump of data_file in function:\n";
print Dumper(@data_file);

#@data_file is well reset each time I call the function but still
#the data from the previous call are in the second call result...
#WHY???
#I think it is the container DATA_CLEANING that is not proprelly empti
+ed
#How to check this?

    open(DATA_CLEANING, $dac03_location."/".$file) or die "Cannot open
+ the file!";

    #Store the text in an array, easier to work with
    while(<DATA_CLEANING>){
        chomp;
        push(@data_file,$_);
        #print "$_\n";
    }
    close(DATA_CLEANING);

    foreach my $index (reverse 0 .. $#data_file){
        foreach my $item (@lines){
            if ($data_file[$index] =~ /$item/){
                splice @data_file, $index, 1;
                #print "The item to be spliced: $item\n";
            }
        }
    }

    return @data_file;
}

#function to generate the new config files
sub CreateConfig{
    my ($file,@data) = @_;
    my $item;

    $file = $file.".new";

    open(NFILE,">$file");
    foreach $item (@data){
        print NFILE "$item\n";
    }
    close(NFILE);

    return $file;
}

###########
#Instances#
###########

#hash of the group with the group name as reference
my %group = DataParsing('group', @group);
#print "$group{jai}[2]\n";

#put the usernames in an array to get info out of passwd
my $item;
my @username = ();
foreach $item (@group){
    for ($a = $i+1; $a > 2; $a--){
        if($group{$item}[$a] ne ''){
            push(@username,$group{$item}[$a])
        }
    }
}

#array containing all of the user that need to be added on system
my @usernames = uniq(@username,@add_machines);

my @special_usernames = uniq(@usernames,@special_users);
#hash of the user in passwd with username as reference
my %passwd = DataParsing('passwd',@special_usernames);

#hash of the password in shadow with username as reference
my %shadow = DataParsing('shadow',@special_usernames);

########################################################
########################################################

my @group_clean_data = CleanFiles('group',@group);

my @passwd_clean_data = CleanFiles('passwd',@special_usernames);

print "Print of the group_clean_data outside of function:\n";
print Dumper(@group_clean_data);

print "Print of passwd_clean_data outside of function:\n";
print Dumper(@passwd_clean_data);
[download]

And the output I get from this code:


Dump of data_file in function:
Dump of data_file in function:
Print of the group_clean_data outside of function:
$VAR1 = 'root:x:0:';
$VAR2 = 'daemon:x:1:';
$VAR3 = 'bin:x:2:';
$VAR4 = 'sys:x:3:';
$VAR5 = 'adm:x:4:';
$VAR6 = 'tty:x:5:';
$VAR7 = 'disk:x:6:';
$VAR8 = 'lp:x:7:';
$VAR9 = 'mail:x:8:';
$VAR10 = 'news:x:9:';
$VAR11 = 'uucp:x:10:';
$VAR12 = 'man:x:12:';
$VAR13 = 'proxy:x:13:';
$VAR14 = 'kmem:x:15:';
$VAR15 = 'dialout:x:20:reyndce';
$VAR16 = 'fax:x:21:';
$VAR17 = 'voice:x:22:';
$VAR18 = 'cdrom:x:24:reyndce';
$VAR19 = 'floppy:x:25:reyndce';
$VAR20 = 'tape:x:26:';
$VAR21 = 'sudo:x:27:';
$VAR22 = 'audio:x:29:reyndce';
$VAR23 = 'dip:x:30:';
$VAR24 = 'www-data:x:33:apache';
$VAR25 = 'backup:x:34:';
$VAR26 = 'operator:x:37:';
$VAR27 = 'list:x:38:';
$VAR28 = 'irc:x:39:';
$VAR29 = 'src:x:40:';
$VAR30 = 'gnats:x:41:';
$VAR31 = 'shadow:x:42:';
$VAR32 = 'utmp:x:43:';
$VAR33 = 'video:x:44:reyndce';
$VAR34 = 'sasl:x:45:';
$VAR35 = 'plugdev:x:46:reyndce';
$VAR36 = 'staff:x:50:';
$VAR37 = 'games:x:60:';
$VAR38 = 'users:x:100:';
$VAR39 = 'nogroup:x:65534:';
$VAR40 = 'crontab:x:101:';
$VAR41 = 'Debian-exim:x:102:';
$VAR42 = 'ssh:x:103:';
$VAR43 = 'reyndce:x:1000:';
$VAR44 = 'mysql:x:104:';
$VAR45 = 'ssl-cert:x:105:';
$VAR46 = 'postfix:x:106:';
$VAR47 = 'postdrop:x:107:';
$VAR48 = 'nagios:x:1001:';
$VAR49 = 'nagcmd:x:1002:nagios,www-data';
$VAR50 = 'bind:x:108:';
$VAR51 = 'ntp:x:109:';
$VAR52 = 'apache:x:72:';
$VAR53 = 'slocate:x:110:';
Print of passwd_clean_data outside of function:
$VAR1 = 'daemon:x:1:';
$VAR2 = 'bin:x:2:';
$VAR3 = 'sys:x:3:';
$VAR4 = 'adm:x:4:';
$VAR5 = 'tty:x:5:';
$VAR6 = 'disk:x:6:';
$VAR7 = 'lp:x:7:';
$VAR8 = 'mail:x:8:';
$VAR9 = 'news:x:9:';
$VAR10 = 'uucp:x:10:';
$VAR11 = 'man:x:12:';
$VAR12 = 'proxy:x:13:';
$VAR13 = 'kmem:x:15:';
$VAR14 = 'fax:x:21:';
$VAR15 = 'voice:x:22:';
$VAR16 = 'tape:x:26:';
$VAR17 = 'sudo:x:27:';
$VAR18 = 'dip:x:30:';
$VAR19 = 'www-data:x:33:apache';
$VAR20 = 'backup:x:34:';
$VAR21 = 'operator:x:37:';
$VAR22 = 'list:x:38:';
$VAR23 = 'irc:x:39:';
$VAR24 = 'src:x:40:';
$VAR25 = 'gnats:x:41:';
$VAR26 = 'shadow:x:42:';
$VAR27 = 'utmp:x:43:';
$VAR28 = 'sasl:x:45:';
$VAR29 = 'staff:x:50:';
$VAR30 = 'games:x:60:';
$VAR31 = 'users:x:100:';
$VAR32 = 'nogroup:x:65534:';
$VAR33 = 'crontab:x:101:';
$VAR34 = 'Debian-exim:x:102:';
$VAR35 = 'ssh:x:103:';
$VAR36 = 'mysql:x:104:';
$VAR37 = 'ssl-cert:x:105:';
$VAR38 = 'postfix:x:106:';
$VAR39 = 'postdrop:x:107:';
$VAR40 = 'bind:x:108:';
$VAR41 = 'ntp:x:109:';
$VAR42 = 'apache:x:72:';
$VAR43 = 'slocate:x:110:';
$VAR44 = 'daemon:x:1:1:daemon:/usr/sbin:/bin/sh';
$VAR45 = 'bin:x:2:2:bin:/bin:/bin/sh';
$VAR46 = 'sys:x:3:3:sys:/dev:/bin/sh';
$VAR47 = 'sync:x:4:65534:sync:/bin:/bin/sync';
$VAR48 = 'games:x:5:60:games:/usr/games:/bin/sh';
$VAR49 = 'man:x:6:12:man:/var/cache/man:/bin/sh';
$VAR50 = 'lp:x:7:7:lp:/var/spool/lpd:/bin/sh';
$VAR51 = 'mail:x:8:8:mail:/var/mail:/bin/sh';
$VAR52 = 'news:x:9:9:news:/var/spool/news:/bin/sh';
$VAR53 = 'uucp:x:10:10:uucp:/var/spool/uucp:/bin/sh';
$VAR54 = 'proxy:x:13:13:proxy:/bin:/bin/sh';
$VAR55 = 'www-data:x:33:33:www-data:/var/www:/bin/sh';
$VAR56 = 'backup:x:34:34:backup:/var/backups:/bin/sh';
$VAR57 = 'list:x:38:38:Mailing List Manager:/var/list:/bin/sh';
$VAR58 = 'irc:x:39:39:ircd:/var/run/ircd:/bin/sh';
$VAR59 = 'gnats:x:41:41:Gnats Bug-Reporting System (admin):/var/lib/gn
+ats:/bin/sh';
$VAR60 = 'nobody:x:65534:65534:nobody:/nonexistent:/bin/sh';
$VAR61 = 'Debian-exim:x:100:102::/var/spool/exim4:/bin/false';
$VAR62 = 'statd:x:101:65534::/var/lib/nfs:/bin/false';
$VAR63 = 'identd:x:102:65534::/var/run/identd:/bin/false';
$VAR64 = 'sshd:x:104:65534::/var/run/sshd:/usr/sbin/nologin';
$VAR65 = 'mysql:x:103:104:MySQL Server,,,:/var/lib/mysql:/bin/false';
$VAR66 = 'postfix:x:105:106::/var/spool/postfix:/bin/false';
$VAR67 = 'bind:x:106:108::/var/cache/bind:/bin/false';
$VAR68 = 'ntp:x:107:109::/home/ntp:/bin/false';
[download]

Comment on Reading Data from a file Select or Download Code

Replies are listed 'Best First'.
Re: Reading Data from a file by moritz (Cardinal) on Apr 03, 2008 at 10:04 UTC
I haven't looked through all of your script, so just two small hints: 1 Use lexical variables as file handles: `open my $handle, '<', $filename or die "Can't open '$filename' for reading: $!"; while (<$handle>){ ... } close $handle` [download] This will avoid trouble if you forget to close a filehandle, or call functions recursively while a handle is still open 2 If you want to dump the output of an array, set a backslash in front of it: `my @list = qw(foo bar baz); print Dumper \@list; __END__ VAR1 = [ 'foo', 'bar', 'baz' ];` [download] This makes it clearer that it's one variable, and you can easily see where it ends.	[reply] [d/l] [select]
Re^2: Reading Data from a file by Anonymous Monk on Apr 03, 2008 at 13:36 UTC
Thanks I did the change but it did not fix my original problem. Does anyone know why I am getting this problem with the second function and not with the first function when they are both build the same. Thanks.	[reply]
Re^3: Reading Data from a file by Corion (Patriarch) on Apr 03, 2008 at 13:49 UTC
I briefly looked at your post, but it contains far too much data and far too much code. Please, reduce the data to two or three lines of data. Also, take out all the code that does not relate to the problem. For example, the reading of `$dac02_location` is useless because we don't know the content of the file. Is the subroutine `CreateConfig` relevant to the problem or not? Also, please provide us with the output you expect and the output you get. It's completely unclear to me why the output you provide is not the output you expect, because all the lines in it are different. And is it really necessary to show us 60 lines of output? I think that three or four relevant lines would be far better. From what the comments seem to say, I infer that you might be trying to find the users on one machine. But it seems to me that the extraction of user names does not work, because the output of your script does not contain usernames but complete `passwd`-style entries. My guess is that possibly your username extraction is wrong or your loop code to find whether a user already exists on the other machine or not. But it's hard to tell in this large pile of code. I suggest you replace the loop for finding missing users with a hash lookup, as perlfaq4 says when searching for "duplicate".	[reply] [d/l] [select]
Re: Reading Data from a file by dwm042 (Priest) on Apr 03, 2008 at 14:00 UTC
Anon, Your suspicions are in fact correct. The subroutine CleanFiles only opens your (temporary) data file and does not remove data from that file. At the end of the run of your program you'll need to do some kind of file cleanup. One way of doing it would be a command like: `system("> $my_temp_file");` [download] A more Perlish (and safer) solution would be to use File::Temp to handle creation and destruction of the temp file for you.	[reply] [d/l]