String searching in file

cbtshare has asked for the wisdom of the Perl Monks concerning the following question:

Replies are listed 'Best First'.
Re: String searching in file by poj (Abbot) on Feb 09, 2018 at 20:34 UTC
if (grep /$line/, $awsLists ) Using perl, grep on the lines in the file not the filename. For example (using test data instead of your command output) #!/usr/bin/perl use warnings; use strict; my $awsLists = '/ansible/awsLists'; my @awsLists = (); if (-e $awsLists){ open my $fh,'<',$awsLists or die "$!"; @awsLists = <$fh>; close $fh; } my $cmd = 'aws ec2 describe-instances --filters "Name=instance-state-n +ame,Values=running" \| grep PrivateDnsNam$'; my @output = <DATA>;#qx/$cmd/; open my $fh,'>>',$awsLists or die "$!"; foreach my $line (@output){ chomp($line); if (grep /$line/, @awsLists ){ print "$line found\n"; } else { print "$line not found - added\n"; print $fh $line."\n"; } } close $fh; __DATA__ test 1 test 89 [download] poj	[reply] [d/l]
Re^2: String searching in file by cbtshare (Monk) on Feb 09, 2018 at 22:06 UTC
Thank you very much, I see what you have done, checking if the file exists, opening it for reading, then opening it again for writing.I was kinda trying to avoid that, and just skip those extra steps and search if it ip address is in the file to begin with, if not write to it. Is there no way to do that, without opening the file for reading and writing?	[reply]
Re^2: String searching in file by cbtshare (Monk) on Feb 09, 2018 at 23:02 UTC
Sorry but looking at your code more closely, I dont see how it will work, sorry if I didnt put all the details. $awsLists isn't populated, so my goal is to populate the file, then check if ip is there, and not do anything, but if it is there write to the file @command=`aws ec2 describe-instances --filters "Name=instance-state-na +me,Values=running" \| grep PrivateDnsName \| cut -d ":" -f2\| cut -d '" +' -f2 \| sort -u `; [download] Result `ip-10-1-68-84.us-west-2.compute.internal ip-10-1-7-183.us-west-2.compute.internal ip-10-1-80-15.us-west-2.compute.internal ip-10-1-80-162.us-west-2.compute.internal ip-10-1-80-184.us-west-2.compute.internal ip-10-1-80-187.us-west-2.compute.internal ip-10-1-80-18.us-west-2.compute.internal ip-10-1-80-57.us-west-2.compute.internal` [download]	[reply] [d/l] [select]
Re: String searching in file by Laurent_R (Canon) on Feb 09, 2018 at 22:07 UTC
I think that the best way is to load your file into a hash (each IP as a key, the hash value doesn't matter), close the file; then add each IP in your array to the hash (a hash will remoce duplicates anyway); once you've finished that, just write the hash keys (i.e. unique IP addresses) to a file (or to the same file if you're confident that it works OK). A quick untested example: `open my $IN, "<", $awsLists or die "could not open $awsLists $!"; my %hash = map { chomp; $_ => 1 } <$IN>; close $IN; chomp @command; # just in case you need it. Note: you could choose a b +etter name for your array $hash{$_} = 1 for @command; open my $OUT, ">", "$awsLists.out" or die "could not open $awsLists.ou +t $!"; print "$_\n" for keys %hash; close $OUT;` [download] Using a hash lookup will be usually be much faster than greping your file or its content many times.	[reply] [d/l]
Re^2: String searching in file by cbtshare (Monk) on Feb 09, 2018 at 23:04 UTC
thanks, will test it out.	[reply]
Re: String searching in file by tybalt89 (Monsignor) on Feb 10, 2018 at 00:05 UTC
Here's one similar to Re: String searching in file with a couple of extra features. If file does not exist, that's OK, will make one. Leaves \n on ip names for ease of handling. Only writes file if new ips are added. Note: filename and @commands have to be fixed, I changed them for testing purposes. #!/usr/bin/perl # http://perlmonks.org/?node_id=1208859 use strict; use warnings; my $awsLists = 'd.1208859'; # change to your filename my @commands = <DATA>; # new ips (with \n) my %old; if( open my $in, '<', $awsLists ) # OK if file not there { @old{ <$in> } = (); # note: keys have \n close $in; } my $oldsize = keys %old; @old{ @commands } = (); # add new ips if( keys %old > $oldsize ) # only write if new ips added { open my $out, '>', $awsLists or die "$! opening $awsLists"; print $out sort keys %old; # sort not needed, but nice touch :) close $out; } __DATA__ ip-10-1-129-212.us-west-2.compute.internal ip-10-1-129-168.us-west-2.compute.internal ip-10-1-129-212.us-west-2.compute.internal ip-10-1-129-158.us-west-2.compute.internal ip-10-1-129-78.us-west-2.compute.internal ip-10-1-1-145.us-west-2.compute.internal ip-10-1-65-215.us-west-2.compute.internal ip-10-1-129-194.us-west-2.compute.internal ip-10-1-1-252.us-west-2.compute.internal ip-10-1-67-133.us-west-2.compute.internal [download]	[reply] [d/l]
Re^2: String searching in file by cbtshare (Monk) on Feb 10, 2018 at 06:03 UTC
thank you very much, my final code and explanation is below, please let me know if I am not understanding something use warnings; use strict; my $line; my $awsLists = '/ansible/awsLists'; my @command; @command=`aws ec2 describe-instances --filters "Name=instance-state-na +me,Values=running" \| grep PrivateDnsName \| cut -d ":" -f2\| cut -d '" +' -f2 \| sort -u `; ##Initializes an empty hash my %old; ##Checks if the file is there to be opened and read into the handle $i +n if( open my $in, '<', $awsLists ) # OK if file not there { ##If file is there then read contents into keys and associating a valu +e of blank or () means new line? ## why wouldn't you do @old{<$in>} = undef; , would it be wrong? @old{ <$in> } = (); # note: keys have \n close $in; } ###This takes the keys from the the hash previously populate from the +file and puts it into scalar my $oldsize = keys %old; ## Populates the hash again, but this time with the values of the @com +mand array ## Does the old keys get overwritten?(I dont think so, eg, an array + values dont get overwritten everytime you push new values) @old{ @command } = (); # add new ips #Does the compare, and if the hash keys from the <$in> and different f +rom the hash keys of @command if( keys %old > $oldsize ) # only write if new ips added { open my $out, '>', $awsLists or die "$! opening $awsLists"; ##print directly to the file the new keys difference print $out sort keys %old; # sort not needed, but nice touch :) close $out; } # system qq{sed -i '/^ *\$/d' $awsLists}; [download]	[reply] [d/l]
Re^3: String searching in file by poj (Abbot) on Feb 10, 2018 at 07:51 UTC
###This ~~takes~~ counts the keys ~~from the the~~ in the hash previously populate from the file and puts it into scalar `my $oldsize = keys %old;` #Does the compare, and if the count of hash keys ~~from the <$in> and different from the hash keys of @command~~ has increased (i.e. new keys added) `if( keys %old > $oldsize ) # only write if new ips added` ## If file is there then read contents into keys and associating a value of blank or () means new line? ## why wouldn't you do @old{<$in>} = undef; , would it be wrong? `@old{ <$in> } = (); # note: keys have \n` In this case no difference but possibly misleading you to think `@old{<$in>} = 1;` would assign the value 1 to all the keys from `<$in>` which is not the case. Try this simple example `#!/usr/bin/perl use strict; use Data::Dumper; my @keys = ('a','b','c'); my %hash=(); @hash{@keys} = 1; print Dumper \%hash; __END__ $VAR1 = { 'c' => undef, 'a' => 1, 'b' => undef };` [download] poj	[reply] [d/l] [select]
Re: String searching in file by marinersk (Priest) on Feb 09, 2018 at 20:08 UTC
I don't see your results in the post; nor what was expected and why it's different. I see an opening backtick on the `@command=`line, but no closing backtick. Is that normal?	[reply] [d/l]
Re^2: String searching in file by cbtshare (Monk) on Feb 09, 2018 at 22:02 UTC
sorry about that: @command=`aws ec2 describe-instances --filters "Name=instance-state-na +me,Values=running" \| grep PrivateDnsName \| cut -d ":" -f2\| cut -d '" +' -f2 \| sort -u `; [download] Sample Results in @command: `ip-10-1-129-212.us-west-2.compute.internal ip-10-1-129-168.us-west-2.compute.internal ip-10-1-129-212.us-west-2.compute.internal ip-10-1-129-158.us-west-2.compute.internal ip-10-1-129-78.us-west-2.compute.internal ip-10-1-1-145.us-west-2.compute.internal ip-10-1-65-215.us-west-2.compute.internal ip-10-1-129-194.us-west-2.compute.internal ip-10-1-1-252.us-west-2.compute.internal ip-10-1-67-133.us-west-2.compute.internal` [download] My issue is that it keeps on printing "not found" even when the ip address is in the file	[reply] [d/l] [select]