MrGibbs has asked for the wisdom of the Perl Monks concerning the following question:

I would like to a search and place, with a list of patterns on one file, and the lines to be worked on in a different file. I believe I could do this in a full script if I had more time, but alas, time is running out. The contents of the pattern file looks remarkably like a list of ip addresses, 1 per line. The file to be worked on is a large script that does things with ip addresses. I would like to find all instances of ip addresses, 1 at a time and if they are not already commented out, comment them out with a leading pound sign. So a one-liner to print would be:

perl -pe /print/ ip_addr_file
A one-liner to do a simple substitution would be:
perl -pi -e s/^/#/ if /66.11.10.20/' script_file
I would like to do both at once. Step through the list of ip addresses, one at a time, and use that pattern to search the script_file. When I find it in there, then comment that line out of the script_file. Something like this:
perl -pi -e "s/^/#/ if /`perl -pe /print/ ip_addr_file` /" script_file
My perl skills are rusty enough that I cannot quite get there. Looking for wisdom. Thanks!

Update:

I was tried seveal different variations and suggestions to try and do this in a perl one-liner. I got close, but could not coax it into doing what I wanted. I reverted to one of the first suggestions and did it in a perl script.

First let me say thanks to the monks, and in particular to jeffa, FreeBeerReekingMonk, Laurent_R, and aaron_baugher for your suggestions and time. It is appreciated.

I've been relunctant to share my code only because my employer is a little parnoid about putting information and data on the Internet. (It has been suggested that we should not share company name and job title on LinkedIn.) I am not quite that paranoid, but I don't need to poke the bear either.

I'll provide some details about this task. We have a script to add static routes on a Solaris box. It was maintained by hand for many years. Over that time, different people did that task different ways. It was decided to move to linux. linux has a routes file that can be used to populating static routes, but the format of the file is different than the route command. Also embedded in the static route file is comments that pertain to a given route (like change request number, customer, etc.) The problen at first was to change the lines in the static routes file that add routes, while not changing the content of the comment lines, and not changing the order of the lines in the file. That became more complicated when it was discovered that there were many variances in the format of the file; it really became a data normalization problem. In Solaris, you can add a route or delete a route. In linux, they only care about the route adds. So if I did this conversion, if there was a delete route on the Solaris box, it would be carried as an add route on the linux box.

I did a lot of the data massaging by hand with grep, fgrep, awk and vi. This perl script was to go into the static routes file, find instances of a stand-alone delete commands, and then comment it out. I was able to do that (thanks!). Originally, I was going to do the data conversion purely in vi. I could go to particular lines in the file, manipulate only those, and leave the others in place. Alas, it got too complicated for that.

Although, I don't feel comfortable sharing the perl script, I can share this nugget I discovered within vi:

:g/XXX/s/^\([^ ]*\) \([^ ]*\) \([^ ]*\) \([^ ]*\)/\1 \4 \3 \2/
That will swap the 2nd and 4th fields on every line with XXX .

Thanks again. MrGibbs

Replies are listed 'Best First'.
Re: one liner with separate input file?
by jeffa (Bishop) on Apr 13, 2015 at 20:08 UTC

    If the substitution doesn't match, then the substitution is not performed, so all you need to do is combine the two one liners that you have into one:

    perl -pi -e's/^(66.11.10.20)/#$1/g' ips.txt

    Now then, stepping through the list of IP addresses is another issue. I would recommend using a script for this rather than a one liner.

    Update: i'm sticking to my case, but this is too much fun. :) Let's say you have 2 text files, one containing the list of all IP addresses (all_IPs.txt) and another containing a smaller list of IP addresses to be commented out in the first text file (decomission_IPs.txt). You could use the shell to create individual strings that contain the Perl one-liners that can in turn be fed back to a shell:

    for i in `cat decomission_IPs.txt`; do perl -le $"print 'perl -pi -e \ +'s/^(\Q$i\E)/#\$1/g\' all_IPs.txt' "; done | sh
    I tried using xargs but it is just too tricky to get that IP address interpolated.

    jeffa

    L-LL-L--L-LL-L--L-LL-L--
    -R--R-RR-R--R-RR-R--R-RR
    B--B--B--B--B--B--B--B--
    H---H---H---H---H---H---
    (the triplet paradiddle with high-hat)
    
      Thanks for responding jeffa.

      I tried your solution with a for..do loop, but it did not work. It appeared to be running for a small amount of time, but did not do any edits. I tried to run it from ksh and bash, and tried to specify ksh and bash in place of sh, but got no joy.

      I thought about trying to do this in xargs, but I too had problems getting it correct. That is why I thought perl might be better.

      There are more details in the back story of how I got to this point, but I wasn't sure it was needed for the perl discussion.

Re: one liner with separate input file?
by Laurent_R (Canon) on Apr 13, 2015 at 22:04 UTC
    I agree with jeffa that this is getting a bit complicated for a one-liner. But a one-liner does not have to be only one line. Or you might want to call it a "two-liner". This compiles cleanly, but I haven't really tested it with actual data. Besides, it certainly does not take care of edge cases. This is just a clue for a possible pure Perl (no Bash) solution.
    perl -pi -e 'BEGIN{ local @ARGV = ("IP_addr.txt"); %IP = map {$_ => 1; +} <>; } $_ = "#" . $_ if /((\d{1,3}\.){3}(\d{1,3}))/ and exists $IP{$1} +;'

    Je suis Charlie.
      Thanks for responding Laurent_R.

      I don't think I need to worry about the edge cases. I did several steps before I got here, with massaging and normalizing the data. Your solution is like jeffa's. It seemed to execute, but did not edit the file to be worked on. I did add the name of that file to the end of the command you suggested.

      I'll keep poking at this as time permits.

        True, I forgot to add the name of the file at the end of the command line. Sorry about that.

        But I can't say what went wrong with what you are saying in your comments.

        Please give full details on what you are trying: exact command line used, sample input files, OS on which you are working, any other thing worth of interest (such as warnings or errors).

        Je suis Charlie.
Re: one liner with separate input file?
by FreeBeerReekingMonk (Deacon) on Apr 13, 2015 at 21:40 UTC

    You want bash/ksh do the hard work, just run this:

    while read ip;do perl -pi -e s/^/#/ if /$ip/ script_file; done < ip_ad +dr_file

      Thanks for responding FreeBeerReekingMonk.

      I tried your suggestion of a while loop, with redirect in. I am using a test file of 5 ip addresses instead of the whole list. I also have a copy of the original script to see how each solution works. The stdout produced:

      Can't open if: No such file or directory.
      Can't open /66.22.11.24/: No such file or directory.
      Can't open if: No such file or directory.
      Can't open /66.22.44.28/: No such file or directory.
      Can't open if: No such file or directory.
      Can't open /66.22.44.18/: No such file or directory.
      Can't open if: No such file or directory.
      Can't open /66.33.22.23/: No such file or directory.
      Can't open if: No such file or directory.
      Can't open /66.33.22.24/: No such file or directory.

      The script did get modified, but it put a "#" at the begining of every line, and repeated it 5 times.

        hmm, somehow the quotes did not survive, I meant:

        while read ip;do perl -pi -e "s/^/#/ if /$ip/" script_file; done < ip_ +addr_file

        Also, if you do not want to match 1.2.3.444 with 1.2.3.4, then \b could help:
        while read ip;do perl -pi -e "s/^/#/ if /\b$ip\b/" script_file; done < + ip_addr_file

        And yes, it modifies the file immediately, and if you run it multiple times, it will keep adding #'s to each line that needs to be REMarked.
        In order to NOT do that, I finally propose:
        while read ip;do perl -pi -e "s/^#?/#/ if /\b$ip\b/" script_file; done + < ip_addr_file

        Drawbacks: It will iterate over the full file for each IP address. There are other solutions proposed, like the one with map, which is more efficient and faster.

Re: one liner with separate input file?
by aaron_baugher (Curate) on Apr 15, 2015 at 21:35 UTC

    I wouldn't do this as a one-liner, but of course you can. But rather than loop through your IPs and check each of them against the other file (or vice versa), put the IPs in a hash, then go through the other file looking for IP patterns, check them against the hash, and comment them if they're found.

    $ cat ips 111.222.333.444 55.66.77.88 $ cat text Line 1 nothing to comment Line 2 111.222.333.444 should be commented Line 3 11.22.33.44 do not comment Line 4 55.66.77.88 should be commented $ perl -pe '@l = `cat ips`;chomp @l; %h = map {$_=>1} @l; s/(\d{1,3}\. +\d{1,3}\.\d{1,3}\.\d{1,3})/$h{$1}?"#$1":$1/e;' <text

    Aaron B.
    Available for small or large Perl jobs and *nix system administration; see my home node.