Parsing ldap attributes into a csv

lvirden has asked for the wisdom of the Perl Monks concerning the following question:

My goal is to ldapsearch for information about the devices registered, and turn the resulting answers into csv format, so that they can be imported into a configuration management database. I have a script that generates data such as:

cn: epcpc5
host: epcpc5
description: research server
l: Narnia Data Center

cn: epcmsp
host: epcmsp
description: Service Processor
l: Narnia Data Center

cn: post62a
host: post62a
description: network printer
keyWords: netprinter
l: Narnia Data Center

cn: tst401
host: tst401
host: tst401.narnia.org
host: srv1ck41
description: Test Application Server
env: test
keyWords: linuxserver
keyWords: nonprod
keyWords: localperl
keyWords: sudoers
l: Data Center, Narnia
sysadm1: aslan
administratorContactInfo: sciops
nsHardwarePlatform: DellR730
[download]

I searched the net and found a perl example that did a subset of what I needed for user information. So I because modifying it to handle my cases - combine repeating attributes, such as keyWords into one csv field, quoted and comma deliminted within the quotes, different column names, etc. My code is currently this:

#!/usr/bin/env perl
use strict;
use warnings;

# Bail out if no file is given as an argument.
my $ldap_file = shift || usage();

# To save output with extension .csv.
my $csv_file = "$ldap_file.csv";

# Preserve order of header labels.
my @header = qw(cn host description env keyWords l sysadm1 sysadm2 adm
+inistratorContactInfo nsHardwarePlatform version);

# Boiler plate to open input and output files handles.
open my $in, '<', $ldap_file or die;
open my $out, '>', $csv_file or die;

# Write the header labels.
print  $out join (',', @header[0..$#header]), "\n";

# Work engine.
{
    # Record separator is a paragraph representation.
    local $/="\n\n";

    # Process one record at a time.
    while(my $record = <$in>) {
        my %data = ();
        chomp $record; # Remove the record separator.
        $data{@header} = ();  # To label the data.
        my @entries = split '\n', $record; # Create data entries.

        # Work with each entry.
        foreach my $entry (@entries){
            if ($entry eq "") {
                next;
            }
           
            my ($key, $value) = split ":", $entry, 2; # split only by 
+first equal.
            if ($key eq "" or $key eq "dn") {
                next;
            }
            if ($value eq "") {
                print "empty value: entry=$entry\n";
                }
            $value =~ s/^\s+//; # Clean leading spaces.
            $value =~ s/\s+$//; # Clean trailing spaces.
            if ( exists $data{$key} ) {
                $data{$key} .= ("," . $value);
            } else {
                $data{$key} = $value;
            }
        }

        # Separate entries with commas.
        foreach my $key (keys %data) {
                print "key=$key, value=$data($key)\n";
            $data($key) =~ s/(^|$)/"/g if $data($key) =~ /,/; 
        }
        my $row = join ',', (map{ $data{$_}?$data{$_}:""} @header);

        # Write to output file.
        print $out "$row\n";
    }
}
# Dismiss file handles.
close $in;
close $out;

# Feed back to user.
print "$csv_file has been saved in the current directory\n";

sub usage {
    print "Usage: $0 ldapfilename\n";
    exit 1;
}
[download]

When I attempt to run this using perl v5.8.4 I get

Global symbol "$data" requires explicit package name at ./ldap2csv.pl 
+line 58.
Global symbol "$data" requires explicit package name at ./ldap2csv.pl 
+line 59.
syntax error at ./ldap2csv.pl line 59, near "$data("
Global symbol "$data" requires explicit package name at ./ldap2csv.pl 
+line 59.
Execution of ./ldap2csv.pl aborted due to compilation errors.
[download]

The line numbers within these messages are these lines

 # Separate entries with commas.
        foreach my $key (keys %data) {
                print "key=$key, value=$data($key)\n";
            $data($key) =~ s/(^|$)/"/g if $data($key) =~ /,/;
        }
[download]

There is a my %data within the while block where all these code is taking place. I don't quite understand what I need to do, when referencing the hash/key to get a value, so that perl strict is happy with the references. The various results I found web searching didn't cover additional safety precautions to be taken in this type of situation. I am hoping someone out there has an answer for me.

Comment on Parsing ldap attributes into a csv Select or Download Code

Replies are listed 'Best First'.
Re: Parsing ldap attributes into a csv by toolic (Bishop) on Mar 02, 2018 at 19:45 UTC
To access a hash key, use curly brackets, not parens. This fixes the errors: `foreach my $key (keys %data) { print "key=$key, value=$data{$key}\n"; $data{$key} =~ s/(^\|$)/"/g if $data{$key} =~ /,/; }` [download]	[reply] [d/l]
Re^2: Parsing ldap attributes into a csv by lvirden (Novice) on Mar 02, 2018 at 20:22 UTC
Thank you for your advice - that helped a lot. After making that change, I now get the message: `Use of uninitialized value in pattern match (m//) at ./ldap2csv.pl lin +e 58, <$in> chunk 7882.` [download] for the line `$data{$key} =~ s/(^\|$)/"/g if $data{$key} =~ /,/;` [download] I did some troubleshooting and find there is a key of "11" which has no value. That appears to be the result of the line `$data{@header} = (); # To label the data.` [download] Originally the line was `@data{@header} = (); # To label the data.` [download] and I didn't think that looked right. I tried changing the line back to that. There seems to still be issues but I will try to figure out why all my quoted values start with a comma. I appreciate your help. I wish this wasn't such a hard task.	[reply] [d/l] [select]
Re^3: Parsing ldap attributes into a csv by AnomalousMonk (Archbishop) on Mar 02, 2018 at 20:49 UTC
`$data{@header} = (); # To label the data.` This expression evaluates the array `@header` in scalar context. An array evaluated in scalar context yields the number of elements in the array. `@data{@header} = (); # To label the data.` This expression is a slice. The elements of `@header` are used as hash keys, to which values are assigned from the list on the RHS of the assignment. Since this list is empty in both statements, all the values assigned in both cases are undef. `c:\@Work\Perl\monks>perl -wMstrict -MData::Dump -le "my %data; my @header = qw(fee fie foe fum); ;; $data{@header} = (); dd \%data; ;; @data{@header} = (); dd \%data; " { 4 => undef } { 4 => undef, fee => undef, fie => undef, foe => undef, fum => undef }` [download] Give a man a fish: `<%-{-{-{-<`	[reply] [d/l] [select]
Re: Parsing ldap attributes into a csv by Tux (Canon) on Mar 02, 2018 at 21:06 UTC
I don't want to sound patronizing, but you should really not generate CSV the way you do. What if a field contains a double quote? Your CSV data will be invalid. There are two very fine (and fast) modules on CPAN that will do this for you: Text::CSV_XS and Text::CSV (which will use the former if installed for speed reasons) Please see the docs for a jump start] Enjoy, Have FUN! H.Merijn	[reply]
Re: Parsing ldap attributes into a csv by 1nickt (Canon) on Mar 02, 2018 at 21:58 UTC
Hi, maybe this is helpful? (Update: modified as suggested by the master) use strict; use warnings; use Text::CSV_XS 'csv'; my @records = map { my %record; for ( split "\n" ) { my ($key, $val) = split ': '; $record{$key} = $record{$key} ? join ', ', $record{$key}, $val + : $val; }; \%record; } split "\n\n", do { local $/; <DATA> }; my @col_names = qw(cn host description env keyWords l sysadm1 sysadm2 +administratorContactInfo nsHardwarePlatform version); my @rows = map { my $record = $_; [ map { $record->{$_} \|\| '' } @col_n +ames ] } @records; csv( in => \@rows, headers => \@col_names, out => '/tmp/foo.csv' ); __DATA__ cn: epcpc5 host: epcpc5 description: research server l: Narnia Data Center cn: epcmsp host: epcmsp description: Service Processor l: Narnia Data Center cn: post62a host: post62a description: network printer keyWords: netprinter l: Narnia Data Center cn: tst401 host: tst401 host: tst401.narnia.org host: srv1ck41 description: Test Application Server env: test keyWords: linuxserver keyWords: nonprod keyWords: localperl keyWords: sudoers l: Data Center, Narnia sysadm1: aslan administratorContactInfo: sciops nsHardwarePlatform: DellR730 [download] Output: `$ cat /tmp/foo.csv cn,host,description,env,keyWords,l,sysadm1,sysadm2,administratorContac +tInfo,nsHardwarePlatform,version epcpc5,epcpc5,"research server",,,"Narnia Data Center",,,,, epcmsp,epcmsp,"Service Processor",,,"Narnia Data Center",,,,, post62a,post62a,"network printer",,netprinter,"Narnia Data Center",,,, +, tst401,"tst401, tst401.narnia.org, srv1ck41","Test Application Server" +,test,"linuxserver, nonprod, localperl, sudoers","Data Center, Narnia +",aslan,,sciops,DellR730,` [download] The way forward always starts with a minimal test.	[reply] [d/l] [select]
Re^2: Parsing ldap attributes into a csv by Tux (Canon) on Mar 03, 2018 at 10:11 UTC
Why complicate the matter more than needed? :) `csv (in => \@records, headers => \@col_names, out => "/tmp/foo.csv");` [download] Enjoy, Have FUN! H.Merijn	[reply] [d/l]