in reply to Net::LDAP::LDIF->new Chokes on large LDIF because of comments every 5k lines

Before posting this, I saw that you have replaced the entire contents of your post with an entirely new question. Please see How do I change/delete my post?, in particular "It is uncool to update a node in a way that renders replies confusing or meaningless". Because I already put some effort into my answer I am posting it anyways, and have considered your node for editing. (Update: It appears you edited your post multiple times and the version I saw was not the original. The same comments apply to those edits too.) Please post your new question in a new thread.

Trying to figure out a way to do an my $ldif = Net::LDAP::LDIF->new ... on the output of an exec qx ...
system qx[...];
Should I be using an System or Exec? I think I should be doing something like this, but use the ldap::ldif->new with EXEC?

Both system and qx// call external commands, so what this code is trying to do is run a command with qx[], get its entire output, and feed that as the arguments to a system command, which is almost certainly not what you want. I of course don't know this custom external dxsearch tool you are using and I am not an expert on Net::LDAP::LDIF, but its documentation does make clear that you can give Net::LDAP::LDIF->new a filehandle instead of a filename. Based on your description, in this case, that filehandle could come from, for example, a piped open, which I describe in my post here along with lots more on running external commands, which I also strongly recommend you check out.

I want to use perl to process the large ldif it creates and format it into a csv report.

Since the code you showed doesn't use it, I recommend Text::CSV (also install Text::CSV_XS for speed).

Replies are listed 'Best First'.
Re^2: open ldif
by 3dbc (Monk) on Dec 27, 2017 at 20:30 UTC
    Thanks, that is what I was looking for. Need to use STRIKE next time instead, but just kept working through it and probably posted too early but that open filehandle way looks more efficient than what I was doing and answered my original question.
    I'm using DXSEARCH to get tens of thousands of directory search results even though the LDAP server has a search limit set of 1k per search and there's no easy way to deal with that with LDIF module (paged client side searching).
    - 3dbc
      I think my original question was more along the lines of all these files I'm opening and closing and how best to manage them. In my script I currently have one system call which uses dxsearch (which is an enhanced ldapsearch) to produce a very large ldif with over 100,000 search results. then I open the same file the dxsearch produces in the next line, (even though I might have it in memory already because I used: my $input = qxdxsearch blahblah, but never use the $input anywhere else in my script). Then I open an output file to convert it to complaint ldif since there are some uncommented bad lines in it. Then I open an output file.csv, I process the ldif generated by dxsearch by reading in the massaged ldif I cleaned up with perl in previous step with net::ldap::ldif->new( so it seems in one script I have three open's, two closes, a file being generated by a system call and net ldap reading in a file I closed right before calling it, not to mention an $input variable probably storing the whole un-massaged ldif in memory which is never used because I wasn't sure that would work with net ldap ldif new. Everything seems to be working fine, just was interested if there's any best practice here or if it's better I open, close files to conserve memory. I guess that is kindof the question I was asking, but once I solved the issue with processing the ldif decided to change it since I didn't want to waste anyone's time.

      I also have a bad habit of compulsively editing my posts after submitting them, (for instance I've probably edited this one 4 or 5 times since I originally posted it ugh) not sure if anyone else has fallen victim to that trap too, just saying ;-)
      - 3dbc