Re: How to select specific lines from a file

I love it. "Taking the last line..." (which is line 570 of the sample input): 66 occurs in the 6th column of the first row and the 293rd row, 16.840 occurs in the 9th column of line 293 from the sample input, and 'B' occurs on the last row and the 293rd, but not on the first. But a little searching through the sample input resolved any ambiguity.

Anyway, this is fixed-width stuff, so you should probably be thinking in terms of unpack or substr rather than regular expressions:

use strict;
use warnings;

while ( <DATA> ) {
  next unless /^ATOM\b/;
  my $chain = substr $_, 21, 1;
  my $position = 0 + substr $_, 23, 3;
  my $Zcoordinate = 0 + substr $_, 47, 7;
  print "$chain, $position, $Zcoordinate\n";
}

__DATA__
ATOM     30  N   HIS A  66       7.514  15.296  11.222  1.00 12.98    
+  A    N  
ATOM     31  CA  HIS A  66       7.318  14.688  12.568  1.00 12.48    
+  A    C  
ATOM     32  C   HIS A  66       8.676  14.309  13.156  1.00 11.62    
+  A    C  
ATOM     33  O   HIS A  66       9.708  14.518  12.545  1.00 11.76    
+  A    O
[download]

Update:

Using unpack is a more computationally efficient alternative, though from a programmer standpoint it always takes me longer to work out the template, which is why I posted the substr solution first. Now that I've had time to work out the template for the unpack solution, here it is:

while ( <DATA> ) {
  next unless /^ATOM\b/;
  my( $chain, $position, $Zcoordinate ) = unpack( 'x21a1xA3x21A7',$_);
  print "$chain, $position, $Zcoordinate\n";
}

__DATA__
ATOM     30  N   HIS A  66       7.514  15.296  11.222  1.00 12.98    
+  A    N  
ATOM     31  CA  HIS A  66       7.318  14.688  12.568  1.00 12.48    
+  A    C  
ATOM     32  C   HIS A  66       8.676  14.309  13.156  1.00 11.62    
+  A    C  
ATOM     33  O   HIS A  66       9.708  14.518  12.545  1.00 11.76    
+  A    O  
ATOM     34  CB  HIS A  66       6.450  13.434  12.442  1.00 12.81    
+  A    C  
ATOM     35  CG  HIS A  66       5.000  13.829  12.378  1.00 13.36    
+  A    C  
ATOM     36  ND1 HIS A  66       4.332  14.002  11.175  1.00 13.57    
+  A    N  
ATOM     37  CD2 HIS A  66       4.073  14.085  13.360  1.00 13.93    
+  A    C  
ATOM     38  CE1 HIS A  66       3.063  14.347  11.461  1.00 14.23    
+  A    C  
ATOM     39  NE2 HIS A  66       2.851  14.410  12.778  1.00 14.47    
+  A    N
[download]

Whether you use substr, or unpack, you'll then be able to feed the input into a data structure as described by Choroba.

Dave

Comment on Re: How to select specific lines from a file Select or Download Code

Replies are listed 'Best First'.
Re^2: How to select specific lines from a file by Laurent_R (Canon) on Apr 29, 2014 at 21:05 UTC
I love it. "Taking the last line..." (which is line 570 of the sample input): 66 occurs in the 6th column of the first row and the 293rd row, 16.840 occurs in the 9th column of line 293 from the sample input, and 'B' occurs on the last row and the 293rd, but not on the first. You love it? So do I. I also spent a quite bit of time trying to figure out which line the OP was really talking about. Anyway, this is fixed-width stuff, so you should probably be thinking in terms of unpack or substr rather than regular expressions. I definitely agree that unpack or substr are the most efficient solutions in terms of computing resources (especially unpack, most probably). But, picking on your remark about the programmer standpoint, and assuming that the file is just a few hundreds or thousands of lines, I might as well consider a regular expression, but not a regex similar to what the OP posted, but a very simple one in a call to the split function. Sometimes, with data looking similar to the OP's data, I find it easier to use something like: `my ($key, $value, $predicate) = (split /\s+/, $line)[0,3,7];` [download] rather than having to compute the exact position of each piece of data (and testing to make sure that I don't have an off-by-one error). But I am doing that only insofar I am reading a relatively small parameter or reference data file before having to process very large or sometimes huge data sets. (Typically, my reference data files have a few hundred or thousand lines, while the real data files to be analyzed have at least dozens of millions of lines, sometimes hundreds of millions lines. In such cases, I really don't care spending a split second more reading the reference data, if I know that processing the main data will take 20 minutes anyway. In other words, I would most probably use the substr or unpack function for the main data, if appropriate, but I don't mind using a slightly slower process for small reference data if it saves me some development time and make the code easier to understand at first glance when I have to maintain it). But this was just a side note about slightly specific situations, I agree otherwise fully with just about everything that you said.	[reply] [d/l]
Re^3: How to select specific lines from a file by davido (Cardinal) on Apr 29, 2014 at 22:14 UTC
I like your comment, and intend to upvote it. In this specific case it appears that the data set is tame enough that the distinction between fixed-width and space-delimited is moot. However, one general principle that I try to adhere to as much as possible is placing the fewest possible demands on a data set as possible. This concept can be generalized from some lessons I learned by reading Effective STL, where Steve Meyers makes some strong cases for why a template container class should place as few requirements on the objects it contains as possible. I'd love to go into the details, but it's a big enough concept that I probably wouldn't do it justice in a simple PerlMonks node. Let's take it as a given, then, that the generalized practice of placing as few demands on an entity that we don't control as possible is "a good thing". In particular, doing so helps to simplify our parser, allows us to unambiguously reject data that is broken, and probably even makes it easier to generate valid data. So what is the simplest, least demanding set of requirements that we can place on our OP's data? As we look it over, it becomes pretty obvious that it is of fixed-width, and that it is space delimited. ...or is it? What if one of those numeric fields (66, for example) extends to four digits? We already see in his data set places where it extends to three digits. A fourth would cause it to run up against our "`[AB]`" field. So there's one requirement we have to place on the data set; no column can become filled to the point that it touches the one next to it. 1000 is illegal for the 6th field. Maybe this is reasonable, but I don't know. I do know that as 66 grows to 100, the field widths haven't shifted, so that field size must always be four or less. But I don't know if four digits is a possible in-range value. What about blank fields? The user's data set example has no blank fields (that I can detect, though there are some big gaps). `\s+` delimited data requires that every field contain something. There's another demand placed on our data set, or if not placed on the data set, another ambiguity that our parser must deal with. Next, by looking at his data it seems obvious that there cannot be embedded spaces. However, that is not just an observation, it's a requirement placed on the data. If a field ever changes such that it allows embedded spaces, our parser breaks. And if that ever happens, we run into all sorts of additional demands for our data; embedded spaces must be escaped or quoted, quotes must be balanced if used, embedded quotes must be escaped, and so on. This will probably never happen with the user's data set; it may never morph into something more complex. Splitting on space may forever be fine. ...it will have to be fine because the parser now demands it. It can never be permitted to morph into something that includes, for example, a notes field (unless it's in the last position, which is another requirement placed on the data and another rule for the parser), completely full fields, or blank fields. So here are the choices for how we can parse fixed-width data: As fixed width: Must be fixed width. As space delimited: No full fields, no blank fields, no embedded spaces. The first rule seems to be the most likely for this data set. If we treat it as If it's fixed width, we impose only one requirement. And probably that requirement is already part of the implementation of the producer. If we treat fixed width data as space delimited, we impose three additional restrictions on the data. Treat fixed width as fixed width for the most robust solution. Dave	[reply] [d/l] [select]
Re^4: How to select specific lines from a file by Laurent_R (Canon) on Apr 30, 2014 at 06:39 UTC
Yes, Dave you are absolutely right. The thing that I did not say is that, usually when I have such cases of parameter file or reference data, I am usually extracting the data myself (or it is done by one of my colleagues) from another system, so that I know exactly what I can demand from the data, or we have something called an interface agreement specifying exactly how the data should look like. When the data comes from unknown source or the exact format cannot be certain, then we are left with trying our best to get the best out of it, and, in the case in point, I fully agree that considering the data as fixed format is the best that can be done on the basis of what the data looks like.	[reply]
Re^3: How to select specific lines from a file by monkey_boy (Priest) on Apr 29, 2014 at 21:29 UTC
Although in the example shown here split would be fine, the fields in these types of file (PDB) are not always defined (e.g. sometimes the chain is blank) so substr or unpack is preferable. See the PDB documentation for more info http://www.wwpdb.org/documentation/format33/sect9.html#ATOM This is not a Signature...	[reply]