Hello Anonymous Monk.

Your code is not perl, it's calls to the cut and sort programs, probably inside a shell script.

But, will a hash be able to handle such a large file?

Probably not, unless you have very little unique values, you are going to end up with several GB of data in your hash. But do you actually need all the values? If you're just trying to get the type of each column you just need to keep a few value:

• if a value that can't be coded as a number has been seen for this column
• the max length (in characters) of this column
• if a non integer value has been seen for this column
• the min and max value for this column (unless you already know that it's a string)

In perl that might look something like that (untested since it's barely more than pseudo code):

use strict;
use warnings;
use Scalar::Util 'looks_like_number';

my @result;
open my $file, "<", "my_big_file" or die "Can't open my_big_file: $!";

while (<$file>)
{
  my @data = split "\t", $_;
  for my $col (0..$#data)
  {
    my $len = $length($data[$col]);
    if (!looks_like_a_number($data[$col]))
    {
      $result[$col]{is_a_string} = 1;
    }
    $result[$col]{max_len} = $len if ($len > $result[$col]{max_len});

    next if $result[$col]{is_a_string}; # Don't check number range if 
+this is a string
  
    # Check number range here
    ...

    # Other stuff
    ...
  }
}
[download]

You might notice the use of looks_like_number of Scalar::Util to help you with detecting numbers. Of course that's if you actually want to use perl, rather than whatever language your script is currently executed as.

---

---

Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!

Titles consisting of a single word are discouraged, and in most cases are disallowed outright.

Read Where should I post X? if you're not absolutely sure you're posting in the right place.

Please read these before you post! —

• How do I compose an effective node title?
• How do I post a question effectively?
• Markup in the Monastery

Posts may use any of the Perl Monks Approved HTML tags:

a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr

You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)

	For:		Use:
	&		&
	<		&lt;
	>		&gt;
	[		&#91;
	]		&#93;

Link using PerlMonks shortcuts! What shortcuts can I use for linking?

See Writeup Formatting Tips and other pages linked from there for more info.