G'day hahazeeq,

Welcome to the Monastery.

"From this I am getting nothing."

You've added checks throughout for failed I/O - this is good. If files weren't created, you would have received messages from these checks. Not posting these is less good: we can't see what went wrong.

The first problem that leaps out at me is @output.

This should probably be a scalar, not an array. A lexical variable (my) would probably be better than an alias to a package variable (our).

Then later in the code

@output = $timestamp .'.sql';
open(my $fh, '>', @output) or die "Could not create file '@output': $!
+";
[download]

becomes

$output = $timestamp .'.sql';
open(my $fh, '>', $output) or die "Could not create file '$output': $!
+";
[download]

There's also potential problems with

open(my $fn, '<', $filename) ...
[download]

which possibly needs to be

open(my $fn, '<', "$directory/$filename") ...
[download]

Add temporary print statements to your code to perform your own troubleshooting. Check if you're reading and writing to the correct pathname, as opposed to the correct filename.

Without seeing your error messages, I'm really just guessing at problems. If you need further assistance, please see these guidelines for what to post such that we can help you most effectively.

Thank you Ken. I've corrected the errors as suggested and confirmed that it is writing to the correct path name which is the current path.

I'm trying to find out how can I execute this exact same script for all of the files(many) in the same directory. The files that I want to include are those with the extension of '.log'. To be specific, the files are log files and I am trying to capture data(succeeded) from each of the file. I have managed to do it for one file now I am trying to do it for each file with a the `.log` extension in the current folder.

The builtin glob function will probably do what you want. Just make sure you check the "Portability issues" noted in that document.

Something like:

my @logfiles = glob "*.log";
[download]

then just iterate @logfiles, processing each in turn.

Because you said "... all of the files(many) in the same directory ..." you may exceed a maximum size. See GLOB_LIMIT in File::Glob for details.

If you are likely to approach that limit, a better option (although, a bit more coding) would be to use readdir.

You'll need to skip everything except normal files with a .log extension. Something like this (untested):

while(readdir $dh) {
    next unless -f and /\.log$/;
    # Process log file here
}
[download]

See File Test Operators if you're unfamiliar with -f.

-- Ken

for my $filename (@list_files)
[download]

Then you have this:

@output = $timestamp .'.sql';
open(my $fh, '>', @output) or die "Could not create file '@output': $!
+";
[download]

For the second one I've edited it. But for the for loops is the part where I'm trying to open each file in the directory. I'm guessing that's the wrong way, any ideas how I can fix this? So far I've only been able to open one file but the file name is hard coded. I want to be able to run the script for each file in the directory(each of the files have the same format).

I suspect you're being distracted by having too much information thrown at you at once.

Examine just this bit of your code:

my @list_files;

for my $filename (@list_files)
{
}
[download]

Do you see the problem? You just declared @list_files, so by definition it has nothing in it.

Your loop therefore has nothing to do.

Perhaps this will help you see the path forward:

my @list_files = readdir $dir;
[download]

you could read all the .log files like this below code

opendir(hand,$ARGV[0]);  #replace ARGV[0] with your DIR
    @files = readdir(hand);
    closedir(hand);

    foreach(@files){
        if(/\.log$/i) {                                               
+      #if the filename has .log at the end
            push(@logfiles,$_);  
        }
    }
[download]

you could then take the logfiles array and do a multithreaded processing like below

while (scalar @Threads < scalar @logfiles) {
        @running = threads->list(threads::running);
    
        if (scalar @running < $n_process) {                           
+         #spawn threads if no of running threads less that MAX proces
+ses defined
            $filename = $logfiles[$fileindex];
            my $thread = threads->new( sub { sorterwriter($filename) }
+);                    #spawning worker threads passing filename to su
+broutine     
            push (@Threads, $thread);                                 
+   #pushing no of threads spawned for checking in outer loop
            my $tid = $thread->tid;
            print "  - starting thread $tid\n";
            $fileindex++;
        }
        
    }
[download]

or you could do regular processing by

$nooffiles = @logfiles;
$fileindex=0;
while ($fileindex <$nooffiles )
{
open hanr ,">", "$logfiles[$fileindex]" or die "could not open logfile
+ .. $!";
$fileindex++;
}
[download]

Hi sandy105 :) readdir is no fun at all, Path::Tiny on the other hand is fun :)

use Path::Tiny qw/ path /;
my @logfiles = path( shift )->children( qr/\.log$/i );
[download]

I rarely use modules because they are not available on the server where i deploy my scripts.

Since i run windows at home , I rarely write perl at home for personal purposes .But yeah i installed some modules including Path::Tiny the other day . I plan to explore that and some XML parsing modules as well dabble with OO programming in perl , which i haven't done before.Infact all the lagacy scripts i maintain at my company are huge ~5k lines of code and most scripts are similar to other scripts.

sub dosomething
{
my($fn,$fh);

}

use strict;
my $DIR;
opendir ($DIR, $directory) or die "Could not open directory '$directory +': $!";

I have found many solutions using either one of those two ways, however I've done trial and error with my codes and can't seem to be able to create the files. Is there an example you could show me using my codes?

glob gives you only existing file names. For new files that you want to have created, you have to give the names yourself, e.g. with

    my @list_files = qw/first.log second.log third.log/;
[download]

forget it, obviously hadn't enough coffee yet