in reply to filesystems & Perl

Hi harangzsolt33,

I'm not sure how Perl implements readdir internally, but if POSIX.1-2008 is anything to go by:

... files may be removed from a directory or added to a directory asynchronously to the operation of readdir(). ... If a file is removed from or added to the directory after the most recent call to opendir() or rewinddir(), whether a subsequent call to readdir() returns an entry for that file is unspecified.

Although I suspect the behavior would vary between operating systems and file systems, I would be surprised if files that have not been added or deleted were skipped by readdir.

However, I am wondering why this is important to you? Have you discovered a case where a file is not being reported by readdir, or are you experiencing some other kind of trouble? Are you, for example, copying files from one place to another and are worried that some might be missed?

Update: This article on Mac OS X says that on HFS, unlinking files while using readdir() may cause some files to be skipped. If you want to delete all files in a directory, the suggestion is to use rewinddir until all files have been unlinked. Update 2: ... and if you look at rewinddir in perlport, you'll see that method won't work on Win32.

On the other hand, search for "readdir" in "The Linux Programming Interface" and on page 354 you'll see the mention that files that have not been added or deleted are guaranteed to be returned.

So again, in order to find the best solution, I think you need to describe the problem you're trying to solve some more.

Regards,
-- Hauke D

Replies are listed 'Best First'.
Re^2: filesystems & Perl (updated!)
by harangzsolt33 (Deacon) on Aug 13, 2016 at 03:21 UTC

    I want to create a Perl program that records a bunch of information about each visitor to the web page. Kind of like a page counter. So, here is how it would work: A visitor clicks on my web page. Somewhere at the end of the HTML page, a JavaScript program will collect a bunch of info and load my Perl script "disguised" as an image. The Perl program reads QUERY_STRING. Opens a file and appends the info to the file. Closes the file and sends back a tiny 1x1 image to satisfy the browser.

    I want to be able to see where my visitors come from, their IP addresses, screen resolution, web browser, operating system, exactly what time they clicked on my site, and everything that can be known. I have started learning Perl in May 2016, so I am just a beginner. And this is a learning exercise. I have already written most of the code, but then I realzied something. Wait a minute. I need to think more about the whatifs. It might not work always! :P

    So, here are some of the issues that I can think of :

    1. My script needs to automatically check the size of this ever-growing data file and make sure it doesn't exceed 300 lines. because let's say, I only want to keep info about the last 300 visitors. I don't want to end up with a log file that's 200MB. lol

    2. Let's say I am giving away free cars and my web page gets 1000 hits per minute. Now, in order to make sure that I can always get write access to my log file, I am going to store each entry in a temporary text file in a folder. So, thousands of files are going to get created there. Every time someone pulls up my website, there's a new text file on my server! Obviously, I need to delete some of them. I only need to keep the last 300 files. So, every now and then, my script will do garbage collection and get rid of some files.

    3. Since I am the admin, I also want to write a Perl program that will read all the temp files that exist at the time and merge them and format them and print me out a nice report. But I am worried about the flexible nature of this folder. Will it mess up readdir? or will I be able to get a full list? Since I do not have such a busy website, there is no way for me to test what would happen. All I can do is think about this in theory and go through all the possible whatifs. :P

    4. Another solution might be to use numbered files 001.txt 002.txt 003.txt and so on. So, I would have 300 text files there at all times. And my script would simply pick out and overwrite the oldest one in the list. That way I would not have to delete and create files. Sounds very simple, but it's not! This might lead to data loss, because two or more instances of my Perl script may want to write to the SAME file all at once. :P

    Help me please! This seems such a simple program, yet it's so difficult. How should I do this? :(

[reply]

      Use a database and let the many developers who created the DBMS worry about the locking and concurrency issues instead. Since you have your arbitrary limit of 300 records it can use in-memory tables too thus avoiding hammering the filesystem.

[reply]

      Hi harangzsolt33,

      Yes, exactly what hippo said - a database can solve all of the problems you named. You'll find many tutorials on DBI on the web, including several on this site.

      Historically one might have solved this using flock to control read/write access, but that can be finicky and hard to get right (assuming it even works on your OS+FS); so I suggest you spend your time learning some basic DBI instead.

      Hope this helps,
      -- Hauke D

[reply]
      for point 2, consider opening a file as "./name.tmp", then write all the lines to it, then close the file so all content gets flushed to disk, and THEN rename it to your valid extension, like .dat with rename $temporal_name, $realfile_name; Keeps operations pseudo-atomic for your readdir's.
[reply]
[d/l]

In Section Seekers of Perl Wisdom