Re: using Linux getdents syscall

The performance problems you're encountering are almost certainly due to the mechanics of the underlying filesystem, or its metadata handling to be more precise.

Choosing the right filesystem and the right options can have tremendous impact is some cases, but be sure to understand the implications. Linux native ext4 with default options is a good all-around choice. For temporary files, the tmpfs is an excellent candidate.

If you must use ext4 filesystem for backing lots and lots of transient files, however, do consider mounting it without journal (-o noload). Or the opposite—perhaps a generously sized journal might save the day. In any case, taking a step back to re-evaluate your optimization approaches would be in order.

Comment on Re: using Linux getdents syscall

Replies are listed 'Best First'.
Re^2: using Linux getdents syscall by glasswalk3r (Friar) on Dec 03, 2015 at 23:55 UTC
You're right oiskuu: I tested the code in a different situation and didn't get a better performance compared to other solutions (`rm -f ` was the better option if all files were to be removed). In the other hand, it did showed faster performance than using `readdir()`. I believed if I retrieve all files names at once before* trying to apply any checking or removing them would be faster, but I still need to benchmark it This is going a bit outside of scope of Perl, but I guess that after all the only way to detect this kind of issue with the filesystem is having a baseline to compare with. Alceu Rodrigues de Freitas Junior --------------------------------- "You have enemies? Good. That means you've stood up for something, sometime in your life." - Sir Winston Churchill	[reply] [d/l] [select]

Replies are listed 'Best First'.

Re^2: using Linux getdents syscall
by glasswalk3r (Friar) on Dec 03, 2015 at 23:55 UTC

You're right oiskuu: I tested the code in a different situation and didn't get a better performance compared to other solutions (rm -f * was the better option if all files were to be removed). In the other hand, it did showed faster performance than using readdir().

I believed if I retrieve all files names at once before trying to apply any checking or removing them would be faster, but I still need to benchmark it

This is going a bit outside of scope of Perl, but I guess that after all the only way to detect this kind of issue with the filesystem is having a baseline to compare with.

Alceu Rodrigues de Freitas Junior
---------------------------------
"You have enemies? Good. That means you've stood up for something, sometime in your life." - Sir Winston Churchill

[reply]
[d/l]
[select]