in reply to Re^8: Finding files recursively
in thread Finding files recursively
fork for as many physically different disks you have. not partitions, not directories.
|
|---|
| Replies are listed 'Best First'. | |
|---|---|
|
Re^10: Finding files recursively
by afoken (Chancellor) on Aug 06, 2019 at 20:19 UTC | |
fork for as many physically different disks you have. not partitions, not directories. When using RAIDs and/or LVM, that rule is a little bit too simple: In case of hardware RAIDs (expensive disk controller with dedicated CPU, dedicated RAM, perhaps battery backup), treat each RAID volume as a single disk. The number of physical disks is irrelevant in this case. In case of software RAIDs (e.g. Linux MD driver, ZFS), things may become compicated. In the most simple case, each RAID volume is composed of several disks containing no or just a single partition, and you can treat each RAID volume as a single disk. If you spread several RAID volumes over several disks (e.g. a /boot RAID-1 using the first partition of each disk and a /data or root RAID-5 using the second partition of each disk), you need to treat the two RAIDs as a single disk. For more advanced setups, things will get successively more complex. In case of fake RAIDs (cheap disk controller with no CPU, no RAM, just a boot ROM, implementing a BIOS-level software RAID), the hardware RAID rules apply if the fake RAID allows only RAIDs of entire disks. If the fake RAID allows to partition the disks into several RAIDs (I've never seen that), the software RAID rules apply. If you use LVM on top of the RAID, or even just on top of bare disks, you need to treat all disks (physical or RAID volumes) shared by an LVM set as a single disk. Alexander
-- Today I will gladly share my knowledge and experience, for there are no sweeter words than "I told you so". ;-) | [reply] |
by bliako (Abbot) on Aug 07, 2019 at 08:49 UTC | |
I was refering to the simpler case of no RAID/LVM at all. Thanks for adding the clarifications for RAID. Can we say that for some RAID/LVM setups, performance of N parallel seeks is unknown? Meaning that some tests prior to forking must be made to establish a good value for N for that specific directory (and also specific action like reading file contents or just listing directory entries)? Or perhaps RAID can be enquired about N before seeking? | [reply] |
by afoken (Chancellor) on Aug 07, 2019 at 15:06 UTC | |
Can we say that for some RAID/LVM setups, performance of N parallel seeks is unknown? Yes, absolutely. Meaning that some tests prior to forking must be made to establish a good value for N for that specific directory (and also specific action like reading file contents or just listing directory entries)? Yes. The general idea is to have one process traversing the directories per set of disks containing the directory tree. It does not matter if that is done via RAID, LWM, ZFS, or something else we haven't thought of. Or perhaps RAID can be enquired about N before seeking? Yes, but implementing an algorithm that considers all edge cases may take some time. Linux software RAID allows really perverted constructs. For example, it does not prevent you from creating four partitions on a single disk and build a RAID-5 on top of that four partitions. It makes absolutely no sense, except for learning and debugging. A quite common setup for software RAID is to have some disks in a RAID-5, but also have a bootable RAID-1 for /boot on the same disks. Linux SW RAID allows that, because it can use partitions instead of the full disks. So you end up with two RAID volumes, each using three or more disks, but using the same disks. Or, you use separate RAIDs for data, root, swap, and boot, because you don't like LVM. This is my home server setup:
Should be easy to parse, and the result should be that all RAID volumes share the same set of disks. In my case, N=1. If you also want to traverse the BD-ROM sr0, N=2. Compare with one of our servers at work:
Six disks, containing two RAID volumes, /boot (RAID-1) and an LVM set. LVM provides /, /var/lib/vz, and swap on top of the RAID-6. N=1, no optical disk. Alexander
-- Today I will gladly share my knowledge and experience, for there are no sweeter words than "I told you so". ;-) | [reply] [d/l] [select] |
by Anonymous Monk on Aug 08, 2019 at 01:29 UTC | |
by afoken (Chancellor) on Aug 08, 2019 at 07:28 UTC | |