Well I've been busy hacking away at it for the last few days but no progress.
I tried using Storable which worked well for arrays but when I tried to put the larger hashes into it (or reload them, I can't remember which) I kept getting out of memory errors and extreme slow downs (needed reisub once).
A couple of weeks ago I found I had originally read the pages out in a slightly wrong order, changing that led to the larger text pages I was able to read. But I realised I scanned the second LBA area using the old addressing scheme (since I was looking for physical addresses rather than LPAs this wold have broken it). Running the scan again once again froze my machine repeatedly due to the extremely large hashes I suppose. Whether I coded it wrong or not I don't know but I gave up using perl for this and wrote it in c with mmap which took a while of course but it executes quite fast so it was worth doing.
However running the scan again yielded nothing. I am looking here at whether the second LBA like array in each LBA block corresponds to any of the other rows sharing the same LBAs. So I find 2 references to a particular LBA in 2 different parts of the image. I then look in the second LBA like area in the LBA block for the physical address that I found the other reference to this particular LBA in. And I find not enough matches to be not from chance only.
This might be because how I am assigning a physical address to each block that I read is wrong (IE the drive thinks of physical addresses differently to how I see it), or because there is no match.
I just can't see what this second field is for though. I've done some more checking and have a decent description of the LBA block contents (each is for 127 data blocks of 16k)
Word refers to a 32 bit word. Each 16k block which is last in an erase size block of 128 blocks consists of:
Word 0 &ed with 0xffff is the superblock number. What the first 2 bytes do is unknown, they are always 8323072 when the superblock matches the second 2 bytes.
Word 1 is the number of bits set in word 2,3,4 and 5 combined.
Word 2,3,4 and 5 are the bit field for the LPN zone that indicate duplicates in this erase size block.
Word 6 to 132 inclusive are LPNs (127 of them).
Word 133 is unknown and suspicious to being a sequence number. This number occupies the position that LPN 128 would take. However it is very frequently a very large number with only a few bits set, like a bit field.
Word 134 is the number of set bits in word 135, 136, 137 and 138 combined. These words (135 to 138 inclusive) are a bit field, marking the subsequent words as valid / existing. Whether 0s in the bit field indicate skipping or invalid words in the second word field (second 128 long zone of address size numbers) is unknown.
Word 139 onwards comprise the mystery fields. There are usually a handful of words here up to around 120ish, rarely more. Their value is in the range 0 to the max LPN found in the first LPN area, an exact match. After this the rest of the block is 0xffffffff.
In the OpenSSD source they state that since the controller can't access the chip spare area they store LPNs in the last block in each erase block, but I recently noticed they say this is for GC, since the GC can compare the LPN with its in memory value and if it does not match it would know the block can be erased. This would be a fast and convenient method, so perhaps this is why the LPN is stored where it is. In the OpenSSD source however they structure they store is a simple single array, the second LBA area is not there. So I am still puzzled as to what it might be for.
So I wonder if this LPN data (the first LPN field) is purely for GC, or might it serve a second purpose as crash recovery. In the first case it seems there will not be a sequence number or way of determining sequence since it is not needed, in the second case a sequence is needed. It would make no sense whatsoever to not record this number in the erase block, since you are writing it anyway and theres many kb free. But the only place I can find thats a possible match for a sequence number is word 133, but it does not seem like one, unless its as you mention not a simple sequence but some kind of derivative of one.
Still, it seems there should be a map block somewhere on the disk for loading at boot time. Its only 64MB or so needed and theres bags of spare unused space. I'm looking around for that, I did find a couple of interesting areas with LBA or PBA range numbers but they are peppered with the odd number an order of magnitude larger. I am wondering whether these could be sequence numbers or similar in blocks of subsets of map data. IE if the drive is writing log style its only concentrating on one region of its address space at once, so only writing updates to reflect that (diffs in essence, addressed to one part of the map).
If map files or fragments thereof are written out, I would imagine there are many of them, scattered around the disk. The blocks I am looking at do seem scattered around the 'smooth grey area' I mentioned. As you say, perhaps this is an area set aside for such use. And it could also include spare blocks so that as the NAND blocks fail it replaces them, and once it is used up the drive fails fully. There is lots of fully 0x00 erase size blocks in this area, why 0x00 rather than 0xffffffff I wonder (better to store cells in the 0x00 state?).
Anyway thats where I am up to, not much to say but I thought I would update. Any ideas appreciated.
In reply to Re^11: Adding cols to 3d arrays - syntax
by peterrowse
in thread Adding cols to 3d arrays - syntax
by peterrowse
| For: | Use: | ||
| & | & | ||
| < | < | ||
| > | > | ||
| [ | [ | ||
| ] | ] |