BrowserUk has asked for the wisdom of the Perl Monks concerning the following question:
If you have two equal-sized buffers and you need to swap the over in memory:
[X0 X1 X2 X3 X4 X5 X6 X7 X8 X9 Y0 Y1 Y2 Y3 Y4 Y5 Y6 Y7 Y8 Y9] becomes [Y0 Y1 Y2 Y3 Y4 Y5 Y6 Y7 Y8 Y9 X0 X1 X2 X3 X4 X5 X6 X7 X8 X9]
It's easy to use a loop that swaps one machine register-sized chunk at a time:
void swapBufferHalves( U8 *buf, U32 start, U32 length ) { register U64 *p1 = (U64*)&buf[ start ]; register U64 *p2 = (U64*)&buf[ start + ( length >> 1 ) ]; const U64 *pend = (U64*)(start + length); register U64 temp; assert( !( length & 1 ) ); assert( ( (U64)buf & 7ull ) == 0 ); while( p1 < pend ) { temp = *p1; *p1++ = *p2; *p2++ = temp; } return; }
But how to deal with the situation where the second 'half' of the buffer is shorter than the first:
[X0 X1 X2 X3 X4 X5 X6 X7 X8 X9 Y0 Y1 Y2 Y3 Y4 Y5 Y6 Y7] becomes [Y0 Y1 Y2 Y3 Y4 Y5 Y6 Y7 X0 X1 X2 X3 X4 X5 X6 X7 X8 X9]
Intuition tells me that this ought to be possible using 2 temporary registers; but the logic won't come?
|
---|
Replies are listed 'Best First'. | |
---|---|
Re: [OT] Swapping buffers in place.
by graff (Chancellor) on Mar 01, 2015 at 06:51 UTC | |
I think the idea expressed in the first reply, involving enough temp storage to hold the difference in size between the two parts, will have the smallest footprint that can be implemented by simply incrementing (or decrementing) pointers. But if you really want to pursue a single-temp-register solution, the basic idea for the pointer arithmetic would go like this: I hope I've got that right (I suspect there may be a better way to describe it)... If it all sounds like a better idea than using more storage, then have fun with that. (Update: Regarding the "completion condition" (item 6), there is a better (more correct) description: it will always happen on an iteration where a value is moved out of the temp register. When D is odd, this happens only once, and when it's even, it happens exactly twice - once for the even-numbered offsets, and once for the odd-numbered ones.) (Another update -- sorry... The problem in steps 5 and 6 is not just an even/odd thing. It has to do with the largest common divisor of D and buffer_count; e.g. if D and buffer_count are both multiples of 4, then you'll need to move values out of the temp register a total of four times; you'll need to add 1 to the pointers after the first three of those, and the fourth one will be the final iteration.) Can't... stop... updating... I improved the phrasing in step 3 (I hope it's clearer), and here is a diagram of the sequence (using a small buffer). Read more... (2 kB) | [reply] [d/l] |
Re: [OT] Swapping buffers in place.
by bitingduck (Deacon) on Mar 01, 2015 at 02:46 UTC | |
Do you know a priori when the second half is shorter than the first? The obvious thing to me is to have a third buffer equal in size to the difference between the two where you store the part that you're about to overwrite. The clever way that struck me while trying to get around that is to reverse the whole thing, then reverse each of the sub-buffers. It takes more time but it does it in place. edit: other than the triple reverse, I can't do it in place (i.e. without a third buffer equal to the difference) without putting in a bunch of rotations to get the ends of the longer buffer in the right place: first, swap the ends of the longer one so you won't overwrite the trailing elements of the long one:
becomes [X8 X9 X2 X3 X4 X5 X6 X7 X0 X1 Y0 Y1 Y2 Y3 Y4 Y5 Y6 Y7]then to a straight swap [Y0 Y1 Y2 Y3 Y4 Y5 Y6 Y7 X0 X1 X8 X9 X2 X3 X4 X5 X6 X7]then you're stuck rotating the trailing elements of the longer one back into place which I think usually takes more time than the triple reverse (which really is only two passes through the whole thing). | [reply] [d/l] [select] |
by BrowserUk (Patriarch) on Mar 01, 2015 at 06:42 UTC | |
I can do it with just one temp and n+m+(m-n) swaps (on paper):
But I'm damned if I can see any pattern to the increments & decrements or steps; to capture that for a generic M+N? the triple reverse (which really is only two passes through the whole thing). That's really quite brilliant, I'd never have thought of that :), and it will certainly be what O shall fall back to if I can't get this working. But as the left buffers are going to be 2GB or 4GB or 8GB or 16GB or 32GB (depending on physical memory in the machine) and the right buffer could be anything from 16 bytes up to the (leftBufferSize)-16 (ie. whatever is left in the dataset modulus the leftBufferSize) you can see why I'd rather avoid (M+N)*2 if I can get M+N+(M-N). (And also why a 3 buffer is impractical.) With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
"Science is about questioning the status quo. Questioning authority".
I'm with torvalds on this
In the absence of evidence, opinion is indistinguishable from prejudice. Agile (and TDD) debunked
| [reply] [d/l] |
Re: [OT] Swapping buffers in place.
by graff (Chancellor) on Mar 01, 2015 at 09:03 UTC | |
(updated include a line inside the while loop to increment iter_count - rather a stupid thing to forget.)
(Updating again to mention that my snippet is bound to have a problem when it hits the last iteration; (final update: last line used to say ... if ( p2 > end_buf ) which was bound to be wrong. Hell, while I'm at. might as well fix the last-iteration problem too. Still, this sort of thing needs really careful testing, which I haven't done.) | [reply] [d/l] [select] |
by BrowserUk (Patriarch) on Mar 01, 2015 at 10:41 UTC | |
Yes. I think you filled in the last piece o fthe puzzle for me...I'll get back once I've coded and tested it. Thank you! With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
"Science is about questioning the status quo. Questioning authority".
I'm with torvalds on this
In the absence of evidence, opinion is indistinguishable from prejudice. Agile (and TDD) debunked
| [reply] |
Re: [OT] Swapping buffers in place. (partial solution:needs more help!)
by BrowserUk (Patriarch) on Mar 01, 2015 at 08:38 UTC | |
I've (possibly as a consequence of reading graff's post, but I'm not sure), come up with two patterns that I think will solve all cases, but I'm still having the devil's own trouble translating them into code! Here are the two graphically, with some (possible, but incomplete) pointer math attempting to describe the swaps. An bigger even/smaller odd case:
The above looks like 3 reps of a loop. (But is that because 6-3 = 3 or just because The smaller is 3; or ... ). And code? A bigger odd/smaller even case:
The above looks like 4 iterations (because the smaller is length 4), but the first iteration requires 3 internal steps and the other 3 only 2? (How to code?) I haven't convinced myself I don't need even/even and odd/odd cases yet. With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
"Science is about questioning the status quo. Questioning authority".
I'm with torvalds on this
In the absence of evidence, opinion is indistinguishable from prejudice. Agile (and TDD) debunked
| [reply] [d/l] [select] |
by bitingduck (Deacon) on Mar 01, 2015 at 09:16 UTC | |
I've copied what you did and added the iterator and reexpressed the righthand sides in terms of the lengths (the lefts are just the rights from the previous line) and the pattern starts to show up (at least if you click on "download" to view it spread out).
The pattern doesn't jump out as easily from your second example. EDIT: I think I got the second one, too. There was a typo that messed me up for a bit, and you solved this one "backwards" so it's easier to reference from the ends of the buffers than the starts, but again there's a pattern: where the addition has to wrap around as graff describes | [reply] [d/l] [select] |
by BrowserUk (Patriarch) on Mar 01, 2015 at 10:35 UTC | |
There was a typo that messed me up for a bit, Sorry! But they aren't easy to construct. (What I need is a program .... :) and you solved this one "backwards" Yes. I eventually noticed that I'd switched horses mid way. I went back and re-did them all consistently. I'm not sure if one way is easier or better than the other; and if it is, if I've chosen the right way. But anyhoo, here are the (now) 4 cases I think are significant; all done the same way. The first two are nicely consistent in that the shorter elements move in left-to-right order; albeit that in the first they always go through the temp reg, and the second, only the first one does so. The other two are neither consistent with the first two; nor each other :( There are definitely two nested loops; the shorter is the outer, the longer the inner. And graff is definitely right that molulo (p1 % p2len) arithmetic is key. But the decision about when to fetch the temp register; or move from the shorter buffer (and which one) is eluding me for the moment.
With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
"Science is about questioning the status quo. Questioning authority".
I'm with torvalds on this
In the absence of evidence, opinion is indistinguishable from prejudice. Agile (and TDD) debunked
| [reply] [d/l] |
Re: [OT] Swapping buffers in place.
by davies (Monsignor) on Mar 01, 2015 at 15:07 UTC | |
One pass, a buffer the size of the difference.
Assumptions: (1) you know the size of the buffer. (2) The "larger half" always comes first. You may want to rewrite the swap algorithm, but it doesn't use a buffer, even of one character. By the time I'd written this, you said you thought you had an answer, but I understand this one & don't know the language of graff's, so I'm posting it anyway. Regards, John Davies Update: the more I look at this, the more sure I am that my if block can be rewritten to calculate the final position of a value from its original position. This means that with a two value buffer in memory, it should be possible to avoid swapping altogether, reducing the number of writes (possibly important for solid state devices). IOW, var1 (using Data::Dumper notation) goes to var8, so read var8 & write var1 there. Var8 goes to var15, so read var15 & write var8 to it and so on through the chain. The question is where to resume when you get to the end of the chain. Once I have sorted this out, I'll post back (or announce failure). | [reply] [d/l] |
by davies (Monsignor) on Mar 01, 2015 at 22:07 UTC | |
Chains seem to matter when the small part is a factor of the length of the large part (possibly also when the diff is a factor of the large). But
Regards, John Davies Update 2015-03-12 (too late to matter): I was concerned that a chain might hit the second position in the starting buffer but not a later one. I have satisfied myself that this is not a problem. If a chain starts at one and hits 2, then a chain that starts at 2 will hit 3 and so on. Therefore, if anything can be missed, it will be the next position. So my algorithm will work for all cases unless there's something wrong with this logic. If so, please tell me. | [reply] [d/l] |
by hdb (Monsignor) on Mar 01, 2015 at 19:40 UTC | |
See here 1118280 for the answers to the questions in your update. | [reply] |
Re: [OT] Swapping buffers in place.
by hdb (Monsignor) on Mar 01, 2015 at 10:58 UTC | |
Here is a program that does the swapping with one temp only:
Of course, the repeated calls to the utility function swapsamesize should be done in a loop until all is done. It also only works if the overlap when swapping (i.e. n1 - n2) is smaller than the rest of the smaller half. I think the logic can be adapted... | [reply] [d/l] [select] |
by BrowserUk (Patriarch) on Mar 01, 2015 at 11:10 UTC | |
Problem:One small change (delete the last entry in the second part of the buffer): and the output is:
With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
"Science is about questioning the status quo. Questioning authority".
I'm with torvalds on this
In the absence of evidence, opinion is indistinguishable from prejudice. Agile (and TDD) debunked
| [reply] [d/l] [select] |
by hdb (Monsignor) on Mar 01, 2015 at 11:51 UTC | |
Which means that the logic also needs to be different for odd length of buffers. | [reply] |
by BrowserUk (Patriarch) on Mar 01, 2015 at 12:23 UTC | |
by hdb (Monsignor) on Mar 01, 2015 at 14:57 UTC | |
| |
Re: [OT] Swapping buffers in place.
by roboticus (Chancellor) on Mar 01, 2015 at 19:10 UTC | |
Here's what I came up with:
The two tricks are: (1) You never have to subtract from your pointers, and (2) you have to detect loops when your pointers have a smaller loop size than the buffer size. Update: It took me a bit of time to get back to this, and I posted it before reviewing the thread. It turns out that graf already described the algorithm, as did hdb. ...roboticus When your only tool is a hammer, all problems look like your thumb. | [reply] [d/l] |
by hdb (Monsignor) on Mar 01, 2015 at 19:44 UTC | |
Seems I found the same independently 1118280. I like your modulus calculation to find the next loaction. | [reply] |
by BrowserUk (Patriarch) on Mar 01, 2015 at 21:06 UTC | |
Thanks. I too like your modulo calculation -- it's what I was trying for when I said "And graff is definitely right that molulo[sic] (p1 % p2len) arithmetic is key." -- but I didn't get the right combination of arguments. With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
"Science is about questioning the status quo. Questioning authority".
I'm with torvalds on this
In the absence of evidence, opinion is indistinguishable from prejudice. Agile (and TDD) debunked
| [reply] |
Re: [OT] Swapping buffers in place.
by Anonymous Monk on Mar 01, 2015 at 12:01 UTC | |
Output:
| [reply] [d/l] [select] |
by BrowserUk (Patriarch) on Mar 01, 2015 at 20:24 UTC | |
Okay. I jiggered and poked it (my apologies if you're sensitive to such things), and added a little trace and got this:
Which when run on the 6/3 testcase produces:
6 swaps is a perfect score. and the logic is very clear and very clean. So clean it is beautiful (which always feels right!):
With a 10/7 testcase things are little more muddy:
I'm not sure what to think about that. It sure looks like that, after the first 7(*) have been moved, then you could recurse to move the 3 to the end. Ie. As your algorithm works regardless of long/short ordering, treat it as a 3/7 input. Of course you'd have to 'lie' about the length of the array, which is easy to do in C, but not so in Perl. I'm not sure if that is optimal -- I haven't (tried to) figured a formula, but it won't be grossly over -- but it sure looks like (that phrase again), that after a clean start, the end seems (perhaps, unnecessarily) busy? I'll need to code yours and hdbs algorithms, (and finish my own/graff/bitingduck version, if I can), in C and run them on some big buffers with awkward ratios: eg. 2GB left buffer and 536870909 & 536870923 (primes either side of 2GB/4) which ought to trigger pathological behaviour if it exists. With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
"Science is about questioning the status quo. Questioning authority".
I'm with torvalds on this
In the absence of evidence, opinion is indistinguishable from prejudice. Agile (and TDD) debunked
| [reply] [d/l] [select] |
by Anonymous Monk on Mar 01, 2015 at 23:56 UTC | |
initially, x points to 1, y points to a: We want to move a b c all the way to the left,so we swap numbers and letters: Now the rightmost elements are scrambled. Note that y at this stage points to the element one off the right end of the array. We restore its position: Now we basically have a smaller array 4 5 6 1 2 3 and we want to swap 4 5 6 and 1 2 3. Hence, recursion. But it's a tail call, aka 'while loop in disguise', so: Same thing... I think :) | [reply] [d/l] [select] |
by BrowserUk (Patriarch) on Mar 02, 2015 at 00:57 UTC | |
by Anonymous Monk on Mar 02, 2015 at 01:59 UTC | |
by BrowserUk (Patriarch) on Mar 02, 2015 at 02:12 UTC | |
by roboticus (Chancellor) on Mar 02, 2015 at 04:17 UTC | |
| |
by Anonymous Monk on Mar 02, 2015 at 00:06 UTC | |
by Anonymous Monk on Mar 02, 2015 at 00:15 UTC | |
by BrowserUk (Patriarch) on Mar 02, 2015 at 01:19 UTC | |
by Anonymous Monk on Mar 02, 2015 at 01:57 UTC | |
by Anonymous Monk on Mar 02, 2015 at 04:28 UTC | |
by BrowserUk (Patriarch) on Mar 01, 2015 at 18:16 UTC | |
Nice! Passes my basket case:
I took out the shorter/longer tests as that's never going to be a requirement. (Cool that it doesn't matter to the algorithm though!) It'll take me a while to wrap my brain around it -- recursion always does -- but so far it ticks all the boxes. Thank you. With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
"Science is about questioning the status quo. Questioning authority".
I'm with torvalds on this
In the absence of evidence, opinion is indistinguishable from prejudice. Agile (and TDD) debunked
| [reply] [d/l] |
Re: [OT] Swapping buffers in place.
by CoVAX (Beadle) on Mar 01, 2015 at 03:21 UTC | |
Having two separate 'length' parameters instead of one would make it straightforward, right?
Searched for donut and crumpit. Found donate and stumbit instead.
| [reply] |
by BrowserUk (Patriarch) on Mar 01, 2015 at 07:24 UTC | |
aving two separate 'length' parameters instead of one For the uneven buffers, you definitely need two lengths, (Or one length and a difference. :) would make it straightforward, right? Unfortunately not. With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
"Science is about questioning the status quo. Questioning authority".
I'm with torvalds on this
In the absence of evidence, opinion is indistinguishable from prejudice. Agile (and TDD) debunked
| [reply] |
Re: [OT] Swapping buffers in place.
by RichardK (Parson) on Mar 01, 2015 at 10:42 UTC | |
For buffers of any significant size this looks like far too much work! Can't you fix your code that uses these buffers to access them through pointers and then you just swap the pointers? | [reply] |
by BrowserUk (Patriarch) on Mar 01, 2015 at 10:53 UTC | |
No. This is to deal with the edge case where the last buffer is partial. Think:end-of-file. "swapping pointers" would not achieve the same result. After the swap, the left (full-sized) buffer will contain the contents of the right buffer, made up the full buffer size, with the left-hand end of the left buffer. And the right buffer contains the right-hand end the left buffer. The buffers are mem-mapped chunks of a disk file. You can't swap chunks of disk around by swapping pointers. With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
"Science is about questioning the status quo. Questioning authority".
I'm with torvalds on this
In the absence of evidence, opinion is indistinguishable from prejudice. Agile (and TDD) debunked
| [reply] |