You'll likely want a search that is strongly sub-linear, ie better than O(n). There are string search algorithms that approach O(log(m)*n/m), on the average.

On modern hardware, the single most expensive avoidable* 'operation' is a cache miss. With instruction cache misses more expensive data cache misses due to pipelining.

*context switches are more expensive, but are, at least in the long term, unavoidable.

The problem with (all?) the analysies of the various sub-linear algorithms; is that they ignore (or were done before), the costs & benefits of 2 and 3 levels of on-chip instruction & data cache, became a significant factor in instrumenting this type of algorithm.

My 7 y/o cpu, with only 2-level caching, can perform upwards of 1000 instructions in the time it takes to fulfill a data-cache miss. The effect for an instruction cache miss is even greater.

Sub-linear algorithms exacerbate both problems, in 3 different ways:

1. They require extra memory in the form of lookup/lookahead tables.

   That space and frequent access to it, directly hits the effectiveness of the data cache's ability to keep the relevant chunks of the haystack and needle in cache.

2. The algorithms call for look-aheads, look-behinds and/or probes; that upsets the caching algorithms ability to keep caches primed with the required data.

   This is especially important, 64 times as important, for a bitstring search, where each bit needs to be compared 64 times each as a part of 64 different groups of 64 bits.

   This is the single most important factor that has come out of my testing of real code, on a real machine. Modern hardware is optimised to scream through memory in a linear fashion; and every single time you do anything that upsets that relentless, mechanical (electronic) forward (or reverse) flow, you pay a huge hit.

3. They also require many more instructions and branch points.

   The number of instructions is significant, because if instructions early in the loop get flushed by instructions later in the loop, then you not only get a stall for the instruction cache when the loop iterates; it also causes a pipeline flush; and invalidates branch predictions. All together; and a very expensive situation.

   The number of branch points is significant because they compound exponentially in their effects on failed branch predictions; with the knock-on effect of instruction cache stalls if you breach the cache size cliff.

The beauty of well-written brute force algorithms, is that they derive absolute maximum benefit from the cache effect, by doing as much work as possible on each cache line before allowing it to get flushed thus minimising the need for reloads:

1. The inner loops are very small with only 1 or 2 branch points required.

   That ensures no instruction cache misses and fullest possible benefits from both branch predictions and pipelining.

2. Because the data is processed strictly serially; and all the work required on each piece is done with just one from-memory load; they work with both the hardware branch prediction and caching algorithms, to best effect.

In the following table of benchmark timings from my implementation:

• The horizontal list of numbers at the top is the length of the needle in bits.
• The major vertical tick numbers are the distance, in quads, into the 128MB/1Gib haystack at which the needle is found.
• The minor vertical tick numbers are the offset, in bits, into the quad, where the needle is found.

Summary. The algorithm is very linear in terms of the distance into the buffer; and sub-linear in terms of the length of the needle. And it is very fast.

C:\test\C>"bvSearch timings.pl"
                64                256             1024            4096
+            16384           65536           262144                  1
+048576
{
  "0" => {
        "0" => ["6.654006e-007", "4.398411e-007", "4.811937e-007", "1.
+030055e-006", "8.608856e-007", "3.048813e-006", "1.130053e-005", "4.4
+66079e-005"],
        1   => ["3.120240e-007", "5.826955e-007", "3.984885e-007", "8.
+834415e-007", "9.736654e-007", "2.864606e-006", "1.202608e-005", "4.8
+30734e-005"],
        2   => ["3.571359e-007", "5.037496e-007", "4.473598e-007", "5.
+150276e-007", "1.007499e-006", "3.342041e-006", "1.193210e-005", "4.8
+18328e-005"],
        7   => ["3.909699e-007", "4.097665e-007", "4.849530e-007", "7.
+029939e-007", "1.789439e-006", "3.026257e-006", "1.190203e-005", "4.4
+42395e-005"],
        17  => ["4.962310e-007", "4.736750e-007", "5.075090e-007", "6.
+766786e-007", "1.697336e-005", "3.022498e-006", "1.170654e-005", "4.4
+40516e-005"],
        31  => ["5.263056e-007", "5.375836e-007", "5.789362e-007", "8.
+232923e-007", "1.199225e-006", "3.383393e-006", "1.645081e-005", "4.4
+43899e-005"],
        47  => ["6.353260e-007", "5.413429e-007", "5.826955e-007", "9.
+511094e-007", "1.248096e-006", "3.210464e-006", "1.159376e-005", "4.4
+51042e-005"],
        61  => ["6.804380e-007", "6.917159e-007", "6.503633e-007", "8.
+533669e-007", "1.281930e-006", "3.169112e-006", "1.171406e-005", "4.4
+99537e-005"],
      },
  10 => {
        640 => ["2.853328e-006", "4.176611e-006", "3.037535e-006", "3.
+086406e-006", "3.665343e-006", "5.428466e-006", "1.421401e-005", "4.6
+91263e-005"],
        641 => ["2.909718e-006", "3.221742e-006", "3.357078e-006", "3.
+116481e-006", "3.514970e-006", "5.360798e-006", "1.403732e-005", "4.7
+22465e-005"],
        642 => ["2.954830e-006", "3.060091e-006", "2.958589e-006", "3.
+184149e-006", "3.533766e-006", "5.477338e-006", "1.418018e-005", "4.7
+44269e-005"],
        647 => ["2.928515e-006", "3.259335e-006", "2.928515e-006", "3.
+217983e-006", "3.514970e-006", "5.428466e-006", "1.419897e-005", "4.8
+20959e-005"],
        657 => ["2.898440e-006", "3.420986e-006", "3.199186e-006", "3.
+499932e-006", "3.695417e-006", "5.439744e-006", "1.400725e-005", "4.6
+90511e-005"],
        671 => ["3.071369e-006", "3.560081e-006", "2.969867e-006", "3.
+345800e-006", "3.669102e-006", "5.537487e-006", "1.414634e-005", "5.1
+20953e-005"],
        687 => ["3.116481e-006", "3.071369e-006", "4.251797e-006", "4.
+966069e-006", "3.661583e-006", "5.608914e-006", "1.424033e-005", "4.9
+96144e-005"],
        701 => ["3.180390e-006", "3.146556e-006", "3.454820e-006", "3.
+274373e-006", "4.285631e-006", "5.785602e-006", "1.415010e-005", "8.9
+72383e-005"],
      },
  100 => {
        6400 => ["2.476644e-005", "2.467621e-005", "2.483786e-005", "2
+.508974e-005", "2.544312e-005", "2.722128e-005", "3.560081e-005", "6.
+958512e-005"],
        6401 => ["2.496192e-005", "2.466118e-005", "2.545439e-005", "2
+.517620e-005", "2.540552e-005", "2.719120e-005", "3.856316e-005", "6.
+902498e-005"],
        6402 => ["2.481531e-005", "2.478148e-005", "2.477772e-005", "2
+.636791e-005", "2.524011e-005", "2.813103e-005", "3.558202e-005", "6.
+867160e-005"],
        6407 => ["2.529650e-005", "2.470253e-005", "2.500703e-005", "2
+.605213e-005", "2.524763e-005", "2.798066e-005", "3.579630e-005", "6.
+880318e-005"],
        6417 => ["2.484538e-005", "2.472884e-005", "2.516493e-005", "2
+.489425e-005", "2.573634e-005", "2.744308e-005", "5.024715e-005", "7.
+004000e-005"],
        6431 => ["2.474012e-005", "2.479651e-005", "2.509350e-005", "2
+.497696e-005", "2.542808e-005", "2.732278e-005", "3.616471e-005", "6.
+890844e-005"],
        6447 => ["2.480027e-005", "2.486794e-005", "2.521756e-005", "2
+.498448e-005", "2.546943e-005", "2.736037e-005", "3.612712e-005", "7.
+171666e-005"],
        6461 => ["2.479651e-005", "2.598070e-005", "2.502583e-005", "2
+.511230e-005", "2.544688e-005", "2.748819e-005", "3.602562e-005", "7.
+065277e-005"],
      },
  1000 => {
        64000 => ["2.421081e-004", "2.417660e-004", "2.421194e-004", "
+2.430592e-004", "2.424840e-004", "2.443787e-004", "2.529011e-004", "2
+.859606e-004"],
        64001 => ["2.457358e-004", "2.437697e-004", "2.429878e-004", "
+2.420592e-004", "2.433675e-004", "2.653144e-004", "2.551718e-004", "2
+.864381e-004"],
        64002 => ["2.462584e-004", "2.422171e-004", "2.418825e-004", "
+2.428111e-004", "2.425818e-004", "2.466118e-004", "2.544763e-004", "2
+.877989e-004"],
        64007 => ["3.002949e-004", "2.450216e-004", "2.418562e-004", "
+2.435254e-004", "2.425743e-004", "2.453486e-004", "2.530139e-004", "2
+.858854e-004"],
        64017 => ["2.421269e-004", "2.419239e-004", "2.420743e-004", "
+2.432246e-004", "2.425066e-004", "2.469802e-004", "2.541267e-004", "2
+.874869e-004"],
        64031 => ["2.430630e-004", "2.421570e-004", "2.421119e-004", "
+2.424239e-004", "2.427622e-004", "2.469839e-004", "2.546041e-004", "2
+.861975e-004"],
        64047 => ["2.419276e-004", "2.421908e-004", "2.423600e-004", "
+2.429878e-004", "2.426269e-004", "2.465291e-004", "2.552507e-004", "2
+.862727e-004"],
        64061 => ["2.420141e-004", "2.421720e-004", "2.437020e-004", "
+2.422585e-004", "2.428148e-004", "2.447960e-004", "2.548259e-004", "2
+.877651e-004"],
      },
  10000 => {
        640000 => ["2.420318e-003", "2.417709e-003", "2.432201e-003", 
+"2.419630e-003", "2.419201e-003", "2.419870e-003", "2.428746e-003", "
+2.479050e-003"],
        640001 => ["2.417886e-003", "2.445092e-003", "2.440434e-003", 
+"2.420611e-003", "2.418863e-003", "2.420397e-003", "2.427712e-003", "
+2.475866e-003"],
        640002 => ["2.438054e-003", "2.583405e-003", "2.428769e-003", 
+"2.417261e-003", "2.454866e-003", "2.421570e-003", "2.427709e-003", "
+2.498681e-003"],
        640007 => ["2.421299e-003", "2.417205e-003", "2.432803e-003", 
+"2.483978e-003", "2.453272e-003", "2.442415e-003", "2.450900e-003", "
+2.462982e-003"],
        640017 => ["2.500963e-003", "2.435400e-003", "2.450648e-003", 
+"2.420261e-003", "2.417946e-003", "2.423333e-003", "2.497985e-003", "
+2.468106e-003"],
        640031 => ["2.444475e-003", "2.417058e-003", "2.417446e-003", 
+"2.422923e-003", "2.419607e-003", "2.420807e-003", "2.586104e-003", "
+2.461452e-003"],
        640047 => ["2.418991e-003", "2.419912e-003", "2.420588e-003", 
+"2.430111e-003", "2.420299e-003", "2.420058e-003", "2.458749e-003", "
+2.461088e-003"],
        640061 => ["2.418036e-003", "2.426551e-003", "2.417190e-003", 
+"2.417540e-003", "2.456092e-003", "2.421130e-003", "2.687839e-003", "
+2.464982e-003"],
      },
  100000 => {
        6400000 => ["2.428315e-002", "2.420329e-002", "2.439345e-002",
+ "2.585758e-002", "2.429269e-002", "4.672148e-002", "2.423426e-002", 
+"2.596225e-002"],
        6400001 => ["2.427041e-002", "2.421823e-002", "2.494646e-002",
+ "2.521549e-002", "2.429425e-002", "2.821954e-002", "4.104935e-002", 
+"2.433770e-002"],
        6400002 => ["2.438305e-002", "2.440626e-002", "2.534497e-002",
+ "2.497522e-002", "3.028730e-002", "2.835039e-002", "2.435997e-002", 
+"2.436076e-002"],
        6400007 => ["2.423938e-002", "2.425249e-002", "2.723007e-002",
+ "2.512355e-002", "2.735322e-002", "2.727200e-002", "2.420713e-002", 
+"2.446515e-002"],
        6400017 => ["2.423709e-002", "4.237703e-002", "2.520556e-002",
+ "2.438899e-002", "2.867939e-002", "4.364147e-002", "2.482842e-002", 
+"2.461508e-002"],
        6400031 => ["2.429650e-002", "2.488188e-002", "2.524961e-002",
+ "2.427564e-002", "2.786244e-002", "2.430260e-002", "2.426029e-002", 
+"2.601774e-002"],
        6400047 => ["2.431900e-002", "2.518372e-002", "2.445157e-002",
+ "2.443820e-002", "4.614855e-002", "2.429054e-002", "4.245227e-002", 
+"2.426723e-002"],
        6400061 => ["4.038383e-002", "2.490072e-002", "2.428324e-002",
+ "2.609846e-002", "2.797442e-002", "2.432375e-002", "2.438290e-002", 
+"2.429224e-002"],
      },
  1000000 => {
        64000000 => ["2.572672e-001", "2.497615e-001", "2.667708e-001"
+, "2.527840e-001", "2.471348e-001", "2.616865e-001", "2.651201e-001",
+ "2.847880e-001"],
        64000001 => ["2.490815e-001", "2.559308e-001", "2.618821e-001"
+, "2.639641e-001", "2.663378e-001", "2.501689e-001", "2.502240e-001",
+ "2.773069e-001"],
        64000002 => ["2.513739e-001", "2.635636e-001", "2.727179e-001"
+, "2.620300e-001", "2.500594e-001", "2.552292e-001", "2.502405e-001",
+ "2.729111e-001"],
        64000007 => ["2.524076e-001", "2.682177e-001", "2.625778e-001"
+, "2.473350e-001", "2.523794e-001", "2.785869e-001", "2.645629e-001",
+ "2.462325e-001"],
        64000017 => ["2.677752e-001", "2.489456e-001", "2.637126e-001"
+, "2.579108e-001", "2.808452e-001", "2.528726e-001", "2.500170e-001",
+ "2.682441e-001"],
        64000031 => ["2.684036e-001", "2.636544e-001", "2.666303e-001"
+, "2.492105e-001", "2.674942e-001", "2.667611e-001", "2.676281e-001",
+ "2.657490e-001"],
        64000047 => ["2.487165e-001", "2.695648e-001", "3.246719e-001"
+, "2.683215e-001", "2.699818e-001", "3.049541e-001", "2.723240e-001",
+ "2.776325e-001"],
        64000061 => ["2.526046e-001", "2.605271e-001", "2.954536e-001"
+, "2.483719e-001", "2.784286e-001", "2.991503e-001", "2.521404e-001",
+ "2.613033e-001"],
      },
}
[download]

Can it be beaten? Maybe, but I very much doubt it will be done using Boyer Moore or similar, jump-around-the-data algorithms. Time will tell :)

Alright. Here's a demo of alpha skip search on bitstrings. The needle size is fixed, but it should be trivial to follow this with a longer compare. A possibility exists that some off-by-one errors have crept in. If that's the case, just deduct from my paycheck. :-)

Alpha skip is quadratic in the worst case and probably not the best choice when nasty matches are expected. Anyway, I'm curious how this might hold up against the optimized brute-force routines.

Note: to change the needle size, just edit the M and logM values. (Compile with -fno-strict-aliasing, or fix the type-punning).

Making minimal changes to your code to allow it compile here, and it simply never finds anything(very quickly), no matter how simple I make the search criteria:

C:\test\C>cl /W3 bitsearch-oiskuu.c
Microsoft (R) C/C++ Optimizing Compiler Version 15.00.21022.08 for x64
Copyright (C) Microsoft Corporation.  All rights reserved.

bitsearch-oiskuu.c
bitsearch-oiskuu.c(49) : warning C4267: '=' : conversion from 'size_t'
+ to 'U16', possible loss of data
bitsearch-oiskuu.c(78) : warning C4267: '=' : conversion from 'size_t'
+ to 'char', possible loss of data
bitsearch-oiskuu.c(80) : warning C4267: '+=' : conversion from 'size_t
+' to 'char', possible loss of data
Microsoft (R) Incremental Linker Version 9.00.21022.08
Copyright (C) Microsoft Corporation.  All rights reserved.

/out:bitsearch-oiskuu.exe
bitsearch-oiskuu.obj

C:\test\C>bitsearch-oiskuu
Result           -1     Took 1.837746e-004 secs
[download]

I've wasted enough time on it; and won't be expending any more, but here is the modified code that compiles clean (apart from warnings that originate from your posted code):

<Reveal this spoiler or all in this thread>

I'd suggest you enter it in an obfuscated C competition; could be a winner.

---

With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'

Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.

"Science is about questioning the status quo. Questioning authority". I'm with torvalds on this

In the absence of evidence, opinion is indistinguishable from prejudice. Agile (and TDD) debunked

'm curious how this might hold up against the optimized brute-force routines.

It's 3 orders of magnitude slower.

More or less, yeah.

[         0.033410] needle 512, haystack 134217216 bytes
[         0.000030] rot 5 at 8000007; bitoff=64000061
[         0.000017] needle stuffed!
[         0.000644] search=64000061
[download]

I see your table lists "2.483719e-001" for the corresponding case. Brute force appears to be >300 times slower (0.2483719/0.000644).

Looks like { M = 16384, logM = 14 } case is the fastestfaster, with tables weighing in at 64KB. Beyond that, slowdown. (0.2784286/0.000236 == ~1180). Even bigger needles/tables may be quicker still.

I'll attempt a brief description of the Alpha Skip Search. Hopefully, this may shed some light on the basic operation of the algorithm.

Consider a pattern of length M. The approach is to quickly locate any fractional substring within the pattern, wherever it may occur. Substrings of length F take offsets from 0 .. M-F, inclusive. Thus there is a set of M-F+1 needle fragments. The preprocessing stage creates an index of those fragments. A perl implementation might read as follows:

push @{$index{ substr($needle, $_, $F) }}, $_     for 0 .. $M-$F;

Search phase proceeds by inspecting the haystack one window at a time. Window shift is the same figure as above, M-F+1. Each time, a substring is extracted at the end of the window. The index now yields all possible alignments where this fragment will match in the pattern.

Theoretical optimum for F is log_sM (log based on alphabet size). That amounts to one entry per index bucket, on the average. For random data, there will be one alignment to test per window, and this is likely to fail at the very start.

In the case of bit-string matching, the substrings are nothing but small integers below (1<<F), which suggests a very simple array-based indexing scheme (instead of trie). Here's a sample implementation in C (just a rehash of the previous demo, main difference being this one handles variable length needles).

Tested with gcc/clang under linux; my apologies if the code is too obscure, or otherwise unworkable.

Update. Simplified above code. Note that it is sometimes difficult to put a name on a string search routine, especially with bit-strings. Minor modification can turn one search into another, and the variants abound. Here's a version that might be called Horspool.

Despite saying I wouldn't look at this further; since you updated, I did. But I didn't get very far.

Once I commented out a couple (non-existent on my system) unix-y headers -- no idea if they are necessary or not:

#include <stdio.h>
//#include <stdint.h>
#include <stdlib.h>
#include <string.h>
//#include <unistd.h>
[download]

I got this:

c:\test\c\oiskuu-search.h(9) : warning C4293: '<<' : shift count negat
+ive or too big, undefined behavior
[download]

Which I fixed by changing 1L to 1ull:

enum { logM = 32, Mmax = 1ull << logM, Mdflt = 65536, Mnil = Mmax - 1 
+};
[download]

Which got me to:

c:\test\c\oiskuu-search.h(10) : warning C4341: 'Mmax' : signed value i
+s out of range for enum constant
c:\test\c\oiskuu-search.h(10) : warning C4309: 'initializing' : trunca
+tion of constant value
c:\test\c\oiskuu-search.h(14) : error C2143: syntax error : missing ')
+' before '('
c:\test\c\oiskuu-search.h(14) : error C2091: function returns function
c:\test\c\oiskuu-search.h(14) : error C2059: syntax error : ')'
c:\test\c\oiskuu-search.h(14) : error C2085: 'bt_off_t' : not in forma
+l parameter list
[download]

From these 3 lines:

// F = logM is theoretically optimal. Mmax is our maximum needle size.
// if logM/Mmax is changed, be sure to also check the bt_off_t below
enum { logM = 32, Mmax = 1ull << logM, Mdflt = 65536, Mnil = Mmax - 1 
+};
enum { F = 16, Ftop = 1 << F, Fmask = Ftop - 1 };

//typedef uint32_t bt_off_t;        // must hold 0..Mnil
enum { BT_OFF_MAX = Mnil } __attribute__((packed)) typedef bt_off_t;
[download]

I hope you'll understand that I'm not being unfair when I say I have simply no clue how to fix that...but I did try.

(Quite why you need to invent your own special integer types all over the place is beyond my understanding.)

---

With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'

Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.

"Science is about questioning the status quo. Questioning authority". I'm with torvalds on this

In the absence of evidence, opinion is indistinguishable from prejudice. Agile (and TDD) debunked

Boyer-Moore is O(1) in memory usage.

Hm? Boyer-Moore prep is the same, O(m) in time and space. A good-shift table of length m is used. It might be more compact in practice.

If the problem is too large, divide it! Code a tuned search routine for fixed length (sub)needles, such that all tables fit into L2. My guess is KB needles can be done. Now the search becomes two-level. Top-level "alphabet" are the subneedles.

As for brute-force. The "Boyer-Moore beating assembler versions that utilize microcoded string search instructions" is a decades-old lesson. Tried to google for some edutaining links, but sadly, google has become ****.

As for brute-force. The "Boyer-Moore beating assembler versions that utilize microcoded string search instructions" is a decades-old lesson

For string search, decades ago (before cpus acquired high-speed, on-chip caches) Boyer Moore was the business, but since the 486 circa. 1990, on-chip caches have tipped the balance again.

"Decades old lessons" were relevant, decades ago.

If you won't take my words for it; and aren't equipped to try it for yourself, take Bjarne Stroustrup's word for it. View http://channel9.msdn.com/Events/GoingNative/GoingNative-2012/Keynote-Bjarne-Stroustrup-Cpp11-Style and skip ahead to timecode 44:40.

A few seconds in he presents a question (on a slide titled "Vector vs. Linked List"), that goes like this:

The task comes in two parts:

1. Generate N random integers and insert them into a sequence so that each is inserted in its proper position in numerical order.
2. Remove the elements one at a time by picking a random position in the sequence and removing the element there.

And the question is: For which N does it become faster to use a linked list than a flat array?

The decades old wisdom on this is that insertions/deletions into/from an array are O(N²), but for a linked list they are O(N log N) (with slightly higher constant factors), so N only need grow to a couple of hundred or a 1000 or so for the linked-list to start outperforming the flat array. Right? Wrong!

Watch for a few more seconds (and wonder in amazement at his non-existent graph) as he shows that the answer is "Never". At no point does the linked list become faster than the flat array. In fact the bigger N gets, the bigger the gap between the two becomes, in favour of the flat array.

As counter-intuitive as it may be, the benefits of the caching on the sequential accesses made by the flat array, far outweigh the costs of moving more data. The non-sequential, scattered around memory nature of linked-lists means they never catch up, let alone surpass the performance of the cache-friendly contiguous memory accesses of the flat array.

Times change; and a lot of the "knowledge" CS students acquired, despite being tortured within an inch of their lives by Knuth; is simply out-of-date. Whilst the proofs of correctness still hold; the analysies of operational complexity -- done in terms of the imaginative, but imaginary MIX (later MMIX) -- simply do not consider the effects of caching, pipelining, branch prediction, memory stalls et al.

Stroustrup concludes that there are "3 orders of magnitude performance" to be gained.

And, the relevance of string search analysis to bit-string search, has to my knowledge, never been demonstrated. They are quite different animals.

---

With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'

Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.

"Science is about questioning the status quo. Questioning authority". I'm with torvalds on this

In the absence of evidence, opinion is indistinguishable from prejudice. Agile (and TDD) debunked