Re: list of unique strings, also eliminating matching substrings

~~When you say "about 300 characters long", what is the actual range?~~ Are there any constraints on where a substring may match a larger string? Can there be exact matches within a set of strings and if so should duplicates be removed?

Update: length question already answered I see.

Update: and the key question I didn't ask: how many strings of the original 100,000 do you expect you might end up with after duplicates and substrings are removed?

True laziness is hard work

Comment on Re: list of unique strings, also eliminating matching substrings

Replies are listed 'Best First'.
Re^2: list of unique strings, also eliminating matching substrings by lindsay_grey (Novice) on May 30, 2011 at 22:42 UTC
i don't think there are any constraints on where a substring may match a larger string. there can be exact matches. we are removing those using the unique function. for the current set of sequences (assuming the program is working correctly), we go from 206,737 sequences to 166,089. not sure how many are removed because they are exact matches and how many are removed because they match a substring of a larger sequence. i will add that check. i think the program is working correctly, but it takes a long time, longer than it seems it should take. (~5 hours on a pretty fast computer) getting the unique sequences is fast, but we have an additional step where we go back and retrieve the sequence name and then do the substring comparison. it is these two steps that are so slow.	[reply]

Replies are listed 'Best First'.

Re^2: list of unique strings, also eliminating matching substrings
by lindsay_grey (Novice) on May 30, 2011 at 22:42 UTC

i don't think there are any constraints on where a substring may match a larger string.

there can be exact matches. we are removing those using the unique function.

for the current set of sequences (assuming the program is working correctly), we go from 206,737 sequences to 166,089. not sure how many are removed because they are exact matches and how many are removed because they match a substring of a larger sequence. i will add that check.

i think the program is working correctly, but it takes a long time, longer than it seems it should take. (~5 hours on a pretty fast computer) getting the unique sequences is fast, but we have an additional step where we go back and retrieve the sequence name and then do the substring comparison. it is these two steps that are so slow.

[reply]