Re: list of unique strings, also eliminating matching substrings

I have hundreds of sets of strings, each containing about 100,000 strings, and each string is about 300 characters long.

A few questions:

You want to eliminate the dups in each of the files? Or across all of the files?
What (roughly) are the maximum and minimum lengths of the strings?
Do they consist soley of ACGT or are the other characters (X N etc.)?

Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.

"Science is about questioning the status quo. Questioning authority".

In the absence of evidence, opinion is indistinguishable from prejudice.

Comment on Re: list of unique strings, also eliminating matching substrings

Replies are listed 'Best First'.
Re^2: list of unique strings, also eliminating matching substrings by lindsay_grey (Novice) on May 21, 2011 at 03:44 UTC
1. I want to eliminate the duplicates within each file. 2. The strings range from 200 to 400 characters. 3. The complete alphabet is A, G, C, T, N. Note for point 1. I want to eliminate not just the exact duplicates but also those that are contained within a longer string.	[reply]
Re^3: list of unique strings, also eliminating matching substrings by BrowserUk (Patriarch) on May 21, 2011 at 05:53 UTC
Presumably you've code the obvious two loops method and it is taking too long. Could you supply a timing for one of your datasets? Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error. "Science is about questioning the status quo. Questioning authority". In the absence of evidence, opinion is indistinguishable from prejudice.	[reply]
Re^4: list of unique strings, also eliminating matching substrings by lindsay_grey (Novice) on May 29, 2011 at 22:43 UTC
7 hours! for about 200,000 starting strings.	[reply]
Re^5: list of unique strings, also eliminating matching substrings by BrowserUk (Patriarch) on May 30, 2011 at 00:18 UTC
Re^6: list of unique strings, also eliminating matching substrings by lindsay_grey (Novice) on May 30, 2011 at 20:07 UTC
Some notes below your chosen depth have not been shown here
Re^4: list of unique strings, also eliminating matching substrings by lindsay_grey (Novice) on May 29, 2011 at 22:51 UTC
7 hours! starting with about 200,00 strings. Thanks!	[reply]


Think about Loose Coupling
	PerlMonks