comment on

I agree that SQLite is quite probably “the thing to do” here. I would put each of the sequences into a table, along with (in a separate column) whatever key value you happen to be looking for ... whatever (I’m not a biologist...) “identifies” the sequence, whether or not it is unique.

If there are many keys that might identify a particular sequence, “okay, big deal, you have a many-to-one relationship.”

Anyhow ... SQLite gives you a quite-robust SQL implementation, in a single file, without a server. (Really, the only “gotcha” that it has is transactions: when you are changing data, you need to be in a transaction while doing so, or you’ll really regret it speed-wise.)

Now, build non-unique indexes.

Once you have the data in this database form, I suspect that a lot of your “difficult processing” will be reduced to SQL queries that you might not even require a Perl program to run.

SQLite is quite the tool ... I really beat the heck out of it once, or tried to. It didn’t flinch. 31 million rows at one point, and it still wasn’t flinching.

In reply to Re: renaming 1000's of FASTA files by sundialsvc4
in thread renaming 1000's of FASTA files by garyboyd

Are you posting in the right place? Check out Where do I post X? to know for sure.
Posts may use any of the Perl Monks Approved HTML tags. Currently these include the following:
<code> <a> <b> <big> <blockquote> <br /> <dd> <dl> <dt> <em> <font> <h1> <h2> <h3> <h4> <h5> <h6> <hr /> <i> <li> <nbsp> <ol> <p> <small> <strike> <strong> <sub> <sup> <table> <td> <th> <tr> <tt> <u> <ul>
Snippets of code should be wrapped in <code> tags not <pre> tags. In fact, <pre> tags should generally be avoided. If they must be used, extreme care should be taken to ensure that their contents do not have long lines (<70 chars), in order to prevent horizontal scrolling (and possible janitor intervention).
Want more info? How to link or or How to display code and escape characters are good places to start.


Problems? Is your data what you think it is?
	PerlMonks