Come for the quick hacks, stay for the epiphanies. | |
PerlMonks |
comment on |
( [id://3333]=superdoc: print w/replies, xml ) | Need Help?? |
For bulk processing like this, I'd sidestep DBI and use command lines tools. Create a file called: getnames.sql containing:
And two perl scripts. 1) partNames.pl:
And 2) uniqNames. pl:
Then run the following pair of commands:
The first line dumps the names and pipes them to the perl script that then writes them to a set of files (called names.?, where ? is the first character of the name). The second script reads those files one at a time and writes the unique names to stdout where they are directed into a file called uniqedNames.txt. The whole (untested) process should take less time than sorting the 34 million names -- inside the DB or with your system sort utility. I'd hazard a guess that it would take less than 20 minutes. Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
"Science is about questioning the status quo. Questioning authority".
In the absence of evidence, opinion is indistinguishable from prejudice.
In reply to Re: Fetch Mysql huge database
by BrowserUk
|
|