Beefy Boxes and Bandwidth Generously Provided by pair Networks
XP is just a number
 
PerlMonks  

Re: grabbing random n rows from a file

by GrandFather (Saint)
on Jul 18, 2006 at 21:46 UTC ( [id://562141]=note: print w/replies, xml ) Need Help??


in reply to grabbing random n rows from a file

Are the lines fixed length? Are there a fixed number of lines per "record"? Is any of this stuff indexed? Is this something that you need to do one off (or occasionally)? Does the "data base" change over time? If it changes can "records" be inserted? Why isn't this in a real database?

Without knowing the answers to any of the above here are a couple of approaches:

  • do an indexing pass through the file to build a hash keyed by "person" containing the file position of the start of the record and positions of lines in the record. Seek through the file pulling out randomly selected j lines from each record as required.
  • Scan through the file identifying record starts and create an array of lines within the current record. At the end of each record spit out j random lines as required from the array.

The first variant is more appropriate if you want to do this multiple times without the "data base" changing between times - keep the index. The second variant is more appropriate for one off use or if the contents of the "data base" is changing and maintaining an index is not fesable.

Update: s/lones/lines/


DWIM is Perl's answer to Gödel

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: note [id://562141]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others cooling their heels in the Monastery: (11)
As of 2024-03-28 09:07 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found