Beefy Boxes and Bandwidth Generously Provided by pair Networks
go ahead... be a heretic
 
PerlMonks  

Re: Reduce RAM required

by jimpudar (Pilgrim)
on Jan 08, 2019 at 22:22 UTC ( [id://1228236]=note: print w/replies, xml ) Need Help??


in reply to Reduce RAM required

Maybe I'm not understanding your requirement, but it seems to me you could just read in one window's worth of sequences, shuffle them, and write them to the file. Then you could empty the window buffer and repeat until you reach EOF. This way, your memory will never grow larger than the window size.

The algorithm is a little trickier that way because you need to deal with sequences that overflow the window, but I don't think it should be too hard.

EDIT: After reading hdb's comment, I would also suggest you go that route. That seems even better than what I was describing.

πάντων χρημάτων μέτρον έστιν άνθρωπος.

Replies are listed 'Best First'.
Re^2: Reduce RAM required
by onlyIDleft (Scribe) on Jan 09, 2019 at 16:04 UTC

    I want to obliterate signal by introducing ample noise by random shuffling. But think about it - if I bring distant regions of the sequences together and then shuffle them (or shuffle them and then bring them together), then it is more of global rather than local shuffling - which is something I prefer. You agree with my approach?

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: note [id://1228236]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others contemplating the Monastery: (7)
As of 2024-04-16 18:19 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found