Beefy Boxes and Bandwidth Generously Provided by pair Networks
Perl-Sensitive Sunglasses
 
PerlMonks  

Re: How to split big files with Perl ?

by sundialsvc4 (Abbot)
on Dec 28, 2014 at 19:58 UTC ( [id://1111585]=note: print w/replies, xml ) Need Help??


in reply to How to split big files with Perl ?

Given that there is a split command (at least, on most Unixes), I would be very strongly inclined to try to use it, hoping that its implementation is most efficient.   (In any case, it is an existing implementation of a classic Thing That Is Already Done.™)

“Splitting a file” is never a thing that should call for recursion:   all that you’re really doing is reading from one file and switching from one output-file to the next one at specified intervals.   I’ve seen lots of ways to do that, probably the most-elaborate ones (not in Perl ...) using memory-mapped files to actually exploit the virtual-memory subsystem’s I/O capabilities as the means of reading from the target and getting the data (in one step) where it needs to go.

Still, in my mind, it all comes back to the same thing:   this is A Classic Thing That Has Already Been Done.™   Search for an existing tool that you can rely-upon, and use it, to avoid having to write-and-debug “yet another” piece of software to do such a trivial task.   Surely you can find one that will meet your project’s performance expectations.

Replies are listed 'Best First'.
Re^2: How to split big files with Perl ?
by karlgoethebier (Abbot) on Dec 28, 2014 at 21:23 UTC
    "...in my mind, it all comes back to the same thing ..."

    Holy shit, nothing but the truth. You got it!

    «The Crux of the Biscuit is the Apostrophe»

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: note [id://1111585]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others sharing their wisdom with the Monastery: (6)
As of 2024-04-18 07:05 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found