Beefy Boxes and Bandwidth Generously Provided by pair Networks
Welcome to the Monastery
 
PerlMonks  

Re^3: How to split big files with Perl ?

by RichardK (Parson)
on Dec 27, 2014 at 17:20 UTC ( [id://1111477]=note: print w/replies, xml ) Need Help??


in reply to Re^2: How to split big files with Perl ?
in thread How to split big files with Perl ?

The most efficient block size will depend on lots of things, but the memory page size of your OS will likely be the most significant. 32 bytes is way too small, I'd start with 4k or 8k and go up from there. Why not try several different multiples of 4K and see which one works best for you?

Also, read returns the number of bytes actually read so there's really no need to use length.

my $len = read($in,$buf,4*1024);
...

[download]

And $len is an integer so it would be better to use the numeric not equal '!=' rather than the pattern match operator.

Replies are listed 'Best First'.
Re^4: How to split big files with Perl ?
by james28909 (Deacon) on Dec 28, 2014 at 03:46 UTC
    Well i did turn up the speed some, but I would watch my memory fill up as it was running. It would punch out a 2GB file in no time (less that 10 secs or so) but then I would see a dramatic slowdown, as in it would only be writing KB/s instead of MB/s. I will try your suggestion as well, thanks

    Also, $len = length $buf; gets the length of $buf, then later on checks and makes sure it is the same size as the read length. If it is not the same size, then that is more than likely the end of the file. I need to figure out a better way to check for end of file actually.
[reply]

In Section Seekers of Perl Wisdom

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?

www.com | www.net | www.org
Node Status?
node history
Node Type: note [id://1111477]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? • Last hour • Other CB clients
Other Users?
Others making s'mores by the fire in the courtyard of the Monastery: (6)
As of 2024-04-24 12:59 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found