I agree that the data should go to the node's scratch, the processing happens, and results are uploaded to NFS again. Never mind NFS, RAID5 is also not helping writes. However, sometimes files become so ridiculously large that the local scratch does not suffice, and I am forced to work off NFS - though I still try to put the products on scratch, and upload them to NFS afterwards.
I did write a simple FullyBuffered module basically doing sysopen, sysread into a large buffer, maintaining a cursor (to avoid unecessary string copies), etc. I was timing this against the original script using Perl's IO, and it performed, to my surprise, a little worse. The Perl IO version takes about 3 minutes for 2^20 lines, my fully buffered approached 4 minutes for 2^20 lines (dang, I tossed its log file). Of course, I do suspect that I am doing something stupidly inefficient.
I thank you very much for showing me the proper utf8 conversions. Should I continue with my module approach, it will come in handy.
-
Are you posting in the right place? Check out Where do I post X? to know for sure.
-
Posts may use any of the Perl Monks Approved HTML tags. Currently these include the following:
<code> <a> <b> <big>
<blockquote> <br /> <dd>
<dl> <dt> <em> <font>
<h1> <h2> <h3> <h4>
<h5> <h6> <hr /> <i>
<li> <nbsp> <ol> <p>
<small> <strike> <strong>
<sub> <sup> <table>
<td> <th> <tr> <tt>
<u> <ul>
-
Snippets of code should be wrapped in
<code> tags not
<pre> tags. In fact, <pre>
tags should generally be avoided. If they must
be used, extreme care should be
taken to ensure that their contents do not
have long lines (<70 chars), in order to prevent
horizontal scrolling (and possible janitor
intervention).
-
Want more info? How to link
or How to display code and escape characters
are good places to start.
|
