While I have no direct wisdom to share, I will tell you
which 2 questions immidiately came to mind when I read
your post. Maybe they will shed some lite:
- You said: "monitoring the output, it
looks like it will take 166 hours to run What are
you baseing this number on? How are you doing this
monitoring? are you sure that your method of monitoring
the output file isn't flawed?
One scenario I
can easily imagine is if you are just checking the
file's size X seconds after starting the program,
dividing into 81GB, and multiplying by X. unless
you turned on autoflush, maybe you just happened
to check the file's size just before it was about
to do a batch write -- completely skewing your estimate.
- Where is your code? A perl program like you
describe sounds extremely simple, but whenever people post
questions about programs that do things "simply"
without posting any code, I tend to wonder what else
is going on. What are "any needed changes" ?
are you sure
there isn't something else you are doing that's taking
a lot of time?
