Preprocessing input files is not something to be afraid of. Of course, it depends on the flow of your data but it is sometimes more efficient to split the job into two distinct, simpler parts than to try and code one more complex solution.
It seems as if you have a (approximately) a set of client system running something like a data entry or data processing system. You are taking the results from these client systems and loading/processing them into some master program. Even if you are just swapping files among systems, a preprocessing step could be very helpful (your milage may vary). The benefit is that, at a certain point in the flow of data, the data will all look exactly the same regardless of originating/destination platform. That can greatly simplify further processing downstream of that point.
Its worth looking at more closely.
PJ
use strict; use warnings; use diagnostics;
