Well yeah. A lot of that I just left out because it's not particularly germane to the problem I'm trying to solve. I've got standard process boilerplate surrounding this thing.

processed/unprocessed doesn't have sustained value over time, as a matching run will produce simple deterministic results and there will either be data or not. Not means failed, and processed is a function of a monthly run being done or not. So there isn't sufficient justification to add additional columns to denote purely derivative information.

Only collect data that can conceivably be useful. I have overall process timing and logging in my boilerplate perl. But most of that stuff isn't really useful.