Granted, I do not understand in detail (yet...) the approach that you are taking, but when you refer to “OLE” instead of a DBI (ODBC...) database driver, well, I rather immediately become nervous.   You do not need to approach MS-Access, the Application, when you can directly approach Jet, the Database Engine.

Nevertheless, your experience is not atypical.   Many database engines (and I do not recall if Jet is of them) provide “fast track” methods for importing large amounts of data from predictably-formatted files.   If they do so, they probably provide a non-standard way to do this in SQL.   (Microsoft, on the other hand, provides a “flat file” database driver, which is a very nifty trick.)

While the completion-times that you are experiencing with your present approach are by no means “a show-stopper,” I agree that they suggest that you have not yet found the most appropriate strategy yet.   “Think outside the box,” and even (if appropriate), “think outside the Perl.”

Please give us all more details: of what you are doing, and of what you find.