First of all, there is no way this could be taking significant amounts of time unless the app is doing basically nothing. However, it is not correct to say that perl "shorts out any further calls for that same module." It at least involves perl running some code that will check %INC for the module in question, and with "use" it would also involve running an import(), although a use statement typically only happens once because it is a virtual BEGIN block.
So, what I'm saying is that require and use definitely do take some time inside of a string eval, even if you have already run them before, and there is a difference in the amount of time they take.