I thought perl specifically tries to hold onto memory so it can reuse it again instead of having to allocate new memory?
No, it defers to the implementation of the C runtime library malloc.
Though I wouldn't recommend it unless you are desperate, it is possible to change the "release memory to the OS" behaviour by building Perl using a custom malloc.
This has been discussed here many times before, see for example: Not able to release memory and How to return unused memory to OS?.
From my testing across a number of OSes, I would expect only Linux and Windows to return "large" blocks of memory back to the OS. I would be interested to learn what operating system the OP is running on.