Yeah, I have had the same experiences.
Ordinarily, I define some kind of (say ...) free() method in such objects and am careful to call it before the object goes out of scope. Also, as you imply, if the object carries-around with it a lot of data that I know is no longer needed, I define another method to explicitly release that data (if it exists). Attempts to do that sort of thing “more automagically” did not seem to work as I had expected, but I also didn’t pursue the matter too much. This arrangement based on self-discipline worked well, and it also gave me definite control over exactly when the operation would happen.
Also, I have had some bad-experiences with freeze/thaw, in which the frozen data either could not be thawed or was not exactly the same. For this reason, and for the added benefit of actually being able to see the saved data on inspection, I stored the data in JSON format within a database table. And ... I actually embedded the freeze/thaw-equivalent functionality into a base class of my own devising. (Now, I had definite control of it ...)
So, in this case, I might write the method so that it accepted either uuid => uuid or data => hashref. If given a uuid, it would look-up itself in the database (and die() if it could not be found. Otherwise, it would instantiate itself, coining a uuid for itself, and snapshot its initial state into the database. (It explicitly checked that the hashref contained all of the required keys, and that the values were plausible, and would die() on the spot if it smelled any smoke.) And, once again, in a base class I did all of these things myself, consciously trading convenience for awareness and explicit control over exactly what was going on. (I never did figure out what was wrong with freeze/thaw on that machine: some data worked just fine, but of course “some” isn’t good enough.)