This isn't just to support caller, it's a very common feature of most languages. If nothing else, it generally contains the information of where to jump to when you return from the current subroutine and the values of the lexical variables which are in that scope but not in the current scope ¹.

It's not to everyone's taste, and it uses a different language, called scheme, but the online textbook "Structure and Interpretation of Computer Languages" or SICP covers this, in this section (and related pages).

Recursion is a powerful tool, and like all such it can be mis-used. In particular, note that the amount of stack space is limited on different platforms. Some Win2k systems give you 2mb of stack (by default). If you have a recursive routine which has a 64k variable in the stack frame, you'll die after 30 recursions or so. (This was a real-world bug in a C program).

¹: OK, so lexicals may be stored seperately from the call stack, since you can have intermediate scopes. The same principle applies though.