|Come for the quick hacks, stay for the epiphanies.|
Per a previous suggestion, I would recommend to any of my customers who ask the same thing to consider simply charging credit cards as the numbers are submitted, thus avoiding the need to store credit card numbers in the first place. It will save you alot of paranoia and development and maintenance time.
For one company I worked for, in which storing credit card numbers was seen as a hard requirement by the business types, here is what I did:
1) https: Of course, credit card numbers must be submitted on a secure page. This is a no-brainer, but I thought I'd mention it for completeness.
2) Encryption: Credit card numbers are stored by the application server in encrypted form in a database via GnuPG::Interface.
3) Decryption: A separate transaction server, which is locked down tighter than, hee hee, well, real tight, has a daemon running that wakes up every 15 minutes. The daemon checks to see if there are any pending transactions, then (after various gyrations) decrypts the credit cards and submits the transactions, after which they are marked as complete.
The key to this is that your private key(s) are on the transaction server. The transaction server has to be made the most secure box you can make, with every unnecessary process turned off and/or ipchains blocking everything except the database, transaction, and monitoring ports out and ssh in. In this particular company, which shall remain nameless, even the CTO was denied an account on the box. (He wasn't particularly astute or technically savvy, as I hear are many CTOs, and denying him an account definitely made waves, though I stuck to my guns.)
I would also recommend reading up on the precise definition of a transaction (particularly 2-stage transactions) and model your transaction daemon on these specifications. All sorts of things can happen between beginning a credit card transaction and completing the transaction, and you should have status markers that indicate each step of the process as well as store every possible bit of information that goes back and forth. In light of the aforementioned caveat, you should also architect your daemon to be able to recover mid-stage through a botched transaction and proceed to completion. Everything from network outages to credit card processing server errors can cause a transaction to terminate unexpectedly, and accounting will at some point want you to be able to complete any partially completed transactions.
Hope this helps!