I'm the author of Finance::Banking::Postbank_de, and in my module I test via two kinds:

• Demo account - the smart folks at postbank.de have a demo account with a well-known account number and PIN, and I run live tests against that demo account
• Saved HTML files - WWW::Mechanize, which I use, can easily load HTML files via file:// URLs or ->set_content(), and I use that to load some canned content into the useragent for running offline tests. This does not protect me against site changes but it makes testing some parsing far easier

I consider interactive test suites as very bad form, especially as I'm running CPAN smoke tests from time to time...

Thanks Corion.

Sadly there's active demo (just a load of flash), so that options out :-(

I like the thought of at least being able to check the parsing is correct using file though, I'll definitely include that.

No worries about the interactive tests - I had a feeling there was bad karma associated and now it has been confirmed I will steer well away :-)

Corion,
I consider it bad form for any test suite to automatically assume it is ok to contact the internet let alone phone home. I also see the interactive is bad argument. The comprimise in my opinion is to prompt the user for input with a sane timeout automatically defaulting to "no".

With that said, I was thinking that it might be good to have continuously updating saved HTML files. I really haven't thought this through but here is the general idea. You have code that periodically logs in and pulls down sample files that are scrubbed (if no demo account available). These files are then place in a public facing location that the test suite points to if the user decides live tests are preferred.

Cheers - L~R

I built a set of testing accounts with known (to my test-package) passwords, and patterned user-data (for example, address1=user1-address1-xxxx, where xxxx is unique test-id (in reality the PID.time() at the start of the test sequence)). This is the only way that I could think of to

• Reliably test the interface in the face of a shifting API (there is only one of me writing the test harness; there are thirty or forty folks in Engineering fixing bugs and extending the feature set)
• Keep from leaking of my personal account and password data.

The method of generating patterned data turned out to have a side-benefit, I don't have to keep re-initializing the dummy accounts with a known data-state. I can just leave the account in the updated state since I know that my next test run will generate a different test-id by virtue of the included time-stamp. I use the fact that the test-id is in the data to validate the test results (is the time-stamp different or the same? Is this what I expect?).

A few years ago, I was working contract for one of the local banks in San Francisco (one with branches all over the state of California and elsewhere). There was a formal standard for building 'dummy' account-numbers, to we could 'test on the live system'.

We had someone run a test on the Check Ordering software and 'forget' to cancel the order to the printer (Rocky Mountain Banknote, as I recall). When the checks were delivered, Someone started using them to pay for purchases at stores in the Los Angeles air-basin. Since the account did have a positive balance (no surprise here), the checks all cleared.

The Bank didn't twig to the scam for over a month, until the quarterly account-reports came out and showed activity against a dummy account that didn't come from a test-id. They initially 'solved' the problem by sending instructions to all of their Printers to report 'suspicious' account numbers while they figured out a real solution. This generated a lot of false-positives....

My contract expired seven months later, and they were still arguing about the best way to prevent the scam from happening again.

UPDATE: added a paragraph tag after the list close; cleaned up the wording in the first sentence.

If only I could get one of those accounts :-)

Depending on your code, take a look at Mock::Object and Mock::Module. The idea is to override what every object/module that you use for getting and receiving data and force it to return something specific. This will separate the requests for data from module that manipulates the data.

1. Your own personal config file does not go into the distro
2. There are clear instructions in the installation docs for the module that make it explicitly obvious to the people who install it that they must provide their own username/password info in the config file in order to run tests.

But then, I think there's still a dilemma. Supposing I wanted to install your module in order to access my own personal bank account (assuming the unlikely possibility that I have an account at the same bank as you, so that everything should work as expected), I don't know how willing I would be to run tests to make sure the module works as intended, using my own bank account as the test bed.

Somehow, that would seem like a bad idea, quite apart from the notion that my account information might be different enough from yours that it might be hard to anticipate what sorts of values and results should be expected.

An ideal solution, similar to what DBI does, would be if the bank would support a "test user" account with stable and reliable data that would always return the same values for the test conditions (unless the tests were failing, of course). But I assume the likelihood of any bank doing that for the sake of a CPAN contributer is nil. So much for the ideal solution.

All things considered, I don't think you can arrive at the appropriate "level of abstraction" to make this sort of thing viable as a CPAN package. You'll have enough to worry about just making sure it continues to work as intended for your own personal needs (as the bank "improves" its presentation strategies), so let that be sufficient.

(You asked for opinions; that's mine.)

Good point about using your own bank account as the test bed... I hadn't really considered the users' perspective (schoolboy error). I've made the test an author test anyway so it shouldn't run for other users unless they explicitly ask for it.

The config file is a tidy idea, so I'll probably go with that for now.

As you say, absolutely no chance of getting a test account with the bank :-(

However this has led me to a problem - obviously to test the basic functionality (e.g. getting my account balance) I need to put in my secure banking details

Don't do that. Distribute your unit tests. They show patterns of use of the API you are writing.

Keep your end-to-end tests closest to your QA group (yourself likely). They are more brittle, unlike unit tests. Unit tests are small, discrete. end-to-end layers your unit tests in ways that can break when integrated, lack a perfect environment or are configured in odd ways.

Persons w/ a large POV will value them more because they are more likely to find edge case bugs.