Users suck at testing and evaluating software.

It's been my experience that the user (and by "user" I mean "the target audience who will ultimately be using the widget in production") make really lousy testers. For the most part they don't know how to stress a program (making choices that the programmer didn't anticipate), or test outcomes (did the the program do what it was supposed to do?), or evaluate the program for completeness (is there something it should be doing and doesn't?) Doing these things takes a LOT of time, getting into a program, understanding what it does, working through scenarios, thinking through possibilities, etc. etc. This is a lot of work, and it takes a special kind of person with special kinds of skills from someone who is very meticulous and linear. In my experience very, very few people in the "user" category are like that, so although I do what you do and toss out new widgets for suggestions/evaluation/bugchecks I rarely get anything more useful than "Huh... looks like it works" which of course leaves the real "evaluation" to when the users are using it in production awhile, try to do X and find the program doesn't do X, or get irritated because function Y doesn't work, or works in a less than optimal way.

These days I still put out my programs for evaluation, but I realize that it's mostly just a formality, and that almost never will anyone spend the amount of time I need that would really be useful. The whole situation makes you very appreciative of the professional QA folks whose job it is to do what you want... it IS a real job, that most people simply don't have the skill or experience to do well.

So, well, good luck to you, but if I were you I wouldn't expect much. The good news is that with a development team of one it'll be relatively easy to make changes after the fact as time goes on after the widget is in production.

Gary Blackburn
Trained Killer

++ajt (for a good question) and ++Trimbach!!

You're right, Trimbach. Users make lousy testers. This is generally because their perspective tends more toward the concrete results they seek, rather than the (what almost certainly appears to them to be) picayune level of detail necessary to adequately test an application (e.g., "Why would anyone want to do THAT?!?").

Of course, the other side of that coin is, "Why doesn't XYZ work," where XYZ is some completely unanticipated sequence that no one, during development, bothered to mention that they sometimes do ("Oh, that'll NEVER happen ...(time passes)... Oh, except when ...").

Nothing replaces solid regression testing, unfortunately, and without another pair of eyes (presumably connected to some form of brain, one would hope) it's difficult to code for users' making choices one didn't anticipate them making.

Good luck to you.

dmm

You can give a man a fish and feed him for a day ...
Or, you can teach him to fish and feed him for a lifetime

I would suggest sitting with five of your users (in person and one at a time and not all together in a group which is bad) as they use it and take notes on what they don't know how to do. See what they find intuitive and find obfuscated. See www.useit.com for more info on user interface issues.

2. What features are missing from it?

Same as above though you really want to do this at the outset when you write up requirements for the project.

3. How to make it faster and other load issues?

That depends...is it slow? If it is then find your bottleneck. Is it the server hardware? The server OS? The client's hardware? The client's OS? The client's browser? The network? The algorithm used to implement function x in your CGI script? Consider timing the various parts of your code to find the bottlenecks and focus on those parts.

As you can see on this site there are many monks who point out faster and simpler alternatives to various perl hacks.

metadoktor

"The doktor is in."

Sounds like you don't want to go with the traditional methods of testing (of which volumes and volumes have been published). Here's my .02 non-traditional approach:

for users finding something wrong: have good logging - don't depend on the users for 100%. Write decent well formatted and consistent error messages so log parsing will be easier.

for missing features: have a good user feedback mechanism - have a "report problem/suggest feature" link on each page. Also make sure the copy "leads" the users to use the link (don't bury it - at least not in the begginning).

How to make it faster. Combine the logging with good timing stats. You should have an inkling of where your bottlenecks will be (database calls, transforming xml, etc). Start logging those hotspots to generate some stats. Maybe they are bottlenecks, maybe not - you'll need some historical data to let you know what's normal and what's not.

I think if you have consistent well thought out logging (that includes having the mail go to some alias with a canned subject) you should have the foundation necessary to programmaticaly monitor the health of your app.

-derby

... it has the functionality I want, and no defects as far as I can see.

Hm, how can you say that without having at least some testing process in place (and be it yourself clicking through the application)?

From your request, you are looking not for testing, but for an evaluation of the application from a user perspective. That is something very different.

For a test, you will have a specification (what must the program do to pass), and a process that (probably automatically) checks the conformance of the program to the specification. The second part ("automatically") tends to be somewhat hard to achieve in web environments, since you have so many components around that you need to run the application. Still, quite a bit can be done here by using the various LWP modules.

For an end-user-evaluation of the application, I would suggest to place a feedback button right there on every page of the application, and make sure that feedback from the users is picked up by somebody and acted promptly upon. If you do this (I mean, act promptly on user feedback), everything else is secondary, and you will soon have a better application (provided you get enough input at all). If you are not prepared to react promptly to user feedback, nothing else will help. A second nice feature is a "See FAQ" button right next to the feedback button, which leads to a document where you put the answers or workarounds for problems that you have already fixed (without that, you will get many duplicated reports).

If you need a *real* user review of your program (e.g., to establish the program fulfills its purpose as part of a contract or so), you better had established a questionnaire and test-run it already. I sincerely hope for you that you are not in this position (it does not sound like it, though).

Christian Lemburg
Brainbench MVP for Perl
http://www.brainbench.com

You have two different type of testing needs listed in your request for help. The first type of testing method is usability testing. The second type is sytematic.

Usability testing is performed by users. In the classes that I took, we constructed web pages out of construction paper and asked users to physically push buttons, fill in text, etc.. after they were given a specific task to perform. This allowed us a gauge on what the user perceived the functionality of the web site was. It also allowed us to ask them questions afterwards. If you get to ask people to use the application, ask them to perform a specific task. (Log on to the application and change your password) While they perform this task sit in the background and watch what they do. If they complete it successfully, ask them to rate it on a scale with the parameter that you are looking for. If they don't complete the task, find out exactly what stopped them from completing it.

Systematic testing is testing the system. Since you know how to use Perl, construct an app that makes demands on the system. Fetch a ton of database material while generating a bazillion web pages.......

I hope that this has helped, since that is my intention.

Good Luck,
fmogavero

I must say that I sleep much better at night since I started doing automated testing on web apps. You still miss stuff, but usually it's pretty obscure. That's one thing about XP programming that I totally agree on.

On the usability front, I try to make it as easy as possible for users to comment on the application as they use it... some sort of feedback link to a form that records as much information as I can get (browser, url, action, time, etc). This gives me a nice database table todo list without having to sift through and reply to dozens of emails.

()-()                            
 \"/                                   
  `