Rather, think about the problems and address them. It is never safe to assume that the client in a web interaction is feeding you safe data. You must validate it on the server, even if you have client side validation for the benefit of manual users. If certain people are overloading your site, protect it from them in some way. Perhaps simply by dumping anyone who feeds invalid data.

Every system you use to try to force people to use a browser manually can, and will, be spoofed, since the protocols are fixed and the browsers are well known. You'll have to protect yourself in some other way anyway. This is not an easy problem as you can see from all the captchas and other schemes people try to use to limit spam bots. If the users in question are genuine I would try to accommodate them, not ban them.

Phil

So you need to 1) validate all input data on the server side and 2) forbid or restrict the script in your robots.txt.

I dont have much of knowledge ...can you tell me something about robots.txt

I could, but thounds others have done the same before: robots.txt

What I am surprised is that no one mentioned defining a Terms Of Service agreement with folks using the site (though robots.txt is a good start). Explain in exactly what ways it is acceptable to interface (API, how frequent are requests, etc). This tells people what you expect and what you will do if they don't abide by those rules.

Then all you can do is protect yourself, identifying abusers, and take other steps to block them.

HTTP Referer checking + hidden field can get rid of 90% nasty user

Some reference:

The left 10% you may consider double cookie (used by blogger), compulsory cookie or session

compulsory cookie (or force cookie)

session

You may be interested in the modulus of Apache:: and *Session in CPAN

The first thing a cracker tries to do is find out what your application responds to by reading your HTML. Then, they try different variations of the parameters. Sound familiar?

Dealing with #2 is harder. Doing validation early can help, since you won't be doing expensive processing on invalid data. But if someone is really out to get you they can flood you with valid traffic too. I wrote CGI::Application::Plugin::RateLimit to deal with this problem for a CGI::App, but it's only as useful as your ability to distinguish one client from another. This problem can also be dealt with at the network level by your firewall or by something like Apache's mod_throttle.

-sam

1) Ask them not to. Your ToS is a means of asking the user not to do this and robots.txt asks the program itself to pass over (parts of) your site. But you have no guarantee that the user will read the ToS nor that the program will read robots.txt and, even if they are read, they may be ignored.

2) Threaten to sue anyone who doesn't use the site in the way that you prefer. While this can be very effective at preventing automated use of your site (if you have the money to spend on lawyers), it is generally more effective at preventing manual use, as many of us prefer not to deal with litigation-happy sites.

Automated use is not the actual problem you're facing anyhow, so you would do better to just accept that and deal with the real problems, namely invalid data and high server load.

For the first, you must, must, must validate the data received from the client. Rule one of designing a networked application is to assume that the other end of the connection may be lying to you and clean up, sanity-check, and otherwise validate all received data. This isn't even just for websites - there's a long history of networked games, from Doom and Quake to the latest MMOs, which have had massive problems with cheating because they foolishly trusted the user's software, probably in the belief that they had a "secret" protocol which only one other program (the game client) knew how to talk. HTTP is very simple and very well-known. Writing a dishonest HTTP client is trivial.

There are several options for dealing with load issues, ranging from limiting the number of server processes allowed to spawn at any one time to caching results and returning them as static pages instead of reprocessing data on every request to blocking IP addresses that issue too many requests too quickly and many other things in between. Or you could just ask the users who are writing robots for your site to please configure the robots to issue no more than, say, one request every 5 seconds. Which option is best for you is highly situation-dependent.