Basically, identify what you're going to be doing with the data, and then figure out how you're going to ensure that this untrusted data is safe.

And no matter how you approach it, don't think of your algorithm as being built to remove bad things. Build it to permit safe things. If this means doing a tr/a-zA-Z0-9_-//cd, then that's what you have to do.

I think the last paragraph should be highlighted. Do not remove bad things. Permit safe things.

A few weeks ago in a reply to someone in beginners@perl.org on similar topic I wrote:

1. There is NO single list of dangerous characters. What characters are dangerous depends on the action you do with the data.
2. If you or someone else creates a list of suspicious characters and test whether the data contain any of them, you are NOT safe. It's for sure you'll forget some character, it's for sure there is something you've never heard of that can go wrong.
3. Always test whether the data DO CONTAIN ONLY ALLOWED characters. And allow only the characters you must.

  Jenda

takes a string and removes all 'unsafe' (meta) characters

And contrary to what I picked up from skimming that long article, the best way to keep the shell from interpreting unsafe characters is to not even use a shell at all! Most child process invocations can use a shell-less invocation (multiple arguments to system or exec), and then there's never a problem with the potential characters in the first place!

So, while I understand what you are trying to do, I don't understand why you are even trying to do it. You're starting at the wrong end of the picture.

-- Randal L. Schwartz, Perl hacker

I think what BrowserUK might be asking for is a library that can be used to handle the "usual" types of data. I'm interested in such a thing, as time and time again, people are untainting the same kinds of data.

Admittedly, such a module would never be strictly perfect, but it would be arguably better than the current condition. Positive progress is better than nothing, isn't it?

What about validators or untainters for such common bits of data as:
• Dates
• Times
• Names
• Integers
• Phone Numbers
• Postal Codes
• URLs
• E-mail addresses
• Free-form comments
These are, of course, vague concepts at best, but as an example, as wild and wooly as telephone numbers get, they usually don't have ampersands, double-quotes, or backslashes in them. Likewise, e-mail addresses can contain quite a lot of hair, but there are certain characters that are just not valid.

The idea is not that things are not necessarily valid names, phone numbers or e-mail addresses, but that they are at least, not dangerous or radioactive. In other words, a name of "Smith'; DROP TABLE foo;" would not be valid.

Just an idea.

URLs and email addresses are never "unsafe" when handled safely. So if you're looking for Email::Valid, it's done.

And why you would be passing a date, time, or name near a shell. I'm still confused. That's still thinking from the wrong end.

As for your DROP TABLE example, if you are using placeholders correctly, that value wouldn't matter.

So, I'm still not convinced that there needs to be a standard "untainting" library. When the data is handled properly, we don't need to "match" "safe" data. Period.

-- Randal L. Schwartz, Perl hacker

My interpretation of the referenced article was that the biggest problems were caused by several factors:

• Not all programmers are, nor could be, 'experts' in understanding all the cracks in the individual or collective amours' of Perl's calls to system functions, or those of all the systems that Perl run's on.
• Even experts make mistakes.
• Re-using (and reusability) are good. CPAN is one of Perl's great strengths. The problem is, when the programmer passes values to modules, he does not know exactly how those values will be used. - Sure, he can read the source, but that negates half the benefit of reuse.
• Doing sufficient screening (or just enough allowing) for the required use of input is fine, but what happens in 6 months or a year when the functionality needs to be extended? Will the maintenance programmer know or understand what sanitising was done and why? The article had an interesting section that showed that how consecutive filtering was order dependant. Programmer 1 gets input, does appropriate filtering for the planned use, and uses it. Programmer 2 comes along a month or 6 later with a requirement for new functionality. Goes in, grabs the value already untainted - and uses it.

  I don't think "only employ competent programmers" cuts it here!

As an example, my current project (its only a learning exercise at this point, so please don't pollute the thread by critiquing the project design), uses .xml files to describe 'things' and these are used to build the HTML for displaying. The user selects the 'thing' of interest by clicking on a menu. The identifier of the 'thing' is passed as a URL search parameter and then that identifier is used to build the filename of the .xml file that is opened. The path/filename constructed is then passed to XML::Simple to read and process. The menu's that the user clicks on are themselves generated by processing input from readdir().

The aim is that new things and groups of things can be added to the site by simply dropping a new .xml file in the appropriate directory and creating new directories respectively. Updates to existing things would be done by editing the .xml (and then validating before putting (back) into the production environment). The idea being that you don't need to code new HTML to add/delete/update 'thing' pages - you edit the .xml, validate it against a custom DTD and move/copy it to the appropriate place and all the layout of the HTML is taken care of by an intelligent Perl script. (Once I get up to speed enough to write such a beast.:)

This simple, small sample of the project implementation raises (at least) the following questions:

The following are only example questions I am not seeking answers to them here!

• What does XML::Simple do with the parameter I pass?
• How restrictive should I be on allowable values for filename chars should I be? What happens if later on this is seen as "too restrictive" and the rules get modified?
• If the user embeds an url-escaped null after the product identifier and I haven't checked for embedded null chars (I hadn't!) and I pass this alone to XML::Simple, will it be used in a way that could be vulnerable?

Again, I've got to emphasize. There is no such thing as "unsafe data". Merely "data used unsafely". So a hypothetical sanitize routine could at best be written as:

sub sanitize {
  die "If you had to call me, you've lost already";
}
[download]

You must fix the behavior of your code, not wrestle your data to the floor.

-- Randal L. Schwartz, Perl hacker

I'm completely stunned that you'd suggest not using regexes for parsing

Did you read the article? if not, please take the time to scan it (search for "s/" to get to the relavent sections) and perhaps you'll see why this method (ord() and unpack())seemed appealing. The way the interpolation that regexs do can be exploited to bypass even the most sophisticated set of multiple passes with regex to sanitise a user supplied path is simply scary.

Peer review and a huge number of test cases, especially those culled from real-world experience

Exactly why I was suggesting development of such code here! There are very few huge corporations and many millions of small companies in the world. The big ones have the budgets for such in-depth, in-house development, review and expertise. The rest have often one or two developers who are responsible for developing and maintaining the code. No possibility of enlisting more than there own expertise in reviewing their own work. And whilst when the big ones make mistakes, they have the have the funds to correct them. When the small ones make mistakes, the finacial costs of correction are often too much for their small net worths to bear and they go under taking the jobs they provided with them. Permenantly.

Expertise takes either time or money. Those that have invested the time, charge substantially to hire that expertise to others. The big guys have the money to grow that expertise internally or buy it externally. They are still making mistakes. The small guy has neither choice.

I don't understand why the idea of utilising the collective resourses of PM to address and simplify the process of handling security--the one thing that (as I have seen all over PM) is at the top of almost every single IT experts', of any flavour, list of major priorities--is so shocking?

BrowserUK (mistakenly posted anonymously)

Added attributio - dvergin 2002-06-28

The small guy has neither choice.

And there's where you're wrong again.

It's the cost of doing business plain and simple. If you can't include security as part of your budget and still make a profit, you've got a bad business plan.

It's just as wrong to cheat on security costs as it is to cheat on paying Uncle Sam his share. If you wouldn't even dream of the latter, why are you even debating the former?

-- Randal L. Schwartz, Perl hacker

Please accept my apologies. I have just realised than when I made the post to which this is a reply, and this earlier post, I was was not logged in. So they have come up as Anonymous Monk not as me. I don't know how (or if) this can be corrected?

If not, I offer this post for any that feel the need for --.

For example if you accept a $q->param and send it back to the browser as a "Hello world" example, save for a 20Mb input, why would you need to untaint it?

If on the other hand, if you are calling the shell, you need to check for naughty characters.

So untainting your data depends completly on the context in which your application is developed.

I think you *may* be getting confused between untainting and data validation. There is a difference, the former making sure there are no naughty characters in your input, the later checking the data conforms to a standard.

Anyway's FWIW, i think a centralised method of data validation would be cool, however i dont think untainting can really be centralised outside of the context of the application for which its developed.

Respectfully, I thought that your "Hello world" example, or more classically, the "Hello Ryzard, welcome back" using your name supplied from a form was benign until I read this - but when you see that by embedding HTML and script tags into the name field can, when returned to the browser for display, open up a wealth of possibilities of cross-site scripting and cookie theft, it made me think again.

Beleive me, I am not mixing data validation and untainting up. Data validation is very much an application specific function. An telephone number or zip code validation routine written for US numbers/ZIP's would have no application here in the UK.

However, sanitising almost any external input has universal application. the same hacks and cracks that would affect your server will (in most cases) affect my server too.

As I wrote elsewhere, there are very few uses of external data that are cause for concern - opens, commands, database entry, re-display, passing to other modules - very few more. The hacks that are possible in each of these cases are limited and the fixes/preventions should be pretty much the same wherever the program is destined to run. Its also much harder, and requires much greater experience to prevent the "Reverse Directory Transversal" vuln than it is to validate a date or a ZIP or telephone number.

The latter is a fairly standard programming problem.

The former, as bugtraq prooves, is a much harder and requires much greater real world expertise.

Hence my beleif that it is a ripe candidate for standardisation.

However, it seems that I am in a minority and/or 'nih' syndrome is at play here :(