Here is what you are missing.

In the example that I provided, there is a general case matching algorithm and a special case matching algorithm. To get correct behaviour for all kinds of regular expressions, you sometimes need to use the general algorithm. To get the specified speed for some common regular expressions, you sometimes else need to use the special case matching algorithm.

The public specification does not say where the boundary between the special case and the general case is. It is a basic fact of life that there is a good chance of finding bugs around that boundary.

What I am saying is that you'll find corner cases along that boundary which are nowhere visible or obvious in the specification, and it is a very good idea to have white-box tests for those corner cases.

In pseudo-code, your implementation will contains something that looks like this:

  if (some conditions here) {
    use special case with performance guarantees
  }
  else {
    use general case that may be slow
  }
[download]

You need white-box testing to be sure that you're testing that all regex features which can show up under both implementations are tested under them. If you just use black-box testing, then your tests that the construct x{3,5} works properly may only test one implementation and your tests won't notice if it fails in the other. This is exactly what happened in Perl 5.6.0.

I really hope that this explanation is enough for you to understand the issue. If it is not then I really don't know how I can say this more clearly.