The parser basically works in a two phase process, first it scans using a very fast non-backtracking regex, to see if it can find any opening tags it recognises. If it succeeds it marks the tag with a control character, then runs a looped slower regex which scans for a complete open and close tag set. When it has something it knows is valid, and does not contain a nested tag set (as determined by negating the control character), it then executes the relavent code and substitutes the return value into the document. The loop continues until there are no more matching sets to process.
The first phase negates the backslash to prevent it from picking up on and marking out the close tags so the tag you mentioned will probably be ignored by both phases and remain untouched in the final output. I'd have to check back on the actual regex used to be certain but I'm pretty sure that is correct.
| [reply] |
Have you considered writing a proper state machine-based lexer/parser? It would positively fly, compared to using regular expressions, and probably end up more amenable to extension (if you wanted to actually support XML, for instance).
| [reply] |
I have made many attempts at building a character by character state-machine for it, all of which failed miserably and ended up breaking the functionality in some way or other.
I do believe it is possible to construct such a thing, and I do believe that if I had time and inclination to do so that eventually I would solve the problem, but since I have now adopted Plack as the basis for the system to sit upon I'm already getting close to a thousand hits a second out of it using a Quad Core Phenom II as it stands. (16,000 hits in just over 18 seconds, last time I ran my stress testing script)
Given that 24 and 32 processor server systems are readily available and 100+ processor core chips are already in production by companies such as tilera, I don't feel at this stage that there is much point to further optimisation as all the "low hanging fruit" has already been had by optimising the regexes and removing backtracking etc where possible.
| [reply] |
