A particularly good tool for this is Ragel http://www.colm.net/open-source/ragel/

It compiles character-fed state machines and is a more modern and flexible design than lex or flex.