There's a neat trick to get around the lookbehind-thinks-backreferences-are-variable-length problem: embed a lookahead in your lookbehind. It doesn't mind if you use backreferences inside the lookahead.

For the problem at hand, it would go like this

    /([QGYN]{2}   # First two characters of the desired class
      (?:          # Followed by the complex expression...
        # Look for a trio starting two characters back
        (?<=(?!(.)\2\2)..)
        [QGYN]     # Then take another of the desired class
      ){1,4}       # ...1 to 4 times
      )/gx
[download]

This was a trick someone posted some time ago, but it was over my head at the time. Now I get it.