You are making the regular expression engine do a lot of extra work by:
1) Using capturing parenthesis instead of noncapturing ones .
2) Even a non-greedy dot-star is bad (see Death to Dot Star!). Use a negated character class like [^\]]* instead.
3) Excessive going in and out from parenthesis levels, as in the [^\ expression following the "|", which ought to have a "+" after it.
These inefficiencies are probably causing the regular expression engine to go wild and run out of memory. By the way, why not use module Text:Balanced instead?
Update: Here is a brushed-up version with my suggestions added:
if ($text =~ /^
(?: # Any amount of:
\[ # An open brace
[^\]]* # Any amount of non-brace stuff
\] # A close brace
| # Or
[^\+ # Anything that's not an open brac
\] # Followed by a close brace
Update2: After recommending Text::Balanced for this problem, I decided to try it for myself.
I would have expected the following code to work, but it doesn't (I get the error: and it does work as long as you test for the following message as an acceptable case: "Did not find opening bracket after prefix: "[^\*", detected at offset 1216". The message just means you have passed all the brackets, which is fine.
while ( $next = (extract_bracketed($text,'','[^\*')) )
print "found matching brackets: *$next*\n";
print "found bracket error: $@\n" if $@;