1) Using capturing parenthesis instead of noncapturing ones .
2) Even a non-greedy dot-star is bad (see Death to Dot Star!). Use a negated character class like [^\]]* instead.
3) Excessive going in and out from parenthesis levels, as in the [^\[] expression following the "|", which ought to have a "+" after it.

These inefficiencies are probably causing the regular expression engine to go wild and run out of memory. By the way, why not use module Text:Balanced instead?

Update: Here is a brushed-up version with my suggestions added:

if ($text =~  /^
                (?:                  # Any amount of:
                  (?:
                    \[              # An open brace
                     [^\]]*         # Any amount of non-brace stuff
                    \]              # A close brace
                  )
                  |                 # Or
                  [^\[]+            # Anything that's not an open brac
+e.
                 )*
                 \]                 # Followed by a close brace
              /xs
[download]

Update2: After recommending Text::Balanced for this problem, I decided to try it for myself. I would have expected the following code to work, but it doesn't (I get the error: and it does work as long as you test for the following message as an acceptable case: "Did not find opening bracket after prefix: "[^\[]*", detected at offset 1216". The message just means you have passed all the brackets, which is fine.

my $next;
while ( $next = (extract_bracketed($text,'[]','[^\[]*'))[0] )
{
   print "found matching brackets: *$next*\n";
}
print "found bracket error: $@\n" if $@;
[download]