Yes, I'm on Windows. I think you really need to stick with single quotes on Ubuntu, otherwise things get a bit weird with the setting and interpolation of $_.
The Windows shell doesn't interpolate on the "$" sign - as "$" isn't meaningful on Windows like it is on Linux.

On my Ubuntu-14.04 with perl-5.18.0, the first time I run the command quoted above, it's fine.
But after that I get errors too.

sisyphus@sisyphus5-desktop:~$ echo $_
PATH
$ perl -le "$_=~/sub mymy\{}/;" # No error here
$ echo $_
PATH=~/sub mymy\{}/;
$ perl -le "$_=~/sub mymy\{}/;"
Backslash found where operator expected at -e line 1, near "mymy\"
syntax error at -e line 1, near "mymy\"
Execution of -e aborted due to compilation errors.
[download]

The actual error messages can vary a bit, depending upon what gets run when - but I've seen the same error as you reported.

Cheers,
Rob