By piecing together the information in the above replies, you can find your answer. But if you are impatient, here is a more direct explanation.

The Windows command shell uses the double-quote character as its quoting character. Anything that you don't want the shell to try and interpret needs to go between quotes, such as the Perl code being passed to -e. The problem you are facing comes in when you want to use a double-quote character in that Perl code. In order to do that without the shell trying to interpret it, you need to escape the character with a backslash, like you did in your second example. The reason that example fails is not because you got the shell quoting wrong, but because the Perl code is nonsensical as written. Perl is interpreting it like this:

print "
[download]

gellyfish points out in his answer that you'll need to quote the character for Perl before it will work:

print '"'
[download]

and translated for the shell:

perl -e"print '\"'"
[download]

In response to your plea to have it "just work," there's no way around it. If you wan't to use a shell's quoting character without the shell interpreting it, it needs to be escaped. This is the same for Cygwin, it just uses a different character. But if, under Cygwin, you wanted to use the single-quote character in the Perl code, you would need to escape it as well.

One last comment on printing quote characters in one-liners. Perl includes a scarcely-mentioned ability to interpret a short sequence of characters directly as a single ASCII character: v-strings. A bareword matching /v\d+/ will be translated to the ASCII character represented by the digits following the v. This is useful particularly for Windows one-liners because you can get a newline without having to double-quote anything (which, as we have seen, is painful). For example, this prints hello world with a newline:

perl -e"print 'hello world', v10"
[download]

And this prints a double-quote character with a newline:

perl -e"print v34, v10"
[download]

Note, however, that v-strings' usefulness is limited by the fact that they only work as barewords, not inside other strings. This would print v34hello worldv34 instead of "hello world":

perl -e"print 'v34hello worldv34'"
[download]

Even with that limitation, I have used them often to put newlines at the end of a print in a one-liner, although this works as well and is somewhat more readable:

perl -e"print qq[Some text ending with a newline.\n]"
[download]