It's hard for me to think of any kind of programming as anything other than a kind of applied mathematics. And my gut feeling is, well, maybe you don't like algebra classes. Perhaps the teacher is old and the material dry. But if you become any good at programming, and you avoid algebra, you'll end up having to teach yourself many of the same concepts.

I can't speak for others, but I learn about twice as fast with help than without it. And given informed help, I learn the subject a bit more thoroughly. So, if it were my decision, yes, I'd suggest you take algebra.