I don't think you will find that a big problem. None of the main ideas of the book require any mathematical knowledge.
Some of the examples are mathematical in nature. For instance, there is about twelve pages in chapter 6 that concerns using an infinite list object to represent a sequence of increasingly accurate approximations to the solution of a certain financial problem. People who have studied calculus will immediately understand the ideas here; people who haven't might understand them anyway, and even if you don't you can always skip those examples; most of them have nothing in particular to do with math.
My own tendency is to write a lot of math stuff, because I find it very interesting, but while I was writing HOP I tried really hard to get rid of the mathematics, because I knew that a lot of people don't like it.