You don't need to install the server (which, in most cases, is the same thing as the database). But you do need to install the Oracle client and the C libraries in order to build a working DBD::Oracle.
Please tell us which version of Oracle you are trying to use and what version of Linux. I have successfully built DBD::Oracle on RedHat Enterprise 3.0 AS (very similar to RHL 9.0) using the Oracle 9.2 client.
Update: This is weird. I meant to respond to the OP's other note below. My bad...