Just set an Expires header and you won't need to deal with the 304 case at all. It will be faster too.