(server-start)
[download]

to your startup file and open a file using emacsclient instead of emacs (I defined a shell function ec for that, and evan made it start emacs if it's not already running).

The only difference now is you close the files with C-x # instead of C-x k, but I wrote the following lisp function

(defun server-edit-or-close (buf)
  "Close the buffer regardless of whether it's a server file or not."
  (interactive "bKill/finish buffer: ")
  (if server-buffer-clients
      (progn
        (switch-to-buffer buf)
        (server-edit))
    (kill-buffer)))
[download]

and bound it to

(global-set-key  "\C-xk"  'server-edit-or-close)
[download]

and I can close a file using the standard way no matter how it was opened.