Q1: In "The truly paranoid programmer would lock the file", which file are the authors referring to?

$old.

Q2. ...

I would do something like:

use Fcntl qw(:DEFAULT :flock :seek);
my $old = "...";
my $new = "...";  # Assume it's safe to clobber this file.
my ($o_fh, $n_fh);
#
# The sysopen doesn't use the O_CREAT flag. If you the 
# task is to *change* $old, it should be an error if the 
# file doesn't exist.
#
sysopen($o_fh, $old, O_RDWR) or die "...";
flock($o_fh, LOCK_EX) or die "...";
# Now we have an exclusive lock.
#
# Open $new in read-write mode. Create if necessary.
#
sysopen($n_fh, $new, O_RDWR | O_CREAT) or die "...";
#
# Get rid of any existing data in the file.
#
truncate $n_fh, 0 or die "...";
while (<$o_fh>) {
    ...
    print $n_fh $_;
}
#
# Go back to beginning of files.
#
seek $n_fh, 0, SEEK_SET or die "...";
seek $o_fh, 0, SEEK_SET or die "...";
while (<$n_fh>) {
    print $o_fh $_;
}
#
# Truncate any remaining garbage.
#
truncate $o_fh, tell $o_fh or die "...";
close $o_fh or die "...";
[download]

Note that you only have to go this way if $old is a large file. Otherwise, you could just suck in the content of $old, modify it, and write it back, without going over the trouble of using $new. You still need to flock $old and do the truncate though.

Q3: Is the problem the fact that $new might exist already because another instance of this script running at the same time had created $new a split-second ago in connection with its own update of $old, and that our process will destroy the contents of that $new due to the way ">" works, thereby causing the other instance (e.g., another web user submitting data via the same page's form) to produce mangled or empty data when that instance renames $new to $old? Yikes, there goes the database.

If you get an exclusive lock on $old before opening $new, and don't let the lock go (note that closing the file lets the lock go), there's no problem.

Q4: In a multi-user environment, does a careful programmer need to use "sysopen/flock LOCK_EX/truncate" every time a script needs to write a file? And now a final wrinkle on the addition of a file lock for $new in the recipe.

No. Only if it's possible more than one process (or thread) that tries to modify the file.

Q5...

The question uses a wrong premisis. It's not $new that needs to be locked - it's $old that needs to be locked.