Well, that's 99% true :-)

If you make the assumption that the user has JavaScript (but code for the case where they don't, as well!), you could open up a new, miniature browser window (with window.open) and set its URL to something like http://www.whatever.fu/upload/upload-status?id=xxx with an HTTP Refresh: header in that script. If you refresh every 3-5 seconds, you could display a 'status bar' and/or percentage/number of bytes/transfer speed...

Just link the JavaScript to the form's submit button. (Sorry, my E-262's really rusty or I'd try to offer some sample code, but I'd probably do more harm than good in this case! :-) ) -- I forget which way it goes, but if you return (true | false ? ) from your window.open call in the onClick (?) handler, the form won't submit, but run your code instead, so make sure to RTFM :-)

With a non-parsed-header (nph) CGI script, one can send output directly to the client while the script is still running. (As opposed to a normal CGI script, where the server waits until the script closes its STDOUT before sending the output to the client).

Thus, a file upload script with a simple progress report could be implemented by alternately reading a chunk of the uploaded file and printing out a status message. The browser would display all the messages that had been sent.

This assumes, however, that the server doesn't read in the whole request itself and then buffer it for the script. That's the part I'm not sure about.

Please allow me to clarify in an attempt to prevent this conversation from devolving into inflammatory rhetoric.

I am aware of the nature of the HTTP communication process, though I did not clearly say so. O'Reilly's "CGI Programming with Perl" provides a pretty clear explanation.

I believe chipmunk has sensed the same possibilities that I have, though I'm not completely convinced that non-parsed headers are the right approach.

Though I have not tested this, I presume that scripts implemented using CGI.pm do not fire until the server has fully received the client's submission, including the file being uploaded--which means it's too late.

However, the book I mentioned goes into great detail about the HTTP and CGI processes, discussing error codes, proxies, content negotiation, and so on.

I wonder many things; I wonder if:

• One can submit a form using "HEAD" as the request method and, if so, if that would include the size of the file attempting to be uploaded.
• A creative use of HTTP 1.1's capabilities can lead to a workable solution without requiring plug-in's, browser specific code, or the introduction of new tools (and their requisite expenses/learning curves).
• Query something within Apache itself to detect incoming requests that have not been completed. If so, is it possible to get enough information that can help solve this?
• Combine error codes, redirects, multiple scripts, links, and other standard tools to achieve this effect, perhaps even a carefully controlled proxy request.
• There is an "outside of the box" solution that will unravel this Gordian knot as effectively as Alexander's approach.
• Anyone's already done this and documented it.

A friend of mine likes to say, "Programmers use absolutes like 'It's impossible' or 'It can't be done' when they mean 'I don't know how to do that and haven't the slightest idea how (or desire) to learn how to do it.' Let's look at impossibilities as challenges and see what we learn while doing so."

In responses to other comments and replies:

• cajun's links are helpful, though it seems overkill to use TCL for just this. If that's the easiest way to do it, though, then so be it.
• I said nothing about resumeable uploads.
• I realize that I'm asking an unusual question, one that will add time to the process. Given the nature of the my intended users (who do not look at the existing feedback devices) and the size of the files being uploaded (2-5MB), I am perfectly willing to spend a few more cycles obtaining those files if it keeps my users from rebooting or stopping the upload.
• baku's approach may be doable, though I would prefer to avoid client-side scripting if possible.
• epoptai post is suggestive, but I wonder if there is a missing link and one or more incomplete sentences.

I have never done anything like this, and indeed my first response would be to say "impossible" meaning "if doable, much harder than you want". However it probably is possible to do this.

Were I to tackle this project (after appropriate moaning and groaning) I would crack open the source-code to CGI, and look for the file-upload part. That seems to be in a function called read_multipart. I would then create (ick) a private version of the module with a hook in that function to, based on a flag, launch a nph response and stream upload information back. Or to stream upload information back to a predictable place (based on a random ID sent to the client and returned in an obvious place - eg PATH_INFO) so that the client can have the second window going. (That is probably the saner approach to implement. Besides which the server may not be expected the CGI script to reply while the upload is going on, so the second window may be the only easy response.)

I would expect that since file uploads can be large, there would be no way that a decent webserver would accumulate the entire upload before starting the CGI.

Anyways figuring how to modify the existing CGI module to have the hook is going to be a lot easier (and better) than rolling from scratch, even though the hook will be so ugly that there is little chance it will be accepted back into the module.

There may be other ways to do it using HTTP negotiation etc. But the hack I described can probably work.

OK, here's the deal. The way clients implement the upload is rather up to the clients. From my personal experience dealing with much this same request I can tell you that you aren't going to have any fun trying to get this to work.

The best I can recommend is that you look at the javascript option of opening a second window. You're real problem will come from the fact that you have a nasty precedence problem. When you throw up the page for the upload, you need to already know the "temp" name of the upload so that you can watch its progress. But you don't yet know it because you haven't had anything submitted. And, when you get something submitted, the window that gets the submission doesn't show it's response till after the upload is complete so once you know the temp name you can't send the user to running display window.

The solution is to pick a "temp" name first when they go to the page and have the submit button pop a window and send the file (I suppose with javascript, somehow). Alternately, you can interpose an entire exchange where when the user hits the submit button, a javascript send with the file name happens, starts the watcher window then actually submits the file.

Any way you look at it, ugly ugly ugly and browser dependent to boot. Worse, you take the chance of making it completely unusable to people without JS or with JS turned off.

Covering your six points quickly:

• HEAD: Try it but it doesn't solve your problem anyway. Most clients won't actually let a form send a HEAD request, anyway.
• HTTP/1.1 sure it has capabilities but not ones you can get at through the client. The client already knows all you need, size of file and how much it has sent. Only the TCP/IP stack knows for a while how much has been ACK'ed though. That is why most clients don't bother. Even FTP clients are a little wonky on upload.
• Apache and CGI.pm already know what you need, your problem is getting the messages sorted out in the right order. After you start the upload you can't send messages back until you finish and you can't start watching easily until you start uploading...
• Combine. Javascript and a couple of perl scripts are all you need to do it but as I said you have a nasty communications order problem.
• "Outside the box?" Yeah, submit a patch to the Mozilla project so they implement a built-into-the-client upload bar. Or have the user use FTP (Perhaps fire up a onetime FTP on a special port with Net::FTP? That might be sexy. There would be a second confirmation box pop up in there though. Maybe write an entire browser in perl so you could add fancy upload progress bars easily.
• Already done? The closest I've seen to what you want is Blogger.com's "hold on while I ftp to your site" javascript. That is just for seeing if a process on the SERVER finishes. It works, mostly, but it doesn't have your communication issue.

As to the post about TCL on the mailing list, that wasn't what you were looking for. That was a browser plug-in that gets the browser to do the work while the upload happens. It is effectively a browser upgrade so you would have to encourage your users to get and install a plug-in that changes how their client displays uploads. That is fixing the client, not a CGI fix.

--
$you = new YOU;
honk() if $you->love(perl)

Excuse my bad grammar. That node may make more sense now.

You said you have:

• users who do not look at the existing feedback devices
• users rebooting or stopping the upload because it takes too long

  While an upload dialog just like the download dialog would be ideal, if you can't manage it, you could still improve the upload success rate by adding some tips to the page with the upload form. It'll give them something to do and educate them about the matrix they find themselves in.

  1. Determine average upload sizes and make a chart showing upload times for common bandwidths.
  2. Tell them how to read the throbber and status bar.
  3. Tell them to notice the size of the file they're uploading so they know how long it takes. (even tell them to right click in the open dialog and view details to see file sizes)
  4. Be extra nice and give them a javascript timer, or tell them "<noscript>since you don't have javascript this timer doesn't work</noscript> Notice the start time to estimate how long this upload will take. Please stay calm, do not reboot or change this browser window until you are told that the upload is complete. If the upload doesn't finish in much more than the estimated time you may want to try again. In the meantime you can browse the web in another window by pressing ctrl+n or just hang out here and listen to a story. Once upon a time..."

  epoptai

I was not, nor am I now, in any way trying to be offensive. If I was, you would've known it for sure.

Sometimes things seem a little different in text, than they do if I were to say them(you can't exactly hear my tone of voice now can you?).

Please note that none of the words in the sentance "You're not listening." were CAPITALIZED, in bold or italic type, or a different color.

As for me misunderstanding your question in regards to 'resuming', I don't think I did.

You said you would like to allow users to upload files in 'in increments, so you can print messages that the user sees?', which seems to indicate to me that you want your users to upload a part of a file, see some kind of message, and then upload another part of a file.

 
"Disclaimer: Don't blame, it came from inside the void"
#--------------------------------------------- crazyinsomniac