[cgiapp] Streaming Status Updates for a Long-running Run Mode

Wed Oct 7 12:35:38 EDT 2009

On 10/07/2009 12:24 PM, eric.berg at barclayscapital.com wrote:

> I wouldn't do that either, but I've had no end of issues with running
> subprocesses from Apache2.

Me: Doctor it hurts when I try to run subprocesses from Apache2.
Doctor: Don't do that :)

What I try to do is have an external process which looks for jobs to do 
and then does them. Then my web app simply puts things in the job db 
table and the job queue picks them up and processes them, putting the 
status (complete, failed, working, etc) back in the db table. Then my 
web processes that check the status of jobs checks to see if it's done 
and then shows that in the browser.

Almost all web applications that I know have some things that should be 
delegated to an external job queue. Even something as simple as sending 
an email shouldn't be done from the web processes (could get stuck on 
DNS lookups or SMTP waiting, etc).

> I just ran across the 'at' solution, which I like quite a bit, however
> it turns out that we do not receive the email for the users that the web
> server runs under and 'at' sends error and other confirmation
> information via email, so I have no idea at this point why my
> subprocesses aren't running.

If you want to go the "at" route instead of writing your own queue (or 
using an existing queue) then have "at" call a Perl script that wraps 
around the real process doing the work. Then it could set a failed 
status and error messages in the DB or something that you could have 
access to. Obviously if this wrapper fails it would have the same 
problems, but that should be pretty infrequent since your wrapper could 
be really simple.

> Understood that you can't do this with Ajax, but how would you do it
> with cgiapp in a non-ajax context?

Something like this rough code should work

$self->header_type('none');
print "Content-type: text/html\n\n";
$print $main_content; # but don't close the <html> tag yet

while($not_done) {
    # do some work
    ...
    # periodically print so the browser doesn't close the socket
    print "\0";

    # or print some progress that the already printed document knows
    # how to deal with
    print "<script type="text/javascript">update_progress(10)</script>";
}

print "</html>";

return;

-- 
Michael Peters
Plus Three, LP