[cgiapp] Streaming Status Updates for a Long-running Run Mode

Wed Oct 7 13:39:18 EDT 2009

> -----Original Message-----
> From: cgiapp-bounces at lists.openlib.org 
> [mailto:cgiapp-bounces at lists.openlib.org] On Behalf Of Michael Peters
> Sent: Wednesday, October 07, 2009 12:36 PM
> To: CGI Application
> Subject: Re: [cgiapp] Streaming Status Updates for a 
> Long-running Run Mode
> 
> On 10/07/2009 12:24 PM, eric.berg at barclayscapital.com wrote:
> 
> > I wouldn't do that either, but I've had no end of issues 
> with running
> > subprocesses from Apache2.
> 
> Me: Doctor it hurts when I try to run subprocesses from Apache2.
> Doctor: Don't do that :)

We've got a ton of legacy code that executes subprocesses for many
things.  I've been migrating them to use the process runner wrapper so
that I can just swap out a back-end for running, but...well...it's
legacy and is in constant conflict with all of the new stuff I have to
do.

> What I try to do is have an external process which looks for 
> jobs to do 
> and then does them. Then my web app simply puts things in the job db 
> table and the job queue picks them up and processes them, putting the 
> status (complete, failed, working, etc) back in the db table. Then my 
> web processes that check the status of jobs checks to see if 
> it's done 
> and then shows that in the browser.
> 
> Almost all web applications that I know have some things that 
> should be 
> delegated to an external job queue. Even something as simple 
> as sending 
> an email shouldn't be done from the web processes (could get stuck on 
> DNS lookups or SMTP waiting, etc).

What do you use for your external job queue?

> 
> > I just ran across the 'at' solution, which I like quite a 
> bit, however
> > it turns out that we do not receive the email for the users 
> that the web
> > server runs under and 'at' sends error and other confirmation
> > information via email, so I have no idea at this point why my
> > subprocesses aren't running.
> 
> If you want to go the "at" route instead of writing your own 
> queue (or 
> using an existing queue) then have "at" call a Perl script that wraps 
> around the real process doing the work. Then it could set a failed 
> status and error messages in the DB or something that you could have 
> access to. Obviously if this wrapper fails it would have the same 
> problems, but that should be pretty infrequent since your 
> wrapper could 
> be really simple.
> 
> > Understood that you can't do this with Ajax, but how would you do it
> > with cgiapp in a non-ajax context?
> 
> Something like this rough code should work
> 
> $self->header_type('none');
> print "Content-type: text/html\n\n";
> $print $main_content; # but don't close the <html> tag yet
> 
> while($not_done) {
>     # do some work
>     ...
>     # periodically print so the browser doesn't close the socket
>     print "\0";
> 
>     # or print some progress that the already printed document knows
>     # how to deal with
>     print "<script 
> type="text/javascript">update_progress(10)</script>";
> }
> 
> print "</html>";
> 
> return;

Great!  Just what I was looking for.  The $self->header_type('none'),
printing your own headers, and using print makes perfect sense for
"getting around" the cgiapp "restrictions".

Thanks again, Michael.
Eric
> 
> -- 
> Michael Peters
> Plus Three, LP
> 
> #####  CGI::Application community mailing list  ################
> ##                                                            ##
> ##  To unsubscribe, or change your message delivery options,  ##
> ##  visit:  http://lists.openlib.org/mailman/listinfo/cgiapp    ##
> ##                                                            ##
> ##  Web archive:   http://lists.openlib.org/pipermail/cgiapp/   ##
> ##  Wiki:          http://cgiapp.erlbaum.net/                 ##
> ##                                                            ##
> ################################################################
> 
> 
_______________________________________________

This e-mail may contain information that is confidential, privileged or otherwise protected from disclosure. If you are not an intended recipient of this e-mail, do not duplicate or redistribute it by any means. Please delete it and any attachments and notify the sender that you have received it in error. Unless specifically indicated, this e-mail is not an offer to buy or sell or a solicitation to buy or sell any securities, investment products or other financial product or service, an official confirmation of any transaction, or an official statement of Barclays. Any views or opinions presented are solely those of the author and do not necessarily represent those of Barclays. This e-mail is subject to terms available at the following link: www.barcap.com/emaildisclaimer. By messaging with Barclays you consent to the foregoing.  Barclays Capital is the investment banking division of Barclays Bank PLC, a company registered in England (number 1026167) with its registered office at 1 Churchill Place, London, E14 5HP.  This email may relate to or be sent from other members of the Barclays Group.
_______________________________________________