David's Blague

Continuing on the riff Brian started, I was reflecting on some stuff I built a while back.  In 2001 and 2002, I was one of the people who worked on Democrats.org.  There were a whole mess of people involved in that project, from a whole mess of different organizations, and it was fun.  I even married one of the people on the project.

One of the parts I was responsible for was the contribution processing code.  I kept working on this through the end of 2004, and during the last part, it was pretty high volume.  The system was written in Perl (HTML::Mason and XML::Comma), ran under mod_perl, and it processed credit cards in real-time.

We used a cool trick to keep browsers from hanging up on long-running processes (sometimes the bank would take up to a minute to get back to us about the credit card).  We immediately checked for easily caught errors (for instance, credit cards have a check digit that can help catch mistyping), then gave the browser an immediate response response.  That response was an HTML page written into a file on the server, which basically told the browser, "Yeah, still working on it, check back here five seconds."  In the background, we ran a forked process to talk to the bank that didn't talk to the browser directly.  When the bank got back to us, the forked process would write our response into that same file, and the next time the browser checked back, they would see a response: "Thanks for Contributing," or whatever.

We spent a lot of time refining the system over the next few years, and by the time the general election came around, we had processed some absurd nine figure sum of money with those couple hundred lines of Perl.

That basic strategy- do something easy, respond immediately, run long processes asynchronously, seems to be generating quite a lot of interest these days.  Node.js does it for I/O, Google Go has it built in to the language, NGINX does it for the Web, AnyEvent does it for Perl, Twisted in Python, EventMachine in Ruby, and the list goes on.  There are differences in the way each runs this process, but that core description seems to hold, and it shares a lot in common with the way we did it nearly a decade ago.  There's even a spec for this kind of thing in the works.

Now, as the inimitable Ryan Dahl points out, there are plenty of reasons not to do development this way.  Cultural reasons and technical reasons.  He rejects the "it's too hard" argument, and gives some pretty convincing demos in counterpoint, and maybe he's right.  Or maybe "cultural reasons" and "it's too hard" are the same thing, and all we really need is someone to come along and hand it to us on a silver platter.  Probably an opportunity there.