Web Architecture

My first introduction to programming was C++ on DOS. That lasted about a day. Over time I looked into other languages, but nothing really interested me until Java and finally HTML/JavaScript. I liked how visual this kind of programming was and was particularly interested in some of the new features that JavaScript was bringing to the early browsers. One of my first attempts was a site where you could assemble beads on an interactive canvas authored for Internet Explore 4.0. I finished back in 1999, and I still like to show people how interactive Web sites could be, even prior to 2000.

Generally, I think Web architecture is made overly complex by underutilizing the processing abilities inherent in the browser. For me, this all came to a head in late 2000, while I was contracting for an investment bank in New York. They were integrating some of their legacy systems with a Web-based architecture that included WebLogic, TopLink, Sybase, and some mix of homegrown object-relational mapping systems. To complicate their technical hurdles, regulations required that the system implement a convoluted security model. After several months of work we came up with a Web site that was pretty typical at the time: click, refresh, click, refresh, click...you get the picture. As bad as I thought the user experience was, the bankers were even more upset. They had been accustomed to using legacy software that was fast and simple. They used controls like tables and trees and grids, not simplistic buttons and textboxes. In the end, the site was slow, difficult to maintain, and a pain for users.

Aware of the impending meltdown I redesigned the security and login procedures for the site. We kept the business logic on the server, but moved the interaction logic to the client. Instead of treating the browser like a dumb terminal, we started to leverage it for logical processing. We had the distinct advantage of being inside the corporate firewall, so we standardized everything on IE 5--which gave us XML and XMLHTTP. I still remember the meeting where we presented the changes. I was wearing a black turtleneck, because this was Manhattan and, more importantly, the long hours (and big dinners) left only those wardrobe choices that "stretched". It was a hit. And for me it was a defining moment in my own archtectural philosophy. We had stumbled upon a design pattern that was easier to develop, easier to scale, and easier to use.

General Interface

By July 2001, I was sufficiently burned out on New York and left for San Francisco. I had heard there were no jobs left in the Bay Area due to the dot-com collapse, and that was fine by me. I really was due for a break. But boredom set in and within a few weeks I was on craigslist looking for work. And somehow I found it. A Web shop in the city had just landed a contract with a bio-tech down in Mountain View named Iconix Pharmaceuticals. They were doing amazing work, aggregating the human genome, the FDA's known compounds database, and numerous in-house experiments. The data was expansive and rich, but the user interface was...a lot like craigslist. And it was a non-starter for the scientists they were trying to sell it to. Within three months we had delivered our first working version. It remains, to this day, the most interesting project I've worked on.

For various reasons, we decided that the only way to deliver the user interface on time was to write a pure, client-side framework that had no server component. We would only integrate with the backend by sending and receiving XML documents. Our technical integration boiled down to a document contract (Schema). It was all new to us back then, so we didn't yet realize the pattern we had stumbled upon. Michael Peachey, the manager on the project, used the term data schematic, but we all know now that what we were talking about was Web Services. It's a much more loosely-coupled approach to Web architectures, and now that Ajax has gained sufficient momentum, it will continue to feed an industry-wide shift.

Ultimately, the work with Iconix ended, and we found ourselves asking why this design pattern couldn't be applied to any number of other industries. So we set out to make a product out of the project (doesn't everyone) and came up with General Interface. It's a client-side framework written entirely in JavaScript with support for XML and Web Services. We even have a lightweight development environment created with the framework itself (which I used to create the alternate craigslist interface).

In that I actually work on the General Interface product, I rarely have time to actually use it. Instead, I'm usually focused on tracking down bugs or adding enhancements, so it was a welcome opportunity to set aside some time in the evening to actually use the product. What I really liked about the process was that without any help from craigslist, I was able to create a parallel interface that compliments (not replaces) their main Web site. In fact, deploying the application was no more complicated than finding an affordable Web host (I use eapps for $9.99 a month) and uploading some files. I've gone through a couple of iterations reworking the layouts, but it's been pretty easy, especially given some of the testing tools that I have at my disposal.

Since General Interface is a purely client-side technology, the only modification I made to the server was to set up a reverse proxy to handle cross-domain security. If viewed from a theory of constraints, the pinch point is not my server memory and CPU, but the speed and capacity of the network itself. In fact, I have an alternate version of this application that uses Yahoo! pipes to broker an RSS feed using JSON in order to completely bypass my server altogether. In the mean time, if you're interested in viewing the applicaton source code (SVN) or downloading your own local copy, it's hosted on googlecode.

Acknowledgments and Disclaimers.

First, all syndicated content belongs to craigslist. I'm just improving how users browse the information. Beyond that I'd definitely like to acknowledge dojo and the dojo.storage widget that I use to persist user data. Using this control allowed me to implement session storage (like favorite posts, blocked posts, settings, phone numbers, etc), without requiring a server component. This means that whatever you choose to save will be persisted on the computer you are using--nowhere else. I also use google maps for the mapping component and skype for integrated phone service.

As for browser support, General Interface runs on Internet Explorer and Firefox. I would bet that Safari support is on the way, but various legal constraints forbid me from mentioning it as fact. If you love some other browser and you somehow feel cheated, feel free to drop me a line and volunteer your services to write the extensions for your particular browser.