Monday, November 06, 2006

AfterBurner Web Server

I am finally getting around to sharing my older code, stuff that really deserves to be free.

Before I had many bad experiences with other people taking code I wrote and re-labeling it and getting credit. Now with Source Forge I feel a lot more confident. Not that this was the only thing holding me back. Part of it was time, management of the project, promotion of the project etc.

So I finally posted my Afterburner web server. What a long long history.

Afterburner on Sourceforge
Home page there

Since it's inception many other groups have chosen to use the Afterburner name. From Macromedia to bware's php plug-in. But I'm not changing the name. It's been that for 10 years now. I find it hard to believe all of these other Afterburners were named independently.

Originally written in 1995/1996 when I had my little consulting company in Venice Beach.
It was discussed what would be needed to get the optimum performance from these early PC Unix servers.

NCSA performance was lacking, this was before many of these other servers existed. My company specialized in high end support for web developers and ISP's just as the Internet was taking off.
So every major web site, when their server would have problems, I was almost always called in to find a solution. When working with DirectNet for hosting the Movie ShowGirls, their server was really having problems. It was an SGI high end something, and we actually had Kernel developers at DirectNet working to handle the 3 Million hits per day approx 3Mbps of bandwidth.

It's at this time, out of frustration with this crappy SGI and bloated Netscape Commerce Server, that Afterburner was hastily written.

The site was only 2MB is size, well small enough to fit into Ram. Only a few years earlier I had been working with 10 Mhz 286's with the Clarkston TCP/IP drivers and had no problem doing a sustained FTP transfers at 9.8Mbps, So I just didn't seem right to have system with 1/2 a Gig of ram, run out of memory and CPU at 3 Mbps.

So I laid out the server and had Aryeh Friedman, code it up, It used a very simple scheme were the whole site it loaded into RAM, and uses a single process, single thread, looping around the select command to service many TCP connection at once.

On a little crappy PC 90 Mhz we easily handled the 3 million hits per day from the Showgirls site. With a very small CPU load, while the same site would easily crash far more powerful servers.

SGI did a bunch of PR on being able to handle the web traffic on that site, since it broke the record at that time.

Eventually my video streaming took off, both Xing Streamworks and Livecam and the single threaded select design from Afterburner found it way into my video servers and even a chat server.

The video lead us into the Adult business from 1996 to 1997, and Afterburner was supporting many large sites, but remained a underground and generally unknown to the Public.
For a while we were one of the largest consumer of bandwidth, but nobody noticed!
We were the First large scale Content distribution network.
We had banks of servers around the earth each pumping out 10 to 40 Mbps, maxing out many of the ISP's that hosted us. We supported free video to the top 2500 web sites and supporting 100's of millions of hits per day and millions of video viewers.

Victoria Secret would get press that they did 37,000 viewers, everyone seemed to have problems viewing, and yet I would pull up our live stats and see 400,000 viewer watching live right now, and no reported would pay us any attention..... Enough Ranting.

The ability to verify the referring page was very important because there were many other site that would just link to your images and you end up paying the bandwidth bill while they make money of membership or advertising.

Afterburner would be able to deliver one image from the correct site and a totally different image when users come in from another site. Same with the video also.

In late 1997 we sold off everything that was associated with Adult to a group of Italians from New York and we left for Korea.

While in Korea, I added improvements to the livecam and we had been promised Investment there, but shortly after arriving, the Korean economy imploded...

In 1998 Digital Video Broadcast Systems was formed, to take the several technologies we developed into a Public Company. The idea was to get back on track with the SDSN business plan I had from 1995. This was to build a Symmetrically Distributed Server Network, what is now called a CDN content distribution network.
I realized there are really four technologies needed to make this whole system fly, and we developed each piece.

1.) High Performance Servers, Afterburner met that need.
2.) Video Compression , Livecam and other video tech we pioneered
3.) Access to Bandwidth. SDSN did that with 100's of Co-lo's
4.) A communications protocol better then TCP/IP that wouldn't stall or load down a server as much. ECIP did that

With these 4 pieces we would be able to compete with the TV broadcast stations.
P2P was also possible with ECIP, and part of that was called "Client Proxy"
What is now referred to as a Web accelerator... Where the servers would gather up content, and compresses it, send it over error corrected UDP and them make it available to the local web browser as regular content from localhost. So would be returned by the DNS lookups for all web hits...

Preliminary US patents were submitted in 1997 on Afterburner but were not followed up on because of financial problems.

Later a patent was filed on the improved version, WIPO PCT Number "WO 00/41455" filled with the World Intellectual Property Organization - official filling date July 20 2000. -- Afterburner Patent Doc
I don't think this patent was ever filed correctly though either because soon after the company fell apart.

In late 1998 we hired Jef Poskanzer, the author of the high-performance server 'thttpd' to clean up Afterburner and benchmark it. He made a lot of improvements.

In early 1999 we paid a company called MindCraft $20K to Benchmark Afterburner using 24 NT servers acting as Load Generators. We got amazing specweb numbers surpassing their best expectations. We found that they couldn't generate enough load to really show the true capacity of our server.

We reach 3600 Ops per sec in the SPECWeb96 benchmark, but that was all the load they could generate, this was about 321 Mbps on a single Intel 450Mhz P4 Xeon processor. In our own internal benchmarking done by Jef Poskanzer we reached 5300 Ops per second.

To put this in to perceptive, the best benchmark to date that was reported was 1500 Ops for that class of processor.

The next day we showed up at MindCraft, the doors were locked, and there was no way to get a hold of them for a about a month.

During those 4 weeks, Microsoft/ISS benchmarked against Linux/Apache and the Microsoft numbers doubled to around 3400 Ops!!!

Dan Kegel mindcraft_redux

The Specweb bench mark was unrealistic and unfair for many reasons.

One was there were only a small number of TCP connections at any one time. usually well under 200 total. In the real world we would see 1000's of slow lingering connections.
But in the benchmark, these were a few connections, that were communicating over a high speed LAN, so connections would start, send and quit very fast over gigabit Ethernet as compared to the real world of 56K modems and 1 minute to get a 100K file rather then some fraction of a second.

Why is this so important. Because each connection in parallel uses resources. Many server had to have a Full copy or at least a process per connection.
500 connections = 500 copies of the server in memory at once.
Even if the OS was smart enough not to duplicate program images in Ram, the context switches between processes and number of open file descriptor will kill many OS's.

Another unfair thing was Time Waits. When a TCP connection closes , the one who initiates the Close has to wait 2 to 5 Minutes to ensure the connection is fully complete.
In Unix these use up on MBUF or SKBUF(in Linux) approx 256 bytes. at 5000 ops per second * 60 second * 5 minutes, this added up really fast. 384 Megabytes of Ram spent on Closed connections!!!
In the real world, web servers were optimized to reduce these Time waits to 20 seconds or something a bit more manageable. But Microsoft changed their code to address the specweb benchmark deliberately, by just keeping a placeholder in Ram, so the timewaits would show up in netstat, but would consume a whole buffer. The Specweb rules required this.

In the Afterburner case, we didn't really use that much ram for keeping processing around or Disk Cache so it didn't really hurt us.
Also we had a real fix. We would pause after sending for .5 seconds and most of the time the web browsers would close the connections for us, the client PC's would get the timewaits. We were required to disable this feature during the benchmarks.

In Early 2000 DVBS had gone public, the Brokerage house or someone involved in take us public was playing games and took a ton of money, killing the company in the process.

With the closure of DVBS, Afterburners ownership had reverted back to me and had since been collecting dust.

I gave a talk at BAFUG the Bay Area FreeBSD users group that meet a Whistle communications in Foster City some time in 2000.

Terry Lambert commented on it with his own Kqueue experiments.

So I am happy to be able to finally share this code with the world.


Anonymous said...

Hi John. Is there any website now that is running on Afterburner??

John Sokol said...

I have had several large companies contact me after having run AB for a while to consult on other things.

I am not sure how many people are running it to be honest. It really only makes sense for super high traffic sites that can have there HTML stripped down to nothing but static content. It was originally written for Motion Picture web sites that get slammed just as a movie comes out.

Anonymous said...

Only HTML ok. I would also want to know about flash content whether that is considered as static content.

Since you have made it open source is there any way anybody realizing Afterburner's potential, would make it work on Dynamic pages?

John Sokol said...

Afterburner will server any "files" so flash would be fine, video clips, or anything that would work off an FTP server.

No CGI's and no scripts running on the server. So if a page is to be dynamic it will have to rely on Javascript and not the server.
Which is the way it really should be done anyhow. I find that there are very few limitations to that. But Ajax goes out the window. Oh well.

Aryeh Friedman said...

Just a note the only reason for not doing CGI's is anything that has control flow variations in it may have a infinitely long run and thus are not suitable for a non-blocking I/O environment

Henk Poley said...

It appears people are reinventing ECIP for mobile video streaming right now:

Henk Poley said...

It appears people are reinventing ECIP for mobile video streaming right now:

(friggin ReCaptcha on this blog..)

Aryeh Friedman said...

On redesicovery of ECIP/Spack even at the time people claimed it was not new but then proceed to show a broken version. The mobile one appears to still be stuck with Shannon's Limit which neither ECIP or any erasure or other methods we came up have that limit.