A few weeks ago MySpace invited me to give a presentation about using Orbited to create scalable Comet applications. While I can’t speak to any plans on the part of MySpace, I did get permission to post my speaking notes and slides. One point to keep in mind is that this presentation was made shortly before the naming of WebSocket was decided in the HTML5 spec, so the slides refer to the then-current name “TCPConnection”.

Slides

Speaking Notes

Slide: Orbited Logo + Michael Carter

-My name is Michael Carter
- Founder Lead developer of Orbited
- 2 years in development
- countless architectures tested
- now we’ve found the best
- javascript work

Slide: Comet

- Here to tell you about Comet
- Everything I’ve learned in two years of Comet work
- You probably already know what Comet is. But i’ll say it again just in case.

Slide: What is Comet

- Its about asynchronous updates in the browser (Pushing data)
- Not necessarily about a particular technology
- could be flash
- polling
- javascript hacks (streaming)
- Its about an interaction model — user is notified asap with new data
- Its about the applications that you can build with that interaction model
- Chat, clearly
- Financials
- Gaming
- Auctions
- Sports reporting
- Collaborative Editing
- Network monitor reporting / system administration
- Election Monitoring (Andrew Betts / London Paper)
- Air traffic control (no joke)
- To name a few
- In practice though, everyone wants Comet without the need for flash or bad performance of polling
- Just plain old js and html, low latency
- Which brings us to Transports

Slide: Transports
- These are the actual mechanisms for delivering data
- There are many

Slide: Iframe

- Originally we had script tag streaming
- but it made sense to put it in an iframe
- So we refer to it now as iframe streaming

Slide: Iframe Diagram

- Application page has a callback “receive”
- Finishes loading, creates and iframe
- Iframe causes an HTTP request to the server
- the server sends headers
- (transfer-encoding chunked for http 1.1)
- And then stops
- The key to this transport is incremental rendering
- Browser will try to display any open/close tag pair that it receives
- This includes displaying invisible elements, like script tags
- So the server can cause the browser to execute arbitrary javascript
-

Slide: Iframe Code
- Easy to implement
- Works in all browsers with the same code
- Suffers from..

Slide: Iframe Usability
- Loading Bars
- Hour glasses
- clicking noises?!

Slide: Iframe Unacceptable
- Modern applications can’t have clicking noises and hour glasses.
- The users would get too confused
- So Iframe streaming isn’t the most popular transport
- Instead, the most popular transport is…

Slide: XHR Long polling

- Long polling is what the cool kids do.
- (Meebo and Facebook, for instance)

Slide: XHR diagram

- Browser says, ‘Give me an event’
- Server says, ‘…’
- Server says, ‘event.’
- Browser says, ‘Give me an event’
- server says ‘…’
- Its cross-browser

Slide: Long polling code
- still easy to implement
- and no usability issues!
- seems like a big win
- but how does long polling performance compare to iframe streaming?
- Consider an island off the coast
- its surrounded by water

Slide: Island
- We want to send conveys to the island, so we build a bridge

Slide: Bridge
- Conveys can now go to the island as they please
- But maybe a big ugly bridge is not what we wanted… its too unsightly
- So instead we build a boat

Slide: A boat
- The convey arrives at the boat, and the boat takes it to the island
- no problem
- But what about the second convey that arrived just after the boat leaves?
- It waits for the boat to go to the island
- It waits for the boat to come back
- It gets on, and waits for the boat to go to the island
- 3x travel duration
- Well Iframe streaming is the bridge, and the boat is long polling
- Meaning, long polling suffers from a 3x latency performance hit
- But thats not all
Slide: Bandwidth comparison
- You also have to send headers all the time
- Your buddy says ‘hows it hanging’
- But the browser says ‘GET /some/url HTTP/1.1\r\nUser-Agent: …etc’
- And the server says ‘HTTP/1.1 200 ok\r\nContent-type….’
- your 15 byte message can easily incur 300 bytes of headers, or more!
- thats a 20x bandwidth increase in that case

Slide: Long polling unacceptable
- 3x latency is too much
- 20x bandwidth is too much

Slide: Strike 2, you’re out
- Strike 2, you’re out
- or so we thought

Slide: Htmlfile Object
- htmlfile object to the rescue

Slide: ActiveX!
- but safe for scripting
- hides usability issues

Slide: Garbage crossout
- leaks live connections on navigations
- But we can fix that by explicitly calling the garbage collector

Slide: Htmlfile works great
- Streaming
- But No side effects
Slide: Explorer only
- partial solution
Slide: Xhr Streaming
- transport for ff and safari
Slide: ready state 3
- don’t do one event per request/response cycle
- do a hundred. a thousand…
- ready state 3 saves the day
- callbacks for download progress from server
- responseText is just plain text without any inherent meaning

Slide: Manually parse
- So invent our own format
- parse it manually
- And we’re set for firefox and safari
- Without loading bars or anything
- and the same bandwidth and latency constraints of iframe streaming
- So we’ve got all the browsers
- Except opera

Slide: SSE

- Opera goes above and beyond the call of duty
- I don’t ask that browser’s make my life easy
- They can keep their nasty forest of terrible hacks
- All I ever asked from browsers is that they make it possible to do comet
- But Opera actually makes comet a first class citizen of the browser
- Create an event source, name a url, add a listener. The end.

Slide: Html5

- Then they went on to propose that the this be included in HTML5
- after discussions with other vendors, gained support, a new spec came up

Slide: deprecated

- so the new one isn’t implemented
- but thats okay
- we can use the deprecated one in opera 8+

Slide: Transport Conclusions 10.18.20
- Ten minutes to explain
- Years to discover

Slide: hacks
- And yes, its hacks, hacks hacks

Slide: aim carefully
- Developing these transports is like trying to throw darts at a board

Slide: moving target
- but the board moves after you’ve thrown the dart
- That is to say, just imagine the most frustrating game you can
- That game is twice as fun as building transports

Slide: sleek bridge
- We can avoid the boat transport (long polling)
- And provide great user experience
- get a nice looking bridge

Slide: sleek bridge image with orbited logo superimosed
- And orbited does just that

Slide: Scalability
- Once you have a way to do comet, you need to scale it
- where transports are frustrating, scaling is hard
- Step 1,

Slide: Vertical Scalability
- How do you put lots of users on one machine?
- Seems like such a hard problem, but really there’s a simple trick
- And once you know the trick, its possible
- Don’t use threads.
- People have tried this with servers like apache
- A reasonable thread count, of 20, will service 20 users
- A better user count is a 1000 though
- But a 1000 threads brings a system to its knees with context switches
- No comet, just thrashing

Slide: Comet + threads != scalable
- So use the little trick of omitting threads, and you’re golden
- Build event-based servers
- You don’t want your comet apps to be like nature preserves
- You don’t want your app to be a lonely desert with the occasional wanderer
- it should be a riot, with users tearing down the walls, sucking your resources dry, demanding more…

Slide: desert -> riot
- So use event based programming for your comet server
- And I guarantee, your app will be a riot, not a desert.

Slide: zzzz
This is the easy problem
The harder problem is the other “half” of scaling,

Slide: Horizontal Scalability
- If one server supports 1000 users, does 2 support 2000? and n, n-1000?
- This is a truly tough problem
- Really though, it depends on the application
- No one answer.
- But a few core rules are in order
- 1. Don’t share state between comet servers
- 2. Only do Comet in the Comet server. no Pub/sub, no jabber/game server, no load balancing, just comet
- 3. Treat comet as the last leg of the journey

Slide: Two Comet servers (sharded by user) doing pub sub (ajaxworld slide)
- For some reason, comet servers are all trying to provide a pub/sub api
- Developers like it, great.
- But it was already a hard problem before Browser’s existed
- And the people solving the problem, RabbitMQ/ActiveMQ, aren’t part of the comet community. (there have been minor inroads)
- Integrating streaming comet sith real publish/subscribe message queues would be great
- But, thats just a specific case of a larger issue
- The real issue with scaling comet applications is that
- Comet is difficult
- Comet imposes architectural constraints on web apploications that we aren’t used to (can’t do round robin balancing per http request)
- Its a moving part that has to be closely tied to the other pieces of the system
- But integrating comet with existing technologies would allow us to solve the horizontal scaling problem however these existing technologies already solve it.
- How could you scale a thick client deployment?
- We want that to be the only question to answer when scaling a comet application.
- So How do you integrate comet servers with X?

Slide: Offboard Comet integrated with Jabber
- This is an example of how a typical comet server is integrated with jabber.
- the browser maintains a connection with the comet server and the “bridge”
- The bridge is an http server (hopefully event based as well!)
- and also an xmpp client
- and also a “Comet client” (dispatch protocol)
- Message flow goes browser -> bridge -> jabber -> bridge -> comet -> browser

Slide: A lolcat tangled up
- This is a bit of a mess
- complex systems are hard to scale

Slide: previous slide (Offbard comet integrated with jabber)
- you can’t scale this with your typical web load balancer
- the mapping between comet servers, bridges, and xmpp servers are almost certainly not giong to be 1-1
- so you have the complex task of figuring out load balancing between each part
- and then you have to implement each client twice
- once for the bridge, once for the browser
- the bridge speaks xmpp to the jabber server, but json-chat (made up protocol) to the browser
- Clearly not the solution…

Slide: Victory march picture, maybe the planting of the american flag in some battle
- Standards save the day.
- HTML5 has specified a TCPConnection standard for the browser

Slide: TCPConnection (word)
- I’ve been working closely with whatwg/w3c to specify the protocol and api that solves all of our problems
- tcpconnection is just like a socket, but for javsacript in browsers
- It will
-traverse forward proxies
- speak ssl
- include cookies (auth integration)
- use a single socket for both directions
- revolutionize web applications
- Think about it — tcp connections revolutionized network programming in the 70s.
Slide: Iron Man screenshot during test sequence
- tcp/ip. Thats how dad did it, thats how america does it, and its worked out pretty well so far
- And we don’t even have to wait for browser vendors

Slide: The future is now (sci-fi-ish picture + words?)
- Orbited provides a TCPConnection to all major browsers, today.

Slide: Browser < - > Comet Proxy < - > Jabber diagram
- Orbited speaks comet transports on the front
- and plain old tcp on the back
- So you put it in front of ANY tcp server
- and the browsers can interact as if they’d connected directly to the backend server over tcp.
- and the server doesn’t need to be altered whatsoever.
- No need for a bridge or dispatch protocol
- Much less complex means easier to scale
- Load Balancing?
- We can simplify this picture still

Slide: Browser < - > ( comet Proxy + Jabber Diagram single machine)
- Treat the comet proxy as part of the server
- One comet proxy per backend server
- Now you can all sigh in relief

Slide: Sigh of relief
- Because Comet isn’t going to get in the way
- Scaling Comet applications horizontally isn’t a twofold problem anymore
- If you know how to scale a network application, then you can scale a comet application.

Slide: Jabber server cloud with half Comet proxies, half normal
- And whats more, there’s no difference between the browser and the desktop

Slide: Auth + persistence (a pair of keys and some kind of storage)
- There are a couple of details: authentication and persistence

Slide: Jabber server with comet proxy on left, web app on the right, sql server behind both
- Just have the network server and the web app coordinate on the back end
- put the sql server behind both
- User creates an account over the web app
- but can log in to jabber
- jabber saves logs to the database
- and the user can see those logs via the web app
- And your problems are over.

Slide:Conclusion (Bullseye (with orbited logo somewhere))
- Transports are frustrating, but Orbited implements them (indeed, a number of projects take our implementation directly)
- Vertical Scalability and threads don’t mix. Orbited use event-based network IO.
- Horizontal scalability is tough because you have two problems.
- If you can solve integration, then you can get a handle on horizontal scalability by normal means
- TCPConnection is the perfect integration point. It works with all past, present, and future tcp/ip servers. Out of the box.
- Orbited provides that functionality today.