Comet Support by SitePen
SitePen Support

The Future of Comet: Part 2, HTML 5’s Server-Sent Events

by Jacob RusJanuary 10th, 2008

HTML 5 and Comet

Comet doesn’t have to be a hack. Currently, as we saw last time, Comet relies on undocumented loopholes and workarounds, each one with some drawbacks. We can make Comet work effectively in every browser, using streaming transports on subdomains of the same second-level domain, or using script tag long polling across domains. But this leaves Comet developers implementing (and more frustratingly, debugging) several transports across several browsers. Traps are numerous and easy to stumble into.

Recognizing the benefit of a straight-forward, standardized Comet technique, the WHATWG’s HTML 5 specification includes sections about server-sent events, and a new event-source element, which aim to de-hackify Comet. For now, only Opera has implemented these, and its implementation remains incomplete, but both Mozilla and Apple have committed to HTML 5, and an implementation is at least in-the-works for Safari.

Every Comet developer should familiarize himself with these specifications, because they provide the best streaming transport for Opera since version 8.5, and can only grow in importance as browsers adopt them. Furthermore, HTML 5 is a work-in-progress, and now is the best time to provide feedback to the WHATWG. Those interested in the future of Comet should comment now, before the specifications have been repeatedly implemented and can no longer be easily modified.

Basics of the server-sent events data format

The server-sent events portion of HTML 5 defines a data format for streaming events to web browsers, and an associated DOM API for accessing those events, by attaching callback functions to particular named event types. The format, which is sent by a Comet server with content type application/x-dom-event-stream, is straight-forward. Each event is made up of key-value pairs. For example:

key 1: this is the value associated with key 1
key 2: this value for key 2 stretches
key 2: over three lines, which are concatenated
key 2: by a browser supporting server-sent events
; these lines, which each begin with a `;`, are
; comments, and are ignored by the browser

key 1: after a pair of newlines, this is a new
key 1: event, with its own set of key-value pairs
; the following key has an empty corresponding value
empty key

...

Each key-value pair is known as a field, and several special fields are defined in the specification. In particular, the Event field names the event as a specific type. The browser can attach different callback functions to specific named event types. If omitted, the event type is assumed to be message. Also, though it is only mentioned in examples, the data field is useful for our event payloads.

So an event stream sent by Orbited might look something like:

Event: orbited
data: this is our event payload for the
data: first Orbited event

Event: orbited
data: this is the payload for the second
data: Orbited event

Event: ping
data: \o/

Event: orbited
data: here's the third Orbited event

...

The specification mandates that all files are UTF-8, but considers any of carriage return (\r), line feed (\n), or both (\r\n) an acceptable newline—in the browser, multiple lines of a single field are joined with \n alone.

The event-source element

Server-sent events are received by objects supporting the RemoteEventTarget interface, which merely means that they support two methods, addEventSource and removeEventSource, each of which takes a URI string as input, and adds or removes it, respectively, from the list of event sources for the object.

In addition to any JavaScript objects supporting server-sent events, HTML 5 defines an event-source HTML element, which declaratively indicates the use of a Comet source in a web page. As well as supporting the addEventSource and removeEventSource methods, the event-source element has a src attribute. When the src is changed, the event-source closes its previous connection, and opens a new Comet connection to the new URI.

Cross-domain usage

HTML 5 allows connections across domains, through use of the Access-Control HTTP header, as defined in a separate W3C specification (which applies identically to normal XHR usage and to server-sent events). A request is made for a resource as usual, but if that resource on the server (in this case, an event stream from a Comet server), includes the Access-Control HTTP header with values allowing the use of the resources, browsers will treat it as if it came from the same domain as the main document. If the header is not found, or if it denies the requested use, browsers will behave as if the resource does not exist (so that denied requests reveal no information about the resource).

Additionally, HTML 5 defines a “cross-document messaging” mechanism, which allows cooperation between documents (perhaps in iframes, etc.) from different domains, using a postMessage function.

Opera’s implementation

Opera has, since version 8.5, implemented a subset of these HTML 5 technologies. Opera versions since 8.5 support the event-source element, and recent versions have pure-JavaScript interfaces as well.

To support versions back to 8.5, we must create event-source elements, set their src attribute, and attach them to the document. Then we can add “event listener” callback functions to the event-source for each type of named event in the event stream. In Orbited, we use the following JavaScript to accomplish this:

connect_server_sent_events: function () {
  var es = document.createElement('event-source');
  es.setAttribute('src', this.url);
  document.body.appendChild(es);

  var event_cb = this.event_cb;
  es.addEventListener('orbited', function (event) {
    var data = eval(event.data);
    if (typeof data !== 'undefined') {
      event_cb(data);
    }
  }, false);
},

Where this.event_cb is some callback function, a property of the Orbited object, which will receive the event payload of every orbited event. By default we send payloads in JSON format, so evaling each yields a JavaScript object.

Also, it is quite easy to test browsers for server-sent events support from JavaScript, using code something like:

if ((typeof window.addEventStream) === 'function') {
  // ... browser supports server sent events
} else {
  // ... no support.  fall back on another transport
}

Caveats

Opera’s implementation differs from the specification in a few key ways, however, so Comet application authors must be careful.

  • Opera ignores events without a defined Event field, rather than assuming its value to be message. It is possible to include multiple named event types, attaching separate event listeners for each type.
  • Event payloads must be in the data field. A callback attached to the event source will receive an event object as input, with event.data set to the value of the data field.
  • At present, Opera only supports linefeed (\n) characters as newlines in event streams, and will silently fail if carriage returns are used in newlines (\r\n and \r are not supported newlines).

Legacy support

Even though Opera is the only browser to natively implement server-sent events, it is possible for a Comet server to treat the last several versions of Safari and Firefox as if they did, with a few caveats. We can use a modified XHR streaming technique (as described in part 1), and implement our own parsing for the application/x-dom-event-stream format (at least the version supported by Opera) in JavaScript. To pull this off we must make one concession to Safari: 256 bytes of dummy data at the beginning of our event stream, so that Safari will begin incremental parsing.

Our Comet server can then be blissfully unaware that these browsers have no real server-sent event support, and we can get away with implementing only two transports on the server side: the iframe transport, supported by most browsers since at least 1999 or 2000, and—using the htmlfile ActiveX object—capable of a flawless user experience back to Internet Explorer 5.01; and server-sent events.

To make them still more compatible, we might be able to build support for event-source into Firefox and Safari using JavaScript alone, creating objects supporting the addEventSource, removeEventSource methods, and capable of dispatching named event types to event listeners. Adding such support would require deeper magic than I currently possess as a journeyman JavaScript hacker; if any masters can shed insight, please comment here, or shoot an email to the Orbited mailing list.

In Orbited, we have not yet tried reducing XHR streaming to server-sent events for Firefox and Safari, but it is on the to-do list.

Arbitrary DOM events and controversy

In addition to this straight-forward Comet transport, server-sent events provide more general, and potentially powerful, capabilities, whose complexity is somewhat controversial. Indeed, the whole server-sent events section of the specification currently includes a notice that it may be removed. I expect that it will change at least somewhat from its current form before the specification is finished.

The specification demands that every element supporting the EventTarget interface should also support the RemoteEventTarget interface. A Comet server can thereby send arbitrary DOM events to page elements. This includes events such as mouse clicks, key presses, etc. As specified, event streams can also include a Target field, which can target events to the top level of the document, or to specific element IDs. And browser vendors could, if they desired, add further objects supporting the RemoteEventTarget interface, potentially enabling declarative Comet applications.

So the theory goes. Many of us Comet developers, including Michael Carter and Alex Russell (in a recent IRC discussion), remain unconvinced that there is a benefit in pushing application logic from browser-side JavaScript to the server side. We expect that real-time applications will always have a significant need for client-side logic, so the specification may as well embrace that—and remain as simple as possible. Specification simplicity benefits not only browser vendors who must build conforming and compatible implementations but also Comet developers who must learn its ins and outs.

But I hope server-sent events are not scrapped altogether. Comet would benefit greatly from specification and intentional browser support. The version supported by Opera strikes a reasonable balance, meeting the needs of Comet developers without going beyond those needs.

Conclusion

With improving browser support, creation of more high-quality open-source Comet servers, better developer resources such as Comet Daily, and the examples provided by big-name Comet applications, Comet’s future looks bright. Any standardization efforts by browser vendors that further reduce barriers to entry will only lead to more and better Comet applications.

HTML 5 is coming, with at least Opera, Apple, and Mozilla committed to its adoption, and one way or another it will include improved Comet support. But the Comet-related portions of HTML 5 are still very much unfinished, and in need of feedback from browser vendors and from us, the community of Comet developers. The discussions are transparent and easy to join, and time for action is now. We should figure out what we need and tell the W3C and the WHATWG about it. They are all ears.

[Slashdot] [Digg] [Reddit] [del.icio.us] [Facebook] [Technorati] [Google] [StumbleUpon]
SitePen, Inc. Comet Services

13 Responses to “The Future of Comet: Part 2, HTML 5’s Server-Sent Events”

  1. Kris Zyp Says:

    Jacob, great article and thanks for your work towards Comet standardization. However, a few comments on this approach: Perhaps the main purpose of standards is to bring interoperability. The fundamental flaw/anachronism of this approach is that it is defining messages in completely different domain than we need for Comet messaging. The DOM is almost always application specific, and not abstracted to the level of the data with which you want to communicated about. With anything more than a simple demo, Comet communication is usually in regard to actual data. With pub/sub architecture, Bayeux defines a mechanism for subscribing and publishing events, however if one where to use server-sent events, one still needs to define conventions for publishing and subscribing to data, a substrate of standards based DOM messaging is nice, but we would still need standards on top of that for real data interchange.
    Server sent events is an approach designed for a server centric applications, which in the world of sophisticated Comet applications, is simply antiquated. Comet applications are generally use significant client-side logic, and I think the server-centric server sent events is outdated before it is even available. In addition, I believe HTML is simply the wrong level to solve this problem. To me this analogous to trying to solve database issues with a graphics card. Comet shouldn’t be standardized at a UI-specific level, it is about data and should be dealt with at that level.
    The approach I proposed also leans on standards, but it uses the HTTP standards and conventions to communicate about what really matters, the actual resources and data that Ajax applications are pulling from. This approach gives Comet meaning at the right level, at the HTTP level. It allow a well-established communication about what Ajax applications already understand, the URLs and the resources that these Ajax applications are already accessing data from.
    Anyway, those are just my thoughts. Thanks again for the great article and continued efforts towards standardization.

  2. Revin Guillen Says:

    “Many of us Comet developers … remain unconvinced that there is a benefit in pushing
    application logic from browser-side JavaScript to the server side.”

    Other than the download/parsing savings of not having to send application logic to end users? That alone seems like a decent win to me.

  3. Jacob Rus Says:

    if one where to use server-sent events, one still needs to define conventions for publishing and subscribing to data

    Sure, which is as it should be. HTML5 is certainly not the place to standardize pub/sub communication.

    Server sent events is an approach designed for a server centric applications, which in the world of sophisticated Comet applications, is simply antiquated.

    Given that all the current Comet transports in use, even when a pub/sub is put on top of them, are “server-centric”, I don’t think that using server-sent events would have any harmful effect. As I see it, the main effects would be simplifying implementation for Comet developers on both server and client side, and allowing web browsers to optimize their behavior (which might include things like adding a user interface element showing the status of Comet connections).

    HTML is simply the wrong level to solve this problem.

    I should perhaps have mentioned that I’m a bit dubious of the event-source HTML element. But I think having an API for this accessible from browser javascript is great.

    The approach I proposed also leans on standards, but it uses the HTTP standards and conventions to communicate about what really matters, the actual resources and data that Ajax applications are pulling from.

    Well, you should get talking to browser vendors as soon as possible if you want to make that happen; just proposing it on the web is not likely to provoke any movement in that direction. To me, your proposal seems more complex than necessary for a Comet transport, but I could be misunderstanding it.

    To be honest, I had 3 main intentions in writing this article: 1) To alert the community to the existence of HTML 5, and hopefully provoke some discussion, 2) To advocate some standardized Comet transport across browsers, so that it is no longer necessary to use a different hack for each, 3) to explain to Comet developers how to implement a server-sent events transport which works in shipping versions of Opera, so that Opera is not marginalized by Comet applications.

  4. Jacob Rus Says:

    Revin: what I mean by that is that it seems to me that there is little benefit in directly invoking DOM events from the server. Because the Comet applications I can imagine need quite a bit of JavaScript logic anyway, the ability of the server to, for instance, generate a click event for a submit button, or a keyboard event, or whatever, seems unlikely to have much practical benefit. I don’t see where that ability would reduce bandwidth by any significant amount.

  5. Kris Zyp Says:

    Jacob,
    Well, you should get talking to browser vendors as soon as possible if you want to make that happen; just proposing it on the web is not likely to provoke any movement in that direction.
    There is a big difference between just me thinking that it is a great idea, and others thinking that this it is something worth pursuing :). Anyway, there isn’t much that needs to be done by browsers, JavaScript can handle this, at least for now. Just like any other current Comet technology, it would make things a lot better if IE provided streaming XHR support (and a little higher connection limit wouldn’t hurt).
    To me, your proposal seems more complex than necessary for a Comet transport, but I could be misunderstanding it.
    Almost all of complexity is in the HTTP itself. My proposal simply leverages the complexity that we already have with HTTP, to do Comet. Beyond just using normal HTTP, the proposal itself consists of just basically adding a simple “When-Modified-After” header and an interpretation of HTTP (albeit with some novel inferences like multipart/digest batching).

    To be honest, I had 3 main intentions in writing this article:
    And you did an excellent job with this. Server-sent events is certainly not without merit. Great article, and thanks for the discussion starter.

  6. Kris Zyp Says:

    Arg, how do you do quoting thing? Anyway, second attempt:

    “Well, you should get talking to browser vendors as soon as possible if you want to make that happen; just proposing it on the web is not likely to provoke any movement in that direction.”

    There is a big difference between just me thinking that it is a great idea, and others thinking that this it is something worth pursuing :). Anyway, there isn’t much that needs to be done by browsers, JavaScript can handle this, at least for now. Just like any other current Comet technology, it would make things a lot better if IE provided streaming XHR support (and a little higher connection limit wouldn’t hurt).

    “To me, your proposal seems more complex than necessary for a Comet transport, but I could be misunderstanding it.”

    Almost all of complexity is in the HTTP itself. My proposal simply leverages the complexity that we already have with HTTP, to do Comet. Beyond just using normal HTTP, the proposal itself consists of just basically adding a simple “When-Modified-After” header and an interpretation of HTTP (albeit with some novel inferences like multipart/digest batching).

    “To be honest, I had 3 main intentions in writing this article:”…
    And you did an excellent job with this. Server-sent events is certainly not without merit. Great article, and thanks for the discussion starter.

  7. Jacob Rus Says:

    Arg, how do you do quoting thing?

    Well this gets a little bit off topic, but what I actually do is type using Markdown, and then run it through markdown.pl, before pasting it in the comment box, so that:

    > Arg, how …

    Well this …

    Is turned into:

    <blockquote>
    <p>Arg, how …</p>
    </blockquote>

    <p>Well this …</p>

    But just typing the HTML tags into the text box yourself should work fine too.

    Almost all of complexity is in the HTTP itself. My proposal simply leverages the complexity that we already have with HTTP, to do Comet. Beyond just using normal HTTP, the proposal itself consists of just basically adding a simple “When-Modified-After” header and an interpretation of HTTP (albeit with some novel inferences like multipart/digest batching).

    Don’t browsers need to be modified to understand this new header, and the new interpretation of multipart/x-mixed-replace, etc.? Or can your proposal be implemented completely in JavaScript on top of current browser capabilities?

  8. Jacob Rus Says:

    … Is turned into:…

    And that apparently doesn’t always quite work out, as Wordpress adds a bit of its own markup as well. :)

  9. Kris Zyp Says:

    Thanks for helping an incompetent commenter :). Yes, it should be implementable in JS. I’d like to do an implementation… maybe part 2 of my article.

  10. Comet Daily » Blog Archive » Not a Fancy Name Says:

    [...] adds nothing new to the world past what we had fifteen years ago. As Jacob Rus discussed, HTML5 has the potential to be quite significant for Comet. It’s even ridiculous to suggest that HTML was somehow revolutionary—it’s just a [...]

  11. Comet Daily » Blog Archive » Buzzword Overload Says:

    [...] events: An HTML 5 tag to allow Comet without any JavaScript or hackery. Championed by [...]

  12. Comet Daily » Blog Archive » Is Comet Becoming Over-complicated? Says:

    [...] consistent way. Some work has gone into this in the WHATWG and W3C HTML working group, with ‘server-sent DOM events‘ a feature of the upcoming HTML 5 specification, and already partially implemented in Opera. [...]

  13. Comet Daily » Blog Archive » Interview with Kaazing Says:

    [...] How do you get Server sent events working anywhere besides [...]

Leave a Reply



Copyright 2014 Comet Daily, LLC. All Rights Reserved