Comet Support by SitePen
SitePen Support

Should Bayeux Be a Communication Protocol?

by Michael CarterJanuary 2nd, 2008

The point of any standardization is to allow interoperation between moving parts in a system. At this moment we have application developers who want to create Comet applications. We have server developers who want to create Comet servers that are responsible for all kinds of data encoding and transportation. But the application developer doesn’t actually care how data departs the server and arrives in the browser. So a Comet protocol should clearly define where these disparate components should meet, and how.

Our Comet protocol is Bayeux, and it defines the separation between moving parts to occur at the network edges. That is, the protocol defines precisely how a server and client should interact; it’s a communication protocol. But the application developers don’t actually care about client/server interaction—that can be pure black magic and it’s fine. The application only needs to know about a few functions: connect, publish, and subscribe, for example. I therefore propose this alternative: Let’s not require any specific communications protocol. Instead, a Bayeux server must serve /bayeux.js which exposes a JavaScript API that supports the appropriate functions.

Many servers will likely want to use a shared communication protocol similar to the current Bayeux proposal. But others will be able to push the envelope and experiment with all manner of server/client interactions. For instance, a server could use the same interaction model that Bayeux proposes now, but completely change the encoding of all the information so as to cut the bandwidth down significantly. We wouldn’t want this to be the “standard” because it’s hard to implement, but we would want it to be possible by defining a looser standard.

The only real reason for defining a communication protocol is to allow non-browser clients to connect. But we should just face the fact that Bayeux is a browser-based standard, and define an alternate spec for non-browsers.

There’s no point in making a developer first choose a Bayeux server, and then choose a Bayeux client. The various Comet server developers can get together and collaborate on interaction models or client implementations, but in the end, all we really want is for application A to run on Bayeux server B, C, or D.

[Slashdot] [Digg] [Reddit] [del.icio.us] [Facebook] [Technorati] [Google] [StumbleUpon]
Orbited

10 Responses to “Should Bayeux Be a Communication Protocol?”

  1. Jörn Zaefferer Says:

    Hi Michael,

    I enjoyed your Comet talk at the Boston Ajax experience and I’ve been reading this blog since then.

    It looks like Comet is moving forward really fast - I can’t rememeber you mentioning Bayeux in your talk, be it due to my memory or just that you haven’t mentioned it.

    I think your proposal here is a very important one. Having an interface on the client-side to work with any Bayeux server seems like a great idea. I’d like to emphasize that the bayeux.js you mention, as a JS library, should provide only the client-server-communication API and nothing else. We already have way too many libraries that do everything from DOM traversing and manipulation to server-communication.

    I guess this isn’t quite the right place for this kind of feedback on the Bayeux protocal, but unfortunateley the document doesn’t mention any way to give feedback, only that its asking for it. Could you point me, and whoever else is interested, at the right place for feedback?

    Regards
    Jörn

    PS: Something about /lift/ and Comet would be very welcome, written by someone who is heavily involved in Comet stuff but not in lift…

  2. Michael Carter Says:

    Jörn: You’re right, I didn’t mention Bayeux in my talk. I was more focused on the general principals behind scaling Comet applications. As it stands now, the Bayeux protocol defines both a client api and a server api. My main point in this musing is that the application only cares about the javascript api exposed by the server via the client. you can visit #cometd on irc.perl.org to provide feedback for the Bayeux protocol as it stands. As for lift, I took a look and it seems like it has a built-in comet solution. I couldn’t find any information out from the developers in their irc channel though.

  3. GregWilkins Says:

    Michael,

    Bayeux and Cometd is a communications protocol for the “last mile to the browser”, which really means over the internet and through firewalls. It is not intended as a general purpose communication protocol and their are plenty of alternatives that have decades of maturity represented in their implementations (eg JMS messages).

    I do agree with you is that the major thing is to get application developers to agree on the semantics of the API. Once developers start using messaging rather than programming to some XHR wrapper, then it is infinitely simpler to change implementations underneath the applications.

    Withing cometd/bayeux there is definitely support for standardizing the API, but given the cross language aspirations it is a difficult task (and you need to support multiple server languages at least - and I do think their are some valid use cases for other language client implementations). Some efforts have started here: for example the java implementation uses the dojox.cometd name space simply to define the server side API. And I have always thought of dojo.js as defining the defactor standard API to the client side standard libraries that we all know we need to have:-) Also the Open Ajax Alliance will be soon be making effort to include standardization of the api for Ajax/Comet messaging to their hub.

    So yes - standardizing on messaging semantics is very important. Standardizing on messaging APIs should also be a target and is already aspirational goal of the project.

    But neither of these aims mean that we should not also aim for some standardization of the protocol. There is still value in having a set of transports that can be negotiated by the Bayeux protocol that will allow interoperability between client and server implementations. Even if all of Bayeux is replaceable under some standardized messaging API, those that do choose the Bayeux family of transports will benefit for interoperability between implementations.

    Eventually the ideal world that I would like to see is one full of choice. A developer working on a Comet application may choose between: changing bayeux transports; changing bayeux implementations on either client or server; changing messaging implementations that implement some standard api from dojo/openAjaxAlliance; changing apis from one messaging abstraction to another (but semantically similar).

    cheers

  4. Alex Russell Says:

    Hey Michael,

    I think you’re dead on the money in highlighting the lack of client API standardization. We probably need an IDL description for the canonical connect, disconnect, publish, and subscribe verbs (and maybe batching). There’s also a similar lack of uniformity in Bayeux server configuration, and getting that nailed-down/portable would also be a huge boon to users.

    That said, the first analogy that comes to mind for me here when we talk about APIs and not protocols are the huge JMS and JNDI design mistakes. Specifically, both of these things *should* have been protocols (with perhaps also APIs spec’d out), but instead they only spec’d out the Java APIs which users would talk to. As a result, interop has been nothing short of pitiful. JMS messages are opaque on the wire (not human readable) and you’re tied not to your client *but to your server*. That means that if someone wants to interoperate with a JMS server from another language or another vendor, they can’t just pack/unpack a protocol-specified format…they need to pack/unpack a particular vendor’s format, forever tying clients to servers. I think this kind of server-specific stuff is probably a huge boon for particular JMS vendors, but it sucks for users and there’s a real chance to avoid this mistake in Bayeux by keeping the semantics of the JSON messages well-defined (at least in the “degenerate” HTTP case).

    Lastly, I would like to note that we’ve already seen that Bayeux-as-protocol has allowed serious experimentation with the wire-level semantics due to it’s ease of wrapping in whatever container you wish to send the messages over. That “loose joint” is probably a good valve as we look to a future where thing like WHATWG’s server-sent event format become viable for some clients and not others.

    Regards

  5. Jacob Rus Says:

    Greg, don’t you think that having a particular set of “blessed” transports inside both the Bayeux specification, and inside the “de-facto API” of dojo.js, discourages Comet server developers who wish to interoperate with Bayeux-based client-side JavaScript (whether dojo.js or something else) from experimenting with different transports, or variations on existing transports?

    This solution Michael is proposing here is an attempt to meet the flexibility goals of your ideal world. Because current Comet transports are so intimately wound up with browser-specific details, and because tweaking them requires changes to both server-side and client-side code, blessing a particular transport or transports in a specification renders those transports “untweakable” by servers or clients wishing to interoperate.

    From my perspective, there are delicate separation-of-concerns questions to be worked out before any standard Comet protocol can successfully meet the goals of the community.

    Three layers which make the Comet communication of a typical Bayeux application are the transport (including both event format and browser-side javascript), a protocol layer for dealing with details such as sessions, authentication, message ordering and delivery guarantees, etc., and finally the semantics for publish/subscribe. Because a Bayeux-using application developer mainly cares about a single API—which exposes the means to connect, subscribe to a channel and designate it a callback function, and publish to a channel—Bayeux, or more specifically here dojo.js, wisely leaves the details of the communication layers as a black box for application developers, allowing them to build on top of it with minimum fuss.

    Unfortunately, because the details of all three of these layers are de-facto specified by the particular dojo.js implementation, Comet server developers wishing to interoperate with dojo.js have no choice but to implement all three layers in a dojo.js-friendly way, which for instance limits the available transports to those implemented in dojo.js.

    We would like as much as possible to separate these various concerns into layers, thereby reducing code duplication by building both pub/sub and non-pub/sub protocols on top of the same transport-specific and session-specific code. Michael’s suggestion, of making the top-level API a standard, exposing it from JavaScript served by the Bayeux/Comet server, and leaving the details of the rest up to the particular server, would allow a server to layer these internal structures—enabling their independent change—and still interoperate with multiple client-side libraries. Michael’s suggestion in this post is one simple way to bring interoperability, but not the only one; an alternate method is the separation of Bayeux into multiple separate specifications, along the lines of the aforementioned layers.

    You want to arrange that “those that do choose the Bayeux family of transports will benefit for interoperability between implementations.” But forcing such a choice of transports on developers could hinder advances even while it aids interoperability: each time dojo.js changes, for instance modifying an existing transport for better browser compatibility, or introducing a new one, or adding stronger message-delivery guarantees, every compatible server will need to simultaneously change. In the worst case, a few years from now, the community could be presented with the choice between stagnation and incompatibility.

  6. GregWilkins Says:

    Jacob,

    I hear what you are saying about “blessed” transports. There has been talk about splitting the spec into a pure transport neutral section and several transport specific documents. But for the moment time limits and expediency has us doing them as one document. Also it is important to have some default transports that can be used as the lowest common denominator for interoperability. However Bayeux remains a negotiated multi-transport protocol and I would encourage alternative transports to be developed. If the current client and server implementations are not sufficiently pluggable to allow alternative transports to be easily plugged - then please raise a bug report / feature request and we can work to provide better pluggability.

    However, from my experience of implementing the java server, there is some conflict between be generic, open and pluggable and of providing optimal and efficient implementations of a specific transport. It may be that alternate transports would be better off having differently architected servers rather than fit a new model into a server designed for long polling with continuations. Of course it is still good to re-use as much code as possible and again I say that we are open to suggestions as to how to make the code more reusable.

    I don’t agree that the layers are defacto specified by dojo.js. We have put a lot of effort into the specification document and it is in no way tied to dojo or dojo.js. Sure the spec is probably deficient in many areas - but we are open to suggestions and contribution as to how to make the document better and suitable as the basis of a clean room implementation.

    I know there are non dojo.js based clients that exist. But unfortunately none of them are open source and/or publicly available. I think the best way to push towards the openess that you describe would be for the development of another opensource implementation of Bayeux client.

    So I’m supportive of the vision that you describe and I think that the only thing that stands in the way of Bayeux/Cometd getting there is time and effort.

    Finally, I would again note that the open ajax alliance is considering messaging from a pure API and open standards view point. I like to think that the two approaches are complimentary, with dojo cometd working the pragmatic lets-get-something-out-there side of the problem, with the OAA looking at the more considered design, separation of concerns, etc. etc.

    PS. Bayeux does not follow your three layer model. We have transports and publish subscribe.
    But there is no support for message ordering or guaranteed delivery etc. we see these as optional extension - either to transports or above the messaging layer.

  7. Alex Russell Says:

    hey Jacob,

    So there are a couple of issues here:

    1.) what’s “blessed” WRT transports in the Bayeux spec.
    2.) how coupled are those things to the CometD project, and how much does that have to do with Dojo
    3.) the ability to experiment in the context of Bayeux

    I’ll address them in turn.

    First, only the two simplest forms of transports are addressed in the Bayeux spec, and they are left relatively un-discussed. No state machines are provided (although the long-polling style will probably be required at some point). You are suggesting that because at least one of these forms SHOULD be provided by Bayeux servers, it’s a burden. And that’s true. It’s the lowest-common-denominator for a Bayeux server to be a Bayeux server, and it means that interoperating clients can expect servers to do AT LEAST that much. It’s by no means the end of the story. Experimentation on transports is alive and well inside of CometD in part because Bayeux is so loose beyond specifying long-polling.

    Secondly, as I mentioned in my previous note, I think we should try to come up with a tighter spec for the client APIs, but I also think you clearly mis-understand the architecture of the dojox.cometd system and the role of transport handshake. Bayeux servers advertise what transports they support in the handshake step as a means to allow better transports than the defaults to be evolved. The dojox.cometd system *explicitly encourages* this type of evolution by providing a plugin architecture for new transport types. While only 2 are provided out-of-the-box, you can easily use the baseline logic in the dojox.cometd client to plug in new, exotic transports and have them be negotiated with the server (in the handshake step). Clients aren’t required to do any of this, but we do it in the Dojo client *specifically* to enable the kind of evolution which you seem to think we are preventing. Developing a new transport today means writing something which handles the well-defined places in the client where protocol events are dished out and registering with the AdapterRegistry which is used to select the right plug-in for a particular connection. It could hardly be easier.

    Lastly, you seem to be suggesting that Bayeux and the Dojo client are tightly intertwined. This isn’t the case, although we do need a reference client and so far, because the folks who have volunteered to work on Bayeux also have worked on CometD, that has been the defacto RI. We welcome and encourage you to get involved in Bayeux’s evolution. It’s an open process, an open project, and the RI’s are also as open as we can possibly make them. If you think we’re *not* being as open as we could, I’d love to know how you think we could improve and work with you to make it happen.

    Regards

  8. Comet Daily » Blog Archive » Is Comet Becoming Over-complicated? Says:

    [...] probably more on the way) standardising on one of them just seems to limit choice. I agree with Michael Carter’s recent post that if we are to standardise anything, it should be the API, not the communication [...]

  9. Comet Daily » Blog Archive » Colliding Comets: Battle of the Bayeux Part 4 Says:

    [...] a previous musing I said that “The point of any standardization is to allow interoperation between moving parts [...]

  10. Comet Daily » Blog Archive » Colliding Comets: Battle of the Bayeux Part 7 Says:

    [...] are even possible to standardize. Instead they’ve chosen a communication protocol (note my previous musing that lacks many necessary features, demands the use of publish/subscribe messaging, and will almost [...]

Leave a Reply



Copyright 2014 Comet Daily, LLC. All Rights Reserved