Colliding Comets: Battle of the Bayeux Part 6

by Alex RussellMarch 26th, 2008

Alex Russell, one of the writers of the Bayeux protocol, says that Michael’s complaints are fundamentally flawed.

Part 1: Greg Wilkins explains the need for Bayeux
Part 2: Michael Carter criticizes the current state of Bayeux
Part 3: Greg Wilkins responds to Michael Carter
Andrew Betts’ thoughts (from a related article)
Part 4: Michael Carter responds to Greg Wilkins
Part 5: Kris Zyp’s thoughts
Part 6: Alex Russell responds to Michael Carter
Part 7: Michael Carter responds to Alex Russell

Vaguely Correct vs. Precisely Wrong

My attention was drawn today to re-reading the Battle of Bayeux series again, and as I re-read Michael’s response to Greg which said that Bayeux has an “unrealistic agenda”, I couldn’t help but feel that Michael (and many others I’ve talked to) have simply missed the entire point of Bayeux, which is to specify enough structure to grow the number of interoperable Comet clients and servers while leaving enough to the implementations to allow integration of piecemeal evolution until such a point as the protocol itself is unnecessary.

Reading many of Michael’s complaints about Bayeux, I’m struck by his ardent desire for the protocol he’s implementing to be the right one. Or rather, for the description to be so air-tight as to enable full automated testability, interoperation, and even the integration of concerns such as URL mapping and eliding handshakes as first-order priorities for any “worthy” spec. Perhaps if the authors-to-date who have contributed to Bayeux could count on the world not changing, we might agree that Bayeux should define client APIs or that it should get super-specific about how to handle multi-frame. But why? And why right now? And who is that good for?

Let’s assume for a moment that Bayeux specified a multi-frame strategy. Today, given the limits of browsers, any such specification will be writing into stone what is essentially a series of browser-specific hacks. Indeed, while the entire Bayeux spec is a reaction to browser deficiencies, it seeks to avoid prematurely writing those bugs into stone tablets. Today, given the limits of browsers, any such specification will be blessing what is essentially a series of browser-specific hacks. Indeed, while the entire Bayeux spec is a reaction to browser deficiencies, it seeks to avoid prematurely standardizing hacks. Had we fully specified a multi-frame solution when we started work on Bayeux clients and servers, the demands of terseness and clarity would have demanded that we write into stone an “IPC over cookies” protocol which uses timers in cooperating frames to send/receive packets of encoded data which is re-constructed and re-evaluated on the other side. Today, the “duh” thing to do would simply be to use the HTML 5 DOM Storage provision and notify library authors to hack around it for old browsers for the time being. In another 18 months, the entire point will likely be moot as browser vendors get wise to Comet and provide us with ways around the 2-connection limit at an HTTP level. When that happens, multi-frame concerns are just legacy features. The point here isn’t that one solution is better. It’s that 18 months ago, we didn’t know the future, but were sure enough of our own fallibility to avoid wild goose chases for what is, essentially, a client concern. The “right” solution has changed in the last 18 months. More to the point, how multiple Bayeux clients in a single process communicate to a server has zero bearing on the semantics of the messages which Bayeux exchanges. Addressability amongst peering tabs is, from one view, purely a client issue which clients should be free to experiment and iterate on. Indeed, much of Michael’s argument seems to be predicated on the supposition that he can do things better. Of course he can! That’s the whole point!

The furthest down the path of picking sides that the protocol currently goes is to specify that servers MUST implement at least the long-polling transport, and leaves the API which is exposed on either end to implementers. Clients and servers may then implement other transports and negotiate which one to use via the specified mechanism (although this can also be elided without violating the spec). This is entirely by design, and we expect that there will be revisions and addendum’s which outline recommended APIs once enough clients and servers exist to know what the right thing to do is. The long-polling transport is the small, critical, core of network-level semantics that when combined with client-level semantics can serve to guarantee interoperability. Not only do we expect that the long-polling transport will be the worst performing transport for a given combination of client and server, we also anticipate that it will be the least likely to be used in large-scale production sites. But you don’t have to care about that if you’re just adding Comet to a small blog….all you want to care about is that your server and your client can talk to each other. It is this entry-level sector of the market which Bayeux is aimed at enabling, while still providing enough play in the joints to enable those with different requirements to handle them within the loose framework of the spec. This isn’t not doing the hard work, it’s being smart enough to know that the hard work we do today will not survive long should we over-adapt to a particular environment only to find that is changing underneath us.

As with many kinds of engineering, the best performance requires the greatest adaptation to the current environmental constraints. Bayeux seeks to enable that evolution to local constraints without over-burdening the market with fundamentally incompatible clients and servers over the long haul. Where Bayeux “waves its hands”, it does so for good reasons; ones which we feel outweigh the short-term costs (which Michael is quick to enumerate).

Bayeux, as a spec, is built with the understanding that the environment is changing, and that competition amongst implementations is good only insofar as it doesn’t cause huge amounts of lock-in. By specifying a thin layer of semantics about messages and transports and by building extension and plug-in points directly into the protocol, Bayeux has lead to successful evolution already. From cross-domain long-polling using JSON-P to Flex clients which speak optimized wire formats to a profusion of interoperable servers in many languages, Bayeux is succeeding by leaving enough play in the joints to meet reality, today’s and tomorrow’s, half way and then letting implementations figure out the rest.

There’s a long way to go in making Bayeux better, and we know we’re not done yet, but let’s be clear that by omitting things from the core of the spec, we are not calling them unimportant, just unsettled and not absolutely required for interoperability.

5 Responses to “Colliding Comets: Battle of the Bayeux Part 6”

  1. Marcus Cavanaugh » Blog Archive » Bayeux: Tightly Coupled Says:

    [...] the “Battle of the Bayeux” Part 6, Alex Russell [...]

  2. Marcus Cavanaugh Says:

    I posted a longer reflection about this on my own site, but I’ll summarize by saying this:

    Bayeux developers have done the Comet world a great favor by opening the door to efforts of interoperability and standardization. Bayeux has paved the way to widespread Comet adoption by creating a standard that several Comet servers have begun to implement.

    However, I do believe there are some fundamental issues with the Bayeux protocol (again, summarizing here), primarily that Bayeux is tightly coupled. Publish/subscribe would be much better separated out from the comet-transport/comet-connection aspects of the protocol. By incorporating publish/subscribe at such a core level, it impacts application and server developers substantially.

    I don’t think we should consider Bayeux an “intermediate” solution to the problem domain. If Bayeux becomes more widespread, it will only become *more* difficult to address issues with the specification, especially if implementations begin to differ in interpreting the specification.

    Bayeux is already implemented by several comet servers, but it seems we all understand and agree that Bayeux isn’t completely ideal as a protocol. But if we don’t address these issues soon, it will be too late because Bayeux will be too widespread to effect much change.

    I think loose coupling and layering, in the case of a protocol like Bayeux, would be a tremendous gain. By placing publish/subscribe in the core architecture of Bayeux, the application behavior becomes irrevocably mixed with the transport and Comet layer. A layered architecture would alleviate those concerns.

    Whether or not Bayeux changes, Comet will continue to grow. But if there’s a better solution out there, either through modification of Bayeux or in a new protocol, I think it’s well worth investigating these concerns in order to create a more ideal solution.

    “Battle of the Bayeux” should not be considered as “us vs. them”; rather, it should be an opportunity to carefully investigate the potential problems Bayeux has and consider how we should best proceed. In my opinion, Bayeux has design issues which need to be solved; not everyone may share those concerns. I just hope — for our sake — that we can be open-minded enough to consider the implications of allowing Bayeux to become incorporated into most Comet servers without solving some of these substantial architectural concerns.

    Layered architectures and loose coupling can solve a lot of these issues. As it stands, Bayeux can’t address them without some substantial refactoring.

    Comet is a new world; we’re all searching for the best way to approach Comet. In the case of Bayeux, I am wary of situations where something becomes a de-facto standard before it has had time (and/or the opportunity) to address its own concerns. A loosely-defined spec runs the risk of becoming ambiguous, and if that spec becomes popular enough, you’re more likely to have uninteroperable implementations — the same problem the spec was trying to solve in the first place.

  3. Martin Tyler Says:

    Interesting points, and I agree with your general message.

    I am not completely convinced that pub/sub cannot be used in some of the use cases people think its not suitable however. Your example of a card game with the dealer/server sending private messages to clients can be done with pub/sub as you point out with the client subscribing to a subject specific to them. This means you are making use of the libraries on the client side to manage multiplexing to callbacks etc.. rather than having to deal with a single ‘receiveMessage’ callback and handling all the dispatch in user code. What bayeux (or bayeux server implementations?) are missing here is better support for private channels.. as you say, with auth and the ability to easily send to these channels in a server API etc.

    In Liberator we use pub/sub and it is used extensively for Trading applications where trades are clearly private messages. Liberator has the ability to map subscriptions to private subjects so people cannot subscribe to other users trade channels - it also has auth/permissioning to control access.

    I agree that a layered approach can have advantages, but i dont believe that pub/sub is actually an issue for many use cases, it still gives you desirable functionality on the client side - the server side just needs better support for these different use cases.

  4. Kris Zyp Says:

    I agree that pub/sub is not the universal answer, and that better transport protocols are needed, but I don’t believe that this is a strike against Bayeux. As I articulated in my article: widespread adoption does not limit our flexibility in evolving better transport protocols. Rather, Bayeux is designed to provide a default set of mechanisms for connecting to a Comet server and participating in meaningful communication (and pub/sub makes a decent common paradigm). However, Bayeux is designed to encourage new transports. A Bayeux compliant server can provide the default transports and service negotiations, but a more advanced futuristic client can use better transports without having actually to use any Bayeux defined transport protocols in any given conversation. Bayeux is designed for us to grow out of it, and that is the beauty of it.

  5. Jacob Rus Says:

    Alex: the point you are making here is exactly the one that Michael’s criticisms are making, which is that a specification cannot be too rigid or conflate too many concerns. The only thing is, Bayeux as it stands is on the wrong side of those distinctions. You and Greg are making exactly opposite arguments about where a spec should stand on these points, and as far as I can tell, neither of you has really carefully read Michael’s suggestions, which are carefully constructed to be both more flexible and more interoperable than Bayeux. A back and forth where one side completely ignores all of the other’s direct answers and criticisms is overall pretty fruitless. I think if you could sit down together for a couple of hours, Michael or someone who has read his proposals could help you understand their construction, and explain the reasoning behind it, and the whole thing would be sorted out quite quickly.

Copyright 2015 Comet Daily, LLC. All Rights Reserved