In part 4 of this series on Bayeux, Michael responds to Greg’s rebuttal from part 3.
Part 1: Greg Wilkins explains the need for Bayeux
Part 2: Michael Carter criticizes the current state of Bayeux
Part 3: Greg Wilkins responds to Michael Carter
Andrew Betts’ thoughts (from a related article)
Part 4: Michael Carter responds to Greg Wilkins
Part 5: Kris Zyp’s thoughts
Part 6: Alex Russell responds to Michael Carter
Part 7: Michael Carter responds to Alex Russell
Part 1: Greg Wilkins explains the need for Bayeux
I approached this dialogue initially by imagining an ideal Comet protocol, and then I pointed out how Bayeux differs from that protocol. Greg believes that Bayeux’s only flaw is a lack of documentation, and that if I fully understood Bayeux, I wouldn’t have any problems. While I certainly don’t pretend to possess the same knowledge as the authors of Bayeux, I think that this line of reasoning doesn’t adequately address my concerns, and so I will explain how and why. My first article was somewhat theoretical, as this one will be. But I will also take a look at specific statements from the Bayeux specification and explain how they relate to this discussion.
Framework or Specification
My main point in the section “Cost Without Benefit” in my previous article is that supporting the Bayeux specification costs developer resources, but most of the features of Bayeux are unnecessary for a Broadcast application. Greg replies, “But broadcast does need a very important feature of Bayeux: transport negotiation!” I concur, transport negotiation is very important, and our final standard should contain it. But pointing out one feature of Bayeux that broadcast applications would need isn’t sufficient to justify using the protocol when it is chock full of unnecessary features.
This point on its own might not be enough reason to rework Bayeux, but it represents an alarming trend in the rhetoric coming from Bayeux supporters. I’ve heard many times, as Greg put it, “Bayeux is not a framework, it is a specification.” And it makes good strategic sense to try to write a specification for a standard as opposed to a framework, as frameworks tend to fracture a community whereas standards can do the opposite.
But when defining a protocol, the designers must be exceptionally cautious of feature bloat. Every required piece of functionality is another hurdle to actually implementing the protocol. The first priority of a specification is to explain to developers how it should be implemented, whereas a framework’s first priority is to provide a slew of features in an easily understandable API. Frameworks are for developers-as-users whereas standards are for developers-as-developers. But while the Bayeux supporters want to call it a specification, many end-developers have gotten the impression that they should treat it like a framework.
In a previous musing I said that “The point of any standardization is to allow interoperation between moving parts in a system…[Bayeux] defines the separation between moving parts to occur at the network edges.” Bayeux’s domain, therefore, is those developers who are implementing the communication layer between the browser and the server. As a Comet server developer, I am the implicit target of Bayeux. Application developers on the other hand don’t care about the communication protocol. They only care about a framework—a convenient API with every feature they could want. This is very similar to the way in which web application developers want a framework like Django or Ruby on Rails, but don’t care about specifics of the HTTP communication protocol.
So which is Bayeux, a specification or a framework? As Greg said, they “don’t expect that many developers will implement Bayeux, unless they have some new idea, language, transport etc.” Besides the fact that we wouldn’t want them to re-implement Bayeux just because they thought of a new transport, I doubt a developer could even have some new idea without first fully understanding the specification, probably because they’d implemented either the server or client side previously. The long and short of it is that if a developer, like me, has a problem with Bayeux, the response is that it’s not really meant for a developer like me to implement. This is a dangerous precedent as it limits Bayeux’s developer base basically to the people who designed it, or developers with identical priorities and an identical set of most-likely use-cases.
The Bayeux authors are trying to skip the middle man and cater directly to end-users. Even though Bayeux’s implicit domain has nothing to do with application developers, the designers have gone out of their way to include feature unnecessary for basic HTTP push. These are framework features, and there has been a general understanding that these features mean that Bayeux is some kind of standard for the masses.
I don’t think this is necessarily an isolated phenomenon. In Greg’s article he seems to imply that we should actually treat Bayeux as a framework when he asks, “And how many of us really use all the features of the frameworks that we have at hand?” The answer is that we often don’t use all the features of a framework. But how about this for a contrasting question: And how many of us really use all the features of Internet Protocol? I certainly use each and every feature of Internet Protocol whenever I do anything on the internet, as do you, the reader, just by reading this article. That is because the designers of IP guarded very carefully against feature bloat, promising nothing but the bare essentials necessary for application/protocol designers to efficiently implement any functionality they required on top of IP. So instead of redefining a network stack from the ground up, the HTTP protocol just bases its communication on TCP and IP.
Bayeux is very clearly trying to play the role of a framework even though it is technically a specification. The result is that it’s a bad communication protocol and a bad framework. A good framework, after all, would have long ago published a standard API and worked out a method for third-party, client-side transport discovery. And a good communication protocol wouldn’t utilize publish and subscribe messages as communication primitives while forgetting to include basic reliability and message ordering guarantees. My entire previous article shows in depth exactly why Bayeux is a bad communication protocol, and I have yet to see any evidence to the contrary. It seems that the general mindset when defining Bayeux functionality has tended towards showing that a feature might be desirable in some situation. This is alarmingly similar to process we’d want when designing a framework, but hardly resembles the cold and rigorous approach of cutting down features when designing a communication protocol. If anything my critique is that Bayeux bites off far more than a single communication protocol could chew.
In the section “Cost and More Cost” I outlined a situation where the communication primitives established by Bayeux are actually a burden to support. My argument is that Bayeux lacks a simple peer messaging system, and building one on top of the publish/subscribe infrastructure will be a big performance hit. Greg clearly agrees with me, but offers the /service/* channels as a way around the problem. Unfortunately, there is a misunderstanding in what the server is, potentially because I say “server” where I mean “application server”.
A bit of history is of import: when I first approached Comet-style communication and began work on the Orbited project, I spent some time in the #cometd IRC channel where the authors of Bayeux coordinate their efforts. The major reason I avoided implementing Bayeux at that time was that there is no good provision for application <--> Bayeux server communication, aside from treating the application as a Bayeux client itself. The only other provision was to treat a Bayeux server like a framework and write the application as a hook into the Bayeux server. This required a complete Bayeux server implementation in the language of the developer’s choice, and precluded the use of modern frameworks like TurboGears, Rails, Django, and others. This wasn’t a problem for the designers of Bayeux though, because they were the ones who implemented the original Bayeux servers, and so they were able to hook any functionality they wanted directly into the server.
But for those of us treating Bayeux as a specification for a communication protocol and not a server framework, the only alternative was to have the application connect to the Bayeux server as a client itself. And this is exactly why I didn’t bring up the /service/* channel in my previous article: it only allows messaging between the Bayeux client and the Bayeux Server, not peer messaging between Bayeux clients, such as the application server and the browser. It doesn’t mitigate the problem in the slightest as there is still no way to efficiently send messages from the browser to the application over Bayeux except to implement peer messaging on top of the publish subscribe mechanism offered by Bayeux.
Greg does point out that I “correctly [identified] that there is unacceptable cost in having user specific channels for user specific messages.” But Bayeux’s solution with /service/* only half solves the problem, as there is absolutely no way for an application implemented outside of the Bayeux process to send a message directly to a client. So as-is, Bayeux imposes unacceptable performance costs. Until we have a better tool to solve this problem, applications should be implemented within the Bayeux process, as if the Bayeux server was a framework. I have noticed that Bayeux supporters seem to rarely assume that the application server is separate from the Bayeux server, such as Greg’s suggestion to just use the /service/* channels, because that’s probably just not how they think about the problem. Once again we have the protocol/framework dichotomy where the demands of each impact the other.
Greg isn’t really thinking about performance in the same way I am though. His proof that Bayeux performs well is that even if you don’t use Bayeux’s features, aside from its ability to push data, it only costs a request (or, as Kris points out in comments, several requests) to setup those unused features.
As much as I am interested in counting the pennies and saving some real money on bandwidth over the years, I’m more interested in being able to scale my solution to many concurrent users. My point is that Bayeux offers a publish/subscribe framework that has many challenges to scaling. I believe there are some good approaches to scaling a publish subscribe network, but the constraints in such a system are at least an order of magnitude greater than those in peer-messaging. By implementing peer-messaging on top of Bayeux we need to first accept the constraints of publish/subscribe. This is, simply put, backwards. It would be like implementing a one-time search algorithm in a language that required all arrays to be stored in sorted order. Instead of incurring an O(n) cost for the search, we incur an O(nlogn) cost for the sort and the search, when we didn’t even need a sorted list at all.
An Unrealistic Agenda
One advantage of my proposed layered approach to Comet is that the lower tiers make no provision for browser -> server communication. There are many ways to accomplish this, and existing application developers already use them without problems. I was very clear on this point in my previous article: “The worst thing we can do is use the promise of Comet as a carrot, and the Bayeux standard as a stick, and try to force developers to adopt our way of communication with the HTTP server.”
But still Greg states that “if Bayeux is successful in weaning developers off XHR (and wrappers), of getting them thinking of messaging (instead of request/response)… then Bayeux will have done its job.”
I need to be emphatic here that this is not Bayeux’s job. And if it is Bayeux’s job, then adopting Bayeux is a strategic catastrophe for the Comet community. As I previously said, it puts us directly at odds with “millions of libraries and billions of lines of code geared towards supporting query-string encoded client -> server communication as either a form submission or an XHR request.” In fact, I’d go as far as to say that it puts us at odds with the HTTP specification and all of those who support that standard.
The lack of real-time applications is due to developers not knowing how to push data from the server to the browser. Comet versus request/response is not a dichotomy we want to create, because we’ll lose. A layered approach to Comet would allow some basic push capabilities at a lower level, thus allowing developers to choose whichever upstream communication model they want. But it still would provide a good platform to reform the current messaging paradigm at a higher level. Bayeux could very well use the publish/subscribe model, provided there was a lower-level option for developers who simply were not interested.
In short, it makes strategic sense for the Comet standard to closely resemble and interoperate with current standards. We want to be saying, “But Comet is basically HTTP, not some new, strange standard.” Then we’ll have a much easier time gaining widespread adoption.
The extent of Greg Wilkins’ arguments in his first article is that specifications can solve many problems, and Bayeux is a specification, so therefore Bayeux can solve many problems. He puts a lot of stock in the idea that Bayeux is ideal because it solves the problem of multi-frame Comet operation, also known as the 2 connection limit problem. But the reality is that Bayeux has at best the most minimal support for overcoming this problem. Section 8 of the Bayeux specification describes “Multi frame operation”. All it says is that the Bayeux server or client should set a cookie as a marker that a particular client already has a connection open to a particular server. Then, if a second connection is necessary, one of the two should fall back to polling. Here is the exact content that describes how a client should handle multi-frame operation:
8.2 Client Multi frame handling
It is RECOMMENDED that Bayeux client implementations use client side persistence or cookies to detect multiple instances of Bayeux clients running within the same HTTP client. Once detected, the user MAY be offered the option to disconnect all but one of the clients. It MAY be possible for client implementations to use client side persistence to share a Bayeux client instance.
It’s very possible that the Bayeux authors have some great ideas about this, but those ideas haven’t yet made it into the publicly published standard where I can read it. As-is, the whole section is mostly a big hand-wave. There is no explanation on how to detect connections across domains. What if I’m connected to ten different Comet servers? I can’t think of any “client side persistence” mechanism that would allow clients served from different domains to all store and read data from the same place, much less work in all major browsers. Furthermore, without the exact description of what data should be stored, all of those Bayeux connections would necessarily need to be the same version of the same client. What’s the point of the standard if we all have to use the same client?
I understand this is a hard problem, and even Greg admits at the end of his first article that “the solutions for multi-tab are still very rudimentary”. But his entire first article does seem to be directed towards solving the two connection limit, when in reality he only briefly alludes to how Bayeux solves the problem: he suggests that publish/subscribe and transport negotiation may be helpful. I flatly disagree with the suggestion that publish/subscribe has any bearing on the problem. On the other hand, transport negotiation is a good idea that will allow connections to degrade to polling. This is a definite strength of Bayeux, and I think our final standard should have transport negotiation.
That brings us to the issue of defining transports, or rather the lack of specification in that area. Greg shares with us that
[Bayeux] recognizes that we are not all going to agree on long polling or streaming, or on stopping innovation on new transports. So transport is negotiable in Bayeux.
Bayeux recognizes that we are not all going to agree on the same language on the server or the client.
While HTTP is the predominant transport protocol used on the internet, it is not intended that it will be the only transport for Bayeux. Other transports that support a request/response paradigm may be used. However this document assumes HTTP for reasons of clarity.
Of course, Bayeux does provide an Unconnected Operation for message dispatch:
OPTIONALLY, messages can be sent without a prior handshake… This pattern is often useful when implementing non-browser clients for Bayeux servers. These clients often simply wish to address messages to other clients which the Bayeux server may be servicing, but do not wish to listen for events themselves.
This provision is a false hope though, as it only allows synchronous message dispatch from external servers because it relies on HTTP. A much saner protocol would allow us to dump as many message requests as we wanted without receiving any of the responses. We’d simply put an ID on each request and the server would respond as it was able. Without this provision, throughput would be severely limited. If you opened 10 connections to the Bayeux server from your app, and each one had a round-trip latency of 200ms, the most messages you could dispatch would be 50/second. You can easily perform this calculation once you realize that the round-trip latency directly affects throughput. The number of messages you can deliver each second is just one second divided by the round-trip latency. So with 200 ms latency you can deliver only 5 messages a second.
Bayeux recognizes that not everybody is a JSON fan, and while the content of the Bayeux messages is specified in JSON, there is no requirement that they are sent as JSON, and implementation of XML (e.g. BOSH), binary, or telepathic transports would be welcome.
This statement is very misleading—the extent of what it says is that any data format can be encoded in any other data format. We can invent telepathic transports, but we still have to encode some part of the communication in the JSON format. Yes, the application can encode the payload as an XML document, but it’s still re-encoded in JSON for transport. See section 1.4.2 of the Bayeux standard:
conforming Bayeux servers and clients MUST still conform to the semantics of the JSON messages outlined in this document.
I’ve outlined many reasons in these two articles as to why Bayeux isn’t ready. Standardization is of the utmost importance, and I think the people behind Bayeux have made great strides towards creating something useful. But the supposed goal of Bayeux seems in reality to be that of a framework, not a protocol. If this is the case, then the Bayeux authors would do well to rework their standard into an API proposal. This would provide many more benefits in line with their goals. If, on the other hand, they really do want Bayeux to be a communication protocol, then they should get serious and drop any efforts to target application developers.
Overall though, there has clearly been a lack of perspective in the design process of Bayeux. It is very easy to start optimizing towards a particular use case, and for many projects that is the right and obvious approach. For a standardized protocol though, the designers must be extraordinarily careful that they create a technology that is robust enough to withstand the test of time. If they do not, then a whole new set of de-facto standards will evolve, ultimately displacing the original standard. If we tackle this problem now then we can avoid multiple years of setback.
The only reasonable approach to this problem is layering the protocol. This is not something that can happen on top of or underneath the Bayeux protocol as is. Layering doesn’t mean you adopt a monolithic standard for Jabber over HTTP (see BOSH), and then put another monolithic standard on top of it. This is one idea Greg puts forth, and indeed it is layering, but what’s the point? You get none of the advantages of having flexible, discrete layers as I outlined in my previous post.
Every single concern I’ve brought up can be solved by simply defining multiple standards with different domains. Bayeux is a perfectly reasonable high-level protocol or framework API, provided we jettison all the specific communication mechanisms it defines. After all, splitting the framework-like features of Bayeux away from the communication protocol features is the only way that Bayeux could possibly satisfy Comet server developers (like me) and application developers who want to use Comet. The crux of the issue is actually pretty simple, as I discuss in Should Bayeux be a Communication protocol?. Pay close attention though, because I don’t actually believe we should have only a protocol, or only an API standard. Rather, Bayeux is trying to look like both, but it should make a choice and leave room for the other. Given Bayeux’s feature set and the general philosophy of its creators, I suspect its place is that of an API standard.