I will be speaking at the upcoming Rich Web Experience conferences in Washington D.C. and San Jose and at the Ajax Experience in Boston on several topics that may be of interest to Comet developers:

• JSON SOA-Based Client/Server Application Development - This session will cover various open protocols that can be used as a basis for service oriented web architecture including Comet protocols Bayeux and HTTP/REST Channels, and how to use these in Dojo.
• Applied JSON: HTTP REST, Ajax databases and beyond - This session will cover using REST storage/database servers, and will cover and demonstrate using Persevere’s RESTful data notification Comet capabilities.
• I will be giving two additional topics at the Rich Web Experience that are less Comet-related but should still be interesting for general Ajax development: Secure Mashups and Persistent Computing on the Web.

I hope to see you there!

HTTP Channels is implemented in Dojo 1.2 with the HttpChannels module. HTTP Channels is intended to provide data synchronization and update notifications by leveraging the ubiquitous HTTP REST semantics. Within Dojo, HTTP Channels are particularly useful, integrating directly with Dojo Data for live data synchronization. The HttpChannels module is very flexible as well; it can be used in several different ways:

Live REST Updates

The easiest and highest level use of HTTP Channels is in conjunction with the JsonRestStore. HttpChannels can be used as an enhancement for the REST service, to enable “live” data (real-time updates that directly alter the data in caches/indexes). To add enable live updates through HTTP Channels messages to the data in JsonRestStore, simply add HttpChannels after loading the JsonRestStore:

dojo.require("dojox.data.JsonRestStore");
dojo.require("dojox.cometd.HttpChannels");

Now, we can create and use a JsonRestStore exactly the same way as you would without HttpChannels:

var myStore = new dojox.data.JsonRestStore({target:"/myTable"});

This is all that is needed on the client side (of course on the server side you need a HTTP Channels capable storage system, like Persevere). All resource/data retrievals made by the JsonRestStore will automatically trigger subscriptions to the resources, and data updates that are sent from the server will automatically trigger the appropriate Dojo Data notifications for updates in widgets.

This means that building an application with Dojo that can load data from a server and receive updates on data changes and update the UI with these changes can be as simple as plugging a JsonRestStore data store into a widget like a grid, and letting Dojo handle all the communication and notification routing. Plugging a “live” data store into a grid can be done like:

<table dojoType="dojox.grid.DataGrid" store="myStore" ...

We can see a very simple example of using a live data store with a grid widget.

Cometd Transport

HTTP Channels is a Bayeux compliant transport, and the HttpChannels module can also be used as a Cometd transport. Simply load HttpChannels after Cometd and it will auto-register itself as a Cometd transport:

dojo.require("dojox.cometd");
dojo.require("dojox.cometd.HttpChannels");
// initialize cometd
dojox.cometd.init("/cometd");
// subscribe to a channel
dojox.cometd.subscribe("/weather/slc", function(){ ... });
// publish to a channel
dojox.cometd.publish("/news/slc", {title:"Hot Weather"});

By loading the HttpChannels module after Cometd, HttpChannels will register itself as an available transport. If the server advertises the ability to handle the “http-channels” transport, the Cometd module will use HTTP Channels for handling subscriptions and publishing messages. It is important to note that HTTP Channels integrates pub-sub messaging with REST resource locators, and therefore publishing messages and subscribing are executing by HTTP requests directly to the target path. With an HTTP Channels compliant server, the example above would result in HEAD request with a subscription header to the relative URL “/weather/slc”, and POST request to the relative URL “/news/slc”.

Standalone Protocol

To use the HttpChannels module standalone, you first get an instance of HttpChannels, the default instance (which is used with the REST services and Cometd by default) can be retrieved:

var httpChannels = dojox.cometd.HttpChannels.defaultInstance;

You can now directly access the HttpChannels API. The primary HttpChannels functions are get, subscribe, unsubscribe, and receive. The subscribe function allows you to subscribe to a resource and provide a callback for notifications. The get function also does a subscribe, but it gets the target resource at the same time as sending a subscription request. The receive function is called whenever data is received from the server. You can use dojo.connect to listen for all messages from the server. For example:

dojo.require("dojox.cometd.HttpChannels");
var httpChannels = dojox.cometd.HttpChannels.defaultInstance;
 
// get the current weather and subscribe to future changes
httpChannels.get("/weather/slc", function(weatherReport){ 
	// this is called for the initial response and all updates
}); 
dojo.connect(httpChannels,"receive",function(message){
	// this is called for all notifications from the server
});

Dojo’s new HttpChannels module provides an efficient implementation of HTTP Channels with strong integration with other modules in Dojo including Cometd and JsonRestStore. HttpChannels can be used for rapidly developing data-centric applications that rely on real-time updates; Dojo provides all the facilities for creating a Comet connection with the server, handling server-sent messages, appropriately updating data, delivering notifications through data stores, and updating widget user interfaces in response to these notifications.

The HTTP Channels introduction primarily defined the protocol, but it may be worth expounding the the goals and motivation for HTTP Channels:

• The primary purpose of HTTP Channels is to provide an application-level Comet protocol for live data synchronization that leverages the widely understood semantics and vocabulary of HTTP/REST. By providing such a protocol for data synchronization semantics, servers and databases can provide real-time access to their data stores, and generic client frameworks can provide data update handling without requiring a custom protocol and custom event handling.
• HTTP Channels is intended to be usable as a Bayeux transport, leveraging Bayeux for service negotiation, and providing compatible capabilities.
• HTTP Channels provides the means for PubSub interaction through defined channels/resources, such that it can properly interact with Bayeux agents. However, HTTP Channels is foremost designed towards application level data synchronization, it is not optimized for PubSub, and the data model may impose restrictions on what channels you may publish messages through for some environments.
• HTTP Channels is intended to work with existing HTTP communications. In a data synchronization model, retrieving and subscribing to data is almost always done together. HTTP Channels allows you to retrieve resources with standard GET requests, and augment those requests with subscription requests, so that retrieve and subscribe requests can be done efficiently with a single HTTP request. HTTP Channels integrates with HTTP rather than layering on top of HTTP. This improves efficiency in many situations and also gracefully degrades for servers that do not have Comet/subscription capabilities. GET requests are still standard HTTP requests, and legacy servers can still handle them correctly (there just won’t be any future Comet notifications of changes to the resources).
• HTTP Channels supports a unified approach to RPC, REST, and PubSub interaction. This allows entities to interact in all three realms without disparate infrastructure.

HTTP Channels may not be perfect for every Comet situation, some applications may require a custom protocol for maximum efficiency. However, HTTP Channels should be well-suited for applications that are focused on displaying live, real-time data, and wish to utilize existing standards (HTTP and Bayeux) for interoperability, while maintaining the flexibility of handling multiple paradigms of messaging.

HTTP Channels is a transport protocol based on the existing messaging capabilities of HTTP and utilizes the concepts of REST for notification of resource changes. HTTP Channels is intended be a Comet solution for RESTful real time data synchronization architecture and can be used as a Bayeux transport, or it can be used stand-alone. HTTP Channels is designed to provide a mechanism for efficient and standards-based retrieval and monitoring of data and for receiving notification of changes to data with meaningful semantics.

HTTP Channels brings strong coherency and interoperability with publish and subscribe architectures by equating the PubSub concept of a topic or channel with the REST concept of a resource locator. This allows HTTP Channels to superset the capabilities of PubSub messaging (consequently it can operate as a Bayeux transport), allowing resources to be conduits of messages as well as mutating data entities. This transport is based on the goals and the refinement of the standards based approach to Comet communication with REST, but with improvements based on suggestions, further research, and an actual implementation in Persevere. HTTP Channels can also use HTTP tunneling in conjunction with XHR streaming where available (Firefox, Safari, and Internet Explorer 8’s XDR) for greater efficiency than long-polling. HTTP Channels provides duplex client/server communication and leverages the broadly understood and interoperable semantics of HTTP.

HTTP Channels is conceptually based around the concept of a full duplex HTTP. However, full duplex HTTP is not possible within the constraints of the browser because of a lack of sufficient control of connections and pipelining. Therefore, the Channels transport also includes the necessary mechanism to emulate a single connection. In order to aggregate requests into the conceptual Channels connection, requests should include a client id. This header can be created by the client and should be a sufficiently randomized string to avoid collisions.

Channels Server Listening Connection

To initiate the long-lived server listening response which provides the means for the server to send messages, a POST request should be made to the /channels URL on the target server with an X-Create-Client-Id header with the client id for the value. Subsequent subscription requests should include an X-Client-Id header with the client id in order to associate the subscriptions with the Channels server listening response. When the client has received a server response/message, the client should immediately resume the connection by making another POST request to the /channels URL again (long-polling). On these subsequent requests, the request should include an X-Client-Id header (not X-Create-Client-Id) with the client id. If the client id is no longer valid (if there was an excessive amount of time between requests, for example), than the server should respond with a 404 status. The client may then reissue the channels request using the X-Create-Client-Id header. At this point the client should assume a new connection, and any subscriptions that should be resumed must be reissued to the server (and the client may want to use the X-Subscription-Since header to resume since the last response, which is discussed below).

The server sent notification message should include a Content-Location header with the URI indicating the resource with which they are notifying about. The message should also include an X-Event header. This header indicates what event triggered this notification. When the notification is the result of a request sent to a header, the event should correspond to the name of the method of the request that triggers the notification. If a PUT request was used to update a resource that triggered a notification, X-Event header should have a value of PUT.

Subscription Requests

HTTP Channels uses an HTTP GET or HEAD requests to establish a subscription. To request a resource and monitor in a single request, you can use a GET request to retrieve a resource and subscribe to it in a single request. To subscribe to a resource without retrieving the current resource, you can use a HEAD request. Subscriptions are made by including an X-Subscribe header. The following are valid values for X-Subscribe:

• * - Subscribes to the resource indicated by the request URI. Only * is allowed for now, but in the future we could allow a comma delimited list of event names, for granular subscriptions.
• none - Unsubscribes the resource

Subscriptions should include an X-Client-Id header to properly correlate the subscription with the channels server listening response.

Retroactive subscriptions can also be made by including an X-Subscribe-Since header. The value of the header must be a date according the HTTP specification of a date format,and it must be in the past. Generally this header should be used when a connection is dropped, and the value should correspond with the Last-Modified date on the last notification or response received over the Comet connection.

Subscriptions should include a Cache-Control header with a value of max-age=0. This header will indicate that the request must be delivered to the origin server for revalidation. This will ensure that the subscription reaches the server. Conditional validation can still be used such that if the client has a cached copy of the resource, the server does not need to send the content (it can send a 304 status).

When a server returns a response to a subscription requests, the response should include a X-Subscribed header with a value of OK if it was successful, and an error message if it was not successful.

Here is a diagram of HTTP Channels interaction, where one client subscribes to a couple resources and listens for notification of changes while another client modifies data that triggers notifications:

Message Ordering

Subscription requests as well as resource modifying requests may optionally include a X-Seq-Id header to ensure deterministic message ordering. Each request should increment providing a value one higher than the previous request. Servers can then reorder requests for proper message ordering if they arrive out of order.

Examples

Here is an example subscription request and response:

GET /customers/ HTTP/1.1
Accept: */*
X-Subscribe: *
Cache-Control: max-age=0
X-Seq-Id: 1
X-Client-Id: 867208037610061686690752</code>
 
 
HTTP/1.1 200 OK
Content-Type: application/json
Last-Modified: Wed, 16 Apr 2008 21:47:44.906 MDT
X-Subscribed: OK
Content-Length: 233
 
...content...

Here is an example of the Channels server listening request and response:

POST /channels HTTP/1.1
Accept: application/http
X-Create-Client-Id: 867208037610061686690752

The response should remain open while waiting for notifications from the server. Now suppose that someone issued a POST request to the /customers/ resource (which this connection is subscribed to). The following notification would appear in the content stream for the channels response:

HTTP/1.1 200 OK
Content-Type: application/json
X-Event: POST
Content-Location: /customers/
Content-Length: 40
 
{name:"new customer"}

Tunneling on HTTP Channels

Ideally, an HTTP client can implement HTTP Channels and fully interact with a Channels server, sending subscription requests and message/modifications requests, and receiving notifications all over a single TCP connection. A single HTTP response remains open and can stream messages from the server. Full asynchronous bi-directional messaging freedom can be achieved by using pipelining, requests can be sent by the client without waiting for the response to finish. This eliminates the usual request/response order of operation typically used by HTTP because the requests will have no corresponding response, a single long-lived response eliminates the need to provide a response, without violating the HTTP protocol. Both requests from the client and messages from the server can be sent at any time using this single TCP connection. Duplex communication on a single TCP connection should certainly be a primary goal for Comet, and this transport is designed for that future.

The ideal of full duplex HTTP on a single TCP connection is not realizable with current browser technology. JavaScript does not have control of pipelining, and in many situations a streaming response is not possible over XHR, so long-polling must be used instead. However, where streaming is often possible, and when streaming is available, HTTP tunneling can be utilized to frame messages and improve performance by eliminating the need for new requests after each server message.

It is recommended that the server use the User-Agent header to determine if the client supports XHR streaming. In addition, it is recommended that the server check for a XDomainRequest header. If it is present, include a XDomainRequestAllowed header with a value of 1, so that the client can utilize XDomainRequest if available (this is the only fully accessible streaming mechanism in Internet Explorer 8).

The Channels server listening request should specify an Accept header which includes application/http as a desired content type. The response should be of the content type application/http, and the response content should consist of a stream of HTTP messages. This is in essence HTTP tunneling. The response is composed of individual HTTP messages in HTTP response format. Each of these responses is a server sent notification. If the client does not support XHR streaming, or if the path to the client includes proxy servers, every message from the server should be the actual server response message itself. The following is an example of HTTP tunneling:

POST /channels HTTP/1.1
Accept: application/http
X-Create-Client-Id: 867208037610061686690752
 
HTTP/1.1 200 OK
Content-Type: application/http
Transfer-Encoding: chunked

The response will remain open, but the server messages will appear in the content stream of the container response above.

HTTP/1.1 200 OK
Content-Type: application/json
X-Event: POST
Content-Location: /customers/
Content-Length: 40
 
{name:"new customer"}

Note that if a browser did support pipelining control, duplex pipelined requests do not have responses, making plain subscription requests (using HEAD) unnecessary. In a duplex pipelined request situation, if a GET request is made and the server needs to deliver a response, it should be sent as a tunneled response message in the application/http message stream with an X-Event value of GET. Pipelined HEAD requests for subscriptions do not necessitate a response unless they resulted in an error.

PubSub on HTTP Channels

HTTP Channels is capable of PubSub messaging. Resources act as topics, and messages can be posted to resources, with the subscription system routing the messages to listeners. When HTTP Channels is used as a Dojo Cometd transport, subscriptions are sent as subscription HEAD requests as described above and publishing messages is done by doing a POST to the target topic/resource. Messages are then routed to clients as notifications with an X-Event of POST. By using resources as topics, a simple PubSub system can be evolved into more advanced resource modification and RPC subscription systems.

Looking Ahead

JSON Channels is another forthcoming transport which is intended to have a direct mapping with HTTP Channels. JSON Channels is intended to be an extension of JSON-RPC with subscription and resource/topic locator capabilities. JSON Channels will be easier to implement in many environments since it does not require HTTP parsing, however it is intended to simply be an alternate format that can work within the same semantic structure of operation. Both HTTP Channels and JSON Channels could be used interchangeably, clients could request that the server sent message stream be JSON messages, but still issue subscription requests with HTTP style GET/HEAD requests.

Design Rationale

Most of the rationale for this transport can be found in my Comet Daily articles A Standards Based Approach to Comet and A Taxonomy of Comet. However, I wanted to note the reasons for changes in the “Standards-Based approach”:

• Content type of application/http for tunneling - Originally it was multipart/digest. application/http is the correct MIME type for multiple/pipelined HTTP messages.
• X-Subscription and X-Subscription-Since instead of When-Modified-After - This is simply to improve the clarity and explicitness of the messages. The When-Modified-After header would require too much overloading of functionality. HEAD requests were also added to make it easy to make a subscription without retrieving the resource.
• Dropping use of Ranges + 206 Partial Content - This was intended to provide a mechanism for reliable messaging with PubSub, but this is difficult and complicated to implement, and with custom methods may be inefficient and confusing. It is much easier to simply allow for the concept of retroactive subscriptions.
• Use of Client Id/Virtual Connections instead of HTTP tunneled requests - This makes it easier to add subscriptions and does not require the client to manually create HTTP messages for tunneling. HTTP tunneled requests may still be valuable for batching, but I have not implemented that.

Implementation

HTTP Channels has been implemented in Persevere, providing an open protocol for real-time notifications of changes to data in the Persevere’s storage system for a data centric architecture for message routing. Persevere also serves as a reference implementation for learning to use the protocol. A Dojo-based implementation of HTTP Channels is also coming, which can both plug into Dojo’s cometd system and provide real time updates to a Dojo Data store.

The latest release of the Persevere Server (0.9.5) now includes Comet support. The Persevere Server is a web accessible database and service side JavaScript server that exposes persistent data storage as RESTful HTTP/JSON web services, which can be queried through JSONPath, and supports distributed method calls through JSON-RPC.

Various underlying sources such as database tables, XML files, web services, and object repositories can plug into Persevere, and such persisted graphs can even span domains. With Comet support, it is now possible for clients to subscribe to persisted objects/resources and be notified of changes through a Comet connection. This means that a client can not only directly access persisted data from the server, it can maintain a live synchronization with the persisted data.

Persevere uses a new HTTP Channels transport protocol I have been developing (based on discussions) which can be used with Bayeux. Persevere supports Bayeux service negotiation, and eventually full Bayeux support through integration with Jetty’s Cometd implementation.

Demonstration

View a demo of a live data grid.

This grid can be opened from multiple browsers/locations. When a change is made and saved by one user, that change will notify other clients that have the grid open, and their grids will automatically be updated.

This demo is using a forthcoming Dojo implementation of HTTP Channels connecting to the Persevere server (running on Jetty). HTTP Channels integrates with the new Dojo REST store, and notifications from the server automatically trigger data store notifications and visual grid updates without requiring any extra coding. I will discuss and demonstrate this new experimental Dojo module in a subsequent article, but for now it is included as a demonstration with the Persevere download.

Interacting with Persevere is straightforward. Normally if you want to retrieve a persisted object/resource, you can simply do a request to the corresponding URL; you could request:

GET /Customer/2

And Persevere would respond with a JSON representation of that object. Now with Comet/subscription support, you can include an extra header X-Subscribe to indicate that you not only want the object, but you want to be notified of any changes to that object:

GET /Customer/2
X-Subscribe: *

Persevere will still respond with a JSON representation of the object, but you can also maintain a Comet connection (with a separate request) that will receive notifications of changes to any of these subscribed objects. If someone else makes a modification, for example:

PUT /Customer/2
 
{"name":"Changed Name",...}

This PUT will provide new values for that object, and a notification will be triggered and sent to the subscriber(s). The notification will include the path/id for the object that was modified (/Customer/2), the new data that replaced the object, and the event that triggered the change (PUT).

With Persevere, a Comet connection is made by connecting to /channels. When an monitored object is modified, Persevere will send a message/response on the Comet connection like:

HTTP/1.1 200 OK
X-Event: PUT
Content-Location: /Customer/2
 
{"name":"Changed Name",...}

Please see the coming HTTP Channels proposal for more detailed information about the format of messages and how connections are correlated. Persevere also supports deterministic message ordering as defined in the HTTP Channels proposal as well.

Users can also send POST requests to add new items to a table of objects, or send DELETE requests to delete objects. These modifications will also result in notifications as well. In addition, you can subscribe to a query, in which case you are implicitly subscribing to all the objects returned in the query. The grid example uses this technique. A single query request (with the subscription header) is made for all the customers in a list, in order to populate the grid. When any object that is returned from the query is modified elsewhere, a notification is sent and the grid can be updated. Notifications will also be sent when new objects are added or objects are deleted. Clients can use these notifications to track changes and maintain synchronized data.

With the addition of a Comet support Persevere can be used not only as RESTful persistent object storage solution, but also for web applications that need to provide a live view of frequently changing data.

In the comments of my proposal for native Comet support in browsers, Stu Charlton pointed me to an excellent paper on extending REST [23 MB pdf] to support subscriptions, routing, and decentralization. The author of the paper is Dr. Rohit Khare, a co-founder of KnowNow and certainly one of the pioneers of Comet. The paper provides a great theoretical foundation for our efforts towards more powerful Comet architectures, and highlights the power of leveraging HTTP capabilities in advancing two-way communication [powerpoint].

Pub/sub is a popular architecture for Comet development. Simple applications may only need to deal with a single pub/sub broker (also known as a hub, which can run on a client or a server), which gives a single point for subscribing to topics, publishing messages to topics, and receiving notifications about topics. However, more complex applications may involve multiple pub/sub brokers. This means that rather than having a single domain or namespace for topic names, there may be multiple domains to consider. Multiple brokers may exist when Comet applications connect to multiple pub/sub servers, when both the client and server have a broker, when multiple frames each of have a broker and they can communicate with each other (through FIM or postMessage), or even when different portions of a single client application have their own brokers. Rather than having a single universe of topics, there are different domains that each have their own topics, and these topics may even have the same topic name in a different domain. Brokers may act as proxies to each other, and when these brokers interact with each other, it is important to track both the topic name and the broker to which it belongs. Multiple brokers can be critical for optimum performance and routing of messages. Different modules within a single page may communicate with each other through pub/sub messaging, and routing all such messages through a Comet server would be absurd.

A naive approach would be to try to replicate all topics as is, across all brokers involved. This is ignores all principles of locality, is inefficient, and is terribly unmanageable. It is the pub/sub equivalent of using global variables for programming.

Broker Alias Path Prefixing

The first reasonable technique for broker interaction is to import broker A into broker B where all broker A topics are treated as sub-topics, with path prefix. For example if we have a Utah broker, and it wants to interact with the California broker, it would treat all the topics from the California broker as topics that start with an arbitrary chosen path. In the figure below, all the California topics are prefixed with “/CA”. The Utah broker can now route all subscription requests and publish events for /CA/** to the California broker. The Utah Broker is than acting as a client/subscriber of the California broker, and when it broadcasts messages of interest to the Utah broker, the Utah broker can then translate these broadcasts to its corresponding sub-topics for the Utah broker subscribers. The Utah broker has then incorporated the California topics into its topics, and can act as a proxy for clients that only can access the Utah broker.

This integration has not required any cooperation on the part of the California broker—the integration takes place completely on the Utah broker side. This does not require that the California broker use the /CA prefix for all its topics internally. Mandating that all brokers uniformly use a prefix internally and externally is a fragile attempt at centrally driven scheme. It does not utilize locality and makes adding new brokers very risky.

In order to achieve full transparent proxying, the Utah broker must forward publish and subscription requests. Brokers can generally forward publish requests by globbed subscriptions requests. However, brokers must be capable of routing subscription requests. This is not a traditional functionality of pub/sub brokers. In this case, importing the California topics necessitates informing the Utah broker that it needs to route subscription requests for /CA/** to the California broker (or an intermediary that can deliver the request).

Dojo uses this approach to integrate Bayeux servers/brokers into the central Dojo broker on the client. However, it doesn’t do full proxying: subscriptions and publications are not routed through the central broker, and instead are made directly against the bridge (the cometd handler). By default all Bayeux messages are imported into the /cometd/** topic space in the Dojo broker. A recent patch allows multiple Bayeux servers to be incorporated into the Dojo broker with different names.

Peered Broker Relationship

The integration just described is completely asymmetrical. The California broker has no awareness of the Utah broker. Subscriptions and event publishing can only flow from the Utah broker to the California broker. Event broadcasting can only flow from the California broker to the Utah broker. Of course, peer-to-peer symmetry can be achieved if the California broker simply imports the Utah broker topics in the same manner. Full pub/sub interaction can be performed in both directions between these two brokers. This symmetrical relationship does require both brokers to import each other’s topics.

URL Paths

Importing foreign topics with path prefixing has a couple of problems. First, the path prefixes encroach on the domain of topics for the integrating broker. That is, if the California topics are imported as /CA/** topics, they may collide with existing topics that already have that namespace. Second, path prefixing creates a tight coupling between the client’s use of topic names and the name selection for importing the foreign topics. In order to subscribe to topics from the California broker with the Utah broker as a proxy, the client must know what the Utah broker will use as a prefix for the California broker (in this case /CA). The client may already know the destination broker, but it also needs to know the routing information. If the prefix for routing changes, the client must be changed as well. If another proxy is added between Utah and California, the prefix may grow longer.

In order to solve this problem, we can treat topics as not only paths, but as URLs. If the California broker is hosted at hub.california.com, the topics may have corresponding URLs where they could be reached, like http://hub.california.com/weather and http://hub.california.com/sports. Now if a module knows that it wants to subscribe to http://hub.california.com/weather, it does not need to know anything about the path to get there. It only needs to know the absolute topic URL. The broker is now responsible for routing the subscriptions and other messages. How this happens, whether it’s through a single proxy or multiple proxies or whether the broker is the final recipient, is all implementation details that the subscribing module does not need to be concerned about. This model is simply based on the model of the web, applied to pub/sub architecture, which has been developed so that resources can be accessed by URLs without knowledge of the routing required to get the resources. Also, plain paths for local topics can still be used (which are analogous to relative URLs). The different syntax for URLs and relative paths precludes collisions.

Path prefixing and URL topics provide two means for integrating multiple pub/sub brokers. Path prefixing minimizes the effort required by brokers and is good solution in small integrations, or when universal location mechanisms are not available. URL topics provide a more robust broker routing mechanism, and can scale more safely and effectively with the addition of new brokers.

I recently blogged about the relationship between REST and RPC. REST and RPC also have a relationship with the publish and subscribe architecture. Joe Walker recently discussed three major API styles for Comet: traditional, message-passing, and data synchronization. These API styles correlate to these three architectural styles, respectively: RPC, pub/sub, and REST. However, the architectures are not unrelated, disparate concepts. There is benefit in understanding how these architectures are related and interact.

RPC architecture is based on sending remote calls without any architecturally defined expectations on the behavior, and RPC alone can be viewed as the most generic request and response architecture. REST can be viewed as a form of RPC, in which the architecture does define expectations on the behavior of the remote calls. REST typically has defined a shared vocabulary of calls (GET, PUT, POST, and DELETE) with commonly understand effects.

A notification is a message that does not require a response. A notification may be viewed as similar to an RPC, in that it invokes a remote method, but it differs in that no response is necessary for the application logic. Some systems may provide an acknowledgment of the notification, but this is more of an artifact of the transport than the application semantics.

The pub/sub architecture bears some similarities to REST and RPC, but it is generally composed of notifications instead of requests and responses. Pub/sub has several key components. First, publishing an event is a type of message with a defined expectation, very similar to REST commands in that respect. The expected behavior of a publish request is that the published message be delivered to all subscribers of the specified topic or channel. The publish action resembles the REST POST command. POST suggests that the provided content or data may be appended to the indicated resource. With the publish command, the provided data is conceptually appended to the stream of messages for the given topic. There is also a notable similarity between a pub/sub’s channel or topic and REST’s resource locator or path: they both have a unique forgeable identifying string with path-style syntax.

The subscribe command can be viewed as a notification form of the GET command. The subscribe command is similar to a GET in that it is “safe” (no other resources or users are affected by a subscription), and it is idempotent (subscribing multiple times to a topic should have the same effect as subscribing once). The primary difference is that a GET command follows the request/response paradigm: a single GET request should be followed by a single response. The subscribe command may be followed by zero or more notifications from the subscribed topic.

Finally, when the published message is broadcast to clients, this is similar to a POST response. The message is informing the client of the result of an action that has been taken on the server. However, this is a notification message because it is not in one-to-one response to a request. Of course, a Comet message that is being delivered to subscribed clients does not necessarily have to be the result of an explicitly published message. A message to subscribers may also be the result of another action with side effects like a REST command, or caused by another asymetrical source.

Below is a diagram of the relationship between these different forms of messages:

Venn Diagram of Pub/Sub, RPC, and REST

SitePen has just launched support services which cover Comet, among other technologies. I’m a fairly new arrival to the company, but SitePen has been doing Comet development, training, and consulting for quite some time. See the press release or the website for details.

SitePen has an open approach to advancing the web, and this is particularly true of our efforts with Comet. There’s no single official SitePen client/server Comet framework. Instead, SitePen employees have created and contributed to various Comet efforts, including DWR with reverse Ajax, Cometd (related to the Dojo Toolkit), the Bayeux Protocol, the twisted Python Cometd server, this web site, and more innovations that are coming soon. In particular, the efforts to construct a Comet standard with the Bayeux protocol reflect SitePen’s commitment to open Comet interaction, rather than a single product-centric approach.

Traditional support services that revolve around a single product can turn into an exercise in shoehorning. When the support team is dealing with problems caused by an ill-fitting client, the tendency is to try and force things to work, because the alternative is losing the customer. Our varied Comet efforts mean this won’t happen. The community will benefit from the service too: fixes for support customers will be committed to the project if appropriate.

Native cross frame communication capabilities are coming, but in the meantime we struggle with various hacks to facilitate communication between pages from different sources. The most commonly used approach is fragment identifier messaging (FIM) with various flavors that relies on setting the fragment identifier in the URL of a frame to expose messages, and having recipients poll for URL changes to receive messages. This approach will be used by the OpenAjax hub 1.1 to facilitate pub/sub messaging across frames. Another library was just released that uses an approach which involves loading a page from a target domain with a query containing a message to be stored in a cookie for retrieval from the target page. Unfortunately both of these approaches rely on polling, which is slow, and each poll has a small limit on data size due to URL size limits and cookie size limits. Polling with these small packets is very slow for large amounts of data. Also, FIM causes URL changes which often triggers a click in IE, and can result in machine-gun sound effects during messaging.

Interestingly, Comet could also be used for cross frame communication. If a page from foo.com wants to communicate with a page from bar.com, bar.com could setup a pub/sub Comet channel specific to the loaded instance of the page, and foo.com could subscribe to it using remote callback scripts. With this setup, bar.com could publish a message to its server, and foo.com would immediately receive a message (through a loaded callback script) from bar.com. Since the message is sent in a script, there is no data packet limit, and no polling required during message transmission.

This would certainly be a rather involved approach and would probably not be optimal in most situations, but in limited cases where large messages need to be transferred from one frame to another, and both frames have high-bandwidth access to the same server, this could be a faster messaging approach. Comet has numerous applications and it is interesting to note that one of them may actually be providing a solution for a certain class of cross frame messaging needs.