Lightstreamer
SitePen Support

Introducing HTTP Channels

by Kris ZypMay 13th, 2008

HTTP Channels is a transport protocol based on the existing messaging capabilities of HTTP and utilizes the concepts of REST for notification of resource changes. HTTP Channels is intended be a Comet solution for RESTful real time data synchronization architecture and can be used as a Bayeux transport, or it can be used stand-alone. HTTP Channels is designed to provide a mechanism for efficient and standards-based retrieval and monitoring of data and for receiving notification of changes to data with meaningful semantics.

HTTP Channels brings strong coherency and interoperability with publish and subscribe architectures by equating the PubSub concept of a topic or channel with the REST concept of a resource locator. This allows HTTP Channels to superset the capabilities of PubSub messaging (consequently it can operate as a Bayeux transport), allowing resources to be conduits of messages as well as mutating data entities. This transport is based on the goals and the refinement of the standards based approach to Comet communication with REST, but with improvements based on suggestions, further research, and an actual implementation in Persevere. HTTP Channels can also use HTTP tunneling in conjunction with XHR streaming where available (Firefox, Safari, and Internet Explorer 8’s XDR) for greater efficiency than long-polling. HTTP Channels provides duplex client/server communication and leverages the broadly understood and interoperable semantics of HTTP.

HTTP Channels is conceptually based around the concept of a full duplex HTTP. However, full duplex HTTP is not possible within the constraints of the browser because of a lack of sufficient control of connections and pipelining. Therefore, the Channels transport also includes the necessary mechanism to emulate a single connection. In order to aggregate requests into the conceptual Channels connection, requests should include a client id. This header can be created by the client and should be a sufficiently randomized string to avoid collisions.

Channels Server Listening Connection

To initiate the long-lived server listening response which provides the means for the server to send messages, a POST request should be made to the /channels URL on the target server with an X-Create-Client-Id header with the client id for the value. Subsequent subscription requests should include an X-Client-Id header with the client id in order to associate the subscriptions with the Channels server listening response. When the client has received a server response/message, the client should immediately resume the connection by making another POST request to the /channels URL again (long-polling). On these subsequent requests, the request should include an X-Client-Id header (not X-Create-Client-Id) with the client id. If the client id is no longer valid (if there was an excessive amount of time between requests, for example), than the server should respond with a 404 status. The client may then reissue the channels request using the X-Create-Client-Id header. At this point the client should assume a new connection, and any subscriptions that should be resumed must be reissued to the server (and the client may want to use the X-Subscription-Since header to resume since the last response, which is discussed below).

The server sent notification message should include a Content-Location header with the URI indicating the resource with which they are notifying about. The message should also include an X-Event header. This header indicates what event triggered this notification. When the notification is the result of a request sent to a header, the event should correspond to the name of the method of the request that triggers the notification. If a PUT request was used to update a resource that triggered a notification, X-Event header should have a value of PUT.

Subscription Requests

HTTP Channels uses an HTTP GET or HEAD requests to establish a subscription. To request a resource and monitor in a single request, you can use a GET request to retrieve a resource and subscribe to it in a single request. To subscribe to a resource without retrieving the current resource, you can use a HEAD request. Subscriptions are made by including an X-Subscribe header. The following are valid values for X-Subscribe:

  • * - Subscribes to the resource indicated by the request URI. Only * is allowed for now, but in the future we could allow a comma delimited list of event names, for granular subscriptions.
  • none - Unsubscribes the resource

Subscriptions should include an X-Client-Id header to properly correlate the subscription with the channels server listening response.

Retroactive subscriptions can also be made by including an X-Subscribe-Since header. The value of the header must be a date according the HTTP specification of a date format,and it must be in the past. Generally this header should be used when a connection is dropped, and the value should correspond with the Last-Modified date on the last notification or response received over the Comet connection.

Subscriptions should include a Cache-Control header with a value of max-age=0. This header will indicate that the request must be delivered to the origin server for revalidation. This will ensure that the subscription reaches the server. Conditional validation can still be used such that if the client has a cached copy of the resource, the server does not need to send the content (it can send a 304 status).

When a server returns a response to a subscription requests, the response should include a X-Subscribed header with a value of OK if it was successful, and an error message if it was not successful.

Here is a diagram of HTTP Channels interaction, where one client subscribes to a couple resources and listens for notification of changes while another client modifies data that triggers notifications:
channels1.png

Message Ordering

Subscription requests as well as resource modifying requests may optionally include a X-Seq-Id header to ensure deterministic message ordering. Each request should increment providing a value one higher than the previous request. Servers can then reorder requests for proper message ordering if they arrive out of order.

Examples

Here is an example subscription request and response:

GET /customers/ HTTP/1.1
Accept: */*
X-Subscribe: *
Cache-Control: max-age=0
X-Seq-Id: 1
X-Client-Id: 867208037610061686690752</code>
 
 
HTTP/1.1 200 OK
Content-Type: application/json
Last-Modified: Wed, 16 Apr 2008 21:47:44.906 MDT
X-Subscribed: OK
Content-Length: 233
 
...content...

Here is an example of the Channels server listening request and response:

POST /channels HTTP/1.1
Accept: application/http
X-Create-Client-Id: 867208037610061686690752

The response should remain open while waiting for notifications from the server. Now suppose that someone issued a POST request to the /customers/ resource (which this connection is subscribed to). The following notification would appear in the content stream for the channels response:

HTTP/1.1 200 OK
Content-Type: application/json
X-Event: POST
Content-Location: /customers/
Content-Length: 40
 
{name:"new customer"}

Tunneling on HTTP Channels

Ideally, an HTTP client can implement HTTP Channels and fully interact with a Channels server, sending subscription requests and message/modifications requests, and receiving notifications all over a single TCP connection. A single HTTP response remains open and can stream messages from the server. Full asynchronous bi-directional messaging freedom can be achieved by using pipelining, requests can be sent by the client without waiting for the response to finish. This eliminates the usual request/response order of operation typically used by HTTP because the requests will have no corresponding response, a single long-lived response eliminates the need to provide a response, without violating the HTTP protocol. Both requests from the client and messages from the server can be sent at any time using this single TCP connection. Duplex communication on a single TCP connection should certainly be a primary goal for Comet, and this transport is designed for that future.

The ideal of full duplex HTTP on a single TCP connection is not realizable with current browser technology. JavaScript does not have control of pipelining, and in many situations a streaming response is not possible over XHR, so long-polling must be used instead. However, where streaming is often possible, and when streaming is available, HTTP tunneling can be utilized to frame messages and improve performance by eliminating the need for new requests after each server message.

It is recommended that the server use the User-Agent header to determine if the client supports XHR streaming. In addition, it is recommended that the server check for a XDomainRequest header. If it is present, include a XDomainRequestAllowed header with a value of 1, so that the client can utilize XDomainRequest if available (this is the only fully accessible streaming mechanism in Internet Explorer 8).

The Channels server listening request should specify an Accept header which includes application/http as a desired content type. The response should be of the content type application/http, and the response content should consist of a stream of HTTP messages. This is in essence HTTP tunneling. The response is composed of individual HTTP messages in HTTP response format. Each of these responses is a server sent notification. If the client does not support XHR streaming, or if the path to the client includes proxy servers, every message from the server should be the actual server response message itself. The following is an example of HTTP tunneling:

POST /channels HTTP/1.1
Accept: application/http
X-Create-Client-Id: 867208037610061686690752
 
HTTP/1.1 200 OK
Content-Type: application/http
Transfer-Encoding: chunked

The response will remain open, but the server messages will appear in the content stream of the container response above.

HTTP/1.1 200 OK
Content-Type: application/json
X-Event: POST
Content-Location: /customers/
Content-Length: 40
 
{name:"new customer"}

Note that if a browser did support pipelining control, duplex pipelined requests do not have responses, making plain subscription requests (using HEAD) unnecessary. In a duplex pipelined request situation, if a GET request is made and the server needs to deliver a response, it should be sent as a tunneled response message in the application/http message stream with an X-Event value of GET. Pipelined HEAD requests for subscriptions do not necessitate a response unless they resulted in an error.

PubSub on HTTP Channels

HTTP Channels is capable of PubSub messaging. Resources act as topics, and messages can be posted to resources, with the subscription system routing the messages to listeners. When HTTP Channels is used as a Dojo Cometd transport, subscriptions are sent as subscription HEAD requests as described above and publishing messages is done by doing a POST to the target topic/resource. Messages are then routed to clients as notifications with an X-Event of POST. By using resources as topics, a simple PubSub system can be evolved into more advanced resource modification and RPC subscription systems.

Looking Ahead

JSON Channels is another forthcoming transport which is intended to have a direct mapping with HTTP Channels. JSON Channels is intended to be an extension of JSON-RPC with subscription and resource/topic locator capabilities. JSON Channels will be easier to implement in many environments since it does not require HTTP parsing, however it is intended to simply be an alternate format that can work within the same semantic structure of operation. Both HTTP Channels and JSON Channels could be used interchangeably, clients could request that the server sent message stream be JSON messages, but still issue subscription requests with HTTP style GET/HEAD requests.

Design Rationale

Most of the rationale for this transport can be found in my Comet Daily articles A Standards Based Approach to Comet and A Taxonomy of Comet. However, I wanted to note the reasons for changes in the “Standards-Based approach”:

  • Content type of application/http for tunneling - Originally it was multipart/digest. application/http is the correct MIME type for multiple/pipelined HTTP messages.
  • X-Subscription and X-Subscription-Since instead of When-Modified-After - This is simply to improve the clarity and explicitness of the messages. The When-Modified-After header would require too much overloading of functionality. HEAD requests were also added to make it easy to make a subscription without retrieving the resource.
  • Dropping use of Ranges + 206 Partial Content - This was intended to provide a mechanism for reliable messaging with PubSub, but this is difficult and complicated to implement, and with custom methods may be inefficient and confusing. It is much easier to simply allow for the concept of retroactive subscriptions.
  • Use of Client Id/Virtual Connections instead of HTTP tunneled requests - This makes it easier to add subscriptions and does not require the client to manually create HTTP messages for tunneling. HTTP tunneled requests may still be valuable for batching, but I have not implemented that.

Implementation

HTTP Channels has been implemented in Persevere, providing an open protocol for real-time notifications of changes to data in the Persevere’s storage system for a data centric architecture for message routing. Persevere also serves as a reference implementation for learning to use the protocol. A Dojo-based implementation of HTTP Channels is also coming, which can both plug into Dojo’s cometd system and provide real time updates to a Dojo Data store.

[Slashdot] [Digg] [Reddit] [del.icio.us] [Facebook] [Technorati] [Google] [StumbleUpon]
Comet Support by SitePen

7 Responses to “Introducing HTTP Channels”

  1. kL Says:

    Sounds *very* much like HTML5 Server-Sent Events:

    http://www.w3.org/html/wg/html5/#server-sent-events

    The latest draft has even id:/Last-Event-ID which are like a mix of your X-Client-Id and Seq-id.

    I urge you to send this proposal to HTML5 WG, otherwise you might end up implementing almost-the-same-but-not-quite feature.

  2. Comet Daily » Blog Archive » Intended Usage of HTTP Channels Says:

    [...] HTTP Channels introduction primarily defined the protocol, but it may be worth expounding the the goals and motivation for [...]

  3. Comet Daily » Blog Archive » Dojo HttpChannels Module Says:

    [...] HTTP Channels is implemented in Dojo 1.2 with the HttpChannels module. HTTP Channels is intended to provide data synchronization and update notifications by leveraging the ubiquitous HTTP REST semantics. Within Dojo, HTTP Channels are particularly useful, integrating directly with Dojo Data for live data synchronization. The HttpChannels module is very flexible as well; it can be used in several different ways: [...]

  4. SitePen Blog » Getting Started with Persevere Using Dojo Says:

    [...] and Persevere support HTTP Channels which allows for addition of live data updates through Comet notifications. Adding Comet capability [...]

  5. Comet Daily » Blog Archive » Persevere Tutorial Says:

    [...] support in Persevere occurs with HTTP Channels. You can start building applications almost instantly; no need to create schemas or table [...]

  6. Comet Daily » Blog Archive » Upcoming Presentations Says:

    [...] can be used as a basis for service oriented web architecture including Comet protocols Bayeux and HTTP/REST Channels, and how to use these in [...]

  7. Comet Daily » Blog Archive » REST Channels: HTTP Channels with JSON support Says:

    [...] HTTP Channels is the proposed Comet protocol for leveraging the semantics of HTTP REST for subscribing to resources and receiving resource/data modification notifications and closely integrates with HTTP. HTTP Channels is most obviously expressed in the MIME-style format used by HTTP, however the MIME format is not necessarily the easiest format to work with, particularly in JavaScript. Fortunately the semantics of REST notifications can easily be expressed in JSON format, which is very easy to parse and handle in the JavaScript environment. This proposed resource modification based notification system, expressible in HTTP/MIME (HTTP Channels) or JSON format will be called REST Channels. JSON-based REST Channels is also well-suited for use with the new Web sockets protocol defined in HTML 5. [...]

Leave a Reply



Copyright 2014 Comet Daily, LLC. All Rights Reserved