A Standards Based Approach to Comet Communication with REST

by Kris ZypDecember 27th, 2007

The HTTP protocol is the basis of the web, and provides many powerful capabilities for building Comet communication. In particular, HTTP is built on the principals of Representational State Transfer (REST). The REST concepts that underlie HTTP facilitate important scalability capabilities for agents that choose to take advantage of them, and enable meaningful communication about resources, which can benefit Comet usage. Comet was not specifically incorporated into HTTP, but HTTP provides a powerful vocabulary that can provide a substantial foundation for Comet communication. I propose an approach for RESTful Comet that would utilize the existing HTTP 1.1 and MIME standards as much as possible. This approach has important implications for future scalability possibilities, proxy service utilization, and REST data/messaging interaction. This approach maintains the critical HTTP principle of statelessness which reduces server resource consumption. This proposal can be integrated into proxy servers to provide increased scalability through distributed messaging with HTTP proxy servers. It could also even be utilized by proxy servers to maintain fresher caches independently of Comet applications.

A Simple HTTP Comet Protocol for Single Resources

For simple single resource HTTP GETs, it is possible to add a single simple header and provide a very intuitive technique for using Comet with a single resource that maintains the principles of HTTP. A “When-Modified-After” header is defined on a GET request that instructs an HTTP server that it should not respond to a request until a modification to a resource is made. The value of the header should be a time stamp, indicated the point in time after which new data should trigger a response. If a modification has already been made since the given time stamp, the server should immediately respond to the request with the current resource using a standard HTTP response. If there have been no modifications to the resource since the given “When-Modified-After” date, then the server should wait until a modification is made before responding to the request (with a standard HTTP response). The response should include a standard “Last-Modified” date, so the client knows what date value to use for the next “When-Modified-After” date to resume monitoring.

Using this simple extension to standard HTTP protocol, a client can create a connection and issue a request that stays open, and the response will not occur until a change has been made to the resource. This is effectively a long-polling Comet connection for monitoring a single resource. Here is an example:

-->
GET /myResource HTTP/1.1
Accept: */*
When-Modified-After: Wed, 26 Dec 2007 16:00:15 GMT

In the meantime, suppose the resource content is replaced with “Hello World!”, which triggers the monitored resource response below:

<--
HTTP/1.1 200 OK
Last-Modified: Wed, 26 Dec 2007 18:00:15 GMT

Hello World!

One very useful application of this approach would be for monitoring an Atom feed. Rather than polling for changes in an Atom feed, a newsreader client could issue a request with a “When-Modified-After” header, and the connection would remain open and waiting for a response until a new Atom entry was added.

REST Methods

While we can simply return a standard HTTP 200 (OK) response to provide an updated copy of the current resource, the REST concepts behind HTTP provide the impetus for a wider vocabulary of HTTP responses, which yields more flexible and efficient communication of resource modifications. When a resource is deleted while monitoring, a 404 (Not Found) response can be used to indicate that the resource has been removed. When data has been appended to a resource while monitoring, the HTTP range capabilities can be utilized to return a 206 (Partial Content) response. A 404 response is very simple, but returning partial content in a response is a bit more nuanced. A response may only return a 206 if the request has included a Range header indicating what range is desired. By including a Range header specifying the range units and range, a request can make it known that a 206 response is acceptable and which range units are acceptable. However, a Range header alone should only receive a successful response of 206 (not 200). In order to allow a 200 or a 206 response, the If-Range header can be used conditionally allow for a 200 or a 206 response, but with resource monitoring, the If-Range header as defined by HTTP is not useful. The condition for selecting between a 200 or a 206 response is based on the date of the resource, where resource monitoring should select the response type based on whether the resource had been modified in such a way that a partial content delivery is applicable. Therefore, I propose that combining the When-Modified-After header with the Range header should indicate that both 200 and 206 responses are acceptable and the selection should be based on the utility of a 206 response in communicating the modification. The HTTP specifications define a range unit of bytes. Below is an example:

-->
GET /myResource HTTP/1.1
Accept: */*
When-Modified-After: Wed, 26 Dec 2007 16:00:15 GMT
Range: bytes 12-/* This indicates that the bytes after byte 12 (the current end of the resource) should be returned

In the meantime, suppose the string “How are you?” was appended to the resource’s content. This action triggers the monitored resource response below:

<--
HTTP/1.1 206 OK
Content-Range: bytes 12-23/24


How are you?

With allowance for conditional 206 responses, monitored resource HTTP response codes have a correspondence to their causal HTTP method verbs:

  • PUT action -> 200 - full resource content replacement results in a 200 full content response for monitoring request)
  • DELETE action -> 404 - deletion of a resource results in a 404 response for monitoring request)
  • POST action -> 206 - if the POST results in simple appending or replacement of data to the resource, and the response modification can be described with a content range, the result is a 206 response. Otherwise a full content 200 response may be necessary.

The byte range units may not be the most appropriate range unit for many modern format types. While the HTTP specification only defines the byte range unit, it allows for other range units. For modern data formats such as XML/Atom and JSON, the alternate range unit “updated” defined by the Atom Publishing Protocol is generally preferable. Let us consider the case of an Atom feed. When a new news entry is added to an Atom feed, it does not result in data being appended to the end of the content; instead the entry is embedded in the XML file. However, we can easily use a updated range unit to efficiently communicate the addition of a new entry:

GET /myAtomFeed HTTP/1.1
Accept: */*
When-Modified-After: Wed, 26 Dec 2007 16:00:15 GMT
Range: updated Wed, 26 Dec 2007 16:00:15 GMT-/*

In the meantime a new entry is added to the Atom feed, triggering this response:

HTTP/1.1 206 Partial Content
Content-Range: updated Wed, 26 Dec 2007 16:00:15 GMT-/Wed, 26 Dec 2007 20:00:25 GMT
<?xml version="1.0" encoding="UTF-8"?>
<feed>
<entry>
<title>REST Comet Protocol Possibilities</title>
<author>Kris Zyp</author>
<published>2007-12-26T20:00:25Z</published>
<content>...</content>
</entry>
</feed>

Streaming for Resource Monitoring

In a simple implementation of the “When-Modified-After” approach, the response will be a standard HTTP response to a GET request, which will end the HTTP conversation. To resume monitoring, a new HTTP request needs to be issued. However, it may often be preferable to keep the HTTP response alive and indefinitely continue to receive updates on a resource rather than only receiving a single update. The HTTP protocol is designed for streaming data (in particular with chunked encoding). However, it is still necessary to have some protocol for the partitioning of update messages from the data stream. Here the MIME standard provides standards-based direction for coherent segmentation. We can use multipart/mixed to break the stream into individual resource updates. In particular, we can use the multiplepart/mixed variant MIME type of multipart/mixed-replace, which indicates that each message is intended to be a replacement for the resource, superseding the previous message. When a 200 response is provided, the HTTP response can remain open indefinitely receiving new updates/replacements to the resource. This MIME type, in particular, has received attention for Comet because of Firefox’s support for it in its XmlHttpRequest object.

The multipart/mixed-replace is limiting, lacking the ability to transfer REST information with each update. The parts of the multipart/mixed MIME type are defined to only contain the content and content-type. Each update must be full replacement; partial content and deletion updates can not be included. However, another MIME type is applicable for this purpose. The multipart/digest MIME type is for sending whole messages as the parts of the outer message. While the MIME protocol defines its usage for email messages as being embedded email messages, it is easily inferred that in a different context such as HTTP, the parts would be the same type as the outer message, in this case HTTP response messages. Therefore a monitoring response could consist of a content stream of the multipart/digest type and contain within it multiple HTTP response messages with their own status codes, headers, and content.

Some browsers engines (Gecko, WebKit, and Opera) support incremental loading through XmlHttpRequest, but Internet Explorer does not. Without incremental loading, using a multipart response to provide a stream of data that remains open for future messages is problematic. If the stream stays open, an IE XHR request will not return until the response is finished. Therefore content negotiation via the Accept header should be used to indicate the browser’s preference for multipart. For example, a client that can handle multipart/digest and multipart/mixed-replace may send an Accept header on Firefox to indicate that multipart/digest (for the purpose of streaming) is preferred:

Accept: multipart/digest, multipart/mixed-replace;q=0.7, */*;q=0.5

And in Internet Explorer it could send:

Accept: */*, multipart/digest;q=0.5

When multipart/digest is a lower preference than the default content type, it can be assumed that no advantage is provided through multipart streaming (because the client does not support incremental loading).

Monitoring Multiple Resources with One Connection

If each resource requires its own HTTP request/response connection, we will quickly encounter scalability problems when attempting to monitor multiple resources. Here again, the multipart/digest type can rescue us with a means to deliver multiple HTTP requests in one HTTP request. The Content-Type of an HTTP request can be set to multipart/digest, and then multiple requests can be included in the body of the HTTP request using the MIME multipart definition. It may be useful when sending a request that encapsulates multiple monitoring requests to designate a single location as the target for these types of requests. For now, I will suggest “/subscribe”. I would also suggest that one can assume that the sub-requests can conceptually inherit headers from their parent request.

This technique does introduce a problem in correlating responses with requests. In HTTP, responses are normally correlated to requests simply by their TCP/IP connection, but when sending multiple HTTP requests embedded within a single HTTP request, this clearly is not adequate. In this case responses must use the HTTP Content-Location header to indicate what resource the HTTP response message is referring to. It is important to note that using multipart/digest type to encapsulate multiple responses in a single response does not necessitate multipart requests, nor vice versa. Request and response multiparts can be used independently or in combination. A multipart request can receive a single response, a single request can receive a multipart stream of response messages, and a multipart request can receive a multipart stream response.

Now let us look at an example incorporating these ideas. The following request encompasses two requests that indicate that the /news and /weather resources should be subscribed to or monitored:

-->
POST /subscribe HTTP/1.1
Accept: multipart/digest, multipart/mixed-replace;q=0.7, */*;q=0.5
When-Modified-After: Wed, 26 Dec 2007 16:00:15 GMT
Content-Type: multipart/digest; boundary=separation

--separation
GET /weather
Range: updated Wed, 26 Dec 2007 16:00:15 GMT-/*


--separation
GET /news
Range: updated Wed, 26 Dec 2007 16:00:15 GMT-/*

--separation

The response will now send parts of the content stream as resource modifications take place.

<--
HTTP/1.1 200 OK
Content-Type: multipart/digest; boundary=separation
Transfer-Encoding: chunked

after a news event occurred:

50; chunked encoding length
–separation
HTTP/1.1 206 Partial Content append to the news stories
Content-Location: /news
Content-Type: text/xml
Content-Range: updated Wed, 26 Dec 2007 16:00:15 GMT-/Wed, 26 Dec 2007 20:00:25 GMT

<?xml version="1.0" encoding="UTF-8"?>
<feed><entry><title>Big news today...</title>....</entry></feed>

--separation

after the weather update:

30
HTTP/1.1 200 OK replace the old weather report
Content-Type: text/plain
Content-Location: /weather

New weather report: Sunny weather ahead.

--separation

if the news feed is deleted, this would occur:

20
HTTP/1.1 404 Not Found
Content-Location: /news

--separation

Message Delivery Reliability with Incremental Time

Rather than relying on a stateful session to reliably receive a stream of messages, this proposal relies on time segments to define message delivery. When an agent subscribes to a Comet resource, the agent provides a beginning timestamp in the When-Modified-After header (which is normally the current time). When a response message arrives, the response message must include a timestamp (in the Last-Modified header or the Content-Range header if it is using a dated range). The agent can then be assured that it has received all (relevant) messages up to that timestamp. If the connection is broken (or ended in the case of long-polling), the agent can send the last timestamp in the When-Modified-After request header when reconnecting to the server. If another message that should be sent to the client was triggered while the connection was broken, the server can immediately fulfill the request; if no messages are ready to be sent, the server can wait for the next available message. If a server sends a message but the client does not receive it due to a broken connection, the client can still resume the connection with the previous timestamp. This technique allows clients to seamlessly receive all messages for a given timeframe even with an unreliable underlying connection. Furthermore this technique does not rely on a session and is therefore not prone to session timeouts and unnecessary server resource utilization.

An implication of this technique is that it is necessary for servers and clients to be capable of giving every message a unique timestamp. Therefore, servers must be able to assign timestamps with decimals for the seconds and provide a means for continually incrementing the timestamp for uniqueness when multiple messages are prepared within a single second.

Publish/Subscribe Messaging

A publish/subscribe event-based messaging system can still be handled with this approach. This type of system is important for some applications like a chat application. A resource can be treated like a pub/sub channel, and subscribing to a channel is done by monitoring a resource as described above. Messages can be published by posting to a resource using the HTTP POST method. A POST, as the HTTP protocol suggests, can imply appending a record/message to the resource. This can easily translate into the message being broadcast to all resource subscribers as 206 Partial Content messages. In this way the message can be published and subscribers will receive it.

Authentication

Since this proposal is basically an interpretation of the HTTP/1.1 specification for Comet, one can naturally use existing HTTP means for authentication, including the authentication scheme defined in the HTTP specification or alternate application level security that may use cookies or other methods for authentication.

Conclusion

This proposal for using existing different definitions in the HTTP tool set with minimal addition provides a standards-based approach for Comet with integrated REST resource interaction. Only one necessary addition is defined here (When-Modified-After)—everything else is leveraging existing standards. While there may be other approaches that are less verbose and more efficient for particular applications, I believe an approach like this that relies on existing standards and includes the capabilities for interacting with highly scalable world of RESTful resources has potential for significantly improving the interoperability of Comet applications and their integration with RESTful data services. I would like to follow up on this with some more discussion of practicalities like browser limitations, optimizations, globbing, and making this work with other transports like streaming iframes and JSONP scripts.

[Slashdot] [Digg] [Reddit] [del.icio.us] [Facebook] [Technorati] [Google] [StumbleUpon]

14 Responses to “A Standards Based Approach to Comet Communication with REST”

  1. Dan Says:

    I think instead of returning a 404 Not Found when a monitored resource is deleted, you’d want to return a 410 Gone. A 410 says that a previously existing resource for a given URI was explicitly removed. A 404 just says that nothing can be found.. there’s no indication that a resource ever existed or not. I suppose you could infer that if the request doesn’t immediately return a 404, that it was deleted, but I still think in most cases 410 communicates server state more clearly.

    Its interesting reading about approaches to Comet using relatively normal HTTP requests/responses, but I’ve always thought there might be a different approach that would work better in many cases.

    Imagine there were two HTTP methods: SUBSCRIBE and UNSUBSCRIBE. The client would perform a SUBSCRIBE method on any resource it wants to monitor. UNSUBSCRIBE would notify the server that the client no longer wants to receive updates on a resource. Multiple subscribes/unsubscribes could be made in a single connection that remains open. When a resource is updated, a simple message is sent to all the subscribers with just the resources URI and nothing else. The client is then responsible to make a normal HTTP GET request to retrieve the updated resource’s representation.

    The benefit of this approach is that it has very nice scalability properties because it allows you to leverage much of what is already available to distribute load across the back end. For example, we can cut down on the server load by caching the response to
    the initial GET request using a high speed HTTP accelerator like Varnish, and serve the cached response to all the other subscribers.

    A client can use content negotiation to retrieve a representation in whatever format they prefer. Clients can also use basic/cookie authentication to identify themselves to the server when retrieving the updated resource. This means that the subscription protocol can be relatively simple and not concern itself with formats or authentication. Simply telling an unauthorized client a resource has changed doesn’t usually constitute a security problem… and if it did, then we can apply the same access/auth/authz checks before allowing the SUBSCRIBE as we do to prevent unauthorized PUT, POST and DELETE requests.

    The notification message can be extremely small (just the URI), which allows more subscribers to be notified in a given time span, and also probably would allow a larger number of clients to be serviced simultaneously.

    With this approach we leverage the entire infrastructure and systems available to help normal HTTP traffic scale. While it does require a formalized protocol, and a server implementation to handle SUBSCRIBE/UNSUBSCRIBE, it might be worth exploring as an alternative to existing implementations.

  2. Kris Zyp Says:

    Dan:

    Regarding using 410 instead of 404, you are absolutely right, good catch.

    Regarding SUBSCRIBE/UNSUBSCRIBE method, I have considered something like this, but a few thoughts:

    For single resources, the adding an extra header to a GET instead of using a separate methods is advantageous because clients can automatically fall back to a standard GET, it is also stays closer to the standard HTTP spec (using existing methods). But I think you are more interested in multiple resources…
    When monitoring/subscribing to multiple resources, the technique I described is certainly a novel interpretation of HTTP. As far as semantics goes, using a SUBSCRIBE method to subscribe to multiple resources has a lot of merit as well. I didn’t include it in the spec simply because I think it is generally easier for app servers to provide special logic around a special URL than a special HTTP method.

    The idea of returning a resource location instead of the resource on notifications itself also has merit. However, it seems like the vast majority of use cases will actually need to retrieve the resource. I don’t think we really lose much capability in terms of content negotiation or HTTP capabilites by actually returning the resource, since we are still basically using a GET (kind of tunneled version of GET). However, you are right the proxy servers (at last right now) would not understand the GET request/response embedded within the multipart content, and performing a second request to actually get the resource would allow proxies to accelerate the process. Your idea is certainly attractive in terms of minimizing protocol overhead as well. On the otherhand, if one was using new proxies that understand the proposal to accelerate and scale the entire process including the notification dissemination (and this proposal is designed specifically for that), these proxies could certainly be aware cached copies and utilize them. And of course returning resource location when a client wants a resource increasing the number of round trips.

    Is the idea of {UN}SUBSCRIBE, that each {UN}SUBSCRIBE is supposed to be embedded requests within a single HTTP request, or are they each their own HTTP request? I would be very leary of this if they are each their own request, because that would break the stateless nature of HTTP, since the server would need to track what resources a client is subscribed to between requests. One of the goals of this proposal is to maintain the stateless nature of HTTP. I don’t think that is what you are suggesting though.

    Anyway, great thoughts Dan, I appreciate it. Of course one of my main goals with this proposal was to try to leverage and stay as close to the existing HTTP spec as possible, but I will hopefully write another article exploring some of this and other ideas more.

  3. A Standards Based Technique for Batching Ajax Requests « Notes from Kris’s Journey Says:

    [...] Based Technique for Batching Ajax Requests January 1, 2008 — kriszyp I recently wrote an article for Comet Daily that was a proposal for a standards based approach to comet communication. One of [...]

  4. Comet Daily » Blog Archive » Aware of Push Says:

    [...] Comet related aspect of Aware is the integration of RSS/Atom feed long-polling capability. In my last article, I proposed the use of the simple HTTP header “When-Modified-After” as a mechanism for [...]

  5. Nicholas Dronen Says:

    Caching in general is a good thing for scalability, but I have to wonder how useful caching would be for the (presumably) frequently-updated resources that Comet applications usually care about. Whether the contents of the resource denotes some current state or a history of states, neither is going to be cacheable for long.

    Let’s just say for the sake of argument that caching matters for that kind of data and that we want to maximize the amount of time a resource can be cached. A maximally-cacheable resource is an immutable resource. In the barely-cacheable case, you have some resource, the contents of which denotes the current value of the stock XYZ. The resource is updated with the value of the last trade for XYZ at the start of each minute:

    http://www.example.com/stocks/XYZ

    So a GET on that resource would be cacheable for 60 seconds. (This is hypothetical, of course; don’t balk at the lack of temporal granularity.) In the maximally-cacheable case, there is not a single resource but a set of resources rooted at the same URI, each identified by something like a date or time stamp. Today’s per-minute prices during one o’clock would be identified by:

    http://www.example.com/stocks/XYZ/2008/01/06/13/00
    http://www.example.com/stocks/XYZ/2008/01/06/13/01
    http://www.example.com/stocks/XYZ/2008/01/06/13/02

    http://www.example.com/stocks/XYZ/2008/01/06/13/59

    And those resources would be eternally cacheable. Obviously there’s a trade-off between resource granularity and cacheability.

    Anyway, just a thought experiment. I’m personally suspicious of any attempt to add SUBSCRIBE/NOTIFY to HTTP. There’s a long-standing resistance to pub-sub mechanisms in HTTP. See for instance Roy Fielding’s discussion of EBI in his dissertation on REST. Another example is Mark Nottingham’s recent post on cache channels:

    http://www.mnot.net/blog/2008/01/04/cache_channels

    If you go that route, browser vendors might as well add support for a protocol that was originally intended to be used for real-time communication, like XMPP or SIP. Kidding, in part. :-)

  6. GregWilkins Says:

    Kris,
    the problem with the When-Modified-After header is that it does not deal with the 2 connection limit. If 2 widgets on the same page both issue When-Modified-After requests then that page is effectively blocked until a modification occurs.

  7. Kris Zyp Says:

    Greg,
    The section “Monitoring Multiple Resources with One Connection” addresses that issue with the multipart/digest request batching. Multiple subscriptions can delivered in one request, while still using HTTP standards. Of course, with any Comet application, there must be some central Comet dispatch that will handle the various subscription requests and funnel them into one request/connection.
    Nicholas,
    It is important to note that I am not proposing a pub/sub architecture like Bayeux. I am actually in agreement as far as being resistant to the idea of adding a stateful (between connection) pub/sub system to HTTP, but this proposal stays very close to the stateless principles of REST.
    You are right that caching of resources is probably not that important for browsers that are dealing with rapidly changing content, but my assertion that the caching principles could be used to create Comet proxy servers that distribute the load of Comet using fairly standard HTTP principles in combination with the proposed “When-Modified-After”.

  8. Technolalia » Blog Archive » Standardizing Comet (Redux) Says:

    [...] this, my comment about avoiding pub/sub was directed to Dan’s proposal to introduce SUBSCRIBE and [...]

  9. Nicholas Dronen Says:

    Hi, Kris:

    I started to respond to your comment, but it got a bit long for a comment, so I made it a post of its own. My response is here: http://technolalia.com/?p=13.

    Regards,

    Nick

  10. Kris Zyp Says:

    Nicholas,
    Thanks for the great discussion! However, I think there is some misunderstanding about how the When-Modified-After header is intended to work. The When-Modified-After header works almost the same way as the standard Is-Modified-Since header, except that when the resource has not been modified since the indicated date, rather than returning a 304 Not Modified, the server should block until the resource has been modified. Therefore the behavior follows the same RESTful behavior of HTTP. This does not break the notion of statelessness any more than the If-Modified-Since header does. No information about prior requests is needed (that is what the stateless nature of HTTP is in reference to).
    I am not sure I understand how a When-Modified header (with no date) would work. If it is looking for any modifications, well every resource has been modified at some point (unless it has not been created yet), so you would not have a blocking response and therefore no Comet. I assume you mean that it is looking for any modifications to a resource after the request is received. However this creates some big problems with reliability. Every time there is a gap in time between one monitoring request and the next monitoring request, there would be a time that no monitoring would be taking place. Most browser implementations would probably be using long-polling which causes a break in the connection after every single message. Having gaps in the monitoring after every single message creates an unacceptable unreliability in monitoring resources. Resource changes can take place without the client ever being notified. Stateful pub/sub architectures like Bayeux deal with this by maintaining a session that has it’s own queue, but that is definitely stateful and anathema to REST. In order to preserve RESTfulness without loss of reliability, the client must be able to indicate when the last time was when it was up to date with all monitoring, that is when the last update was recieved. The client can resume monitoring from that last time stamp without creating gaps in monitoring.
    In regards to feed paging, I believe the server can and should use feed paging whenever possible with this approach.
    In regards to channel caching, I have looked at it before with consideration for how it could fit in with my goals, but I will have to look at it more closely, I think.
    In regards to the pseudocode for the HTTP cache, your pseudocode was not correct. An HTTP cache should not immediately return a cached resource to a client with a When-Modified-After date that is newer than a cached resource, but should block and wait for a new resource from the server. I can write up a more detailed pseudocode a little later. Thanks again for the feedback.

  11. Comet Daily » Blog Archive » Proposal for Native Comet Support for Browsers Says:

    [...] my last article, I pointed out the technique of encapsulating HTTP messages into content. I originally suggested [...]

  12. JSON » Blog Archives » HTTP as the Basis for Comet Says:

    [...] couldn’t agree more! Posted in Uncategorized | Trackback | del.icio.us | Top Of [...]

  13. Comet Daily » Blog Archive » Colliding Comets: Battle of the Bayeux Part 5 Says:

    [...] does build upon the foundation we have in HTTP, rather than fighting it. I think my proposal for Comet communications using HTTP semantics is a good candidate, but it does need more [...]

  14. Comet Daily » Blog Archive » Introducing HTTP Channels Says:

    [...] as well as mutating data entities. This transport is based on the goals and the refinement of the standards based approach to Comet communication with REST, but with improvements based on suggestions, further research, and an actual implementation in [...]

Leave a Reply



Copyright 2014 Comet Daily, LLC. All Rights Reserved