Scalable Real-Time Web Architecture, Part 1: Stomp, Comet, and Message Queues

by Michael CarterOctober 8th, 2008

What is Stomp?

Stomp is an incredibly straightforward message queue protocol. Stomp derives its name from an acronym with the same letters that stand for Streaming Text-Oriented Messaging Protocol. As you can probably guess, Text-Oriented means that the protocol is human readable. Like HTTP, A stomp frame consists of three parts: 1) Frame type 2) frame headers, 3) body. Here is a typical Stomp message frame:

MESSAGE\n
destination:/queue/a\n
message-id: 234\n
\n
hello world\0

The above frame illustrates a MESSAGE frame with two headers (the destination and message-id), and the frame body “hello world”. The official specification contains all of the details of the Stomp protocol.

The Upside

Stomp is a great protocol for a few simple reasons.

  1. It is Human Readable and incredibly simple to implement (a reasonable parser might be 20-100 lines depending on your language)
  2. Stomp clients already exist in 14 different languages, including Java, C#, Python, Ruby, Perl, PHP, and JavaScript.
  3. There are an increasing number of servers/brokers that support Stomp; it is the most widely inter-operable MQ protocol at this point, with support in at least five servers.
  4. Stomp clients can easily consume JMS via ActiveMQ or StompConnect, and easily consume AMQP via RabbitMQ.
  5. Writing a light-weight Stomp server is not prohibitively difficult. (such as MorbidQ). This type of server is great for embedding so as to ease development and can generally make life easy for small/medium deployments.

The Downside

There are some problems with Stomp as well, though.

  1. For such a simple protocol, the official specification is poorly written. In email correspondence, a core contributor to a well known Stomp client told me: “it’s almost as if the spec. thinks that ‘whitespace on an HTML page’ is a valid ASCII character”
  2. To worsen matters, the reference implementation (ActiveMQ) contains a long standing protocol bug! Now 75% of the existing clients and half of the servers have all reproduced that bug. The solution at this point is to engineer servers to optionally accept the bug without crashing, as ActiveMQ and MorbidQ currently do.
  3. The protocol uses a “\0″ delimiter which is incompatible with binary mode, so you need to specify content-length for binary data. This is confusing and makes the spec. worse; it should always require content-length to size-prefix frames instead of using the “\0″ delimiter.
  4. Not all MQ features map onto STOMP in the same way. For instance, ActiveMQ/Stomp will implicitly map destinations of the form “/topic/*” to topics, and “/queue/*” to queues. This isn’t in the Stomp spec.; its just a feature of ActiveMQ. Without at least specifying these features as optional extensions, it is more difficult to migrate an application from one stomp server to another.

Stomp is Good Enough for Now

Despite the downsides of STOMP, its so straightforward to implement and gain immediate browser support of back-end AMQP and JMS systems that it is well worth any costs. In the Battle of the Bayeux series by Greg and I, we dissented on a number of points, but Greg really convinced me that messaging is a much, much better architecture paradigm than HTTP’s request/response. The future wave of Internet application architectures will have a strong showing of messaging style architectures over the traditional request/response methods. They will use HTTP to serve only static content, with the GUI written completely in JavaScript, and the application/business logic implemented as an MQ producer on the server-side. Each user will be act as a browser-based MQ consumer, using a protocol probably not unlike Stomp.

Conclusion

In the near future you can probably expect to see Greg, myself, and the other Bayeux authors work out some revisions of Bayeux to create a solid separation of transport and messaging such that Bayeux can easily be used on top of HTML5’s WebSocket or Orbited’s TCPSocket. The end goal here is to have a solid messaging protocol that is tailored to modern browsers. The best fit at the moment, in my opinion, is Stomp; but Bayeux is solidly situated to be a serious contender soon.

In my next article in this series, I’ll provide a step-by-step tutorial that illustrates sending real-time financial data to the browser with Orbited, MorbidQ, and Python.

7 Responses to “Scalable Real-Time Web Architecture, Part 1: Stomp, Comet, and Message Queues”

  1. James Strachan Says:

    Great post!

    Hopefully version 1.1 of Stomp can fix all of its shortcomings. Incidentally the reason for 3 was to make it even easier for developers to work with stomp; so they don’t have to pre-parse their payloads for example so you can telnet into the message broker and just type messages. I don’t think having an optional header is that confusing?

    BTW one minor correction - you can talk to *any* JMS compliant message broker with Stomp by just deploying StompConnect with any JMS provider - which basically means you can talk to all MOMs with any kind of market penetration

    http://stomp.codehaus.org/StompConnect

  2. Comet Daily » Blog Archive » Tutorial: Rails + Orbited (With js.io Stomp) Says:

    [...] on using Orbited with Rails. He takes a similar approach to that which I outlined in my previous article on Stomp. The tutorial is a great introduction that takes you through everything you need to get off the [...]

  3. Comet Daily » Blog Archive » Scalable Real-Time Web Architecture, Part 2: A Live Graph with Orbited, MorbidQ, and js.io Says:

    [...] application we are building will use the Message Queue Architecture as outlined in the part 1 of this series. In particular, this tutorial showcases a data broadcast use-case where data is [...]

  4. Michael Carter Says:

    @James:

    There is an additional and concrete reason why I am not a fan of the ” frame delimiter — Browser’s can’t send ” in XHR requests, so we end up having to not just tunnel tcp over HTTP, but to also base64 encode it. Granted, if we want to deal with binary data at all we need to base64 encode the payloads, but for the more common case of sending json around.

    On the other hand, I see what you mean about the ease of testing the protocol out in telnet. I would propose requiring the content-length header in all cases, but only use ” if the content-length header value is “variable”. I suppose having optional headers isn’t so bad, but what I’m more interested in is having a trailing ” be optional, and preferably not the default.

  5. Michael Carter Says:

    For the record, if you see an empty ” in the above post, I was trying to post ’slash zero’ but apparently Wordpress sanitizes a \ and 0 together… Yet another reason to remove them from the spec ;-)

  6. Alexander Goldstone Says:

    @Michael:

    Thank you for a great post.

    To back-up the first response from @james, not having to pre-parse payloads to generate a content-length header is a big win for light-weight (embedded) hardware systems.

    Also, having been exposed to network connections that bill per-byte, I ‘d like to see optional headers remain optional.

    That being said, whilst its great for one use-case it it sounds like the null-terminator is far from ideal for another.

    When documenting the content-length header, the STOMP specification says:

    “The frame still needs to be terminated with a null byte”

    One option, which I think you are heading towards based on your comments, is that the null-terminator should only be needed when a content-length is not specified. This sounds like a reasonable compromise to me and appears to suit both use-cases. Are there any obvious downsides to this in terms of server implementation?

  7. MichaelCarter Says:

    @Alexander:

    I like where you’re going with the optional null byte, though I’m not completely convinced on your premise.

    “To back-up the first response from @james, not having to pre-parse payloads to generate a content-length header is a big win for light-weight (embedded) hardware systems.”

    What are we trying to optimize for on the embedded systems? If its cpu, I’d think that keeping a running count of the payload length and sticking it in a header would be much less intensive than parsing all incoming payload content for a null byte.

    “Also, having been exposed to network connections that bill per-byte, I ‘d like to see optional headers remain optional.”

    Stomp is pretty clearly not the protocol you want in this situation. Its not optimized for low bandwidth usage, its optimized to be human readable. I hardly think changing content-length from optional to required will be the straw that breaks the camel’s back.

    “One option, which I think you are heading towards based on your comments, is that the null-terminator should only be needed when a content-length is not specified. This sounds like a reasonable compromise to me and appears to suit both use-cases. Are there any obvious downsides to this in terms of server implementation?”

    Would this actually help your embedded system scenario? Your embedded client won’t have control over which the server decides to use.

    It seems to me that there is an obvious dichotomy missing in the Stomp spec, and that we’ve yet to explore: Streaming versus non-streaming. Right now the protocol name begins with “Streaming” and in opposition to my recommendation, you can’t really do streaming when you have to specify the payload size up front. It seems like this really should be handled with an optional “mode” header which can be set to “streaming” or “message”. If its “streaming” then no content-length header should be required and it should append the null byte to each payload.


Copyright 2015 Comet Daily, LLC. All Rights Reserved