Comet Gazing: Memory Usage

by Comet DailyJanuary 3rd, 2008

Comet Gazing is a semi-monthly feature on Comet Daily. We pose a question to our Contributors and post the answers.

This time, we’ve asked the question: “Open connections consume memory, so how do you minimize and estimate memory usage effectively with a large-scale Comet-based application with your Comet server of choice?”

Roberto Saccon

ErlyComet is a Comet server written in Erlang. Memory usage per connection starts at 43 KB, measured from the ErlyComet (development snapshot) demo application. That allows more than 20,000 connections per GB of RAM. Not included in this view is the in-memory database, which maps connection identification to process identification and might contain other shared information. Its size with the demo application is marginal, but with a complex application, its memory consumption can easily grow beyond the total memory consumed by the connection processes. How memory consumption increases when all connections are simultaneously exchanging messages needs to be determined with stress testing.

To reduce overall memory consumption, there are some Erlang specific tricks:

  • For text, binaries should be used whenever possible for the internal representation, instead of lists.
  • For the in-memory database, partitioned tables (across nodes) can be used instead of per-node replicated tables.

Kris Zyp

I am developing Comet support for Persevere, which is a JSON REST server written in Java, and runs on J2EE servers. I am currently developing on Tomcat, and utilizing its Comet API (that uses Java NIO). To estimate memory usage, I wrote a JavaScript unit test that creates 50 concurrent Ajax requests to my Comet resource. In order to have a large number of concurrent connections in Firefox, you must increase the maximum connection limit which is done by increasing the “network.http.max-*” values in the “about:config” page (Firefox seems to have a hard limit of 50 concurrent connections).

In Tomcat, using the Comet API (which does not require a thread for each connection), each open connection takes 87KB. This seems disappointingly high, especially when considering that using the standard servlet API which uses a thread for each connection takes 115KB per connection. It seems the Tomcat Comet API does not provide a very substantial memory advantage (unless I missing something).

How do I minimize memory usage? Perhaps I will switch to Jetty.

Martin Tyler

Caplin Liberator is a custom built server written in C and uses asynchronous IO based on poll and epoll. Some Comet servers based on existing web/application servers have an overhead per connection because they were originally designed for transient connections, rather than the long lived connections a Comet server has to deal with.

In Liberator a connection uses minimal memory on its own, but the more subscriptions each connection has the more memory it uses. If 1,000 connections are subscribed to 100 items each that means 100,000 data structures representing that subscription, so careful attention is paid to subscriptions to keep them lean. The number of objects, or channels, cached within Liberator can also be a key concern, but the significant part of that is the data payload so tweaking the data structures has less effect.

Calculations have to be made which take into account these aspects; the number of objects and the number of subscriptions being key.

Alessandro Alinone

Lightstreamer leverages Java NIO API to handle TCP sockets. The usage of this API requires a very careful design and implementation, in order to avoid deadlocks and memory leaks. So, obviously, the first requirement for efficient memory management is to get rid of any kind of leak. When a Comet session terminates, all the related resources must be freed up at all levels (from low-level system resources to high-level application resources). Lightstreamer tends to keep all the buffers small in order to control the outbound bandwidth and the size of the TCP packets. For many types of pushed information, conflation algorithms are acceptable, resulting in the possibility of dropping or merging events, rather than buffering them indefinitely. This allows predictability in memory consumption.

Greg Wilkins
Cometd + Jetty

Jetty takes great efforts to minimize the resources assigned to connections, specifically threads and buffers. We have previously discussed the threading model, where asynchronous continuations are used to allocate threads only to active requests and not to those that are simply waiting for events.

Thread allocation mostly gets the headlines, but it is also buffer allocation that greatly affects the scalability of a Comet server. Big buffers can consume too much memory, but small buffers can result in too much blocking, content switching and inefficient network usage. Jetty takes an approach that allows a mix of many small and a few large buffers to be used intelligently to create a scalable Comet server.

Jetty does not allocate any buffers at all to an idle connection. The connection is simply placed in a SelectSet that will detect any incoming IO activity. Once IO is detected, Jetty allocates only a small header buffer that is large enough only for typical HTTP request headers (e.g. 4kB configurable). This buffer is used to asynchronously read the request.

If the request contains a body (e.g. for a form or a file upload) that did not fit in the header buffer, Jetty may then allocate a much larger buffer to use to receive the data content. Because these buffers are infrequently allocated, they can be large (64kB or more) and thus blocking can be avoided as Jetty asynchronously fills a buffer before dispatching to a servlet.

For Comet requests, a significant proportion of requests will simply wait for an event before sending a response. Therefore Jetty does not allocate any output buffers (or character converters and convert buffers etc.) until they are required. Thus during a long poll only the small request header buffer is held.

Only when a response starts to be generated are output buffers allocated. Jetty also uses different buffers for response headers and content that are joined together with an efficient NIO gather write to be written with a single system call. When sending static content (such as images and JavaScript), Jetty can use memory mapped files that are “gather” written with a small response header buffer. The memory mapped files use the operating system buffer space (and DMA) and do not consume any user space memory or java heap. Therefore only a small response header buffer is needed in user memory.

For comment responses, typically a content buffer is allocated to hold the dynamically created response. Once written by the Comet servlet, this buffer can be asynchronously flushed and then returned to the buffer pool. As the response content buffer is only allocated while it is actively being used, fewer buffers are needed and thus they may be larger and reduce the chance of the dynamic content generation blocking in order to flush a buffer. Note that for this reason, Jetty discourages the use of forever frame responses that require output buffers to be held while waiting for events. I’ve written a musing about why long-polling is as good as forever frame.

In summary, Jetty only allocates buffers to active connections and then only the minimal request header buffer during comet event waits. Large content buffers can be efficiently brought into action only for the short periods that they are actually needed and used.

4 Responses to “Comet Gazing: Memory Usage”

  1. GregWilkins Says:

    In recent benchmarking, I have scaled Bayeux Cometd running on Jetty to 20,000 users per node. Each additional bayeux connection required 88k of server memory to be allocated.

    The small 4KB header buffer that Jetty allocates per connection is only 5% of this. If Jetty were to instead allocate a full input buffer of 8KB and an output buffer of 32KB for all waiting connections, then total memory usage would be increased by 40% and the buffers would represent 32% of total per user memory.

    This shows that the smart buffering of Jetty really does significantly reduce memory footprint.

  2. Kris Zyp Says:

    Greg, do you know how much memory Jetty uses in a do nothing open connection? I assume that Cometd is responsible for some of the 88K of server memory in your Cometd tests.

  3. GregWilkins Says:

    Kris, I just did a quit test of open a couple of hundred connections to a Jetty server that opened a connection, made a request and suspended on a continuation. The incremental cost appears to only be less than 10k per connection, but I’d need to do more tests in better controlled circumstances to pin that down to an exact number.

    Of course, if the connections were truely idle, ie no request and no continuation, then the memory requirements would be even less. At this level it will also depend a lot on what data structures are expanded and at what point. It may be that you can add 512 connections for little or no cost, but that the next one will cost a few extra KB as hashsets and select sets are resized.

    I’ll try to find the time over the next few days to come up with some more definite numbers.

  4. Kris Zyp Says:

    Greg, Wow.

Copyright 2015 Comet Daily, LLC. All Rights Reserved