The Five Key Metrics of a High-Performance Comet Server

by Mihai RotaruJanuary 17th, 2011

Today, many Comet servers come with vague information regarding their performance. However, performance is one of the most important features of a Comet server, a primary reason to exist besides a traditional web server. The purpose of this article is to present five important metrics and explain why they are essential when looking at a Comet server from a performance perspective. Some concrete values for each metric are also provided, which can be used as a good basis for comparison.

The concrete values of the metrics are obtained from a recent benchmark of Migratory Push Server. A complete kit to replay the benchmark can be freely obtained from Migratory.

High Vertical Scalability

The scalability of a Comet server is determined by the number of users it can handle concurrently from a single instance.

A high-performance Comet server should achieve high scalability when running on a less expensive machine and it should achieve even higher scalability when running on more expensive machines. When this is true, one can say that the high-performance Comet server scales vertically.

Another method to achieve scalability is to use multiple instances of a Comet server. This is called horizontal scalability. While support for horizontal scalability is a must for any enterprise Comet server – because it usually represents the basis for other important features of a Comet server such as load-balancing, fault-tolerance, and guaranteed message delivery – horizontal scalability alone could increase significantly the total cost of ownership or be simply insufficient in some situations.

At Migratory, it is not unusual to receive inquires asking for HTTP streaming to millions of concurrent users. A Comet server handling thousands of concurrent users is clearly not an option, even if it scales horizontally. Obviously, in such situations, multiple instances of a Comet server must be used on multiple machines, but each instance must be vertically scaled as far as possible before scaling horizontally by adding more machines.

Example — Migratory Push Server is able to handle 1 million concurrent users from a single instance running on an entry-level machine (see the section “Extreme Number of Clients” of the benchmarks).

Low Message Latency

The latency of a Comet server is defined as the time for a message to propagate from the server side to the user. It can be accurately computed as the delta between the time a message is created on the server side and the time the message is received by the user.

Concerning the latency of a Comet server, the most important statistics to look at are its average and standard deviation.

Example — Migratory Push Server is able to push real-time data to 10,000 users where each user receives 100 messages/second, each message of 50 bytes, with average latency of 143 milliseconds and standard deviation of 79 milliseconds, while running on an entry-level machine utilizing a 1Gb Ethernet network (see the section “Very High Update Rates” of the benchmarks).

Low Bandwidth Consumption

To achieve low bandwidth consumption, the protocol used by the Comet server to deliver data to its users must be as efficient as possible. It should add only a small overhead to the actual data payload.

Example — Migratory Push Server adds to each message the following overhead to the actual data payload: 42 bytes when pushing data to the Internet Explorer, and 20 bytes for all the other browsers. The overhead is constant and also includes information used for the guaranteed delivery of data.

The bandwidth used to push real-time data with Migratory Push Server to 16,000 users where each user receives 5 messages/second, each message of 1 kilobyte, is 85 megabyte/second, while the actual data payload is about 79 megabyte/second (see the section “Large Data Messages” of the benchmarks).

High Throughput

The throughput is the average rate of data delivered by a Comet server.

Example — Migratory Push Server running on an entry-level machine saturates 1 Gb Ethernet being able to push real-time data with a throughput of 106 megabyte/second to 10,000 users where each user receives 5 messages/second, each message of 2 kilobytes (see the section “2-Kilobytes Messages” of the benchmarks).

High Connection Rate

The connection rate of a Comet server is defined as the rate of new, established connections it can accept.

This metric is particularly important for an application where many users connect frequently to it. Also, in a fault-tolerant deployment using multiple instances of a Comet server, after the failure of one instance, the remaining instances should be able to rapidly absorb the users of the failed instance.

Example — Migratory Push Server is able to accept new users with an average connection rate of 5000 users/second.

Conclusion

The five dimensions of performance above are essential for any high-performance Comet server. They can be measured independently of the specific architecture of a particular Comet server. In this way, they could be used for benchmark comparisons of the Comet servers.

[Slashdot] [Digg] [Reddit] [del.icio.us] [Facebook] [Technorati] [Google] [StumbleUpon]
Comet Support by SitePen

5 Responses to “The Five Key Metrics of a High-Performance Comet Server”

  1. Martin Tyler Says:

    Good post Mihai, and good results. Good to see you are still basing your tests on my scenarios. I actually changed the medium and high scenarios recently, to increase the backend update rate, so they are not all the same at the backend.

    Medium is now 4000 subjects, therefore 2000 msgs/sec.
    High is now 10000 subjects, therefore 5000 msgs/sec.

    Something to consider next time you run some benchmarks.

    Do you have customers using 1k or 2k messages?

  2. MihaiRotaru Says:

    Thanks much for your feedback, Martin.

    As I said in my post, one can often see random claims about performance of Comet servers as currently there is no benchmarking standard in the Comet server industry.

    Caplin Liberator is one of the few Comet servers that published comprehensive performance results and we benchmarked Migratory Push Server based on the existing performance results using comparable scenarios and hardware. Also, we added new scenarios for large/huge number of users and large data messages. Scenarios for the connection rates will be also added soon.

    Concerning the streaming of large messages. Streaming of large data can be necessary for many applications when batching is enabled in the Comet server. We at Migratory also received demands for streaming of large data messages. So, I think benchmark scenarios for large messages is useful.

    Let’s hope the metrics listed above will help other Comet vendors or independent testing companies to produce new comparative benchmarks.

    Best regards,
    Mihai

  3. Ahmed Charfeddine Says:

    hi Mihai,
    Thank you for your precious article.

    Regarding message latency, is it not better to consider
    the time difference between when the message is sent by client
    and reception time ? Because, I think it is impossible to synchronize
    clocks to a precision lower than the range of latency values. Maybe I am wrong on this.

    However, also when there are many connected users, this latter way of measuring
    can include the server ability to handle the load.

    of course request should contain timestamp which the server copies into
    the corresponding response.

  4. MihaiRotaru Says:

    Hi Ahmed,

    Thanks for your question and sorry for delayed answer (not sure why I wasn’t notified of your comment).

    In our tests, the benchmark publisher and one instance of the benchmark client (we used up to seven client instances each running on a different machine to simulate up to 1 million users) ran on the same machine.

    Each message is timestamped by the publisher at the creation time, then that message is sent to the push server (running on another machine) which distributes it to the benchmark client. The benchmark client computes the message latency as the difference between the time when it receives that message and the timestamp of the message. Because both the client and the publisher are on the same machine, the latency is accurately computed.

    Here you can find the setup diagram:

    http://migratory.ro/documentation/doc/html/Benchmark/Benchmark.html#SECTION00420000000000000000

    But unless one is interested in microsecond latencies, a time synchronization with ntp can be acceptable too.

    Regards,
    Mihai

  5. billig tandblekninng med laser Says:

    billig tandblekninng med laser…

    [...]v Thanks for the sensible critique. Me …

Leave a Reply



Copyright 2014 Comet Daily, LLC. All Rights Reserved