Comet Gazing is a semi-regular feature on Comet Daily. We pose a question to our Contributors and post the answers.
This time, we’ve asked our contributors the question: “In your experience with Comet development, what solution to a problem are you most proud of, and how did you accomplish it?”
There are lots of things we are proud of with Liberator. It has various high level features that solve problems common to financial data, but could also be useful in other areas. However, there are two areas that we are proud of.
Firstly, with Liberator itself, we are proud of the raw performance, in benchmarks we have achieved 1 million client updates per second. When Liberator was first written one of the main reasons was to achieve better performance than our previous solution which was a Windows C++/Java hybrid solution with a thread per user architecture. We chose to start from scratch and write a C application primarily on Unix (Linux and Solaris). We still used threads, but decoupled from the number of users, each thread running an asynchronous event loop based on select (later to become poll and then epoll and devpoll on Linux and Solaris respectively). With this low level base the application built on top was written with performance as a main criteria and continuously benchmarked.
Secondly, Caplin Trader. Comet technology is not much use without applications using it. At Caplin our main focus is on our Caplin Trader product, this is a complete framework for building financial trading portals. Out of the box it comes as a complete demo showing FX and FI trading and it is intended that this is then developed, using the integration points and configuration, to meet the business needs of the project. The reason we are proud of it is that it shows what is possible in a browser - using native browser technology to create a desktop-like interactive application.
Frankly, in my eight-year experience with Lightstreamer, I had to face so many challenges, that it is difficult now to recall them all or classify them by importance… I will pick some random memories.
When we started, in 2000, the market required us to provide full Comet support on Netscape Navigator 4 and Internet Explorer 4… No XHR, no JS exception management, no DOM manipulation, etc. It was a pain, but it compelled us to keep our client code extremely portable. I am still proud of how we manage to use Netscape Navigator’s proprietary LAYER and ILAYER tags to update the page in real time. For the records, you can still check out a demo that works on Netscape 4!
In 2004, we had to face the server-side scalability issue. Until that time, we relied on a traditional architecture based on the “one-thread-per-connection” model. Then, a new customer put a very demanding benchmark requirement in their contract with us. If we passed their scalability test, they would buy, otherwise… “let’s be just friends”. We spent all our energy in re-designing the Kernel of Lightstreamer server to decouple thread and connections, adopting Java NIO, and optimizing the event processing engine. We passed the benchmark.
But Sun’s Java NIO lib was not stable enough for quite a long time. I remember that early in the customer’s test reported periods of 100% CPU consumption on Linux without any significant load on the Server. It was not easy to nail the problem, but we ended up discovering a bug in the NIO implementation for Linux. We implemented our own work around, until Sun fixed the issue.
I think I could go on recalling anecdotes of problems and solutions for hours, but I will stop here. I will just report the feeling that these experiences left me with. In eight years, we have faced at least ten tipping points, that is, times when the technical issue was so difficult to analyze and/or solve that we had to consider the option of giving up. But persistence has always paid off, bringing fresh energy and enthusiasm to our team after seeing the solution work like a charm…
Ah, here I focused on the technical problems only. Another chapter would be talking about the marketing/sales/communication/evangelism problems with Comet!
The platform provides for a variety of home control services and applications. The embedded software, ControlPoint, provides a best-of-breed networked user-interface available on home computers, televisions, mobile phones, and touch screens, aka the “4-screen” architecture. The server software, Portal Server, completes the distributed software system, while also providing carrier-grade remote management for technical support personnel and remote access for end-users.
The implementation wasn’t trivial at all, as the client used Comet for plain and ssl connections, and several JavaVM had been in use (embeded, normal). They all pose challenges in term of Comet support. For example, they all close a connection differently. With GlassFish, we have an API that delivers the remote close information to a Comet application, and based on VM, browser, connection, etc., it was a challenge to unify all use cases and make sure the 4homemedia application recovers properly when a Comet connection is closed.
The application uses Flash and AJAX for communicating with GlassFish. And just seeing what they sell, it’s amazing to think this is Comet under the hood, and without Comet, it would not have worked.
Cometd + Jetty
Three solutions common to mind:
- Scaling to over 25,000 simultaneous long-poll requests within the servlet model
- The inclusion of suspend/resume semantics in the Servlet-3.0 specification
- The Cometd implementation’s ability to self-heal after network problems