On December 22nd, the popular Spanish Christmas Lottery took place. This lottery is arranged by the Spanish Government and has a very long tradition in Spain, especially at Christmas when the rewards are the greatest.
The Spanish Lottery
The best prize is big, but the probability to win this reward, “el Gordo” (”the biggest”), is disappointing, 1/165.750.000, if you just buy one ticket (”décimo”) for 20 euros (around $28 USD).
While I would prefer to waste this money in a casino game where the probability to win is not so low, it is very difficult to escape from this well established and popular tradition (beyond gambling).
The raffle is itself a show because nobody knows when the number with “el gordo” is going to be chosen from a hopper containing all of numbers because concurrently another hopper contains the prizes (containing the big ones and many small prizes). Because many more numbers are taken out randomly with minor prizes, the show takes several hours.
An introduction to the Spanish gambling culture is not the objective of this article, the subject is of course Comet.
Several million people follow this Christmas special event over the radio, television and of course the Internet, especially online newspapers. Are these newspapers using a Comet technique to notify users in real-time when the best prizes are known?
I have watched the most important Spanish online newspapers (elpais.com, elmundo.es, abc.es, 20minutos.es, elperiodico.com, cadenaser.com) to check if the number with “el Gordo” is shown in real-time (listening to the radio is an effective way to know when this occurs). When it comes to technology, Spanish online newspapers are very advanced and have no reason to envy any other international newspaper.
In spite of this, the answer to the question is… NO!!
No newspaper was using any kind of Comet technology or even simple Ajax polling. They all are using a typical full page reload using a timer or a refresh meta tag, typically with an interval of 10 – 15 minutes. This interval is too large and is definitively not real-time in this case.
When “el Gordo” was removed from the hopper, people listening to the radio, usually at work (yes, listening to the radio is usually tolerated this day), compulsively reloaded their preferred online newspaper to check the exact number, because this number, obviously, was not shown in their outdated web page. Consequence: some of these newspaper web sites were stalled for a while.
Can Comet help the Spanish Gambling Culture?
Do not take this question too seriously. This “experiment” is an excuse, based on a concrete and real example, to speculate about how Comet technologies can be introduced to online newspapers or similar web sites with expected, but unpredictable, spikes in traffic.
Comet in Online Newspapers
Full reload of the newspaper’s main page is especially ugly for users and expensive for the server because it contains a substantial amounts of information. Reloading completely takes time, bandwidth and server resources to rebuild the web page because usually newspapers make use of template processors and sometimes querying databases, as with any typical web application.
We know there are several non-Ajax techniques for Comet, like the forever page, forever frame or script tag long-polling. These techniques are not as clean as Ajax-based alternatives, so I will limit this analysis to Ajax polling and long-polling techniques for this article.
Ajax polling and long-polling can improve the real-time experience of online newspaper readers without excluding users for technological reasons (users with very old browsers are in fact already excluded).
Before trying to add Comet to the news publishing world, one limitation is very important to take into consideration: because most online newspapers are free for users and financed by advertisements, we cannot expense much server power and memory per user because this revenue model only scales with million of readers with a very small cost per user. Web applications for online newspapers are basically stateless. Bandwidth is not a problem because any Comet technique used is a bandwidth saver and usually Comet is a power saver too, but this cannot be said for memory consumption with Comet connection usage.
Comet (Ajax polling or long polling) could be used on two levels:
- To provide an almost real-time updating service of something specific
- As a replacement of the old-fashioned full page reload technique
Specific Real-Time Services Based on Comet
The Spanish Christmas Lottery is a very good example of a specific service that could be done with Comet. This event is short in duration and occurs over a reasonably small time frame, so it is worth the effort to spend a bit more server power (it depends on the Comet technique used and the update interval).
For instance, using Ajax polling with an interval of one minute or less, the server can send to the client new reward numbers in almost real-time. The return of most of Ajax requests will be empty (almost no server power and bandwidth used) and will help to avoid the issue of compulsive user reload. What is more expensive: X requests with empty responses, or one request with a useless full page reload? It depends on X and how much a full page reload costs.
Long polling can be more bandwidth effective and can offer the user instant notification, but the server cost can increase because it needs more server resources (connections and memory).
Other similar use cases are notification events of breaking news or during sports games.
The ItsNat Approach
On the server, because the dynamic zone is small, most of the server DOM will be declared as static, cached as text (DOM serialized as markup) and shared between all users reducing to the minimum the server memory used per user (client page). ItsNat caching is very important because the memory used by a user must be very small to keep the business need of low small cost per user.
Replacing the Full Page Reload Technique with Comet
This problem is, by far, more complex.
With Comet, the web page would change partially and only if necessary, avoiding full reloads, reducing server power and bandwidth. However, adding Comet may be analyzed with care because in this context one important piece of this puzzle is server memory. The web application of a newspaper is basically stateless, there is not very much personalization and all users usually see the same content. Adding too much state to the server can increment the server costs (as said before in a service financed with ads), that is to say the web application must remain mainly stateless.
A) Client-centric Approach:
But in this case, what is the application? The main page of a newspaper is an interesting example of a highly flexible web application where data and visual appearance are changing frequently. Sometimes these changes are unpredictable when something very important occurs.
Maybe the most effective action in this case is to notify the client for a full page reload (for instance sending a window.location.reload(true)). So the promise of no full reload is not fulfilled in all cases.
B) Server-centric Approach:
Usually Ajax based server-centric frameworks keep track of the visual state in the server, any change in data model is rendered and the client is updated accordingly. If any part of the web page can change, the page is fully dynamic and must be fully saved in the server. This visual state can use too much memory (maybe valid in any typical web application but not for a newspaper). ItsNat is no different because it uses DOM to save this state in server and in theory any part of the page can change (no caching).
Another problem: if any part of the web page can change, how do you modify the visual state within the server memory programmatically without discarding the current state? In theory ItsNat can fully change the page in server and in client with no reload, so an interesting approach would be to change the original file template and automatically update the DOM tree of the live page with changes. This is an interesting approach to investigate.
A newspaper is not a typical predictable and fixed web application. Comet (Ajax polling or long-polling) can be used to partially update some concrete and predictable parts of the web page with client or server-centric strategies, and the cost per user may be acceptable.
If the visual structure of the page randomly changes the most effective strategy is a full reload because the “application” has changed and the visual state saved in server and in client is no longer valid and is very hard to update partially. In this case Comet (polling and maybe long-polling) can be used to detect when the page must be fully reloaded, saving significant bandwidth and server power, but the promise of partial rendering of any part of the page is not fulfilled, so typical Ajax approaches, both client and server-centric may be too inflexible and/or consume too much memory.
This article speculates about how to introduce Comet in the main page of online newspapers, this problem is relatively new and open to ideas and experiences.
What do you think?
What kind of strategy would you choose?