Comet in Newspapers: The Spanish Lottery Experiment

by Jose Maria ArranzDecember 22nd, 2009

On December 22nd, the popular Spanish Christmas Lottery took place. This lottery is arranged by the Spanish Government and has a very long tradition in Spain, especially at Christmas when the rewards are the greatest.

The Spanish Lottery

The best prize is big, but the probability to win this reward, “el Gordo” (”the biggest”), is disappointing, 1/165.750.000, if you just buy one ticket (”décimo”) for 20 euros (around $28 USD).

While I would prefer to waste this money in a casino game where the probability to win is not so low, it is very difficult to escape from this well established and popular tradition (beyond gambling).

The raffle is itself a show because nobody knows when the number with “el gordo” is going to be chosen from a hopper containing all of numbers because concurrently another hopper contains the prizes (containing the big ones and many small prizes). Because many more numbers are taken out randomly with minor prizes, the show takes several hours.

An introduction to the Spanish gambling culture is not the objective of this article, the subject is of course Comet.

Several million people follow this Christmas special event over the radio, television and of course the Internet, especially online newspapers. Are these newspapers using a Comet technique to notify users in real-time when the best prizes are known?

The Experiment

I have watched the most important Spanish online newspapers (elpais.com, elmundo.es, abc.es, 20minutos.es, elperiodico.com, cadenaser.com) to check if the number with “el Gordo” is shown in real-time (listening to the radio is an effective way to know when this occurs). When it comes to technology, Spanish online newspapers are very advanced and have no reason to envy any other international newspaper.

In spite of this, the answer to the question is… NO!!

No newspaper was using any kind of Comet technology or even simple Ajax polling. They all are using a typical full page reload using a timer or a refresh meta tag, typically with an interval of 10 – 15 minutes. This interval is too large and is definitively not real-time in this case.

When “el Gordo” was removed from the hopper, people listening to the radio, usually at work (yes, listening to the radio is usually tolerated this day), compulsively reloaded their preferred online newspaper to check the exact number, because this number, obviously, was not shown in their outdated web page. Consequence: some of these newspaper web sites were stalled for a while.

Can Comet help the Spanish Gambling Culture?

Do not take this question too seriously. This “experiment” is an excuse, based on a concrete and real example, to speculate about how Comet technologies can be introduced to online newspapers or similar web sites with expected, but unpredictable, spikes in traffic.

Comet in Online Newspapers

People like to keep open the main page of their preferred online newspaper to check any change. The main page of an online newspaper is usually (ever?) updated using a simple full page refresh with a JavaScript timer or a refresh meta tag directive.

Full reload of the newspaper’s main page is especially ugly for users and expensive for the server because it contains a substantial amounts of information. Reloading completely takes time, bandwidth and server resources to rebuild the web page because usually newspapers make use of template processors and sometimes querying databases, as with any typical web application.

Online newspapers are designed for the desktop using JavaScript and large screens. Usually there are alternative mobile versions, but most of them do not use JavaScript.

Excluding from this analysis mobile based versions, newspapers make some use of JavaScript and DHTML because web site designers know that virtually 100% of visitors are using modern browsers like Internet Explorer 6+, or those derived from Gecko (FireFox, Mozilla, Camino…) or WebKit (Safari, Chrome). All of these browsers of course have support for Ajax.

We know there are several non-Ajax techniques for Comet, like the forever page, forever frame or script tag long-polling. These techniques are not as clean as Ajax-based alternatives, so I will limit this analysis to Ajax polling and long-polling techniques for this article.

Ajax polling and long-polling can improve the real-time experience of online newspaper readers without excluding users for technological reasons (users with very old browsers are in fact already excluded).

Before trying to add Comet to the news publishing world, one limitation is very important to take into consideration: because most online newspapers are free for users and financed by advertisements, we cannot expense much server power and memory per user because this revenue model only scales with million of readers with a very small cost per user. Web applications for online newspapers are basically stateless. Bandwidth is not a problem because any Comet technique used is a bandwidth saver and usually Comet is a power saver too, but this cannot be said for memory consumption with Comet connection usage.

Comet (Ajax polling or long polling) could be used on two levels:

  1. To provide an almost real-time updating service of something specific
  2. As a replacement of the old-fashioned full page reload technique

Specific Real-Time Services Based on Comet

The Spanish Christmas Lottery is a very good example of a specific service that could be done with Comet. This event is short in duration and occurs over a reasonably small time frame, so it is worth the effort to spend a bit more server power (it depends on the Comet technique used and the update interval).

For instance, using Ajax polling with an interval of one minute or less, the server can send to the client new reward numbers in almost real-time. The return of most of Ajax requests will be empty (almost no server power and bandwidth used) and will help to avoid the issue of compulsive user reload. What is more expensive: X requests with empty responses, or one request with a useless full page reload? It depends on X and how much a full page reload costs.

Long polling can be more bandwidth effective and can offer the user instant notification, but the server cost can increase because it needs more server resources (connections and memory).

Other similar use cases are notification events of breaking news or during sports games.

The ItsNat Approach

Using ItsNat, this can be easily achieved using an Ajax timer (polling).

On the server, because the dynamic zone is small, most of the server DOM will be declared as static, cached as text (DOM serialized as markup) and shared between all users reducing to the minimum the server memory used per user (client page). ItsNat caching is very important because the memory used by a user must be very small to keep the business need of low small cost per user.

Another alternative is having a fully-static server-side DOM (hence fully cached and shared between users) and dynamic changes served from server as JavaScript code (calling ItsNatDocument.addCodeToSend(String)) to the client to render the dynamic zone usually with the help of a JavaScript framework. In spite of ItsNat, this case is a client-centric solution and ItsNat can be replaced with another technology on the server and toolkits like Dojo can provide Ajax polling.

Replacing the Full Page Reload Technique with Comet

This problem is, by far, more complex.

With Comet, the web page would change partially and only if necessary, avoiding full reloads, reducing server power and bandwidth. However, adding Comet may be analyzed with care because in this context one important piece of this puzzle is server memory. The web application of a newspaper is basically stateless, there is not very much personalization and all users usually see the same content. Adding too much state to the server can increment the server costs (as said before in a service financed with ads), that is to say the web application must remain mainly stateless.

Investigating strategies:

A) Client-centric Approach:

In this kind of problem a JavaScript framework with Comet capabilities like Dojo seems to shine because “the application” is brought to the client and only the strictly necessary user data lives in the server because the client is responsible to keep track of visual rendering and state. For instance the server can instruct the client using Comet (polling or long polling) to render new news, to change the ads showed and so on.

But in this case, what is the application? The main page of a newspaper is an interesting example of a highly flexible web application where data and visual appearance are changing frequently. Sometimes these changes are unpredictable when something very important occurs.

Partially updating this kind of page in an unpredictable “application” is hard to drive and code using JavaScript, because the client code may be outdated, and new code should be injected replacing the old code (the client needs to be “reprogrammed”).

Maybe the most effective action in this case is to notify the client for a full page reload (for instance sending a window.location.reload(true)). So the promise of no full reload is not fulfilled in all cases.

B) Server-centric Approach:

Usually Ajax based server-centric frameworks keep track of the visual state in the server, any change in data model is rendered and the client is updated accordingly. If any part of the web page can change, the page is fully dynamic and must be fully saved in the server. This visual state can use too much memory (maybe valid in any typical web application but not for a newspaper). ItsNat is no different because it uses DOM to save this state in server and in theory any part of the page can change (no caching).

Another problem: if any part of the web page can change, how do you modify the visual state within the server memory programmatically without discarding the current state? In theory ItsNat can fully change the page in server and in client with no reload, so an interesting approach would be to change the original file template and automatically update the DOM tree of the live page with changes. This is an interesting approach to investigate.

The easier solution in ItsNat is to update the file template in server accordingly, notify the server of this change and with a long-polling event, or when the client queries the server to detect any new change (Ajax polling), the server will send a window.location.reload(true) JavaScript instruction or similar to fully update the page. In this case the visual state in the server is almost useless because the page is fully reloaded when something important changes, and the partial update requisite is not fulfilled. So again the promise of no full page reload is not fulfilled.

Conclusion

A newspaper is not a typical predictable and fixed web application. Comet (Ajax polling or long-polling) can be used to partially update some concrete and predictable parts of the web page with client or server-centric strategies, and the cost per user may be acceptable.

If the visual structure of the page randomly changes the most effective strategy is a full reload because the “application” has changed and the visual state saved in server and in client is no longer valid and is very hard to update partially. In this case Comet (polling and maybe long-polling) can be used to detect when the page must be fully reloaded, saving significant bandwidth and server power, but the promise of partial rendering of any part of the page is not fulfilled, so typical Ajax approaches, both client and server-centric may be too inflexible and/or consume too much memory.

This article speculates about how to introduce Comet in the main page of online newspapers, this problem is relatively new and open to ideas and experiences.

What do you think?
What kind of strategy would you choose?

4 Responses to “Comet in Newspapers: The Spanish Lottery Experiment”

  1. Alessandro Alinone Says:

    Jose,

    A very nice example of a Spanish online newspaper that uses Comet in an extensive way is “El Economista”: http://www.eleconomista.es

    They integrated Lightstreamer in all the pages that show real-time prices and indexes, including the home page.

    So, in the home page you can see indexes “IBEX 35″, “DAX 30″, etc. pushing live updates. Click on one of them to drill down into the full list of stocks that compose the index, again updated in real time.

    I agree with you that when it comes to technology, Spanish online newspapers are very advanced.

  2. Jose Maria Arranz Says:

    Interesting example of “some concrete and predictable” application using Ajax polling.

  3. Alessandro Alinone Says:

    Well, it does not use Ajax polling at all…
    It tries with streaming (forever frame) and falls back to long polling if streaming is blocked. But traditional Ajax polling is not used.

  4. Comet « Bloc d’en Sergi Grau Says:

    [...] alguns usos de la tecnlogia COMET. He publicado un artículo en CometDaily.com con el título Comet in Newspapers: The Spanish Lottery Experiment, el artículo reflexiona sobre las posibilidades de introducir tecnologías de tiempo real en los [...]


Copyright 2015 Comet Daily, LLC. All Rights Reserved