This is the third article in a series aimed at illustrating how to develop Lightstreamer v.3.x Adapters based on various technologies:

1. “Hello World” with Lightstreamer: An introduction to Lightstreamer’s data model, Web Client API, Java Data Adapters, and Server deployment, through the development of a very basic “Hello World” application.
2. “Hello World” for .NET with Lightstreamer: The .NET version of the Data Adapter used in the “Hello World” application, showing both a C# and a Visual Basic port.

In this third installment, I will show a Data Adapter that communicates with the Lightstreamer Server through plain TCP sockets, instead of leveraging higher level abstractions as I did with the Java API and the .NET API. The rationale for this is to enable the development of Data Adapters based on technologies other than Java and .NET. This way, it is possible to inject real-time updates into Lightstreamer Server from programs written in C, PHP, Ruby, or any other language that allows client TCP socket programming.

The Architecture

On the client side, we will keep the same exact HTML front-end used in the two previous installments. For an explanation of the HTML/JavaScript code, please see the “Creating the front-end” section of the first article.

On the server side, we will leverage the Lightstreamer Adapter Remoting Infrastructure (ARI):

This is the same architecture used in the .NET example, but in this case the Remote Data Adapter is not a .NET application, but any process that opens two sockets with the Proxy Data Adapter and implements the ARI Protocol over TCP. As in the previous examples, we will not code a custom Metadata Adapter, but will use a default one. So the Remote Metadata Adapter will not be present.

Let’s recap. On the server side, this new example will be comprised of:

• Proxy Data Adapter: A ready-made Data Adapter, based on Java, which is provided as part of the Adapter Remoting Infrastructure (available in the free and commercial distributions of Lightstreamer Server v. 3.x).
• Remote Data Adapter: The subject of this article…
• LiteralBasedProvider: A ready-made Metadata Adapter, based on Java, which is provided as part of all the free and commercial distributions of Lightstreamer Server v. 3.x. The LiteralBasedProvider replaces the Proxy Metadata Adapter in the diagram above.

So, what about the Remote Data Adapter? We could implement it in any language that supports socket programming, but in this article we will do something more interactive, which does not require any programming. We will use a very normal telnet client to connect to the Proxy Data Adapter and will manually play the ARI Network Protocol. Should be fun…

The Proxy Data Adapter listens on two TCP ports and the Remote Data Adapter has to create two sockets. One socket is used for interactions based on a request/response paradigm (this is a synchronous channel). The other socket is used to deliver asynchronous events from the Remote Adapter to the Proxy Adapter (this is an asynchronous channel). Therefore, our Remote Data Adapter will be comprised of two telnet windows.

Setting Up the System

If you didn’t already do it in the previous installments, download and install Lightstreamer Moderato.

Now, let’s deploy the Adapter pair (Proxy Data Adapter + LiteralBasedProvider) in a new folder. Go to the “adapters” folder of your Lightstreamer Server installation. You could already have a “StockList” folder (containing a pre-installed demo Adapter), plus a “HelloWorld” and a “ProxyHelloWorld” folders (if you followed the previous two articles). In any case, you don’t need these three folders, as we are going to create a new one for this example.  Let’s call this new folder (created inside “adapters”) ”ProxyHelloWorldSockets“. Then create a “strong>lib” folder inside “ProxyHelloWorldSockets” and copy the “ls-proxy-adapters.jar” file from “Lightstreamer/DOCS-SDKs/sdk_adapter_remoting_infrastructure/lib” to “Lightstreamer/adapters/ProxyHelloWorldSockets/lib”.

Create a new file in “Lightstreamer/adapters/ProxyHelloWorldSockets”, call it “adapters.xml“, and use the following contents:

<?xml version="1.0"?>
<adapters_conf id="PROXY_HELLOWORLD_SOCKETS">
 
  <metadata_provider>
    <adapter_class>com.lightstreamer.adapters.metadata.LiteralBasedProvider</adapter_class>
  </metadata_provider>
 
  <data_provider>
    <adapter_class>com.lightstreamer.adapters.remote.data.NetworkedDataProvider</adapter_class>
      <param name="request_reply_port">7001</param>
      <param name="notify_port">7002</param>
      <param name="timeout">36000000</param>
  </data_provider>
 
</adapters_conf>

I have chosen TCP port 7001 for the request/response channel and TCP port 7002 for the asynchronous channel. Feel free to use different ports if such numbers are already used on your system.

Notice the “timeout” parameter. It sets the maximum time the Proxy Adapter will wait for a response from the Remote Adapter after issuing a request. The default value is 10 seconds, but in this case the Remote Adapter is played by humans, so I have configured a very high value (10 hours), to do relaxed experiments without the pressure of any timeouts.

Finally, let’s deploy the simple Web resources. Create a “HelloWorldSockets” folder under the “Lightstreamer/pages” folder. Put in this new folder the “index.htm” file available from this archive (the page is very similar to those of two previous articles; we just changed the Adapter name to “PROXY_HELLOWORLD_SOCKETS” and added the “pushtable.setSnapshotRequired(true)” line to always get the current state of the fields). Now we have to deploy the JavaSscript libraries used by our page. Let’s create an “LS” folder under “HelloWorldSockets” and copy to it all the files located in “Lightstreamer/DOCS-SDKs/sdk_client_web/lib”. For further details on the page deployment, please see the first article.

Let’s Play

Start Lightstreamer Server from a command or shell window (I will call this the “log window“). You should see something like this:

[...]
28.Jul.08 17:56:01,437 < INFO> Lightstreamer Server starting in Moderato edition
28.Jul.08 17:56:01,468 < WARN> JMX management features not available with the current license
28.Jul.08 17:56:01,500 < INFO> Started HTML Adaptor for JMX on port 6666
28.Jul.08 17:56:01,531 < INFO> Started JMXMP Connector for JMX on port 9999
28.Jul.08 17:56:01,578 < INFO> Loading Metadata Provider PROXY_HELLOWORLD_SOCKETS
28.Jul.08 17:56:01,578 < INFO> Loading Data Provider PROXY_HELLOWORLD_SOCKETS
28.Jul.08 17:56:01,593 < INFO> Connecting...
.

The Server is waiting for the “PROXY_HELLOWORLD_SOCKETS” Remote Data Adapter to connect to the two TCP ports we specified above (7001 and 7002). The Server initialization will complete only when these connections succeed.

Now open other two command or shell windows.
In the first one (which I will call the “request/response window“) type:

     telnet localhost 7001

In the second one (which I will call the “async window“) type:

     telnet localhost 7002

The Server initialization will complete and in the log window you should see something like this:

[...]
28.Jul.08 17:56:01,437 < INFO> Lightstreamer Server starting in Moderato edition
28.Jul.08 17:56:01,468 < WARN> JMX management features not available with the current license
28.Jul.08 17:56:01,500 < INFO> Started HTML Adaptor for JMX on port 6666
28.Jul.08 17:56:01,531 < INFO> Started JMXMP Connector for JMX on port 9999
28.Jul.08 17:56:01,578 < INFO> Loading Metadata Provider PROXY_HELLOWORLD_SOCKETS
28.Jul.08 17:56:01,578 < INFO> Loading Data Provider PROXY_HELLOWORLD_SOCKETS
28.Jul.08 17:56:01,593 < INFO> Connecting...
28.Jul.08 18:31:05,078 < INFO> Connected
28.Jul.08 18:31:05,093 < INFO> Loading Metadata Provider STOCKLISTDEMO
28.Jul.08 18:31:05,093 < INFO> Loading Data Provider STOCKLISTDEMO
28.Jul.08 18:31:05,234 < INFO> Pump pool size set by default at 1
28.Jul.08 18:31:05,265 < INFO> Notify receiver '#1' starting...
28.Jul.08 18:31:05,265 < INFO> Events pool size set by default at 1
28.Jul.08 18:31:05,265 < INFO> Lightstreamer Server 3.4.8 build 1398 starting...
28.Jul.08 18:31:05,281 < INFO> Request sender '#1' starting...
28.Jul.08 18:31:05,281 < INFO> Reply receiver '#1' starting...
28.Jul.08 18:31:05,312 < INFO> Server "Lightstreamer HTTP Server" listening to *:8080 ...
.

Now we are ready to interact with the Server.

Let’s connect a client to activate a subscription. Open a browser window and go to: http://localhost:8080/HelloWorldSockets

In the log window you will see some information regarding the HTTP interaction between the browser and the Lightstreamer Server.

In the browser window you will see:

loading...
loading...

The “greetings” item has been subscribed too by the Client, with a schema comprised of the “message” and “timestamp” fields. The Server has then subscribed to the same item through our Remote Adapter (due to the fact that Lightstreamer Server v.3.x is based on a “Publish On-Demand” paradigm). This subscription will manifest itself as a request in the requets/response window, similar to the following:

10000011b6a823e31|SUB|S|greetings

The first string is the unique ID of that request and will change every time. Let’s respond saying that we accept such subscription. We can do this by typing the following string in the requests/response window and hitting Enter:

     10000011b6a823e31|SUB|V

Note: Replace “10000011b6a823e31″ with the actual ID you received, otherwise the subscription will not succeed and you will see a warning in the log window.

Our Remote Data Adapter has now accepted to serve events on the “greetings” item. It’s time to inject some events by hand, through the async window. With most telnet applications you will not see anything will typing in the async window, so it is better to use copy and paste. Paste the following string, then hit Enter:

     0|UD3|S|greetings|S|10000011b6a823e31|B|0|
     S|timestamp|S|Now is the time|S|message|S|Hello socket world!

Note: Make sure to paste everything on a single line (the text above was split on two lines to fit in the page). And again, replace “10000011b6a823e31″ with the actual ID you received.

Now look at the browser window and enjoy the results of this effort:

Hello socket world!
Now is the time

We can push more events on the “greetings” item, leveraging the same two fields (”message” and “timestamp”) ans sending arbitrary data. For example, paste this in the async window (always on a single line and replacing the ID):

     0|UD3|S|greetings|S|10000011b6a823e31|B|0|
     S|message|S|What do you call a fish with no eyes?|S|timestamp|S|A fsh

The Network Protocol

In the examples above we scratched the surface of the ARI Network Protocol. By delving into deeper details, you will see that it is quite straightforward. The full specification is available in the “ARI Protocol.pdf” document, which is available in the “Lightstreamer/DOCS-SDKs\sdk_adapter_remoting_infrastructure\doc” folder of any distribution of Lightstreamer Server v.3.x.

The Remote Data Adapter can only receive two synchronous requests: subscribe and unsubscribe. It can send three asynchronous events: update, end of snapshot, and failure.
The Remote Metadata Adapter (which was not covered in this article) can receive more synchronous requests, as its interface is a bit more complex than the Data Adapter, but it does not send any asynchronous events at all (in fact it uses one TCP socket only).

Final Notes

In reality, of course, you will never implement a “human-driven” Adapter as we did in this article, but this approach was useful, from a didactive perspective, to illustrate the basic principles of the Lightstreamer Adapter Remoting Infrastructure (ARI). If you need to develop an Adapter based on technologies other than Java and .NET, the ARI makes the trick.

Should you develop any Adapter in PHP, Ruby, Python , Perl, or any other language, feel free to let us know, by posting a comment here on the Lightstreamer Forums.

Gianluca Bertani has developed an open-source library that implements the Lightstreamer client/server protocol. The library, written in Objective-C, is comprised of a small set of Cocoa-native classes and is targeted to Mac OS X platforms.

The project, named iStreamLight, is hosted on Google Code and can be reached at http://code.google.com/p/istreamlight/

A quite bizarre aspect is that Gianluca cannot officially state whether the library has been tested with the new iPhone SDK too… In fact, there is still some controversy on what Apple’s license agreement allows or denies. Gianluca wrote:

I can’t reveal whether this library works or not under iPhone OS 2.0… guess? At the moment nobody can say anything about any aspect of the iPhone OS 2.0 SDK, as you can read from these discussions:

• Open Sourcing an iPhone Application acceptable under the License Agreement?
• The iPhone SDK and free software: not a match
• Apple’s iPhone contracts leave developers speechless

If you are a registered developer for iPhone, try it for yourself. But beware: don’t reveal it to anyone…

It seems that many developers are waiting for a clarification from Apple, or for a change in that license agreement, now that the SDK is out of the beta stage. In any case, I’m sure that as soon as the legal aspect are clear enough, Gianluca will show us some nice Comet applications for the iPhone.

Yesterday Finextra published the article “Facebook users want Web 2.0 banking“.

New research has found that around half of Facebook users would use Web 2.0 applications for online banking, while a quarter would even consider switching banks to obtain Web 2.0 services.

Even if the article does not mention Comet directly, it is evident that Comet capabilities would benefit several banking applications. Being able to update the contents of the various finance widgets is real time does improve the Web 2.0 experience.

Commenting on the survey findings, David Lavenda, VP, marketing and product strategy, Worklight, says: “To lose a quarter of your customers to competitors who have provided secure Web 2.0 banking would bear a significant impact on existing business. This is particularly true at a time when financial services companies are struggling to retain and acquire customers in a market that saturated with new competitors and countless new offerings.”

Comet could become a key feature, for financial services companies, to increase the level of customer retention and acquisition.

News for our readers from India. I will give a talk on Comet at Great Indian Developer Summit 2008.

Bangalore, India, May 19-23 2008
www.developersummit.com

Ajax and Comet: Implementing the Real-Time Web
May 21st, 11.25-12.25 Main Hall

Ajax has proven a successful means to revamp the Web and facilitate the migration of many applications from “thick” clients to browser-based and zero-install clients. But several applications require real-time data to be effective (for example: financial market data visualization; online auctions; online gaming; messaging systems and social networks). The Comet paradigm, as an extension to the “traditional” Ajax techniques, allows any web page to receive real-time data pushed by a specialized server. This talk will introduce Comet from both a technical and historical perspectives. Several examples will be shown and a simple application, based on the Lightstreamer Comet framework, will be developed from scratch.

Hope to see you there!

Today a customer asked me if Lightstreamer works on Firefox 3.0 beta 5. I had tested previous beta versions of Firefox 3.0 but not yet beta 5. I was so impressed with its scalability (in terms of concurrent Comet sessions opened on multiple tabs) that I had to write this musing.

Basically, I was able to open 32 instances of the Stock List Demo on 32 tabs, with no significant slackenings. I stopped, but I could definitely open many more. [I did this test on a local network. Connecting to our server from the Internet could be slower. Does anybody feel like doing this test, that is, opening more tabs until the browser hangs? If so, Install Firefox 3.0, put "http://app.lightstreamer.com/StockListDemo/" in your clipboard, then start typing control-t,control-v (or command-t, command-v for Mac users of course), Enter. Please comment below on the results].

This test is possible because Lightstreamer does not suffer from the 2-connection limit. Multiple instances of a Comet application or even different Comet applications connecting to the same server can automatically share the same physical connection. In fact, after opening these 30+ instances, a netstat confirmed that one single socket was opened to the server.

Notice that this Stock List Demo is configured with a bandwidth limit of 30 kbits/s for the Comet socket. This means that all the instances will share this bandwidth limit. After opening many instances, you should notice a reduction in the update frequency (notice I said frequency reduction, not delivery delay. This is a key point of Adaptive Streaming and Congestion Control. In other words, you will see less updates, not old/delayed updates. I will probably return on this topic in a future article).

Besides, notice that each instance does its own subscriptions, that is, only the Comet channel is shared, while the subscriptions are completely independent. This is useful because each application could want to subscribe to the same items with different schemas and/or with a different quality of service (in terms of frequencies, buffering, etc.).

Let me know your thoughts.

After publishing the article “Hello World” with Lightstreamer, I received many requests to release a version of that tutorial for .NET developers. Therefore, in this installment we will work on a .NET port of the Java Data Adapter that we illustrated in the previous article. In particular, both a C# version and a Visual Basic version will be shown.

Let’s get started

First, please take a look at the previous installment, which provides some background and the general description of the application. Notice that the front-end will be exactly the same. We created a very simple HTML page that subscribes to the “greetings” item, using the “HELLOWORLD” Adapter. Now, we will replace the “HELLOWORLD” Adapter implementation based on Java with C# and Visual Basic equivalents. On the client side, nothing will change, as server side Adapters can be transparently switched and changed, as long as they respect the same interfaces. Thanks to this decoupling provided by Lightstreamer Server, we could even do something different. For example, we could keep the Java Adapter on the server side and use Flex, instead of HTML, on the client side. Or we could use the C# Adapter on the server side and use Java, instead of HMTL or Flex, on the client side. Basically, all the combinations of languages and technologies on the client side and on the server side are supported.

As in the previous installment, these examples will be based on Lightstreamer Server v. 3.4 and Lightstreamer Web Client v. 4.2.

Give me some .NET interfaces!

Lightstreamer Server is written in Java, so its native Adapter interfaces are Java based. The .NET interfaces are added through the Lightstreamer Adapter Remoting Infrastructure (ARI). Let’s have a look at it.

ARI is simply made up of two Proxy Adapters and a Network Protocol. The two Proxy Adapters implement the Java interfaces and are meant to be plugged into Lightstreamer Kernel, exactly as we did for our original “HELLOWORLD” Java Adapter. There are two Proxy Adapters because one implements the Data Adapter interface and the other implements the Metadata Adapter interface. Our “Hello World” example uses a default Metadata Adapter, so we only need the Proxy Data Adapter.

What does the Proxy Data Adapter do? Basically, it exposes the Data Adapter interface through TCP sockets. In other words, it offers a Network Protocol, which any remote counterpart can implement to behave as a Lightstreamer Data Adapter. This means you can write a remote Data Adapter in C, in PHP, or in COBOL, provided that you have access to very standard TCP sockets.

But—here is some magic—if your remote Data Adapter is based on .NET, you can forget about direct socket programming, and leverage a ready-made library that exposes a higher level .NET interface. So, you will simply have to implement this .NET interface. Ok, let’s recap… The Proxy Data Adapter converts from a Java interface to TCP sockets. The .NET library converts from TCP sockets to a .NET interface. Clear enough?

Creating the C# Data Adapter

Let’s start with C#. Feel free to skip this section if you are only interested in the Visual Basic example (I will replicate the same comments on the code).

We should implement two classes. One (which we will call DataAdapterLauncher) contains the application’s Main and initializes the DataProviderServer (the provided piece of code that implements the Network Protocol). The other (which we will call HelloWorldAdapter) implements the actual Adapter interface.

1. Create a new C# project (I used Microsoft’s Visual C# 2008 Express Edition).
2. From the “New Project…” wizard, choose the “Console Application” template. Let’s use “adapter_csharp” as the project name.
3. From the “Solution Explorer”, delete the default Program.cs, then add a reference to the Lightstreamer .NET library: go to the “Browse” tab of the “Add Reference” dialog and point to the DotNetAdapter_N2.dll file, which you can find in the “Lightstreamer\DOCS-SDKs\sdk_adapter_dotnet\lib\dotnet_2.0″ folder of your Lightstreamer installation.

DataAdapterLauncher

Add a new class (let’s call its file DataAdapterLauncher.cs). Replace the source code with this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37

using System;
using System.Net.Sockets;
 
using Lightstreamer.DotNet.Server;
 
public class DataAdapterLauncher
{
  public static void Main(string [] args)
  {
    string host= "localhost";
    int reqrepPort = 6661;
    int notifPort = 6662;
 
    try
    {
      DataProviderServer server = new DataProviderServer();
      server.Adapter = new HelloWorldAdapter();
 
      TcpClient reqrepSocket = new TcpClient(host, reqrepPort);
      server.RequestStream = reqrepSocket.GetStream();
      server.ReplyStream = reqrepSocket.GetStream();
 
      TcpClient notifSocket = new TcpClient(host, notifPort);
      server.NotifyStream = notifSocket.GetStream();
 
      server.Start();
      System.Console.WriteLine("Remote Adapter connected to Lightstreamer Server.");
      System.Console.WriteLine("Ready to publish data...");
    } 
    catch (Exception e)
    {
      System.Console.WriteLine("Could not connect to Lightstreamer Server.");
      System.Console.WriteLine("Make sure Lightstreamer Server is started before this Adapter.");
      System.Console.WriteLine(e);
    }
  }
}

This code creates a DataProviderServer instance and assigns a HelloWorldAdapter instance (which we will define below) to it. Then, it creates two TCP client sockets, because the Proxy Data Adapter, to which our remote .NET Adapter will connect, needs two connections (but as I said, after creating these sockets, you don’t have to bother with reading and writing, as these operations are automatically handled by the DataProviderServer). Let’s use TCP ports 6661 and 6662. Assign the stream of the first socket to the RequestStream and ReplyStream properties of the DataProviderServer. Assign the stream of the second socket to the NotifyStream property of the DataProviderServer. Finally, you start the DataProviderServer.

HelloWorldAdapter

Now add another class (call its file HelloWorld.cs). Replace the source code with this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59

using System;
using System.Collections;
using System.Threading;
 
using Lightstreamer.Interfaces.Data;
 
public class HelloWorldAdapter : IDataProvider
{
  private IItemEventListener _listener;
  private volatile bool go;
 
  public void Init(IDictionary parameters, string configFile)
  {
  }
 
  public bool IsSnapshotAvailable(string itemName)
  {
    return false;
  }
 
  public void SetListener(IItemEventListener eventListener)
  {
    _listener = eventListener;
  }
 
  public void Subscribe(string itemName)
  {
    if (itemName.Equals("greetings"))
    {
      Thread t = new Thread(new ThreadStart(Run));
      t.Start();
    }
  }
 
  public void Unsubscribe(string itemName)
  {
    if (itemName.Equals("greetings"))
    {
      go = false;
    }
  }
 
  public void Run()
  {
    go = true;
    int c = 0;
    Random rand = new Random();
 
      while (go)
      {
        IDictionary eventData = new Hashtable();
        eventData["message"] = c % 2 == 0 ? "Hello" : "World";
        eventData["timestamp"] = DateTime.Now.ToString("s");
        _listener.Update("greetings", eventData, false);
        c++;
        Thread.Sleep(1000 + rand.Next(2000));
      }
  }
}

The HelloWorldAdapter class implements the IDataProvider interface (which is the .NET remote equivalent of the Java DataProvider interface).

Implement the SetListener method to receive a reference to the server’s listener that you will use to inject events.

Then, implement the Subscribe method. When the “greetings” item is subscribed to by the first user, the Adapter receives that method call and starts a thread that will generate the real-time data. If more users subscribe to the “greetings” item, the Subscribe method is not called any more. When the last user unsubscribes from this item, the Adapter is notified through the Unsubscribe call. In this case we stop the publisher thread for that item. If a new user re-subscribes to “greetings”, the Subscribe method is called again. As already mentioned in the previous installment, this approach avoids consuming processing power for items nobody is currently interested in.

The Run method is executed within the thread started by Subscribe. Its code is very simple. We create a Hashtable containing a message (alternating “Hello” and “World”) and the current timestamp. Then we inject the Hashtable into the server through the listener. We wait for a random time between 1 and 3 seconds, and we are ready to generate a new event.

Save and build

From the File menu, choose “Save All“. Type a location, for example: “c:\”.

From the Build menu, choose “Build Solution“. Your C# Data Adapter is now compiled. If you used the path above, you will find your executable file under “C:\adapter_csharp\adapter_csharp\bin\Release”. It is called “adapter_csharp.exe“. But be patient and don’t start it now, because you have to configure and start the Lightstreamer Server first. You can skip the section below, dedicated to VB, and go straight to the “Deploy the Proxy Adapter” section.

Creating the Visual Basic Data Adapter

Now let’s work with Visual Basic instead of C#.

We will create a module (DataAdapterLauncher) which contains the application’s Main and initializes the DataProviderServer (the provided piece of code that implements the Network Protocol). Then we will create a class (HelloWorldAdapter) that implements the actual Adapter interface.

1. Create a new VB project (I used Microsoft’s Visual Basic 2008 Express Edition).
2. From the “New Project…” wizard, choose the “Console Application” template. Let’s use “adapter_vb” as the project name.
3. From the “Solution Explorer”, delete the default Module1.vb, then add a reference to the Lightstreamer .NET library: go to the “Browse” tab of the “Add Reference” dialog and point to the DotNetAdapter_N2.dll file, which you can find in the “Lightstreamer\DOCS-SDKs\sdk_adapter_dotnet\lib\dotnet_2.0″ folder of your Lightstreamer installation.

DataAdapterLauncher

Add a new module (let’s call its file DataAdapterLauncher.vb). Replace the source code with this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31

Imports System.Net.Sockets
 
Imports Lightstreamer.DotNet.Server
 
Module Module1
  Sub Main()
    Dim host As String = "localhost"
    Dim reqrepPort As Integer = 6661
    Dim notifPort As Integer = 6662
 
    Try
      Dim server As New DataProviderServer()
      server.Adapter = New HelloWorldAdapter()
 
      Dim reqrepSocket As New TcpClient(host, reqrepPort)
      server.RequestStream = reqrepSocket.GetStream()
      server.ReplyStream = reqrepSocket.GetStream()
 
      Dim notifSocket As New TcpClient(host, notifPort)
      server.NotifyStream = notifSocket.GetStream()
 
      server.Start()
      System.Console.WriteLine("Remote Adapter connected to Lightstreamer Server.")
      System.Console.WriteLine("Ready to publish data...")
    Catch e As Exception
      System.Console.WriteLine("Could not connect to Lightstreamer Server.")
      System.Console.WriteLine("Make sure Lightstreamer Server is started before this Adapter.")
      System.Console.WriteLine(e)
    End Try
  End Sub
End Module

This code creates a DataProviderServer instance and assigns a HelloWorldAdapter instance (which we will define below) to it. Then, it creates two TCP client sockets, because the Proxy Data Adapter, to which our remote .NET Adapter will connect, needs two connections (but as I said, after creating these sockets, you don’t have to bother with reading and writing, as these operations are automatically handled by the DataProviderServer). Let’s use TCP ports 6661 and 6662. Assign the stream of the first socket to the RequestStream and ReplyStream properties of the DataProviderServer. Assign the stream of the second socket to the NotifyStream property of the DataProviderServer. Finally, start the DataProviderServer.

HelloWorldAdapter

Now add a class (call its file HelloWorld.vb). Replace the source code with this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49

Imports System.Collections
Imports System.Threading
 
Imports Lightstreamer.Interfaces.Data
 
Public Class HelloWorldAdapter
  Implements IDataProvider
  Private _listener As IItemEventListener
  Private go As Boolean
 
  Public Sub Init(ByVal parameters As IDictionary, ByVal configFile As String) Implements IDataProvider.Init
  End Sub
 
  Function IsSnapshotAvailable(ByVal itemName As String) As Boolean Implements IDataProvider.IsSnapshotAvailable
    Return False
  End Function
 
  Public Sub SetListener(ByVal eventListener As IItemEventListener) Implements IDataProvider.SetListener
    _listener = eventListener
  End Sub
 
  Public Sub Subscribe(ByVal itemName As String) Implements IDataProvider.Subscribe
    If itemName.Equals("greetings") Then
      Dim t As New Thread(AddressOf Run)
      t.Start()
    End If
  End Sub
 
  Public Sub Unsubscribe(ByVal itemName As String) Implements IDataProvider.Unsubscribe
    If itemName.Equals("greetings") Then
      go = False
    End If
  End Sub
 
  Public Sub Run()
    go = True
    Dim c As Integer = 0
    Dim rand As New Random()
 
    While go
      Dim eventData As IDictionary = New Hashtable()
      eventData("message") = IIf(c Mod 2 = 0, "Hello", "World")
      eventData("timestamp") = DateTime.Now.ToString("s")
      _listener.Update("greetings", eventData, False)
      c += 1
      Thread.Sleep(1000 + rand.Next(2000))
    End While
  End Sub
End Class

The HelloWorldAdapter class implements the IDataProvider interface (which is the .NET remote equivalent of the Java DataProvider interface).

Implement the SetListener subroutine to receive a reference to the server’s listener that you will use to inject events.

Then, implement the Subscribe subroutine. When the “greetings” item is subscribed to by the first user, the Adapter receives that subroutine call and starts a thread that will generate the real-time data. If more users subscribe to the “greetings” item, the Subscribe subroutine is not called any more. When the last user unsubscribes from this item, the Adapter is notified through the Unsubscribe call. In this case we stop the publisher thread for that item. If a new user re-subscribes to “greetings”, the Subscribe subroutine is called again.

The Run subroutine is executed within the thread started by Subscribe. Its code is very simple. We create a Hashtable containing a message (alternating “Hello” and “World”) and the current timestamp. Then we inject the Hashtable into the server through the listener. We wait for a random time between 1 and 3 seconds, and we are ready to generate a new event.

Save and build

From the File menu, choose “Save All“. Type a location, for example: “c:\”.

From the Build menu, choose “Build adapter_vb“. Your VB Data Adapter is now compiled. If you used the path above, you will find your executable file under “C:\adapter_vb\adapter_vb\bin\Release”. It is called “adapter_vb.exe“. As for the C# Adapter, don’t start it now, because you have to configure and start the Lightstreamer Server first.

Deploying the Proxy Adapter

Now that our remote Data Adapter is ready, we need to deploy and configure the provided Proxy Adapter within Lightstreamer Server.

Go to the “adapters” folder of your Lightstreamer Server installation. You could already have a “StockList” folder (containing a pre-installed demo Adapter) and a “HelloWorld” Adapter (if you followed the Java tutorial of the previous installment). Now create a “ProxyHelloWorld” folder inside “adapters”, and a “lib” folder inside “ProxyHelloWorld”.

Copy the “ls-proxy-adapters.jar” file from “Lightstreamer/DOCS-SDKs/sdk_adapter_remoting_infrastructure/lib” to “Lightstreamer/adapters/ProxyHelloWorld/lib”.

Create a new file in “Lightstreamer/adapters/ProxyHelloWorld”, call it “adapters.xml“, and use the following contents:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

<?xml version="1.0"?>
 
<adapters_conf id="PROXY_HELLOWORLD">
 
  <metadata_provider>
    <adapter_class>com.lightstreamer.adapters.metadata.LiteralBasedProvider</adapter_class>
  </metadata_provider>
 
  <data_provider>
    <adapter_class>com.lightstreamer.adapters.remote.data.NetworkedDataProvider</adapter_class>
    <param name="request_reply_port">6661</param>
    <param name="notify_port">6662</param>
  </data_provider>
 
</adapters_conf>

You have just deployed a new Java Adapter pair, where the Metadata Adapter is a default one (called “LiteralBasedProvider”) and the Data Adapter is the Proxy Adapter (called “NetworkedDataProvider”). This Adapter pair will be referenced by the clients as “PROXY_HELLOWORLD“.

As a final configuration, let’s tell our Web client to use this new Adapter pair, rather than those we developed last time. So just edit the “index.htm” page of the Hello World front-end (last time we deployed it under “Lightstreamer/pages/HelloWorld”) and replace:

  engine.connection.setAdapterName("HELLOWORLD");

with:

  engine.connection.setAdapterName("PROXY_HELLOWORLD");

Running the application

Now we have all the pieces ready. Let’s enjoy the results.

Start your Lightstreamer Server. When it’s up, run either the C# or the VB Remote Adapter.

Open a browser window and go to: http://localhost:8080/HelloWorld/

Final notes

Starting from these very simple code fragments we have developed in these two installments (covering basic JavaScript, Java, C#, and VB), I encourage you to experiment and create more complex applications. The full API references for all these languages are available from the links below:

• .NET API reference for Adapters (save and unblock this file from the Properties dialog)
• Java API reference for Adapters
• JavaScript API reference for Clients

All the source code described in this article is available for download.

As I already outlined in the previous installment, we used the free Moderato edition of Lightstreamer for this tutorial. Its main limitation, with respect to the commercial editions, is that it only allows 1 update/sec for each subscribed item (but you can subscribe to as many items as you want). If you need to experiment with higher-frequency events for the same item, you can use a free evaluation license of Lightstreamer Presto or Vivace.

Yesterday I found yoonew.com, a site that implements a full-blown stock market for sports tickets and leverages Comet technology.

From their “How it works” page:

Yoonew is an open market that gives true fans the opportunity to lock in ticket prices to some of the most sought-after events in the world. Buying Fantasy Seats for your favorite teams allows you to guarantee yourself a ticket to the big game if your team makes it. yoonew is the only place in the world where you can score tickets at prices BELOW face value.
Suppose you’re a die hard Giants fan and want to go to Super Bowl XLII next year. However, you only want to go if the Giants make it and you can’t afford to spend $2,000+ per ticket. Instead of buying a ticket, you can buy a Fantasy Seat for the Giants in Super Bowl XLII. If you bought 2 Fantasy Seats for $200* a piece and the Giants reach the Super Bowl, you just scored a pair of super bowl tickets and saved $3,600! If the Giants don’t make it to the Super Bowl, however, your Fantasy Seats cannot be redeemed for tickets and you lose $400. At yoonew, you only pay ONCE. There are no additional payments or hidden fees.

Now, it is interesting to notice that Yoonew employs Lightstreamer and DWR to implement real-time streaming of the market data and bidirectional communication with the server. I see they use Flash charts to display the state of prices over time and these charts are fed with streaming data too.

This really looks like a cool Comet application. Congrats to the guys who developed it!

BOSH (Bidirectional-streams Over Synchronous HTTP) is an XMPP/Jabber extension proposal.

It has already been mentioned several times on Comet Daily. For example:

• November 5th, 2007 - zeank
• February 11th, 2008 - Greg Wilkins
• January 20th, 2008 - Greg Wilkins

While re-reading the BOSH specification, I was a bit surprised to see that the authors define BOSH in explicit opposition to Comet, even though BOSH seems to be based on the Comet paradigm…

For example, in the Introduction:

BOSH achieves this efficiency and low-latency by avoiding polling without resorting to chunked HTTP responses (i.e. without using the technique which is now commonly known as “Comet”).

Then, in the Requirements (note the sentence I’ve put in bold):

The following design requirements reflect the need to offer performance as close as possible to a standard TCP connection.
1. Compatible with constrained runtime environments** (e.g., mobile and browser-based clients).
2. Enable browser-based clients to make bidirectional cross-domain connections.*
3. Compatible with proxies that buffer partial HTTP responses.*
4. Efficient through proxies that limit the duration of HTTP responses.*
5. Fully-compatible with HTTP/1.0.*
6. Compatible with restricted network connections (e.g., firewalls, proxies, and gateways).
7. Fault tolerant (e.g., session recovers after an underlying TCP connection breaks at any stage during an HTTP request).
8. Extensible.
9. Consume significantly less bandwidth than polling-based protocols.
10. Significantly more responsive (lower latency) than polling-based protocols.
11. Support for polling (for clients that are limited to a single HTTP connection at a time).
12. In-order delivery of data.
13. Guard against unauthorized users interjecting HTTP requests into a session.
14. Protect against denial of service attacks.
15. Multiplexing of data streams

*Not possible to fulfill these requirements with the Comet technique.

Finally, in “The BOSH technique” section:

Each block of data pushed by the connection manager is a complete HTTP response. So, unlike the Comet technique, the BOSH technique works through intermediate proxies that buffer partial HTTP responses. It is also fully compliant with HTTP/1.0 - which does not provide for chunked transfer encoding. Also unlike Comet, Web clients can use BOSH to make cross-domain connections.

This misunderstanding seems to derive from the fact that by “Comet” the authors only mean the “forever-frame technique”. In reality, BOSH is based on long polling, which has been part of Comet since the very beginning. In fact, from “The BOSH technique” section:

The technique employed by this protocol achieves both low latency and low bandwidth consumption by encouraging the connection manager not to respond to a request until it actually has data to send to the client. As soon as the client receives a response from the connection manager it sends another request, thereby ensuring that the connection manager is (almost) always holding a request that it can use to “push” data to the client.

So, BOSH is a very interesting specification, but it is not that new from a paradigmatic point of view, and certainly not opposed to Comet.

After a couple of abstract installments, I would like to delve into some more practical details. Let’s see how to build a “Hello World” Comet application with Lightstreamer (and I really mean “Hello World”… not a chat application, which is often considered the “Hello World” of Comet). We will start from scratch and with zero knowledge of the framework. I will introduce some terms and concepts while explaining the code.

What do we want our application to do?

Let’s keep the application very basic. We want to push the alternated strings “Hello” and “World”, followed by the current timestamp, from the server to the browser. Yes, a very exciting application…

What data model should we use?

In the Lightstreamer framework, you subscribe to items. You will find different terms for this concept in other frameworks. For example, in Cometd you subscribe to channels. An item is made up of a number of fields whose values change over time. Here are some examples of possible items:

• An item in Lightstreamer could represent an item on eBay, say, a pair of “Nike Air Jordan” shoes. The item name would be “NIKE-AIR-JORDAN-XX3-XXIII-23-PREMIER-Limited-sz-10”. Some fields would be: current_bid, total_bids, and high_bidder. (By the way, I see that the current bid is $1,300.00 right now; not so bad…). When a field changes, the new value is pushed to the browser and displayed in real time.
• An item could represent a weather probe. The item name would be, for example, “Mt_Everest_Probe.1” (this probe was left by MIT after the 1998 Everest Expedition). Some fields would be: temperature, barometric_pressure, and light_level.
• In the most classical Comet application, finance market data dissemination, an item often represents a stock quote. The item name would be, for example, “TIBX.O” (TIBCO Software Inc. on Nasdaq). Some fields would be: TRDPRC_1, TRDTIM_1, BID, and ASK.

That said, how can we represent our very complex Hello World messages? Of course through an item… The item name will be “greetings”. It will have two fields: message and timestamp.

Let’s get started

This application will be based on Lightstreamer Server v.3.4 and Web Client v.4.2, which are the current production versions. First, download and install Lightstreamer Moderato. After you see the Stock List Demo running, you will be sure that the Server is ready to host our application.

Now we need to develop two components: the front-end (on the client side), and the Data Adapter (on the server side).

Creating the front-end

The front-end of this oh-so-cool application will be a simple HTML page that displays the real-time data pushed by the Server. Here is the result:

First, we should include a couple of Lightstreamer libraries:

<script language='JavaScript' src = 'LS/lscommons.js'></script>
<script language='JavaScript' src = 'LS/lspushpage.js'></script>

Then, we can create two div elements that will host the pushed fields:

<div source="lightstreamer" table="hellotable" item="greetings"
   field="message">loading...</div>
<div source="lightstreamer" table="hellotable" item="greetings"
   field="timestamp">loading...</div>

The source=”lightstreamer” property binds a div element to Lightstreamer. The item and field properties identify the data to be subscribed to and displayed. The initial value displayed on the page will be “loading…”, which will be replaced by the real-time data after the subscription has been done.

Finally, let’s add some JavaScript code to tie things together:

<script>
   var page = new PushPage();
   page.onEngineCreation = function(engine) {
      engine.connection.setAdapterName("HELLOWORLD");
      engine.changeStatus("STREAMING");
   }
   page.bind();
   page.createEngine("HelloWorldApp", "LS", "SHARE_SESSION");

   var pushtable = new OverwriteTable(null, null, "MERGE");
   page.addTable(pushtable, "hellotable");
</script>

We call any page in the Web application that will receive real-time data a Push-Page. Every Push-Page should contain a PushPage object. One of the Push-Pages must be a Master Push-Page, meaning that it must contain the LightstreamerEngine, a JavaScript object that abstracts all the Comet functionalities of Lightstreamer. The code above creates a PushPage object and binds it to the current page. Then it creates the LightstreamerEngine, naming it “HelloWorldApp”. (This is a name that identifies the application; more instances of the same application can automatically share the same engine, and therefore the same connections, thanks to the “SHARE_SESSION” flag). The onEngineCreation callback is invoked on the Master Push-Page after the engine has actually been created. This callback is the place to configure the server details (not necessary here, because we will connect to localhost) and to start the Comet session by calling changeStatus(”STREAMING”).
The OverwriteTable object is used to encapsulate the subscription details and the visual binding to the HTML widgets. When calling addTable, we specify that this OverwriteTable should be bound to the HTML elements that we tagged with the property table=”hellotable”.

The full source code of this page is shown below (and is included in this archive):

Creating the Data Adapter

Good, we have a front-end. Now we need to create the server-side code that will pass the data to the Lightstreamer Server, which in turn will pass it to the front-end. This is done by writing a Data Adapter, a plug-in module that injects data into the Server. Let’s choose Java to write our Data Adapter (the other options would be to use .NET or to work at the socket level).

First, we need to implement the SmartDataProvider interface:

public class HelloWorldDataAdapter implements SmartDataProvider {

We will be passed a reference to a listener that we will use to inject events:

private ItemEventListener listener;

public void setListener(ItemEventListener listener) {
  this.listener = listener;
}

Then we should be ready to accept subscription requests (remember that Lightstreamer is currently based on asymmetric Comet):

public void subscribe(String itemName, Object itemHandle,
     boolean needsIterator)
     throws SubscriptionException, FailureException {
   if (itemName.equals("greetings")) {
      gt = new GreetingsThread(itemHandle);
      gt.start();
   }
}

When the “greetings” item is subscribed to by the first user, our Adapter receives that method call and starts a thread that will generate the real-time data. If more users subscribe to the “greetings” item, the subscribe method is not called any more. When the last user unsubscribes from this item, our Adapter is notified through the unsubscribe call:

public void unsubscribe(String itemName)
     throws SubscriptionException, FailureException {
   if (itemName.equals("greetings") && gt != null) {
      gt.go = false;
   }
}

We can then stop publishing data for that item. If a new user re-subscribes to “greetings”, the subscribe method is called again. This approach avoids consuming processing power for items nobody is currently interested in.

Now, let’s see what the GreetingsThread does. Its run method is pretty straightforward:

public void run() {
   int c = 0;
   Random rand = new Random();
   while(go) {
      Map data = new HashMap();
      data.put("message", c % 2 == 0 ? "Hello" : "World");
      data.put("timestamp", new Date().toString());
      listener.smartUpdate(itemHandle, data, false);
      c++;
      try {
         Thread.sleep(1000 + rand.nextInt(2000));
      } catch (InterruptedException e) {
      }
   }
}

We create a HashMap containing the message (alternating “Hello” and “World”) and the current timestamp. Then we inject the HashMap into the Lightstreamer Server through the listener (and the itemHandle we were passed at subscription time). We do a random pause between 1 and 3 seconds, and we are ready to generate a new event.

The full source code of this Data Adapter is shown below (and is included in this archive):

Let’s deploy the Adapter

We should now compile the Data Adapter and plug it into the Lightstreamer Server. To compile it, we need to include the Adapter interface in the Java compiler classpath. We can find this interface in the ls-adapter-interface.jar file located in “Lightstreamer/DOCS-SDKs/sdk_adapter_java/lib” within your installation. If compiling succeeds, you will get two classes:

- HelloWorldDataAdapter$GreetingsThread.class
- HelloWorldDataAdapter.class

To deploy a Data Adapter, we need to create a folder under “Lightstreamer/adapters” (you can see there is already the “StockList” Adapter there). Let’s call it “HelloWorld”. Create a “classes” folder inside “HelloWorld” and put the two .class files in it.

The final step is to create a deployment descriptor for this Adapter. This file should be called “adapters.xml” and put in the “HelloWorld” folder. Its content is very simple:

<?xml version="1.0"?>
<adapters_conf id="HELLOWORLD">
   <metadata_provider>
      <adapter_class>
         com.lightstreamer.adapters.metadata.LiteralBasedProvider
      </adapter_class>
   </metadata_provider>
   <data_provider>
      <adapter_class>
         HelloWorldDataAdapter
      </adapter_class>
   </data_provider>
</adapters_conf>

We assign an ID to our Adapter: “HELLOWORLD”. This is the same that we used in the setAdapterName method of the client. Then, we define a default Metadata Adapter (a Metadata Adapter is responsible for managing authentication, authorization, and quality of service; we don’t need any custom behavior for our application). And we define the main class of our brand new Data Adapter.

Let’s deploy the client

We only need to deploy our Web page and we are done. To make this fast, we decide to use the Lightstreamer Server as a Web server too, meaning both the static resources and the real-time data will be delivered by the Lightstreamer Server. But the typical production architecture has an external Web server (whatever it is) in addition to the Lightstreamer Server. Everything is downloaded from the Web server except for the real-time data, which comes from the Lightstreamer Server. This separation improves both flexibility (you are free to use whatever Web/application server you want) and performance (you can isolate the power-demanding Comet connections to a separate box, without impacting your existing Web infrastructure).

So, let’s create a “HelloWorld” folder under the “Lightstreamer/pages” folder. We name the HTML file created above as “index.htm” and put it in the “HelloWorld” folder. Finally, we have to deploy the JS libraries used by our page. Let’s create an “LS” folder under “HelloWorld” and copy to it all the files located in “/Lightstreamer/DOCS-SDKs/sdk_client_web/lib”.

Ready to go

Let’s start our Lightstreamer Server, then open a browser window and go to: http://localhost:8080/HelloWorld/

Is it working? Hope so…

Final notes

This example was really very basic and exploited only a minor portion of the features offered by the Lightstreamer API. To delve a bit more into the API used above (JavaScript and Java), you can take a look at the online API reference:

- JSdoc of the Web Client API
- Javadoc of the Adapter API

We used the Moderato edition of Lightstreamer. Its main limitation, with respect to the commercial editions, is that it only allows 1 update/sec for each subscribed item. There are no limits on how many items you can subscribe to at the same time, but if you want to push higher-frequency events for the same item (like our “greetings” item) you will see that no more that 1 event/sec is delivered to the client. If you need to experiment with higher frequencies, you can use an evaluation license of Lightstreamer Presto or Vivace.