Recent discussions on Comet server performance have focused on sockets and IO:

• A Million-user Comet Application with Mochiweb, Part1, Part 2 and most interestingly Part 3
• A Gazillion-user Comet Server with libevent
• And here on Comet Daily: Is it Raining Comets and Threads?

Liberator was designed with performance in mind from Day 1. In fact, before Liberator we had a win32 C++ server which started life as a proof of concept. Liberator came about when we realised that we needed better performance.

I have talked about Benchmarking Comet Servers before and described how there are many variables involved, the key ones being the number of users and the amount of data. However, different usage profiles can cause some very different results.

A key part of the discussion recently has been low level socket handling, in Java with IO or NIO, and in C (or Erlang with C underneath) using techniques supported by libevent. Liberator has an event abstraction similar to libevent, with different implementations all providing the typical read, write and timed events with callbacks. This event manager started its life using select(), but quite quickly moved to use poll() which showed some improvements. The select() implementation was kept for the win32 build (the event manager is part of the library that Liberator uses along with our other products, but Liberator itself does not have a win32 build these days).

A few years ago we moved on to use epoll on Linux and /dev/poll on Solaris, both of which give a more efficient API as they allow the OS to give you back a handle when an event occurs, as opposed to select or poll where you generally have to loop through all your open sockets finding which one the event matches. This is clearly going to be better for high numbers of sockets.

One quirk we found under testing resulted in the default configuration of Liberator using epoll for client-side sockets, where there could be high numbers, and standard poll for the server-side (Data Sources) where there are typically only a few sockets.

Liberator runs an asynchronous event loop per thread which means it can take advantage of multi cpu or multi core machines. There is a thread per server-side connection (of which there are typically only a few) and a configurable number of threads to handle clients. It depends on this usage profile, but typically you would have a client handling a thread per CPU core. With very high numbers of server-side updates, you may want fewer client handling threads as the server-side threads require more CPU time—the Operating System can obviously handle this, but fine tuning can improve performance. Multiple sources of data can be configured either for performance reasons, for example load balancing, or to partition data, for example different back end applications or data feeds.

Memory and Abstractions

One important area of a C application is memory management. Liberator has evolved over the years in this area. In the beginning handling of messages internal and external aimed for a zero copy architecture, but after performing tests it was clear that in our case the extra burden of the management of this was not a good trade off.

There are also areas where memory is pre-allocated and cached on a per thread basis, which can avoid the mutexes used by most malloc libraries. However, this has never actually been proven to provide any significant benefit under testing.

There are still various parts of Liberator which avoid unnecessary copying and also unnecessary allocating of memory which leads into abstractions. As mentioned, Liberator was designed for performance and this led to a design which does not abstract away too much functionality behind interfaces, giving direct access where beneficial. There are still internal APIs and abstractions, but that requires intimate knowledge in these areas.

Protocol

There are various protocols used between clients and Comet servers, I have posted about bandwidth before, as this is a critical area when considering server performance, especially to high numbers of clients. Liberators protocol has always aimed to be as brief as possible.

There are two areas specifically where this has been achieved. Firstly, the subject of a subscription is not sent for every update, it is mapped at subscription time onto a shorter identifier which can considerably decrease the size of update messages, especially in the common case of payload size not being particularly big. Secondly the payload itself, for simple name/value pair type messages, the field names are also mapped onto shorter identifiers. This is not as simple for some Comet servers which aim to let the payloads be completely free form, i.e. a JSON data type, but for common usage a streamlined name/value pair data type saves bandwidth. Some other Comet servers have implemented similar techniques.

Batching and Throttling

Two techniques that help out with performance are batching and throttling (or conflation).

Batching is fairly common with network programming. When lots of small messages are to be sent, batching them together to send in one request can be beneficial in a number of ways. Although you are introducing latency by delaying the send of the first messages, overall latency can improve due to better use of TCP/IP. Operating systems actually do this for you, using Nagle’s algorithm, however, application level knowledge usually means improvements can be made to this when implemented by the application.

Batching sends all of the same data, just packaged up differently, but throttling reduces the data you actually send. Again, this is not really possible with completely free-form messages, but in Liberator the record data type is intended for use with financial market data. If the stock price of MSFT is updating 10 times a second, the client may not need to receive all of those updates. So the updates are conflated so, for examples, every half a second the latest value of any field that has changed will be sent out. For benchmarking purposes or in scenarios where the client needs each individual update, throttling can simply be turned off. The time period can also be configured on a subject-space basis, so some data is treated differently than other data.

Data Structures

There are some basics about publish/subscribe servers that you have to get right, and I would hope everyone does. An example: when a message needs to be sent to subscribers, you don’t want to have to loop through all the users finding which ones are interested, you should be able to traverse a list of subscribers directly. Over the years, various other data structures in Liberator have been changed as performance issues arose - when customers are let loose on a project they can do things you didn’t really think they would, and this generally comes back to the ‘variables’ involved in benchmarking. A simple case of removing a subscription can be an expensive operation when the user has thousands of subscriptions if the right data structure is not used, but if you never test a scenario like that, you might not realise you have an inefficient operation.

This is something that Google is very hot on: finding the right algorithms and data structures means you can scale, optimising specific areas of code can help too, but often can yield insignificant gains once you scale things to large numbers.

Conclusion

There are many aspects to the performance of a Comet server, I have touched on a few and how Liberator tackles them.

Richard Jones’ Million Comet Users interested me a lot. Liberator has focused on small numbers of users, relative to Richard’s goals, but with much higher update rates to each user. I would like to test Liberator with a million users, but that does require a fair amount of time and effort - since the core basics of Richard’s test are similar to Liberator I would think that it could cope in a similar way.

Performance of a rich internet application (RIA) can manifest itself in a number of ways, such as responsiveness of user interaction and speed of rendering. This article is a follow-up to Performance of Grids for Streaming Data and covers Firefox 3.

These tests all use Liberator as a source of data and use three implementations of grids:

1. A simple HTML table using SL4B for real time data.
2. The Caplin Grid using SL4B for real time data. This grid is part of the Caplin Trader product and is designed specifically for financial trading applications.
3. A Flex grid using a native RTTP (connection to Liberator) implementation.

Each grid subscribes to a number of subjects to populate the grid. The tests use a test feed that sends updates once per second for each subject subscribed to and provides six fields per update.

These tests are intended to compare Ajax and Flex across different browser versions.

Browser speed

First up, we have a test of different browser versions using the simple HTML table grid implementation

It is clear that Firefox 3 has pushed the performance of JavaScript a long way.

Ajax and Flex

This test compares the Caplin Grid in various browsers against the Flex grid. Since the performance of Flex is not heavily dependent on browser version we only show a single plot for Flex.

This test goes up to 100 rows in the grids. The grids themselves are displaying all 100 rows, but the browser window is not big enough to display the whole grid and only 40 are in view.

This chart shows a few things:

1. The different browser versions have the same effect on the Caplin Grid as they did on the HTML table with Firefox 3 leading the way.
2. The Flex grid performs better than the Caplin Grid on Internet Explorer 7 and Firefox 2, but on Firefox 3 the Caplin Grid performs better than Flex.
3. Both Flex on Firefox 2 and the Caplin Grid on Firefox 3 appear to have some optimizations for updates not on the screen, although this is small and may just be down to accuracy of the tests. This is shown when the rows increase above 40.

Larger Grids

This test compares the HTML Table implementation against the Caplin Grid and the Flex grid for large amounts of data. The test has only 25 rows in view, this time using whatever means each implementation has to handle this. The Caplin Grid and Flex grid both handle this themselves and present scrollbars, whereas the HTML Table implementation simply puts the table in a scrollable div element.

The Caplin Grid does not subscribe to data that is not in view, but makes subscriptions when necessary as the user moves the scrollbar. The chart shows how this leads to perfect scalability as it does not matter how many rows are in the data set.

The Flex grid, although it still subscribes to all the rows does not attempt to render them which is clearly a large part of the work involved since it also has very good scalability, although the CPU Usage still goes up for out of view rows since it is still handling the updates coming in, just not displaying them.

The HTML Table plots show that the scrollable div element does not really help with out of view rows as the CPU usage increases linearly with the rows.

The chart also shows that Flex is not faster than the JavaScript implementations on Firefox 3.

Conclusions

From a performance perspective it is clear that Flex performs well, but the fast moving world of browsers is improving the performance of Ajax applications all the time. The grid test results show Ajax running on Firefox 3 outperforms Flex for this kind of application.

Google finance now has real-time stock quotes from Nasdaq.

Previously all free stock quotes on the internet were 15 minute delayed - so even if they were being streamed, which might give a better user experience, they weren’t much good for anything serious.

I have taken a look under the covers and, when using Firefox at least, it is using a streaming transport using chunked encoding. The contents itself are JSON.

A typical chunk looks like this:

166
[[16,[{"s":"NASD:MSFT","l":"27.88","c":"-0.44","cp":"-1.55","ltt":"11:40AM ET"}]]
,[17,[{"s":"NASD:EBAY","l":"29.15","c":"-0.86","cp":"-2.87","ltt":"11:40AM ET"}]]
]

The subscriptions are part of a separate request, which indicates you could change subscriptions. In other words it would support an interactive GUI unlike some very simple streaming solutions based on a single request.

Since this is Google, you would imagine that this scales well, and they can obviously throw lots of hardware at the problem too if necessary.

Does anyone know any more about this technology?

In addition to some of our earlier coverage on .Net and Comet, it looks like there may be some Comet-like things happening in the .Net/Silverlight world based on Eugene Osovetsky’s recent article, Pushing Data to a Silverlight Application:

• An Instant Messaging / Chat application. When a user types a message into your Silverlight app, you use WCF to make, for example, a SendChatMessage() call. But what about receiving messages that others type back? Sure, you could call something like ReceiveChatMessage() continuously in a loop, and it could return null if there are no messages waiting. But this is a lot of work, not to mention the severe implications this would have for the performance of your app. Wouldn’t it be nice if it “just worked”?
• A Monitoring application. A lot of people are writing applications that monitor something that can be changed at any time on the server. Maybe it’s a stock ticker / charting app, or maybe it’s monitoring some physical sensor… Again, you could continuously poll the server to see if anything changed, but this is inefficient.

To solve problems like this, we are trying to ship a “duplex” feature in Silverlight 2 Beta2. (There is a very good chance that the feature will make it in, but of course no guarantees, and anything I say about the design is subject to change by the time we ship). I’ll first give you a quick overview and then the details:

Obviously it is fairly light on details, but from what is stated you can see roughly how the interface might work. It will be interesting to see if they build anything into the Silverlight plugin to give them an advantage over people with .Net APIs to their Comet servers.

There is a lot of confusion about guaranteed messaging. Many products claim some level of guaranteed or reliable messaging, but what does it really mean? What are you guaranteeing, and to whom?

As a software vendor all you need to do is have the words ‘Guaranteed Messaging’ in your literature and for a lot of customers their box ticking exercise will be satisfied. It’s not until the customers technical team start asking awkward questions that you start to think about what ‘Guaranteed Messaging’ really means and whether the related features you have built into your product really fall under that banner.

In some sectors and products the words Guaranteed, Reliable and Certified have specific meaning, but I am just going to look at the basic concepts surrounding these.

There are a few things about sending messages that could be guaranteed:

• In general you want to be sure a message gets to the intended recipients.
• In some cases you also want to be sure that successive messages are received in the same order as they were sent.

Once you start thinking about solutions for this, a third requirement might spring up:

• You want to be sure a message is received or, more importantly, processed only once.

Point to point

When messages are sent between a single producer and a single consumer this is often called point to point messaging. This is usually the area where guaranteed messaging is required as the message relates directly to a particular consumer rather than being a general message for many interested consumers.

Most solutions to this are based around sending acknowledgements. The basic idea here is that the producer sends a message and the consumer sends an acknowledgement when it receives the message. If the producer does not receive the acknowledgement within a specified timeout period then it will resend the message.

This is where we meet our first problem. If the producer does not receive the acknowledgement, is it because the message did not get to the consumer or because the acknowledgement did not get back to the producer? If the producer simply resends the message we run the risk of the consumer processing the message multiple times. To prevent this from occurring, the consumer needs to remember which messages it has processed, and this requires a unique message id or sequence number to be present on all messages. When the consumer receives a message it has previously processed, it can resend the acknowledgement without processing the message again. It is fairly common for a resent message to be flagged as such, so that the consumer can optimise checking whether it has previously seen a message.

So with this in place we can cope with the message being lost in transit, and also with the acknowledgement being lost in transit. To take it to the next level we need to think about what the messaging API might look like and who is being given the guarantee.

The simplest approach is for the send() method to block until the message has definitely been consumed (received and acknowledged). It may not be acceptable for the sending application to be held up in this way, so instead the API could asynchronously notify the producer that the message has been consumed. Without such a notification your guarantee does not cope with software failure. You cannot simply call a non-blocking send() method and be guaranteed that the message will be consumed. This is a key point, and means that the user of the API must also be involved in the guarantee.

At the consumer end, you can imagine an onMessage() callback. There could be two ways to implement the acknowledgement. The simple way is for the messaging library to send the acknowledgement after the callback has returned. This assumes that the user code is processing the message synchronously and the callback returning implies the message has been processed successfully. Alternatively you can provide a method on the message to explicitly acknowledge it. This gives the user code more flexibility in how it processes messages. Without this, the consumer library could be sending acknowledgements when the message has not been processed, which would open up holes in the system.

Coping with software failure is an interesting topic. Platforms and libraries can only provide so much; the application itself needs to decide its own requirements and how to handle some errors. For full resilience the application needs to be involved in the guarantee and cannot rely just on the platform being used.

Multiple consumers

In many Comet solutions there are a number of consumers all subscribed to the same subject and therefore receiving the same messages for that subject. A single producer may send a message which the Comet server then sends out to multiple consumers.

In this scenario, typically the producer does not require an acknowledgement from each consumer, since the producer is not even aware of all the consumers. In most platforms there will be a component between the producer and the consumer, in this instance it is usually the Comet server. The producer may actually be another Comet client, or it may be a server side component generating messages. The producer may require an acknowledgement from the Comet server, which would state that the server has processed the message and all subscribers will receive the message. However, there are caveats with that acknowledgement; it is really saying that all subscribers will get the message as long as the Comet server does not fail and as long as the consumer does not fail and not reconnect.

Without an acknowledgement from each client this is the best guarantee you can get. For most applications it is impractical for consumers to be sending acknowledgements for messages with multiple consumers. However, if this is required then a similar implementation to the point-to-point acknowledgements can be used, but this time only between the server and the consumer.

An alternative to acknowledging every message is to implement a lazy acknowledgement, which acknowledges batches of messages. A server might store all messages sent to a client, to enable it to resend them in the case of a reconnection - lazy acknowledgements would allow the server to clear out this cache of messages periodically, preventing excessive memory usage.

Message Order and Missing Messages

Guaranteeing the order of messages a consumer receives can be easy. Depending on the implementation, it may be that nothing special is needed.

In a system where all the components are connected via persistent TCP/IP sockets, as long as the Server and libraries do not change the order of messages due to threading, all messages should be received in the order they were sent. However, Comet solutions do not generally have the luxury of a persistent TCP/IP socket, certainly not in both directions. Even with a bidirectional socket, coping with reconnections opens you up to some of the issues that other transports may have to cope with during normal operation.

To handle message ordering each message needs a sequence number. In the simple case of a bidirectional socket, on a reconnection a handshake can take place to ask for all messages since the last received sequence number.

This is not just about message ordering, its about missing messages. The methods used to guarantee message delivery between producer and consumer will obviously prevent missed messages, but they are generally more end to end (producer to consumer) rather than just server to client at the transport level. End to end guarantees are not always used, or not used for all types of data, but you still want a transport that will not let messages go missing.

Due to the nature of Comet it is often at the transport level that preventing missed messages has to be implemented. Although this could be layered on top of the transport, an understanding of the connection semantics of the transport is beneficial.

With a bidirectional socket, it is really only reconnection time that is an issue. Other streaming transports typically have an open socket for server to client messages, but use transient sockets for client to server. This usually means an HTTP request for every client-to-server message, which means an acknowledgement is built in through the HTTP response. The server-to-client messages are effectively a socket connection, so again it should only be an issue at reconnection time. However, it is common for ‘hidden iframe’ based streaming transports to force a reconnection periodically so it is not just unforeseen disconnections that occur. Other, less connected, transports might introduce other issues since new HTTP requests and responses are used continuously throughout the lifetime of the Comet session.

As long as sequence numbering is maintained and checked then handling of message ordering and missing messages in a layer above the transport can work. The system needs to be able to resend messages that might have been missed when asked to do so. However, in some cases knowledge of the transport could make this more efficient.

Conclusion

This article only touches on some of the areas of guaranteed messaging with respect to Comet platforms and hopefully is food for thought. Many Comet applications will not need guaranteed messaging, but will obviously still want a level of reliability where some of the above issues still apply. Hopefully this illustrates that the application itself must be involved in guaranteed messaging rather that relying solely on the libraries or platform being used.

I have just added a JavaScript Tutorial to the Liberator documentation page. This is something written mainly by a colleague to help Caplin’s development team, but I felt it could be useful to anyone developing Ajax or Comet applications.

It covers all kinds of aspects of JavaScript programming, mainly focusing on the kind of details that might not be obvious. It includes an interactive section which is useful as you can edit the code and execute it if you want to try out a variation of what is presented.

The page on JavaScript Equality, specifically the == operator, is quite interesting, with a few results you might not expect. It demonstrates that you have to be very careful with the use of the == operator, especially when the values might contain empty strings or arrays.

People seem to like code snippets and tutorials, so here is a quick run through of the basics of using Liberator to create a streaming Comet application.

Liberator provides you with a JavaScript library called StreamLink for Browsers (or SL4B). This is used to build real time web pages that talk to Liberator. SL4B can be used in two ways: one requires no JavaScript programming at all and the other provides a rich JavaScript API based on subscriptions and callbacks.

No JavaScript

SL4B can make a web page live by simply adding attributes to HTML tags. A typical example of this is making an HTML table contain live FX prices. To achieve this there are two things to change on your HTML page.

Firstly, add a script tag to load the SL4B library:

<script id=”sl4b”
        type=”text/javascript”
        src=”/sl4b/index.js”
        configurationfile=”common-attributes.js”
        includertml=”includertml”>
</script>

And secondly, add some attributes to the HTML:

<table>
  <tr>
    <th>Currency</th>
    <th>Bid</th>
    <th>Ask</th>
  </tr>
  <tr>
    <td>EUR</td>
    <td id=”quote” symbol=”/FX/EUR” field=”dBestBid”>-</td>
    <td id=”quote” symbol=”/FX/EUR” field=”dBestAsk”>-</td>
  </tr>
  <tr>
  <td>GBP</td>
    <td id=”quote” symbol=”/FX/GBP” field=”dBestBid”>-</td>
    <td id=”quote” symbol=”/FX/GBP” field=”dBestAsk”>-</td>
  </tr>
</table>

The above HTML has been enhanced with three extra attributes to the <td> tags:

The standard id attribute has been set to "quote". This allows the SL4B library to find all the HTML elements that might require real time data. A custom attribute called "symbol" is used to specify the subject to subscribe to, and another custom attribute called "field" specifies the name of the field to display in this HTML element.

Behind the scenes, all this information is collated and subscribed to by the SL4B library. When data is received back from Liberator, the relevant HTML elements are updated with the real time values.

You can see an example of this in action.

Add in some JavaScript programming

The previous example shows a very simple way to get live data into a web page. There are more things you can do using that style, but if you want more control over what is happening you will probably want an API to program with.

SL4B allows you to make subscriptions and handle the data in callbacks.

The same script tag is needed to load the SL4B library:

<script id=”sl4b”
        type=”text/javascript”
        src=”http://liberator.company.com:8080/sl4b”
        configurationfile=”common-attributes.js”>
</script>

To make subscriptions in SL4B you need to create a subscriber:

function ExampleSubscriber()
{
    // This is the constructor of our object.

    // Any subclass of SL4B_AbstractSubscriber must call
    // initialise in its constructor. Once initialisation
    // is finished, SL4B will call ready() on this object.
    this.initialise();
}

// Declares that the ExampleSubscriber is a subclass of
// SL4B_AbstractSubscriber
ExampleSubscriber.prototype = new SL4B_AbstractSubscriber();

ExampleSubscriber.prototype.ready =
  function() {
    // Once the SL4B Object has completed
    // this.initialise() it will call ready()
    alert(”SL4B has connected and is ready”);
  };

var Subscriber = new ExampleSubscriber();

The above code is a bare subscriber, but to be useful it needs to make subscriptions and implement a callback:

ExampleSubscriber.prototype.ready =
  function() {
    // The fields to subscribe to are in a comma separated
    // list as the third argument of getObject.
    SL4B_Accessor.getRttpProvider().getObject(this,
                    “/Examples/Records/FX/GBP”,
                    “dBestBid,dBestAsk”);
  };

ExampleSubscriber.prototype.recordMultiUpdated =
  function(sObjectName, oFieldData) {
    var oElement = document.getElementById(’messages’)

    for (var i = 0, size = oFieldData.size(); i < size; i = i + 1) {
        oElement.innerHTML = sObjectName+” “+
                             oFieldData.getFieldName(i)+” = “+
                             oFieldData.getFieldValue(i)+”<br />”+
                             oElement.innerHTML;
    }
  };

The additional code above makes a subscription for a subject called /Examples/Records/FX/GBP and implements a callback to handle the returning data. In this case it is simply inserting the data into an HTML element.

You can see a more detailed version of this tutorial, which adds a few more steps to this example and has working examples.

Providing some data

Both the above examples assume there is some data available for the subjects subscribed to, e.g. /Examples/Records/FX/GBP. This data has to get into Liberator from somewhere. In some Comet applications the clients provide the data as well as consuming it, but in most financial Comet applications the data is provided on the server side.

The Liberator server receives data from DataSources, a DataSource being a separate process that communicates with Liberator via a socket connection. How this data is created in the DataSource is application specific, but in a typical financial application the DataSource is also connected to a third party data feed and is acting as a protocol converter.

DataSources are usually written in Java and a simple API is provided to do so.

The following code snippet shows the creation of a DataSource. Connections are handled internally and are specified by the configuration file.

    // Configures the datasource's connectivity - amongst other things
    final File dataSourceConfig = new File("conf/DataSource.xml");
    // Configures the mapping of field numbers to field names
    final File fieldsConfig = new File("conf/Fields.xml");

    // Create our local DataSource
    DataSource datasrc = new DataSource(dataSourceConfig,fieldsConfig);

    // Start the DataSource
    System.out.println(”Starting DataSource …”);
    datasrc.start();
    System.out.println(”DataSource started”);

To provide data to a Liberator, callbacks must be registered to handle subscriptions and interfaces must be implemented.

public class Stage2 implements DSRequestListener, DSDiscardListener
{
    public static void main(String[] args)throws IOException, SAXException
    {
        // Configures the datasource’s connectivity - amongst other things
        final File dataSourceConfig = new File(”conf/DataSource.xml”);
        // Configures the mapping of field numbers to field names
        final File fieldsConfig = new File(”conf/Fields.xml”);

        // Create our local DataSource
        DataSource datasrc = new DataSource(dataSourceConfig,fieldsConfig);

        // Create a listener for the relevant DataSource message types
        // so we may respond appropriately
        final Stage2 listener = new Stage2();
        // Listen for requests
        datasrc.setDSRequestListener(listener);
        // Listen for discards (un-requests)
        datasrc.setDSDiscardListener(listener);

        // Start the DataSource
        System.out.println(”Starting DataSource …”);
        datasrc.start();
        System.out.println(”DataSource started”);
    }

    public void receiveRequest( int peerIndex, String[] subjects )
    {
        for(int i=0;i<subjects.length;i++)
        {
            System.out.println(”Received request for “+subjects[i]);
        }
    }

    public void receiveDiscard( int peerIndex, String[] subjects )
    {
        for(int i=0;i<subjects.length;i++)
        {
            System.out.println(”Received discard for “+subjects[i]);
        }
    }

In this code we are implementing two interface methods and registering them with the created DataSource object. The callbacks are simply logging when requests and discards are received.

To actually respond to a request, a data message must be created and sent back to the Liberator. The following code shows this:

public void receiveRequest( int peerIndex, String[] subjects )
{
    for(int i=0;i<subjects.length;i++)
    {
        // Create the new record
        DSRecord update = DSFactory.createDSRecord(dataSource, subjects[i]);
        // Populate the new update’s fields
        update.addRecordData(”Counter”, 1234);
        // Send ‘Hello, World!’ in the ‘Message’ field
        update.addRecordData(”Message”, “Hello, World!”);
        // Send the update to the peer that requested it
        update.sendToPeer(peerIndex);
    }
}

Now we have sent back some initial data for the request, but there isn’t much point to Comet if you can’t send further data to a subscription. The same code seen above that sends the initial response can be used to send data at any point. If a discard (or unsubscribe) is received then you should stop sending any data for that subject.

How you send that data is obviously up to you and your application, but purely as a demonstration, the following code will send some subsequent updates every two seconds on a separate thread.

public void run()
{
    int counter = 0;

    while(alive)
    {
        try
        {
            Thread.sleep(2000);

            DSRecord update = DSFactory.createDSRecord(dataSource, symbol);
            update.addRecordData("Counter", ++counter);
            update.addRecordData("Message", "Hello, World! "+counter);
            update.sendToPeer(peerIndex);
        }
        catch(InterruptedException e)
        {
        }
    }
}

A full version of this DataSource tutorial is available in the Liberator kit.

The code above, and much more, is available in the online examples and as part of the free download for Liberator.