Orbited
SitePen Support

Scalable Real-Time Web Architecture, Part 2: A Live Graph with Orbited, MorbidQ, and js.io

by Michael CarterOctober 10th, 2008

Introduction

This tutorial will show you how to build a simple broadcast application for numerical data and represent that data graphically in the browser. The resulting user interface is very simplistic so that we can focus on the important parts of real-time applications. The methods from this tutorial could be extended to include any type of real-time data including monitoring, financial, or otherwise.

Completed Tutorial Screenshot

Completed Tutorial Screenshot

The application we are building will use the Message Queue Architecture as outlined in the part 1 of this series. In particular, this tutorial showcases a data broadcast use-case where data is produced on the server and sent to each connected client. Every browser connected to the application will display synchronized, live updates of the numerical data. While we have the flexibility of publishing different kinds of data to different channels, for the purposes of simplicity we are sending all data to the same topic in this tutorial.

Tutorial Architecture Diagram

Tutorial Architecture

Prerequisites

There are a few packages that we first need to install. You can find detailed instructions on this process in the Orbited Installation Guide

We are also going to use the js.io Stomp client implementation for the tutorial, but Orbited pulls all of the js.io clients into each build/checkout via an svn:external definition, so you don’t have to download the clients separately. Orbited 0.7.x+ also ships with the embedded message queue MorbidQ which will be installed automatically when you install Orbited.

Configuring and Starting Orbited and MorbidQ

Now create a directory called graph_demo. Inside this directory create an a empty graph.cfg and index.html file. We first need to edit the graph.cfg file. Paste in the following content:

[listen]
http://:9000
stomp://:61613
 
[access]
* -> localhost:61613
 
[static]
graph=index.html
 
[global]
session.ping_interval = 300

The [listen] section specifies that we want to listen for incoming connections from the web on port 9000, and stomp connections on port 61613. This is done with the embedded MorbidQ daemon.

The [access] section allows our users to have their tcp connections proxied to localhost:61613. The ‘*’ refers to the fact that it doesn’t matter what the Host header is. Orbited supports Hostname access control on the LHS and destination access control on the RHS of each rule.

The [static] section specifies that our “index.html” file can be accessed by the virtual name “graph”, so if we navigate to http://localhost:9000/graph, it would attempt to serve the “index.html” resource from the file system. Note that we aren’t required to have Orbited serve our static files; we can serve them from another web server, or even test the application by directly viewing the index.html file from the hard drive (Firefox only, in development mode.) For the purpose of this tutorial, we’ll simply serve the application html files with Orbited.

The [global] section specifies a 300 second interval between server -> client pings. This is set to a high value because we don’t really need immediate notification if a client times out. If we were writing a chat room, we might want to lower this value so we can alert other users quicker.

Now put “Hello World” in the index.html file. Turn orbited on by running: orbited --config graph.cfg

You should see the following output:

10/09/08 05:58:58:494 INFO   orbited.start    proxy protocol active
10/09/08 05:58:58:494 INFO   orbited.start  Listening http@9000
10/09/08 05:58:58:497 INFO   orbited.start  Listening stomp@61613

Now navigate to http://localhost:9000/graph in your web browser, and it should say “Hello World”. Orbited is successfully installed and running.

The User Interface

Open up the index.html file and get rid of the “Hello World”. Add this boilerplate to the top of the file:

<!DOCTYPE html>
<title>Orbited + Morbid + js.io Graph</title>
<script src="http://localhost:9000/static/Orbited.js"></script>
<script>
TCPSocket = Orbited.TCPSocket
</script>
<script src="http://localhost:9000/static/protocols/stomp/stomp.js"></script>

The above should include the Orbited.js library, and the js.io Stomp client. Because js.io depends on having a TCPSocket object, we need to specify TCPSocket = Orbited.TCPSocket.

Next we will create an empty <script> tag for the application code itself. The first thing we add is an initialization function. Note that none of our user interface code in this example depends on JavaScript toolkits or frameworks beyond Orbited and the js.io Stomp client.

var bars = [];
var num_bars = 10;
 
// Make ten bars that are blue
var init_graph = function(num_bars, bars, container_name) {
 
    var container = document.getElementById(container_name);
    for (var i=0; i< num_bars; i++) {
        var bar = document.createElement('div');
        bar.className = "bar";
        bar.style.width = "100px";
        bars.push(bar);
        container.appendChild(bar);
    }
    // some pizazz... one bar gets to be red.
    bars[3].style.backgroundColor = "red";
};

This code will find the “container” div and insert 10 blue graph bars (divs), and make one of them red. We expect to update the width values of these bars to represent the changing data. So when we get an array of new data values in real-time, we want to make the width of each bar match. The next piece of code does just that:

// Change the length of the bars to match given numerical data
var modify_graph = function(bars, payload) {
    var vals = JSON.parse(payload);
    for (var i=0; i<bars.length; i++) {
        var bar = bars[i];
        bar.style.width = vals[i] + "px";
    }
}

Finally, we need to hook it all up with the following code:

// In production use your js toolkit's onload system, or event listeners
onload = function() {
    init_graph(num_bars, bars, "container");
 
    stomp = new STOMPClient();
    stomp.onopen = function() {
    };
    stomp.onclose = function(c) { alert('Lost Connection, Code: ' + c);};
    stomp.onerror = function(error) {
        alert("Error: " + error);
    };
    stomp.onerrorframe = function(frame) {
        alert("Error: " + frame.body);
    };
    stomp.onconnectedframe = function() {
        stomp.subscribe("/topic/graph");
    };
    stomp.onmessageframe = function(frame) {
        // Presumably we should only receive message frames with the 
        // destination "/topic/graph" because thats the only destination 
        // to which we've subscribed. To handle multiple destinations we 
        // would have to check frame.headers.destination.
        modify_graph(bars, frame.body);
    };
    stomp.connect('localhost', 61613);
}

This code will create a new Stomp client, and hook up the onmessage frame to the modify_graph function so that received message frames will update the bar graph on our screen. A few other callbacks are attached, including onclose so we know when the connection is lost with the server.

We also need to do a bit of styling to make our user interface more presentable:

<style>
    body, html, h1 {
        margin: 0px;
        padding: 0px;
        font-family: sans-serif;
        background-color: #f3ffe2;
    }
 
    .bar {
        background-color: #38f;
        height: 12px;
        width: 200px;
        margin: 3px 0px;
    }
 
    #container {
        margin: auto;
        border: 2px solid #000;
        width: 400px;
        background-color: #fff;
    }
</style>

And finally, we need to create the container div:

<div id="container"></div>

This is everything we need for the client-side of this demo. You can refresh the browser pointing at “http://localhost:9000/graph” and you should see the ten blue bars and one red bar. Keep the browser pointed at this location, and leave the Orbited server running through the next step.

Server Component

The server is a fairly straightforward piece of code. This tutorial focuses on Python and MorbidQ, but the next installment in this series will look at Java with ActiveMQ. Create a file called “data_producer.py” in the same directory as the other files. Inside this file, add this code:

#!/usr/bin/env python
from stompservice import StompClientFactory
from twisted.internet import reactor
from twisted.internet.task import LoopingCall
from random import random
from orbited import json
 
DATA_VECTOR_LENGTH = 10
DELTA_WEIGHT = 0.1
MAX_VALUE = 400 # NB: this in pixels
CHANNEL_NAME = "/topic/graph"
INTERVAL = 1000 # in ms
 
class DataProducer(StompClientFactory):
 
    def recv_connected(self, msg):
        print 'Connected; producing data'
        self.data = [ 
            int(random()*MAX_VALUE) 
            for 
            x in xrange(DATA_VECTOR_LENGTH)
        ]
        self.timer = LoopingCall(self.send_data)
        self.timer.start(INTERVAL/1000.0)
 
    def send_data(self):
        # modify our data elements
        self.data = [ 
            min(max(datum+(random()-.5)*DELTA_WEIGHT*MAX_VALUE,0),MAX_VALUE)
            for 
            datum in self.data
        ]
        self.send(CHANNEL_NAME, json.encode(self.data))
 
reactor.connectTCP('localhost', 61613, DataProducer())
reactor.run()

The code block is very straightforward. We are simply using a python Stomp client called stompservice, and having it connect to the MQ to send random data values every second. We give a name to each magic number so you can experiment with them as you like. Probably the most interesting is the INTERVAL value.

Save the file, and run “python data_producer.py” in a separate shell from where Orbited is running. Look back at your browser and you’ll see that the graphs are now changing in real-time. You can stop the data producer and then start it again without causing any problems with your Stomp connection. If you stop Orbited, you should see an alert with about losing the connection with the code 201, which is the Orbited.Statuses.ServerClosedConnection code. All codes can be found at the TCPSocket Documentation page.

Complete Code

You can download the entire project as a tarball: graph_demo.tgz

Questions or Comments

If you have a complicated question, please use the Orbited-users Google Group. Alternatively, you can try #orbited on irc either with an IRC client, or with the embedded LiveHelp at the bottom of this page.

Conclusion

This concludes the tutorial. With this knowledge you should be able to create a stompservice that publishes arbitrary data to the browser, and user interfaces that represents it however you please. Keep in mind that messaging is a good architecture for many applications, but not all. For instance, it is poorly suited for chat rooms, as most MQ protocols lack a mechanism to track presence.

The next installment in this series will be very similar to this tutorial, but will show how to produce the data server-side using a Java-based JMS or Stomp client with ActiveMQ.

Embedded LiveHelp

Ask questions now on the Orbited IRC channel:

[Slashdot] [Digg] [Reddit] [del.icio.us] [Facebook] [Technorati] [Google] [StumbleUpon]
Webtide

33 Responses to “Scalable Real-Time Web Architecture, Part 2: A Live Graph with Orbited, MorbidQ, and js.io”

  1. Orbited Blog » Blog Archive » Announcing Orbited 0.7.0 Says:

    [...] marked the release of Orbited 0.7.0, along with the first 0.7.0 tutorial over at CometDaily. Thanks to everyone who helped with the [...]

  2. Chad Lung Says:

    This line:
    Turn orbited on by running: orbited –config graph.cfg

    Should be (at least for me):
    Turn orbited on by running: orbited -–config graph.cfg

    Chad

  3. Chad Lung Says:

    Something else to note for people typing this in. The config file in this tutorial says port 8000 but the code shown here is using port 9000 (the code in the tarball is all using port 9000). So if your typing this directly in from the web page make sure to fix the port numbers.

    If the article is updated/corrected you can ignore this.

    Chad

  4. nuggien Says:

    I downloaded the graph_demo.tgz, ran orbited –config graph.cfg, and ran python data_producer.py. Then I went to the index.html page in my browser. The graph never changes. All the original bars just stay unchanged.

    Any ideas?

  5. Richard Maher Says:

    Hi,

    Sorry to appear negative again but I can’t see an additional HTTP (session, connection, long-poll) as being “the best tool” for receiving asynchronous data/messages from a server(s). I personally believe UDP messages to be a much better PUSH fit (possibly Multicast for intranets) in combination with a TCP/IP (or Ajax) data PULL channel.

    Anyway FWIW, if you’d like to see an example of a bog-standard VMS server that sends UDP messages to Web-client subscribers, then please follow these instructions: -

    1) Click on the following link and read the garb:
    http://manson.vistech.net/~tier3/tier3pager.html

    2) Telnet to manson.vistech.net. (If you don’t already have an account on the Deathrow cluster then please use Username: DEMO Password: USER) and then:

    $set term/width=132
    $run sys$users:[users.tier3.web]demo_udp_msg

    3) Enter the IP address of your client node. Your “stock-monitor” from step 1 should now spring into life with ransom stock-prices generated at 2sec intervals. (NATed clients will find this bit problematic :-)

    4) Enter any adhoc messages that you wish to appear in the seperate Java Frame on the client.

    OPCOM messages to web-subscribers? CHAT conferences? Stock-Watching? Alarm Monitoring?

    Cheers Richard Maher

    PS. The code for Tier3Pager.java is below, but all can be found at SYS$USERS:[USERS.TIER3.WEB]

    Tier3Pager.java
    ===============

    /**
    * Copyight Tier3 Software. All rights reserved.
    *
    * Author: Richard Maher
    *
    **/

    import java.applet.Applet;
    import java.awt.*;
    import java.net.*;
    import java.io.IOException;
    import netscape.javascript.JSObject;
    import netscape.javascript.JSException;

    public class Tier3Pager extends Applet
    {
    private String hostName;
    private JSObject browser;
    private static MessageThread socketThread;
    private static Tier3Talk chat;

    public class MessageThread extends Thread
    {
    private DatagramSocket socket;
    private DatagramPacket packet;
    private String threadData;

    public MessageThread(String name, String txt) throws Exception
    {
    super(name);

    byte[] buffer;
    threadData = txt;

    String port = getParameter(”PORT”);
    String maxBuf = getParameter(”MAXBUF”);
    try
    {
    if (port == null)
    socket = new DatagramSocket();
    else
    socket = new DatagramSocket(Integer.parseInt(port));

    if (maxBuf == null)
    buffer = new byte[512];
    else
    buffer = new byte[Integer.parseInt(maxBuf)];

    packet = new DatagramPacket(buffer, buffer.length);
    }
    catch (Exception e)
    {
    e.printStackTrace();
    System.out.println(”Unable to create UDP Socket”);
    throw new Exception(”Message thread could not be created”);
    }

    setDaemon(true);
    start();
    }

    public void shutdown()
    {
    socket.close();
    }

    public int getLocalPort()
    {
    return socket.getLocalPort();
    }

    public void run()
    {
    System.out.println(”Started Message thread. ThreadData = ” + threadData);
    String args[] = {”Started Message Thread ” + threadData};
    browser.call(”alert”, args);
    boolean stopThread = false;

    readLoop:
    while (!stopThread)
    {
    try
    {
    socket.receive(packet);
    String received = new String(packet.getData(), 0, packet.getLength());
    processMessage(received);
    }
    catch (SocketException e)
    {
    System.out.println(”Shutting up shop”);
    stopThread = true;
    continue readLoop;
    }
    catch (IOException e)
    {
    e.printStackTrace();
    System.out.println(”Unable to retrieve UDP message”);
    }
    }

    System.out.println(”Thread run() unit terminating”);
    }

    public void processMessage(String msgText)
    {
    int msgType = Integer.parseInt(msgText.substring(0,2));
    switch (msgType){
    case 1:
    chat.append(msgText.substring(2));
    break;
    case 2:
    String args[] = {msgText.substring(2)};
    try {browser.call(”priceUpdate”, args);}
    catch (JSException e)
    {
    System.out.println(”Error when calling JS priceUpdate()”);
    }
    break;
    default:
    System.out.println(”Unknown rec type “+msgText);
    }
    }
    }

    public void init()
    {
    System.out.println(”Initializing. . .”);
    hostName = getCodeBase().getHost();

    chat = new Tier3Talk(”Tier3 Messages”);
    requestFocus();

    browser = JSObject.getWindow(this);

    if (socketThread == null)
    {
    try
    {
    socketThread = new MessageThread(”MsgDaemon”, “SomeData”);
    }
    catch (Exception e)
    {
    e.printStackTrace();
    System.out.println(”Could not init Tier3Pager”);
    }
    }
    }

    public void alert(String alertText)
    {
    String args[] = {alertText};
    browser.call(”alert”, args);
    }

    public void destroy()
    {
    if (chat != null)
    chat.dispose();

    boolean stillDying;

    if (socketThread != null){
    socketThread.shutdown();
    do
    {
    stillDying = false;
    System.out.println(”Joining MessageThread”);
    try {socketThread.join();}
    catch (InterruptedException e){
    System.out.println(”Interrupted Join”);
    stillDying = true;
    }
    } while (stillDying);

    socketThread = null;
    }

    System.out.println(”Tier3Pager Applet Rundown complete”);
    super.destroy();
    }

    }

  6. Michael Carter Says:

    @Richard: No one (at least not me) is arguing that we shouldn’t use UDP on the back-end. In fact, I believe that RabbitMQ, among other message queues, currently can use (or has plans to use) UDP as a back-end replication mechanism, particularly in scenarios when multicast makes sense. I would highly recommend using UDP back-end for message queue coordination, though that sort of optimization is wholly outside of the scope of this introductory article.

    But why use PULL (read, polling) for the final leg of the journey to the browser? That just makes no sense. The performance, scalability, and latency issues associated with using a PULL model on the web are well documented, not just on this site, but in every published analysis of real-time web applications that depend on that model and end up with serious scalability issues. Refer to Greg Wilkins post on “Comet is always better than polling” for more information. (http://cometdaily.com/2007/11/06/comet-is-always-better-than-polling/)

  7. Richard Maher Says:

    Polling? Who said anything about POLLING?

    I think we’re all agreed here that polling (long, short, or tall) is anathema to us!

    If the “2sec intervals” threw ya, that’s just my dodgy “pretend someone made a trade” or “something to broadcast” trigger. I could’ve randomized the delay I suppose but it is just a Mickey Mouse event generator. *The client DOES NOT poll* Whenever the data is pushed, the separate client thread will receive it immediately and process it. (See code for more)

    If you were concerned as to why I suggest a PULL channel as well, it is because UDP is inherently unreliable. Messages may arrive out of sequence, not at all, or multiple copies of same. (And heartbeats, sequence-checkers, and timeouts were left out of the example for brevity) Also I don’t believe you can protect UDP traffic with SSL (But you can with IPsec) So at the end of the day, I am of the opinion that whatever is received on the UDP socket really ought to be verified and only acted upon via a connection-oriented (possibly encrypted and authenticated) TCP/IP socket.

    Once again, “No Polling!” :-)

    Where I believe our paths do diverge is your distinction between “back-end” and “final-leg”; I would not introduce an additional orbited server, proxy, or whatever between client and originating server. As per my example, I’d expect the originating server to keep a list of active subscribers and publish the data *directly* to them; no Mr. In-Between.

    Cheers Richard Maher

  8. Michael Carter Says:

    @Richard: I guess I just don’t understand what you mean when you say “in combination with a TCP/IP (or Ajax) data PULL channel”. Pull, by definition, means that the client initiates any kind of data transfer, and as such it is a polling model.

    I think you are missing the entire point of Orbited, and every other comet system featured on this site. We don’t *want* to put a “Mr. In-Between” in the middle of our architecture, we *have* to do it. Your sockets, udp, tcp, or otherwise simply *will NOT work over most IT deployments*. Without the use of Comet (long polling, streaming, or otherwise) you can’t guarantee traversal over forward proxies that most companies, schools, and government organizations use as the single gateway to the outside world. See more on my reply to your comment in the Flash/comet article: http://cometdaily.com/2008/09/30/why-flash-must-adopt-comet/#comment-2304

    Its not like we all woke up one day and wanted to spend five years of our lives hacking together Comet implementations. We (I probably speak for all of us) hated doing it; We tried flash sockets, java sockets, and browser plugins, but found out that the only way for guaranteed delivery was and is with Comet, though hopefully this will change with the introduction of HTML5 WebSocket.

  9. Richard Maher Says:

    Hi Michael,

    > Pull, by definition, means that the client initiates any
    > kind of data transfer, and as such it is a polling model.

    Maybe we’re separated by a common language here; if you see any and all client-initiated data-transfers as polling then so be it. Or if it would help to replace my use of “pull” with “request/response” or “client/server” then I’m good with that also.

    In my example code, whenever an asynchronous UDP “stock-price-change” message event has been received, a JavaScript function is called to display same, and either automatically, or user-driven, a buy-order would be placed on my TCP/IP pipe. Anyway, let’s move on. . .

    > See more on my reply to your comment in the Flash/comet article:
    > http://cometdaily.com/2008/09/30/why-flash-must-adopt-comet/#comment-2304

    Thanks; I responded over there.

    > We tried flash sockets, java sockets,

    Just for the record, Silverlight now also supports Sockets (still beta I think). So we have Sun/Java, Adobe/Flash, and Microsoft/Silverlight all saying that Socket support may be of some limited use to browser clients (outside of WebStart, Air and .NET)? A formidable line-up indeed!

    If only they knew how useless these Sockets are :-)

    > though hopefully this will change with the introduction of HTML5 WebSocket.

    Judging by an e-mail from the WHATWG list a couple of weeks ago it appears that you have solved the same problems with proxy/firewall navigation long ago, that are now bedevilling the WebSocket effort? (Whole e-mail below, please snip it if it’s too big)

    Cheers Richard

    > Date: Tue, 14 Oct 2008 10:22:01 +1100
    > Subject: [whatwg] WebSocket and proxies
    >
    > In the process of testing my WebSocket proposal I discovered the CONNECT
    > method has a major restriction. Most proxies disable CONNECT to anything
    > but port 443.
    >
    > The following is from “Squid and the Blowfish”:
    > ——————
    > It is very important that you stop CONNECT type requests to non-SSL
    > ports. The CONNECT method allows data transfer in any direction at any
    > time, regardless of the transport protocol used. As a consequence, a
    > malicious user could telnet(1) to a (very) badly configured proxy, enter
    > something like:
    > … snip example …
    > and end up connected to the remote server, as if the connection was
    > originated by the proxy.
    > ——————-
    >
    > I verified that Squid and all public proxies I tried disable CONNECT by
    > default to non-SSL ports. It’s unlikely many internet hosts will have
    > 443 available for WebSockets if they also run a webserver. It could be
    > done with virtual IPs or dedicated hosts but this imposes complex
    > requirements and costs over alternatives like CGI.
    >
    > The availability and capabilities of the OPTIONS and GET protocols also
    > varied from proxy to proxy. The IETF draft related to TLS
    > (http://tools.ietf.org/html/draft-ietf-tls-http-upgrade-05) has this to say:
    >
    > ——————-
    > 3.2 Mandatory Upgrade
    >
    > If an unsecured response would be unacceptable, a client MUST send
    > an OPTIONS request first to complete the switch to TLS/1.0 (if
    > possible).
    >
    > OPTIONS * HTTP/1.1
    > Host: example.bank.com
    > Upgrade: TLS/1.0
    > Connection: Upgrade
    > ——————-
    >
    > So according to this draft spec OPTIONS is the only way to do a
    > *mandatory* upgrade of our connection. Once again this failed in testing
    >
    > ——————-
    > => OPTIONS * HTTP/1.1
    > => Proxy-Connection: keep-alive
    > => Connection: Upgrade
    > => Upgrade: WebSocket/1.0
    > => Host: warriorhut.org:8000
    > =>
    > ——————–
    >
    > Other proxies gave different errors or simply returned nothing. The
    > problem may be related to the Upgrade and Connection headers rather than
    > OPTIONS, since I had similar issues using Connection: Upgrade with GET.
    >
    > I had the most success using GET without a Connection: Upgrade header.
    > It seems that the proxy thinks the header is directed at it so it does
    > not pass it on to the remote host. In many cases it will abort the
    > connection. Using the Upgrade: header without Connection allows the
    > Upgrade header through to the actual websocket service.
    >
    > It seems to me that whatever we try in many cases the connection will be
    > silently dropped by the proxy and the reasons will be unclear due to the
    > lack of error handling. There seems to be a wide variation in proxy
    > behaviour for uncommon operations. I suppose proxy developers could fix
    > these issues but whether a significant rollout could be achieved before
    > HTML5 is released is questionable.
    >
    > Given that an asynchronous connection cannot be cached the only reasons
    > remaining for going through a proxy are anonymity and firewall
    > traversal. Automatically bypassing the users proxy configuration to
    > solve the issues above has the potential to break both of these. It
    > would be a significant breach of trust for a UA to bypass the users
    > proxy and some networks only allow connections via a proxy (for security
    > and monitoring).
    >
    > It seems that we’re stuck between a rock and hard place here. In light
    > of this I reiterate my earlier suggestion that the time could be better
    > spent providing guidelines for communication via an asynchronous CGI
    > interface. This would allow reuse of existing port 80 and 443 web
    > services which would resolve the cross-domain issues (the CGI can relay
    > the actual service via a backend connection) and most of the proxy
    > issues above (since proxy GET and CONNECT are more reliable on these ports).
    >
    > Shannon

  10. Stefan Schoeman Says:

    It would be really nice to be able to get UDP directly to my browser as Richard suggests, and even nicer if that were directly supported in the DOM and not via ActiveX, Java, Flash, Silverlight or whatever else. No one can doubt that this would be top notch in the scalability stakes.

    In my case at home, Richard’s solution is no problem to implement. After all, here I control the firewall and doing UDP port forwards to my private net IP are no problem (unless I have multiple browsers that need the UDP feed). It is however a major problem when I need to write an application that is to be used by John Citizen who could be behind a simple cable router “firewall” or a fully-blown restrictive corporate firewall. In the case of the latter, communication is often restricted to HTTP/HTTPS only and even then, the HTTP is typically run through a Proxy server. It gets worse. I don’t know how this applies to others, but in my country the mobile phone networks don’t issue NIC IP’s to subscribers, so our entire mobile population looks as if they are behind a firewalled network :( No UDP forwards for sure, and bargain pretty much on a transparent HTTP Proxy.

    It is in these scenarios where a proxy solution like Orbited running on port 80 or 443 works great for me. Yes sure, the long polling is problematic and it has scalability issues server side. It is a pain to get it to work with all browsers, Opera being my nemesis because it is used on so many Nokia mobiles and I’ve never got them to play nice with Orbited. And Google Chrome doesn’t seem to like Orbited’s TCPsocket onclose event. The closest I can get as an alternative to Orbited is to run a Java applet or Flash swf that makes a socket connection to port 443. This port is often left alone in less restrictive firewall setups, so it has a better chance of getting through than say a connection to port 32080. But if the user is explicitly using a proxy server, i.e. it is set in his/her browser, as is enforced by Windows domain policy in many corporations, this does not work. By using Orbited, I just simply don’t care about this - it just works - at least, I think so, I haven’t tested that ;)

    All in all - for me, Orbited makes for a pretty decent solution without me having to worry about providing clients with port forward instructions to cater for their networking setup, or having to speak to some corporate IT department hawk about why someone should have a specific port opened or forwarded. Of course there are alternatives, and when you control the network, better and more efficient ways of getting the same result (like just simply writing old-school client/server desktop apps ;). Richard, your solution is cool, but that’s been tried and tested and pretty much common knowledge. I do not thing that Orbited aims to be this super-efficient way of getting the same result - it attempts to be universally usable across people’s different network policies without change.

    As for the comments on the Squid Proxy CONNECT issue, yes, I agree with Richard that this is pie in the sky stuff. Most proxies that I’ve encountered will not allow CONNECT to anything other than port 443 out of the box. Anyone that administers a SQUID server and actually looks at their logs, will confirm that CONNECT attempts against proxies, especially port 25 for spam relaying, are rife, so it’s unrealistic to assume that network admins are going to change the restrictions. From that perspective, I don’t think the work on W3C Websocket to use CONNECT if a proxy is detected, is going anywhere.

    Hence, even more reason to use Orbited. Michael, I suspect there might be a much longer lifetime to Orbited. I always get the impression that Orbited is seen as a temporary solution until HTML5 Websocket gets adopted. With the proxy restrictions on CONNECT, I see Orbited being around for quite a while longer ;) Please don’t give up on what I consider to be an excellent initiative, or at least a solution that works for me. Just chuck the Python and write this in C please (–duck–) ;)

  11. Eugene Yunak Says:

    Thanks for this great article!

    I’m trying to make it work, but with no success. Can you help me? I just installed latest orbited from trunk, and stompservice with easy_install. “echo -e ‘import twisted\nimport orbited\nimport stompservice\n’ | python -” shows no error, so i expect all the modules to be correctly installed. I had to change line 7 in index.html to look like “”, because it seems that orbited has moved the file stomp.js into this location. with WebDeveloper plugin for Firefox i can see that with this URL stomp.js is correctly found. The problem is that after starting data_producer, the bars aren’t changing =(
    Then, i changed js in index.html to include
    “stomp.onopen = function() {
    alert(’got connection’);
    };”
    instead of
    “stomp.onopen = function() {
    };”
    But i wasn’t alerted after i started data_producer again.
    I would greatly appreciate any help to sort this out, as i’m only a beginner with orbited, mostly being a python developer with no experience with js. If i can do any additional traces that would help you, i would obviously like to.
    Sorry for my English.

  12. Norppa Says:

    To everyone having problems like nuggien and Eugene Yunak, that is the bars not changing, check that you really used address http://localhost:9000/graph in your web browser,
    not http://127.0.0.1:9000/graph or http://0.0.0.0:9000/graph.

    I first tried the numerical addresses and had no luck, but then switched to ‘localhost’ and it works.

    Has anyone a clue why there is a difference?

  13. Robse Says:

    Works fine with http://127.0.0.1:9000/graph for me

  14. Robse Says:

    Oops no sorry it doesn’t actually, Norppa is right

  15. Don Moir Says:

    I am using flash to demonstrate the use of sockets over port 80 and if needbe, use HTTP to the same server. When connected thru HTTP 1.1, no polling is required. I also have support for HTTP 1.0 which is used by some proxies such as squid. At the moment I have support turned off for HTTP 1.0 simply because I am not yet satisfied with it and will require more testing. Another option is Java 1.6. Both Flash and Java 1.6 allows you to do cross-domain connections which is just like way important. Also SilverLight is expected to follow suit.

    I am still beta testing this and my terminology is not yet complete.

    Here is an example of real time browser connections in the form of a real time blog.

    http://sms.pangolin.com/rtblog/

    Don

  16. Alex Says:

    I followed every step of the tutorial and eventually got stuck with an error on the client side. In the line: “var vals = JSON.parse(body);”

    My JavaScript debugger says: “Uncaught ReferenceError: JSON is not defined”

    What could be the cause and how might I fix it?

  17. Mike Says:

    I had the same Problem as Alex, I added the following to index.html

  18. Tvrtko Says:

    How to make the web page from this example to reconnect after the orbited is restarted (or for any other reason that caused the connection to be lost)?

    I expected stomp.js/orbited.js to reconnect automatically but it doesn’t.

    More specifically, the underlying transport (Orbited.CometTransports.XHRStream) doesn’t reconnect:
    1. Orbited.js, onreadystatechange function, switch case 4, doesn’t meet the xhr.status === null
    2. if I modify Orbited.js so that doReconnect = true, then the url is old (I get 404)

    What I did in the end was:

    var logger = Orbited.getLogger(”example”);
    var reconnectTimer = null;
    stomp.onopen = function() {
    if(reconnectTimer) window.clearTimeout(reconnectTimer);
    };
    stomp.onclose = function(c) {
    logger.debug(’Lost Connection, Code: ‘ + c);
    reconnectTimer = window.setTimeout(reconnect, 1000); // TODO: autoincrease with cap
    };

    The same thing about reconnecting holds for python data_producre.py - it doesn’t reconnect!

  19. Starting Out With Comet (Orbited) Part 1 « Things I Learned Says:

    [...] basically chose Orbited because it was the first one I could get up and running thanks to this tutorial, because I love python, and because my implementation will be working with [...]

  20. Starting Out With Comet (Orbited) Part 2 – Installation and STOMP « Things I Learned Says:

    [...] start I’d like to quote my source.  I’ve learned most of what I’m sharing from Michael Carter’s Tutorial and many hours of playing [...]

  21. Jeff Zhang Says:

    @nuggien and Eugene Yunak:

    I met the same issue with you. Lastly I found the root cause is that my browser does not support JSON. I downloaded a newest one, firefox 3.5 with native JSON parser. It worked fine. You can try that.

    Hope that it is useful for you. I’m also a beginner with Orbited.

    Jeff

  22. Anthony Says:

    I had the JSON problem too. Rather than upgrading to Firefox 3.5 to get the demo going I replaced:

    var vals = JSON.parse(body);

    with

    var vals = eval(’(’ + body + ‘)’);

    Other than that, everything else worked for me, really helpful article.

  23. David Horner Says:

    I added a script tag for http://localhost:9000/static/JSON.js and it worked for me in 3.0 firefox.

    Good luck!

    –dave

  24. Chad Whitacre Says:

    RE: the JSON issue, you can also replace that line with:

    var vals = Orbited.JSON.parse(body);

  25. Stephen Mattison Says:

    You need to import JSON.js, which is stored in the same static directory as Orbited.js

  26. Steven Woo Says:

    Did Norppa ever get an answer as to why numeric addresses did not work? I have to use numeric addresses since the server is not localhost, but everything works except for the data_producer.py step, it never updates. I did have to change my .cfg file to have [access] * -> 10.100.200.20:61613

  27. Alfredo Says:

    Hi all,
    i’m a beginner with Twisted, Stomp and Message Queue Architecture.
    I looked at the tutorial (Simple Real-Time Graph), where data is produced in a synchronous way (using Twisted’s LoopingCall object) every second.
    Now, i want to modify the code to realize monitoring session that broadcasts data generated from “data_producer.py” in asynchronous way as soon as they were received from the outside world, and immediately processed towards the stomp channels.
    I tried to modify “recv_message” and “send_data” defs and make it work, but with no success: data were generated and shown onto the stdout but not send towards connected clients, with orbited, python stomper’s or stompservices’ example programs (stompbuffer-rx.py), etc.

    Can you help me to undestand with simple examples how handling Twisted functions to implement these asynchronous capabilities without using LoopingCall object?

    There is my example code:
    class DataProducer(StompClientFactory):
    def recv_connected(self, msg):
    self.data = “Initialize…”
    # What goes now at this point? <————
    ??????????? # <—————————

    def send_data(self):
    try:
    while 1:
    # Read data (this is a string) from outside world
    frame=RecData(smon)
    # Show it towards stdout
    WriteLog(frame)
    #
    self.data=frame
    self.send(”broadcast/monitor”, json.encode(self.data))
    except …

    reactor.connectTCP(’localhost’, 61613, DataProducer())
    reactor.run()

  28. Tom Leys Says:

    Hi Alfredo.

    Please email me at tom - at - widget5 - dot - co.nz for a detailed example of how to get this to work. As an alternative, repost your question at

    http://stackoverflow.com/questions/tagged/twisted

    Long story short, you create another class that is a server, who receives your incoming events. See this Twisted tutorial for more…

    http://twistedmatrix.com/projects/core/documentation/howto/tutorial/index.html#auto1

    Then you change the line

    reactor.connectTCP(’localhost’, 61613, DataProducer())

    to

    data_producer = DataProducer()
    reactor.connectTCP(’localhost’, 61613, DataProducer())

    Finally, you call methods in data_producer from outside to kick off message sending

    def send_something(message):
    data_producer.send(’/topic/graph’, simplejson.dumps(message, ensure_ascii=True, encoding=”ascii”))

    Of course, you should clean this up (and wrap the message sending inside data_producer), but hopefully it will get you started.

    Thanks for the fantastic tutorial Michael, it was invaluable.

  29. Ivan Says:

    Dear Michael

    Very nice tutorial. Demo wors out of the box for me.

    Best wishes

    Ivan

  30. confiq Says:

    orbited.org:8000 is down….
    And i’m tring to learn it!

  31. Yarko Says:

    hmmm…. orbited.org (still?) down / unresponsive; and twistedmatrix.com domain seems to have expired on 18 June…

  32. Orbited demo as a service in Centos 5.5 | WebDev Says:

    [...] Demo http://webdev.my:9000/graph based on the orbited tutorial. [...]

  33. user Says:

    Hi,

    I have to use comet with Django, I chosen Orbited, but I’ve heard that Orbited is no longer under development? Is it true? Why Orbited site is broken? ;/

    Maybe some advices about best library (server) for comet + django (needed for chat).

Leave a Reply



Copyright 2014 Comet Daily, LLC. All Rights Reserved