Why Flash must adopt Comet

by Michael CarterSeptember 30th, 2008

I constantly get people saying “Comet, pftt… Flash already has a socket.” I agree with the basic sentiment; I’d rather have a socket than Comet, but I’ve always been uneasy about depending on Flash. At first, because it wasn’t installed on all computers, then because new platforms weren’t adopting it, such as the iPhone, and probably foremost because I get poor Flash performance on 64-bit Ubuntu.

Of course, in the real world, 90+ % of people have Flash, and the iPhone has its own SDK for applications which has nothing to do with the web. I would almost say its viable to deploy a real-time web application that uses Flash for its communication. Almost. There is still a major, show-stopping problem with Flash for real-time communication on the internet: forward proxies. If you were to make an HTTP request from Flash, it would implicitly send the connection request through the proxy based on settings in the browser. But Flash doesn’t expose that information to the developer. So if you open a socket, the connection request is not sent via the proxy, and it is therefore never made at all.

This is an enormous problem for developing real-time applications, particularly when the target demographic includes the enterprise, government agencies, or students. An application for this demographic that depends on Flash sockets won’t just be unreliable; it will practically never work due to the prevalence of forward proxies.

There have been attempts at solving this problem, such as Christian Cantrell’s RFC2817Socket class, but it isn’t a sufficient solution. As Christian puts it, “Unfortunately, this may not be the entire story, though. The reason I chose such a clumsy name for the class is that it will only work with proxy servers who adhere to RFC 2817.” This is itself a major issue, but there is a second, more pressing problem with the solution: You have to seed the RFC2817Socket with apriori knowledge of the proxy. This is, in almost all cases, wholly infeasible&emdash;it only works in very niche deployments where the application is maintained by the IT department that installed the proxy in the first place.

Comet suffers from none of these problems. The browser implicitly routes Comet requests through the proxy because they are HTTP. As I’ve noted previously Comet on its own isn’t the best developer-facing API to deal with; a socket would be much more ideal. This is why Orbited tunnels TCP communication over Comet and Ajax, and provides a TCPSocket API to developers.

A number of Flash developers have become interested in using Orbited because they want that socket API that they’re used to, but with proxy support. So much so, in fact, that we’ve started porting Orbited.TCPSocket to Flash. That’s right, we are making Comet/Ajax (HTTP) Requests to Orbtied from Flash, and proxying the data to normal TCP servers. As an added bonus, there is no need to deal with cross domain policy files; that can be configured in Orbited’s access control.

In conclusion, Flash’s Socket class is useless because it ignores proxy settings. A pure-Flash implementation of Orbited.TCPSocket will allow flash applications to connect to arbitrary TCP servers without worrying about forward proxies, or cross-domain policy files. It is ironic that Comet, a technology developed so as to avoid dependence on plug-ins, is the only good way to provide solid real-time communication in Flash.

18 Responses to “Why Flash must adopt Comet”

  1. Geoffrey Lee Says:

    Michael, have you heard of Flash Media Server? It’s capable of HTTP tunneling to allow your Flash applications to get past firewalls and HTTP proxies. There is also Red5 which aims to provide an open source alternative to Flash Media Server.

    http://kb.adobe.com/selfservice/viewContent.do?externalId=tn_16631&sliceId=2

    And an example implementation of a chat room:
    http://www.sephiroth.it/tutorials/flashPHP/flashcomm_AMFPHP/index.php

  2. Michael Carter Says:

    Geoffrey: I am at least peripherally aware of Flash Media Server, though I automatically discount technology that has a 5-digit-per-cpu-license price attached.

    More to the point, Red5 and Flash Media Server both just implement a tunneled version of adobe’s proprietary RTMP (real-time messaging protocol), namely RMTPT and RMTPS. If you look closely at these protocols, they are actually just long polling over HTTP — They already are Comet!

    But most importantly, Red5 and Flash Media Server don’t give you a plain old socket api that you can connect to a plain old tcp server. You can’t use Flash Media Server to connect flash to an xmpp server. You can probably implement some method of bridging the two together, but thats not at all what I’m talking about.

    So you’re right, Flash has already adopted Comet for their media server. Now they need to open up their specification, so we can connect to something besides Flash Media Server or the community’s best effort at reverse engineering the Media server (Both of which are just jumped up message queues with remoting and media streaming thrown in.) But even if they don’t, its okay, Orbited solves the problem of tunneling data or HTTP reliably, efficiently, and with completely open protocols.

  3. iongion Says:

    There are others besides red5 and fms that provider better, more native to the flash player, real time communication middlewere.

    Choose:
    1) blazeds - free (java),
    2) weborb - (.net,ruby,php)
    3) rubyzumi (ruby)
    4) rtmpy (python)
    5) flourine (.net)
    6) haxevideo (actionscript like haxe)

    So a flash developer can also leverage on such an easy rtmp api flash player provides without dealing with proxy objects

  4. Mike Says:

    The spec is open, has been for months, and implemented in several open source projects, BlazeDS and GraniteDS on the Java front, and PHPAmp for LAMP. I’m sure there are more, but even I can’t keep track of EVERYTHING! ;) What still needs to be done?

  5. Cosma Colanicchia Says:

    Hi Michael..

    take a look at BlazeDS, an open source product from Adobe. It implements a number of protocols to allow communications between flash and a java servlet container, including a “near real time” channel.. very similar, AFAIK, to comet technique.

  6. Franck Wolff Says:

    Hi,

    Few precisions about Comet support in GraniteDS: it uses HTTP requests only (based on flash.net.URLStream/flash.net.URLRequest) and it implements (roughly) the Bayeux protocol (long-polling connections only).

    This should be considered as a true Comet adoption in a Flash environment…

    Regards.

  7. Michael Carter Says:

    Its great to hear about all of these Flash products/projects that are currently using Comet for proxy interoperation. Also good to hear that RTMP(T/S) is now an open specification. (Someone should update Wikipedia on that point)

    For the record, I wasn’t trying to say that there isn’t Comet already in use with Flash, rather that Flash’s socket isn’t actually useful, and that functional bi-directional communication still requires Comet. So perhaps a better title would have been “Why Flash Applications Are Adopting Comet”

    My secondary point is the same one I’ve been making for a while, which is that sockets are great, and for them to work they need to be tunneled over HTTP. So it would be nice to have a Flash socket server that simply acts as a TCP/HTTP -> TCP/IP proxy. Then we can write software around a Socket, not an RTMP message queue.

  8. RIA Revolution » Flash and Comet Says:

    [...] Carter of Comet Daily writes: Why Flash must adopt Comet, recommeding that Comet is the best way for real-time Flash applications. The blog post is an [...]

  9. Shashank Tiwari Says:

    Folks! Please get your facts right. RTMP is not an open sourced protocol, AMF is. BlazeDS does not use RTMP, it uses AMF long polling to achieve data push. LCDS (the paid product supports RTMP), so does Flash Media Server. Red5 implements a reverse engineered version of RTMP. LCDS (the paid product) uses Java NIO based Comet style connections. BlazeDS can be easily refactored to support Comet. The primary piece of software in BlazeDS is a Servlet — what is called the MessageBrokerServlet — and it can be refactored to support Comet style connections. TCP Socket is not an answer to all issues around real-time web and data push.

  10. oscar Says:

    hi Shashank Tiwari,

    BlazeDS also supports http streaming feature.

    [quote]
    BlazeDS can be easily refactored to support Comet. The primary piece of software in BlazeDS is a Servlet — what is called the MessageBrokerServlet — and it can be refactored to support Comet style connections.
    [/quote]

    Have you tried the refactoring?
    I am trying to do it. Any idea are welcome.

  11. Richard Maher Says:

    Hi Michael,

    > I constantly get people saying “Comet, pftt…
    > Flash already has a socket.”

    I did read at least one comment, to an earlier blog entry, saying just that. But personally, I find Flash’s Socket support to be rudamentary at best. (No UDP support, no connection-timeout, no OOB-character support, etc) and much prefer the full-blown Socket functionality one can easily obtain via Java Applets and the java.net.Socket class. (This preference extends even to those cases where Flash is already available and in-situ on the page!)

    For an example of database rows being retrieved from a Java Socket and then used to populate Flex dataProviders (in this case ArrayCollection(s) via the Adobe FABridge), for pesentation via Flex Pie-Charts and DataGrids, please see: -

    http://manson.vistech.net/t3examples/demo_client_flex.html

    Username: TIER3_DEMO
    Password: QUEUE

    Don’t enter anything for the name, and just hit the “go” button. Then hover over the pie charts and click for drill-down functionality into the datagrid.

    All source including Java, Javascript, MXML can be found at: -

    http://manson.vistech.net/t3examples/

    > So if you open a socket, the connection request is not sent via
    > the proxy, and it is therefore never made at all.
    : : : :
    > In conclusion, Flash’s Socket class is useless because it
    > ignores proxy settings.

    I don’t follow your thinking here, or maybe I just don’t understand the problem because I’ve never used Flash’s sockets, but I can’t find anything in flash.net.socket.connect() that says “it will go via a proxy”. Who cares that Sockets ignore HTTP proxy settings? They’re not (normally) sending HTTP.

    Perhaps your next blogg should be entitled “Why WebSockets must adopt Comet”. (I certainly can see the very valid argument for those who are fixated with HTTP as a quasi-middleware protocol, or who’d like to preserve the benefits of proxy-server functionality.) But I think you’ll find that Flash (and most definitely Java) is doing just fine thanks very much :-)

    > My secondary point is the same one I’ve been making for a
    > while, which is that sockets are great, and for them to
    > work they need to be tunneled over HTTP.

    Can you explain exactly why you believe this?

    . Socket Cacheing - No Thanks
    . Limited client IP addresses - IPV6
    . Anonymity - Not always a good thing
    . Firewall - Open up connections to/from valid hosts/ports
    . Monitoring/filtering - Requirements spec for binary data

    Maybe a SOCKS proxy-server is more appropriate here, who knows? But please be advised that when it comes to full-duplex binary data transmission over Sockets, there are many out there (and here) who couldn’t care less about proxy-servers.

    HTTP *is* the box; let’s think outside!

    Cheers Richard

  12. Richard Maher Says:

    Hi (again),

    For some reason the dollar sign “$” has been removed from the links that I posted. I don’t know off-hand what the & like syntax for a dollar-sign might be or why it disappeared, but please be advised that their is a dollar-sign between the “T3″ and the “EXAMPLES” portion of the links that I posted. For example: -

    http://manson.vistech.net/t3DOLLAR-SIGN-HEREexamples/demo_client_flex.html

    Cheers Richard

  13. Michael Carter Says:

    @Richard

    I completely agree with you that we want to think outside the box of HTTP, in fact thats what this entire blog is about. The issue isn’t a question of HTTP being the right fit or not, its that we have *no choice* but depend on HTTP because of the corner into which we’ve painted ourselves with forward proxies.

    Consider a java.net.Socket, it opens a normal socket directly to the server and you can send bytes in either direction. Unfortunately, most IT departments do not allow arbitrary socket connections to the outside network. In fact, they only allow HTTP on port 80, and only through a proxy. So, the only way you’ll end up connecting with that remote server is if you play nice with the proxy. Neither flash sockets, nor java sockets, do play nice with the proxy. When native Java and flash sockets work, they’re great. When the user is behind a locked-down forward proxy, those solutions just don’t work *at all*. So my point is that the only sure-fire bet for this kind of is to tunnel through the proxy.

    The lack of UDP support is a big deal, though i think Flash 10 has added that. The problem with UDP though is that it is *impossible* to guarantee connectivity on most networks. It just won’t work because many intranets only allow HTTP/TCP on port 80/443 out. UDP packets are just thrown away.

  14. Richard Maher Says:

    Hi Michael,

    Thanks for your replies!

    > Unfortunately, most IT departments do not allow arbitrary
    > socket connections to the outside network. In fact, they
    > only allow HTTP on port 80, and only through a proxy.

    Look, I’m more than willing to concede “most” here, and I think Orbited is a very useful, and maybe even essential, tool for tunnelling messages in a http-cloak through the firewall and on to any arbitrary TCP/IP server. (if I’ve understood what it does correctly and I promise to investigate more) I applaud what’s been done, and look forward to seeing more and more deployments in the near future. (Any reverse-proxy potential as well?)

    Having said all that, what I’d like you (and ideally the WHATWG people) to concede/acknowledge is that there are very real customer sites out there that have applications that are either strictly inside the firewall, or who are more than willing to punch a hole through the firewall for known applications, hosts, and ports. VPNs and more and more IPsec playing a big part here too!

    Pluralism is good; the best tool for the job, eh? Orbited for when your stuck with 80/443, and Java, Flex (and now Silverlight) Sockets for when no such restrictions apply, and the full-blown underlying socket API and functionality is what you crave.

    > In conclusion, Flash’s [Java's, Silverlight's] Socket class is useless
    > because it ignores proxy settings.

    I guess it’s just that, to me, statements such as that are like a red-rag to a bull :-)

    Cheers Richard

    PS. When it comes to the “Flash Applications adopting Comet” bit, with the FABridge functionality available, what’s left to do?

  15. Michael Carter Says:

    @Richard

    It very rarely makes sense develop against a network technology that only works when you control every aspect of the network topology. I grant that there are some uses of Flash and Java sockets in the browser, and I’m glad that we have those options for the occasional moments when they work.

    I have one, single, argument here: Socket APIS are nice, but they don’t work through proxies, so lets engineer a socket that looks, acts, and feels like a socket, but works with a proxy. I want to write socket servers and socket clients, and for everything to *just work*. And I don’t even mind if our new socket uses a direct connection when available as a slight performance boost.

    At this point I’m not exactly sure what you take issue with. Clearly its not some kind of API difference, because I propose we keep the APIs *identical*. So then, is it that you think performance will be hindered in a substantial way?

    > (Any reverse-proxy potential as well?)

    Yes, Orbited works great with reverse proxies. I default to mentioning forward proxies because its the one part of the network which you are almost guaranteed to never have control.

  16. Don Moir Says:

    In my case, I eliminate the traditional back end services such as apache etc. I have a custom server that listens on port 80. This server accepts socket and HTTP connections. As noted by Michael, sockets do not work with proxies. If a socket fails, I default to an HTTP connection. Since both Flash and Java 1.6 accept cross-domain connections I can connect either way to an outside server.

    Sockets are simply more efficient(server and client) and less probamatic then HTTP connnections. Browser HTTP connections can be limited to just 2 and those are shared with resource loading etc. Michael you do not have control over this. I admit that most browsers can support 6 or more connections but still the 2 connection limit can be imposed and it does happen.

    Additionally, XMLHttpRequest is not cross domain although it can be tricked with more overhead. Also, it is text only cross browser. For me all my messages are binary for both socket and HTTP connections.

    Server side for me is very efficient even with HTTP connections. Mostly I see about 10 percent HTTP connections and the rest are sockets.

    Don

  17. Stefan Says:

    One wonders why Adobe would even release sockets if 10% people can’t use them. In addition, nowhere is it talked about that flash sockets don’t play nice with proxies. It really is quite irresponsible on Adobe’s part

  18. Don Moir Says:

    The fact that sockets are blocked by a proxy is not Adobe’s fault. It’s the proxy intercepting the socket attempt and refusing it.

    90 percent success rate is great. The good thing about flash is you can do sockets over port 80 or http. Other than more overhead with http connections it is pretty much seamless.

    One real problem still exist with proxies or at least one that I know of- Squid…. Much of it is still http 1.0 and that is problematic. http 1.1 is not a problem.

    Yes Adobe is quite irresponsible for many things. I spend alot of time getting flash to behave. But flash’s support for socket and http connections is pretty good.

    Can anyone speak to the situation with proxies and http 1.0 ? Mostly I get http 1.1 request but there is still the occasional http 1.0 request. Depending on browser, I can sometimes send http 1.1 data even though squid is requesting http 1.0.

    Polling seems to be the only sure fire method for http 1.0, but still investigating.


Copyright 2015 Comet Daily, LLC. All Rights Reserved