written on Monday, September 24, 2012 + + +
Out of curiosity I taught the Fireteam presence +server websockets as a protocol in addition to the proprietary protocol it +speaks out of the box. I don't really have a usecase for websocket per-se +with the server, but I thought it might make it more useful for +customers that want to use the event based parts of the API with HTML5 +games. This post focuses on implementing websockets on the server, not so +much about how you would use them from the client and basically collects +all thing I wish I would have known before.
+So let's start with that part first. What are websockets and why would +you use websockets? Basically only if you want to have a bidirectional +communication between an actual web browser and some server. Websocket is +not necessarily a good protocol if neither of the endpoints is an actual +browser. Websockets suffer a lot under the restrictions and +implementation details that were forced upon the protocol to make it work +with existing HTTP infrastructure.
+Websockets in the current iteration as specified by RFC 6455 do a bunch +of things differently to what a raw TCP connections does. The name +websocket gives the impression that it's a traditional socket. In +practice it combines the parts of UDP and TCP: it's message based like +UDP, but it's reliable like TCP. So assuming you know what TCP is, here +is what websocket adds on top of that:
+Websockets make you sad. There, I said it. What started out as a really +small simple thing ended up as an abomination of (what feels like) needles +complexity. Now the complexity comes for a reason. The protocol went +through many iterations and basically had to be changed multiple times +because of unforeseen security problems that came up with misbehaving +proxies. The protocol I created for the internal communication of our +server is upgrading from HTTP just like websockets do, but without the +“secure” parts. And here is why it does not matter:
+Everybody knows about HTTP proxies. We have proxies that do load +balancing on the application side, we have proxies that do SSL offloading, +we have proxies for all kinds of things. Unfortunately outside our +internal infrastructure everyone of us also has to deal with HTTP proxies +in corporate networks, and worse, on mobile network connections. The +amount of shitty HTTP middleware installed all around the world is just +staggering. And this pretty much has shown me that the only way you can +do anything useful these days is by putting TLS on top of everything and +just force people to stop with their shenanigans. On O2's mobile networks +you cannot use websockets unless they are encrypted. You cannot get +websocket connections through Amazon's ELB (Their load HTTP/TCP balancer). +Heck, you can't even get PATCH as an HTTP method through the ELB.
+Everything that Fireteam will be doing will most likely always be behind +an encrypted connection. It guarantees me that nothing can do funny +things with the data I'm sending around. And as a positive side effect I +don't have to mask my communication like websocket does because I know the +software stack on my side until it hits the internet. The communication +gets encrypted and I know nobody is going to change my data on the way to +the client.
+In fact, I would also recommend to always use websockets through TLS. +Even if you don't care about the security side of things you will still +benefit from the fact that your websocket connections succeed in many more +cases. Not encrypting your connection is definitely something you will +regret sooner or later.
+Alright. After this fairly long disclaimer you're still there, which +probably means you still want to do websockets. Fair enough. Now let's +start with the basics, the handshake. This is where everything starts. +It upgrades your connection from HTTP to something else. For the internal +protocol we recommend to customers we upgrade our HTTP connection +basically to a raw TCP connection. Websockets are not an upgrade to TCP, +it's an upgrade to a message based communication.
+To begin: why would you upgrade from HTTP instead of directly starting +with TCP as a protocol? The reasons for why Fireteam's protocol starts +with HTTP not all that different from why websockets upgrade from HTTP.
+Websockets upgrade from HTTP because it was believed that people would +develop servers that serve both websocket connections as well as HTTP +ones. I don't believe for a second that this will be what people do at +the end of the day however. A server that handles stateless requests to +answer with a reply has a very different behavior than a server that keeps +TCP connections open. However the advantage is that websockets use the +same ports as HTTP and HTTPS do and that is a huge win. It's a win +because these are privileged ports (< 1024) and they are traditionally +handled differently than non privileged ports. For instance on a linux +system only root can open such ports. Even more important: ELB only lets +you open a handful of these privileged ports (25, 80 and 443 to be exact). +Since ELB also does socket level load balancing you can still do +websockets on Amazon, just not through their HTTP local balancer.
+We're handling our persistent presence protocol very differently than our +HTTP webservice but we still benefit from the HTTP upgrade in some edge +cases. That's mainly where we have to tunnel our communication through a +HTTP library because arbitrary socket connections are not possible for +security or scalability reasons. If you have ever used Google Appengine +or early Windows Phone you will have noticed that HTTP connections are +possible where regular socket connections are not.
+You will also see that many corporate networks only allow certain ports +outgoing. The fact that websockets use the same port as HTTP/HTTPS make +this much more interesting. If anyone has ever used the flash socket +policy system will know that pain. Currently it's entirely impossible to +use flash sockets behind Amazon's ELB because the Flash VM will attempt to +connect to port 843 to get authorization information. That's a port you +can't open on the ELB. So the idea of starting with HTTP is pretty solid.
+HTTP always supported upgrades, but unfortunately many proxies seem to +have ignored that part of the specification. The main reason for that +probably is that until websockets came around nobody was actually using +the Upgrade flag. There was an SSL upgrade RFC that used the same +mechanism but I don't think anyone is using that.
+Alright. So what does the handshake look like? The upgrade is initiated +by the client, not by the server. The way Fireteam upgrades the +connection is by following the old SSL RFC and looks like this:
++OPTIONS / HTTP/1.1 +Host: example.com +Upgrade: firepresence +X-Auth-Token: auth-info-here ++
The server then replies by upgrading:
++HTTP/1.1 101 Switching Protocols +Upgrade: firepresence/1.0 +Connection: Upgrade ++
If the upgrade header was missing, the server instead answers with 426 +Upgrade Required:
++HTTP/1.1 426 Upgrade Required ++
What's interesting about this is that the upgrade require status code is +defined, but it does not show up in the HTTP/1.1 RFC. Instead if does +come from that SSL RFC.
+Websockets upgrade very similar, but they are using 400 Bad Request to +signal a missing upgrade. They also transmit a special key with the +upgrade request which the server has to process and send back. This is +done so that a websocket connection cannot be established with an endpoint +that is not aware of websockets. Here is what the handshake looks like +for the client:
++GET / HTTP/1.1 +Host: example.com +Upgrade: websocket +Connection: Upgrade +Sec-WebSocket-Key: dGhlIHNhbXBsZSBub25jZQ== +Origin: http://example.com ++
The websocket key here are random bytes. The server takes these bytes and +appends the special string 258EAFA5-E914-47DA-95CA-C5AB0DC85B11 to it, +then creates the SHA1 hash from it and base64 encodes the result (the +bytes, not the hexadecimal representation). The magic string looks like a +UUID and also is one, but that's completely irrelevant because the exact +string needs to be used. A lowercase representation or braces around the +string would obviously fail. That value is then put into the +Sec-WebSocket-Accept header. When the server has computed the value it +can send an upgrade response back:
++HTTP/1.1 101 Switching Protocols +Upgrade: websocket +Connection: Upgrade +Sec-WebSocket-Accept: s3pPLMBiTxaQ9kYGzzhZRbK+xOo= ++
The handshake can also include a protocol request and the websocket +version information but you can't include arbitrary other headers. If you +compare the websocket upgrade with our own upgrade you will notice that we +can't transmit the authorization information. There are two ways around +that. You can either transmit the authorization information as the first +request or put it into the URL as query parameter.
+Also notice that the Sec-WebSocket-Accept header brings it's own +grammar for the value. Normally you would expect you can quote the value +but the specification specifically requires a base64 value there:
++Sec-WebSocket-Accept = base64-value-non-empty +base64-value-non-empty = (1*base64-data [ base64-padding ]) | + base64-padding +base64-data = 4base64-character +base64-padding = (2base64-character "==") | + (3base64-character "=") +base64-character = ALPHA | DIGIT | "+" | "/" ++
Alright. As if websockets were not painful enough as they are, someone +had the amazing idea to also introduce a new URL scheme. Two in fact. +ws:// and wss://. Sounds like a tiny change from http to +https but unfortunately that's not the case. URLs have scheme +specific grammar. For instance FTP URLs can have authorization +information in the netloc part (ftp://username@server/) whereas HTTP +can't. mailto URLs don't have the leading slashes etc. Websocket +URLs are special in that they do not support anchors (#foo). Now why +would that matter? It matters because whoever created the URL parsing +module in Python also decided that they should stick as closely as +possible to the strict standard that you cannot throw arbitrary URLs at +the module. For instance if you would try to parse websocket URLs you +quickly realize that the results are just wrong:
+>>> import urlparse
+>>> urlparse.urlparse('wss://foo/?bar=baz')
+ParseResult(scheme='wss', netloc='foo', path='/?bar=baz',
+ params='', query='', fragment='')
+The reason why websockets have a separate URL is beyond me. I suspect it +stems from the fact that the RFC hints towards eventually dropping the +upgrade from HTTP so the HTTP URL would not make much sense. In any case +it's just a very annoying example of where we now have to things now that +were previously unnecessary.
+Also since it's a different protocol, protocol relative links will +obviously not work. You will have to switch between wss and ws by +hand.
+Otherwise the same rules as for HTTP style URLs apply. Namely that ws +is unencrypted and has port 80 as default port and wss requires TLS +encryption and port 443 as default.
+Before we dive further into websockets let me throw yet another problem at +you. Since proxying is a word that comes up with websockets so very often +we should probably talk about how proxies work there. HTTP in the +original RFC does not really talk much about TCP and IP addresses don't +show up in there either. However it has become clear over the last couple +of years that there is a lot of value behind knowing which IP address +connects to your server. Traditionally this is doing by looking at the IP +address that is stored in the individual IP packets. Unfortunately this +falls short the moment you start proxying HTTP through another server.
+There are two headers that have become popular to remedy that situation. +The first one is X-Forwarded-For which can contain one or multiple IPs +of servers that took part in the request. Each proxy server would add the +remote address of the request to the header. So if you have a user agent +at 192.168.0.100 connect to 192.168.0.200 which acts as a proxy for +192.168.0.201 you end up with an X-Forwarded-For of 192.168.0.100. +Bring in yet another proxy server that one would append it's own IP +address to that list.
+Now there has to be a limit to this. Usually what you do is you configure +the first HTTP server to reject any already set X-Forwarded-For +headers or users can just use a browser extension or a proxy to inject a +forged value in there. And people do that. I remember that people did +exactly that to get American streaming content available in Europe by just +injecting an American IP into that header.
+With websockets it's worse. Odds are that your frontend proxy is doing +TCP level proxying. If it would be doing HTTP level load balancing and +it would understand the Upgrade header it you could make it inject an +X-Forwarded-For header and then read that on the server. However +Amazon's ELB as mentioned only works with websockets if you set it to TCP +level proxying for instance. And with that you lose your remote address. +Maybe not a problem for you, it definitely is a problem for us. Now I +know that protocols exists to inform a peer about the IP address on the +other side of a proxy connection, but ELB does not speak it so it's a +rather uninteresting thing for me.
+The way our server works is pretty similar to how the node.js juggernaut +server used to work. You connect to a presence server that holds your +socket connection open and acts as a central hub that uses other means to +communicate with backend services. We have a pretty lightweight server +and we don't want to make it too complicated to authorize requests against +it. Namely we don't want to implement OAuth a second time for that server +when we already use it for the strictly request/response based webservice +infrastructure.
+The trick we use for that is that we let a user authorize against the +webservice infrastructure through OAuth and then call an endpoint that +gives you a ticket that is valid for a few seconds. That ticket is stored +in redis. It contains your account ID and a few important internal +attributes, but we also store the IP on it which you used at the time the +ticket was generated.
+With that ticket you go to the presence server and redeem it. The +presence server only needs to connect to redis and check if such a ticket +exist and delete it. Since we already use redis anyways in that server +it's a pretty simple undertaking. We obviously assume here that nobody +takes the ticket between IP addresses. We can't gurantee that this does +not happen but we don't care about the implications either. Someone could +obviously create the ticket through a VPN and then disable the VPN +connection when it's time to connect to the presence server. But to be +honest: I don't care. It's as far as I am concerned, no security problem.
+In theory the spec says you can also use any of the HTTP authorization +systems (basic auth, cookies etc.) but since you can't customize the +headers with the JavaScript API that are being sent with the handshake you +are basically very limited to implicit auth or auth through one of the +first requests / URL based.
+Now that we know how we can connect to the websocket server, how to +upgrade to the websocket protocol and how authorization can be handled +without losing the IP address information even if we do TCP level load +balancing. The next thing you have to know is how websocket transfer +works. As mentioned earlier websocket is not a stream based protocol like +TCP, it's message based. What's the difference? With TCP you send bytes +around and have to make sure (for the most part) that you can figure out +the end of a message. Our own protocol makes this very easy because we +send full JSON objects around which are self terminating. For naive JSON +parsers (like the one in the Python standard library) that cannot parse of +a stream we also add a newline at the end and ensure that all newlines in +JSON strings are escaped. So you can just read to the newline and then +hand that line to the JSON parser.
+Websockets makes this easier because it puts a frame around everything. +Upside: easier to handle from the JavaScript side, downside: much harder +to handle on the server side because you now need to wrap everything in +frames.
+So let's have a look first how the frames are defined. This is what the +RFC provides us with:
+++-+-+-+-+-------+-+-------------+-------------------------------+ + 0 1 2 3 + 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 ++-+-+-+-+-------+-+-------------+-------------------------------+ +|F|R|R|R| opcode|M| Payload len | Extended payload length | +|I|S|S|S| (4) |A| (7) | (16/64) | +|N|V|V|V| |S| | (if payload len==126/127) | +| |1|2|3| |K| | | ++-+-+-+-+-------+-+-------------+ - - - - - - - - - - - - - - - + +| Extended payload length continued, if payload len == 127 | ++ - - - - - - - - - - - - - - - +-------------------------------+ +| | Masking-key, if MASK set to 1 | ++-------------------------------+-------------------------------+ +| Masking-key (continued) | Payload Data | ++-------------------------------- - - - - - - - - - - - - - - - + +: Payload Data continued ... : ++ - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + +| Payload Data continued ... | ++---------------------------------------------------------------+ ++
Good news first: as of the websocket version specified by the RFC it's +only a header in front of each packet. The bad news is that it's a rather +complex header and it has the frighting word “mask” in it. Here are the +individual parts explained:
+fin (1 bit): indicates if this frame is the final frame that +makes up the message. Most of the time the message fits into a +single frame and this bit will always be set. Experiments show that +Firefox makes a second frame after 32K however.
+rsv1, rsv2, rsv3 (1 bit each): it wouldn't be a proper +protocol if it did not include reserved bits. As of right now, they +are unused.
+opcode (4 bits): the opcode. Mainly says what the frame +represents. The following values are currently in use:
+(As you can see, there are enough values unused, they are reserved for +future usage).
+mask (1 bit): indicates if the connection is masked. As it +stands right now, every message from client to server must be masked +and the spec wants to to terminate the connection if it's unmasked.
+payload_len (7 bits): the length of the payload. 7 bits is not +enough? Of course not. Websocket frames come in the following length +brackets:
+0-125 mean the payload is that long. 126 means that the following two +bytes indicate the length, 127 means the next 8 bytes indicate the +length. So it comes in ~7bit, 16bit and 64bit. I don't even have +words for this. My browser fragments off after 32K of payload +anyways, when would I ever send a package of 64bit size (oh well, the +most significant bit must be null at least)? 32bit would have been +plenty but oh well. This also means there is more than one way to +represent the length but the spec is very clear about only using the +shortest possible way to define the length of a frame.
+masking-key (32 bits): if the mask bit is set (and trust me, it +is if you write for the server side) you can read for unsigned bytes +here which are used to xor the payload with. It's used to ensure that +shitty proxies cannot be abused by attackers from the client side.
+payload: the actual data and most likely masked. The length of +this is the length of the payload_len.
+Why websockets are frame based and not stream based? Probably because +JavaScript lacks the tools to deal with stream data. Personally I take a +stream based socket over a frame based one every day because if I really +need frames, I can express them on top of a stream ones easily.
+Let's start talking about the data on the frames. As mentioned before the +data can be fragmented into multiple frames. The first frame that +transmits the data has a opcode on it that indicates what sort of data is +being transmitted. This is only really necessary because JavaScript has +pretty much nonexisting support for binary data at the time the +specification was started and now we have to live with it. 0x01 +indicates utf-8 encoded text data, 0x02 is binary data. Most people +will transmit JSON over that thing anyways in which case you probably want +to chose the text opcode. When you emit binary data it will be +represented in a browser specific Blob object.
+Ah, the best part. Payload data can be split up into multiple individual +frames. The receiving end is supposed to buffer them up until the fin +but is set. So you can transmit the string Hello World in 11 packages +of 6 (header length) + 1 byte each if that's what floats your boat. +However fragmentation is not allowed for control packages. However +the specification wants you to be able to handle interleaved control +frames. You know in case TCP packages arrive in arbitrary order :-/.
+The logic for joining frames is roughly this: receive first frame, +remember opcode, concatenate frame payload together until the fin bit +is set. Assert that the opcode for each package is zero.
+The reason for fragmentation supposedly that you can generate bits and +pieces of information on the server and send them to the client to buffer +up instead of the server and vice versa. It's annoying to handle on both +client and server but I can see how it makes it easier for a JavaScript +programmer to handle. The dirty bits of the transport protocol are +entirely hidden away. Since the server's native protocol was entirely +based on the concept of streaming JSON there is zero value in the +messages. I suppose that will be a common thing for people adding +websocket support to servers that previously spoke some other protocol on +top of TCP.
+But when dealing with the payload we not only have to concatenate frames +together, we also have to unmask them. The unmasking is pretty simple +once you have the mask key:
+uint8_t payload[payload_len];
+read_bytes(payload, payload_len);
+for (i = 0; i < payload_len; i++)
+ payload[i] ^= mask[i % 4];
+Masking is the best part because it makes debugging so incredible fun.
+Why is there masking at all? Because apparently there is enough broken +infrastructure out there that lets the upgrade header go through and then +handles the rest of the connection as a second HTTP request which it then +stuffs into the cache. I have no words for this. In any case, the +defense against that is basically a strong 32bit random number as masking +key. Or you know… use TLS and don't use shitty proxies.
+In the case of our proprietary protocol that's not even a problem because +we only allow JSON requests. As such if you would attempt to attempt to +submit a HTTP request in place of a JSON payload the server would respond +with a generic error message which is not very useful attacking purposes. +But really… use TLS and don't use shitty proxies.
+Heartbeating is useful, I can agree with that. First of all certain +things (like ELB \o/) will terminate idle connections, secondly is it not +possible for the receiving side to see if the remote side terminated. +Only at the next send would you realize that something went wrong. With +websockets you can send the ping opcode at any time to ask the other side +to pong. Pings can be sent whenever an endpoint thinks it should and a +pong is sent “as soon as is practical”. Someone also decided that +something like ping and pong are too simple so they were “improved” so +that they can carry application data and if you pong you have to send the +payload of the ping back. Sounds easy enough to implement but this +actually can make it fairly annoying to deal with because now there is +something you have to remember from a ping to a pong (and that application +data an be up to 125 bytes).
+Since you can't send pings/pongs yourself anyways from JavaScript I chose +to ignore the application data part for the moment until I find a browser +that actually sends something there. (If it does, I will be very sad and +complain loudly on Twitter)
+Lastly: closing connections. Now to go with the rest of the pattern +websocket rolls its own thing here. In theory a TCP disconnect should +work as well but it looks like that at least Firefox just reconnects on +connection drop. Instead a connection is terminated by sending the close +opcode (0x08). There the pattern seems to be to exchange close +opcodes first and then let the server shut down. The client is supposed +to give the server some time to close the connection before attempting to +do that on its own. The close can also signal why it terminated the +connection. The lazy person I am I just did not care and just close. +Important however is that you do send the opcode around, otherwise at +least Firefox will not really believe that you closed the connection.
+It should be said that the specification does not introduce a close opcode +just because to make the protocol more complex. It does have actual use +in that it makes the disconnect more reliable. Anyone that ever had to +deal with TCP disconnects will know that this can be a somewhat tricky +thing to do and behaves differently on different environments. That being +said, I don't believe that websocket implementations will get disconnects +right either which now leaves developers on both sides hope that the +implementation is correct. It's too far from TCP that you could fix the +problem yourself if you are programmer that writes the client-side +implementation.
+There are also differences between how browsers respond to websockets. +For instance if you do not provide an application level protocol with +Chrome but the server emits the Sec-WebSocket-Protocol header Chrome +will loudly complain about a protocol mismatch whereas Firefox does not +care a single bit. Safari 5.1 (the one I have installed) does not speak +the current protocol of websockets and sends different headers altogether.
+What's also interesting is that Firefox treats the upgrade as an actual +HTTP request and sends the regular headers (User agent, DNT flag, accept +headers, cache control, etc.). On the other hand Chrome will just submit +the bare minimum and special cases the cookie header. As such you won't +be able to do browser based detection just from the handshake.
+Websockets are complex, way more complex than I anticipated. I can +understand that they work that way but I definitely don't see a value in +using websockets instead of regular TCP connections if all you want is to +exchange data between different endpoints and neither is a browser. If +you do want to make them work I recommend the following things:
+Looking at the RFC it's pretty clear that websockets won't be getting any +simpler any time soon. It hints towards adding multiplexing support and +dropping the remaining HTTP upgrade parts. Also there is a builtin +extension system that will soon be used to negotiate per-frame +compression and that's definitely not the end of it. I also did not +mention that you can negotiate version numbers and application level +protocols through the websocket handshake or that it defines termination +codes when the connection closes. There is an awful lot of stuff in the +specification and even more to come which gives me the impression that we +will see some broken implementations of websockets in the future.
+Maybe a simple policy file like Unity3D and Flash are using would have +been a better idea and just let people speak TCP themselves. At least it +leaves you as application developer with more options to fix problems +yourself instead of hoping the websocket implementation is 100% correct. +But well, that's what we're stuck with now and since it already took three +or four major revisions of the specification and god knows how many +browser updates it's probably not a very wise decision to revisit the +protocol now. I do however believe that when browsers finally get CORS +running for SSE this might be a better solution for many use-cases where +people might want to use websockets. And that is definitely easier to +implement.
+