Cutting Down Bandwidth with JSON Alternatives

Dou­glas Crock­ford wrote RFC 4627, describ­ing the spec­i­fi­ca­tions for JSON, a “text for­mat for seri­al­iza­tion of struc­tured data.” As a language-agnostic, human-readable open for­mat that has native sup­port for encoding/decoding in browsers, JSON has become the de facto stan­dard for data seri­al­iza­tion on the web. There are draw­backs to using JSON, which became evi­dent when we started to write a net­worked game using Web­Sock­ets. (Check out our pre-alpha teaser if you didn’t get a chance to see us at PAX!)

The Setup

In our 3D game, play­ers have a posi­tion and rota­tion that are updated every sim­u­la­tion step. I will refer to the pair of posi­tion and rota­tion together as a trans­form. The server noti­fies clients of these updated trans­forms over Web­Sock­ets. Here is an exam­ple of a trans­form we might have at a given moment:

var transform = {
  origin: {
    x: 15.290663048624992,
    y: 2.0000000004989023,
    z: -24.90756910131313
  },
  rotation: {
    w: 0.12015604357058937,
    x: 0.32514392007855847,
    y: -0.8798439564294107,
    z: 0.32514392007855847
  }
};

Here, we rep­re­sent the rota­tion using a quater­nion, rather than a 3×3 matrix, because it has only four val­ues ver­sus the nine val­ues in the matrix. When using socket.io’s emit func­tion­al­ity, the data argu­ments you pro­vide are seri­al­ized using JSON.stringify and then sent out. Here is the pre­vi­ous trans­form seri­al­ized to JSON:

{"origin":{"x":15.290663048624992,"y":2.0000000004989023,"z":-24.90756910131313},"rotation":{"w":0.12015604357058937,"x":0.32514392007855847,"y":-0.8798439564294107,"z":0.32514392007855847}}

The result­ing string is 190 char­ac­ters long. So what does that mean for us?

The Prob­lem

This is a mul­ti­player game, so let’s assume we have the bare minimum—two play­ers. The result­ing mes­sage that goes out has 380 char­ac­ters, plus the 3 used by JSON for the array brack­ets and comma, giv­ing us a total of 383 char­ac­ters. Let us assume that our game runs at 30 steps per sec­ond on the server. We are trans­fer­ring 11,490 bytes every sec­ond per player. If we assume a rate of 18¢/GB of data trans­fer from our server provider, we have 0.683¢ per hour of gameplay.

Unfor­tu­nately, we didn’t set out to make a two player game. If we assume a max­i­mum of 12 play­ers per game, we see a cost of roughly 4.15¢ per player per hour, a lin­ear increase in cost per max player cap. This may prove pro­hib­i­tive, so what can we do about this?

Obser­va­tions

The data that the client receives from the server is not truly arbi­trary. We know to expect as many trans­form val­ues as there are play­ers. We also know that a trans­form value con­sists of exactly seven floating-point val­ues. We could rep­re­sent a trans­form with an array of seven num­bers that we would then process on the client to recre­ate the trans­form object. This slims our mes­sage down to 138 char­ac­ters, and our costs for 12 play­ers down to 3.02¢ per player per hour.

[15.290663048624992,2.0000000004989023,-24.90756910131313,0.12015604357058937,0.32514392007855847,-0.8798439564294107,0.32514392007855847]

We imme­di­ately see that this is much less read­able. With­out con­text, this JSON string has very lit­tle meaning—the data is no longer struc­tured. This is a trade­off that we begin to see regard­ing opti­miz­ing the net­work traffic.

We also observe that dou­ble pre­ci­sion floating-point num­bers are rep­re­sented as up to 19 char­ac­ters as a human-readable string. In binary, these are only 8 bytes. How do we use this infor­ma­tion to our advantage?

A Solu­tion

We can start to drill down into the binary rep­re­sen­ta­tion of a floating-point num­ber using typed arrays. Here is what node.js shows on a little-endian machine:

var arr = new Float64Array([15.290663048624992]);

// { '0': 15.290663048624992,
//   buffer: 
//    { '0': 0,
//      '1': 0,
//      '2': 128,
//      '3': 201,
//      '4': 209,
//      '5': 148,
//      '6': 46,
//      '7': 64,
//      byteLength: 8 },
//   BYTES_PER_ELEMENT: 8,
//   length: 1,
//   set: [Function: set],
//   slice: [Function: slice],
//   byteOffset: 0,
//   byteLength: 8,
//   subarray: [Function: subarray] }

In the under­ly­ing Array­Buffer, you can see the 1-byte chunks that com­prise the JavaScript num­ber, for a total of 8 bytes. We can then take each of these bytes, and con­vert them into a sin­gle char­ac­ter that rep­re­sents it.

// Store number in Float64Array.
var arr = new Float64Array([ 15.290663048624992 ]);

// View the underlying ArrayBuffer as unsigned bytes.
var bytes = new Uint8Array( arr.buffer );

// Serialize to a string. Pay attention to endian in production code.
var str = [].map.call( bytes, function( byte ) {
  return String.fromCharCode( byte );
}).join('');

// Or more bluntly put:
// var str = String.fromCharCode(
//   bytes[0], bytes[1], bytes[2], bytes[3],
//   bytes[4], bytes[5], bytes[6], bytes[7]
// );

str.length; // 8

Page 1 of 2 | Next page