Simulating Network Lag for Testing Games

In devel­op­ing a net­worked game, some­times one needs to be able to test fea­tures run­ning over spe­cific net­work con­di­tions. How does the game hold up under high latency and/or packet loss? What about in cases of vary­ing net­work jit­ter, where the latency is ever-changing? Answer­ing these ques­tions is cru­cial to test­ing net­work code and mak­ing sure that it will per­form well against what­ever chaos the Inter­net might throw at you.

As a pro­gram­mer rel­a­tively new to net­work­ing, my first thought was to build a net­work sim­u­la­tion that could be directly inte­grated into our bud­ding engine. For exam­ple, a quick perusal of Valve’s devel­oper wiki shows built-in net_fakelag, net_fakejitter, and net_fakeloss vari­ables for con­trol­ling arti­fi­cial lag, jit­ter, and packet loss, respec­tively. Being able to con­fig­ure these para­me­ters within the engine cer­tainly has its advan­tages, but rolling your own sim­u­la­tion prob­a­bly requires a good bit of time and effort, and nobody likes rein­vent­ing wheels.

For any­one look­ing to get some­thing up and run­ning quickly, a bet­ter option is to use an exter­nal tool to alter traf­fic to and from your game with­out mod­i­fy­ing the code. I’ve found Dum­mynet to be pretty use­ful for my own pur­poses, and I’ll be dis­cussing some exam­ples of how it might be used. It comes pre­in­stalled on FreeBSD and Mac OS X, and is avail­able for down­load on other systems.

The Basics

Dum­mynet is con­trolled mostly through ipfw, a command-line util­ity for fire­wall con­fig­u­ra­tion. The ipfw tool itself is very com­plex, and I do not claim to have any real degree of mas­tery over it, but a few sim­ple com­mands and a bit of script­ing is all we really need to get a decent sim­u­la­tion. Be warned, how­ever, that mess­ing with the fire­wall can lead to acci­den­tal lack of Inter­net, so you might want to make sure you have a com­fort­able level of doc­u­men­ta­tion at hand before doing any­thing hasty.

The ipfw tool processes pack­ets accord­ing to a con­fig­urable list of num­bered rules called a rule­set. We can start by view­ing the cur­rent rule­set using the list command.

Note that your out­put may dif­fer based on your fire­wall con­fig­u­ra­tion. We can see that by default the rule­set con­tains a manda­tory rule num­bered 65535, which in this case allows pack­ets match­ing any IP pro­to­col from any source address to any des­ti­na­tion. Pack­ets are processed by com­par­ing them to each rule in ascend­ing order by num­ber and exe­cut­ing the action of the first rule for which there is a match.

Adding Latency

We can con­trol traf­fic by adding rules that match spe­cific packet data, and then send the pack­ets through a Dum­mynet pipe with added delay or packet loss. For exam­ple, we could add a 100 ms delay to all traf­fic head­ing to xkcd.com like so:

The first com­mand here con­fig­ures a Dum­mynet pipe with id 1, cre­at­ing the pipe if it did not already exist. The sec­ond com­mand cre­ates a rule num­bered 100, which matches pack­ets with the desired des­ti­na­tion address and routes them through the pipe. Note that any rule that attempts to pass pack­ets through a non-existant pipe will block traf­fic and throw an error, so do not for­get to con­fig­ure pipes before using them. Using list again allows us to view the updated ruleset.

Ping­ing xkcd.com should show a round-trip time of 100 ms plus what­ever the actual cur­rent latency is across the connection.

In most cases, we would want to sim­u­late a full duplex con­nec­tion by adding a cor­re­spond­ing rule for pack­ets com­ing from the remote host, so that the delay is applied in both direc­tions. When we are done play­ing with our new rule, we can delete it and the pipe as follows.

Sim­u­lat­ing on the Same Machine

One nice thing about using Dum­mynet is that we can actu­ally sim­u­late con­di­tions on con­nec­tions between processes run­ning on the same machine. For exam­ple, we could cre­ate a rule match­ing pack­ets trav­el­ing over the loop­back inter­face like so:

As stated in the doc­u­men­ta­tion, this has its share of caveats. Ping­ing our local­host will show the most obvi­ous prob­lem. Because the rule­set is applied to both incom­ing and out­go­ing pack­ets, using the loop­back causes the pack­ets to be processed four times, result­ing in a round-trip latency of approx­i­mately four times the delay value. A sim­ple fix is to instead use two sep­a­rate rules that will match pack­ets for each direc­tion. The fol­low­ing exam­ple adds latency to only the incom­ing packets.

The result is a round-trip latency of twice the delay, which is what we expect in a full duplex con­nec­tion. Note that this solu­tion might not over­come other issues caused by the repeat pro­cess­ing of packets.

Net­work Jitter

Adding a con­stant, con­sis­tent latency to our sim­u­lated con­nec­tion is use­ful in some cases, but most real-world con­nec­tions are cer­tainly not so well behaved. Unfor­tu­nately, Dum­mynet does not seem to pro­vide a way to cre­ate a pipe with vary­ing latency. Instead, we can do a lit­tle work to set up a pro­gram that gen­er­ates delay by a method of our choos­ing and recon­fig­ures the pipe over time. As a toy exam­ple, I threw together the fol­low­ing bash script, which changes the pipe delay on a set inter­val, each time choos­ing a ran­dom value between 25 and 50 mil­lisec­onds. The script also pings the local­host and logs the out­put to a file.

Not sur­pris­ingly, I was able to find an exist­ing research pub­li­ca­tion by Grenville Armitage and Lawrence Stew­art that inves­ti­gates this exact method and makes some inter­est­ing obser­va­tions. After run­ning tri­als under sim­i­lar con­di­tions, I find that my own results closely match theirs.

Data and Conclusions

The above graph shows the num­ber of pings recorded for each round-trip latency value for two dif­fer­ent tri­als. In the first trial, the delay was recon­fig­ured every 100 ms, while in the sec­ond it was recon­fig­ured every 500 ms. From the data, one can see that the dis­tri­b­u­tion of delay is con­sid­er­ably less uni­form for the 100 ms inter­val, which has a higher con­cen­tra­tion around the mean ping time of 75 ms.

What causes the dis­tri­b­u­tion to be skewed? Con­sider a packet being sent across a con­nec­tion from A to B and back to A. At con­stant latency, each leg of the trip incurs a 100 ms delay, for a total round-trip time of 200 ms. Now sup­pose instead that after the packet has reached B, the delay increases to 200 ms, or 400 ms round-trip. Then by the time this packet returns to A, its total travel time is 100 ms + 200 ms = 300 ms, which is the aver­age of the old and new round-trip times.

As Armitage and Stew­art point out, chang­ing the pipe delay causes any pack­ets cur­rently being processed by the pipe to undergo a sim­i­lar effect. If the pipe is recon­fig­ured at a higher fre­quency, then nat­u­rally a larger per­cent­age of the total pack­ets will be affected, skew­ing the data more severely.

Whether or not the non-uniform dis­tri­b­u­tion is a good or bad approx­i­ma­tion of an actual net­work con­nec­tion is another issue entirely. Log­i­cally one could expect that, for any real con­nec­tion, chang­ing latency at such a high fre­quency would incur the exact same effect, with mid-flight pack­ets expe­ri­enc­ing dif­fer­ences in inbound and out­bound latency. More impor­tantly, would a truly uni­form dis­tri­b­u­tion serve as a more accu­rate approx­i­ma­tion? What kinds of dis­tri­b­u­tions are actu­ally the most use­ful for test­ing? For me at least, this is a topic for fur­ther inves­ti­ga­tion, and per­haps, for another post.

Andrew Dolce

Andrew has a back­ground in com­puter graph­ics and aug­mented real­ity, and is excited about mak­ing games that look and feel awe­some. He also owns too many board games for his own good.

More PostsTwit­ter

2 thoughts on “Simulating Network Lag for Testing Games

  1. Hi,
    I’m try­ing to sim­u­late jit­ter and dropped pack­ages for test­ing, and found out that you can con­fig­ure a ipfw pipe with a CDF graph. Here is the exam­ple from the man page:
    ””“
    Exam­ple of a pro­file file:

    name bla_bla_bla
    sam­ples 100
    loss-level 0.86
    prob delay
    0 200 # min­i­mum over­head is 200ms
    0.5 200
    0.5 300
    0.8 1000
    0.9 1300
    1 1300
    #con­fig­u­ra­tion file end
    ””“
    It’s pretty cool, and I man­aged to repro­duce with it prob­lems that were not affected by delay only.
    Look at the man page (http://www.freebsd.org/cgi/man.cgi?query=ipfw&sektion=8) under sec­tion “TRAFFIC SHAPER (DUMMYNET) CONFIGURATION

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>