Why Instagram is worth big bucks

To ensure a great online game-play experience, your job as a game designer is to minimize the perceived latency between initiating an action and having that action actually take effect in-game, even though there may be 300-400 hundred milliseconds of actual latency between the game client and game server. Actually, I might be dating myself a bit — early Real-Time Strategy (RTS) games had to expect such huge latency because most players were using modems. Internet latency is much lower now but user expectations are higher, so every trick we can use to hide that latency is still useful.

First, it’s important to recognize why latency is so high in RTS games. Games like Warcraft, Starcraft, and Age of Empires are built using a “lockstep model”. All of the computers running a game simulation perform player actions “at the same time”. Well, not at exactly the same time, but during the same game “turn”.

It works like this: when you tell your Ogre Mage to attack my Peasant, your computer sends a network message to my computer. And then your computer ignores the attack action for a while. What?!? Yeah, your computer can’t act on that action yet because, if it did, your computer and my computer would be “out of sync”. Your computer has to wait until it receives my message for that game turn, and then both of our messages can be processed at once. Here’s a diagram that shows what’s going on:

Delaying execution of actions allows time for packets to arrive from remote players

If you’re interested in implementation details, Mark Terrano wrote 1500 Archers on a 28.8: Network Programming in Age of Empires and Beyond. The Warcraft and Starcraft RTS games used the basic same network model a couple of years prior, but Mark gets big props for writing the first article. And here I am just starting to blog 15+ years later!

So anyway, what this means is that, even though your game engine knows your Ogre Mage is going to attack, it can’t do anything to modify the state of the game world or the simulation will be de-synchronized.

Which means that you’re not getting any feedback that you’re about to kick my ass. No fun!

So here’s the solution: provide instantaneous visual and auditory feedback, without making changes to the game simulation.

Remember all those funny voices in Warcraft: “Yes, milord”; “At once, sire”, “Daboo”, “Zug Zug”? While quite entertaining in and of themselves, their fundamental purpose is to immediately reassure players by confirming that their orders are being obeyed.

In addition to playing a sound effect ‘Craft games also played an animation effect on the map-screen which, while it had no effect on the game-play, showed the target of the action.

So there you go, another tool to add to your game-designer toolbox. Improving online game responsiveness is not just about writing better networking code to minimize Internet latency, it’s about finding ways to minimize perceived latency by providing instantaneous feedback to user requests.

Unfortunately, I can’t! At least I can’t recommend one from personal experience because the projects I’ve worked on — Warcraft, Diablo, Starcraft, battle.net and Guild Wars – were all built on proprietary, “from the ground up” network code I developed or co-developed.

But I’ve been meaning to do some research into 3rd-party network libraries so I don’t have to write something from scratch next time; it just takes too long. The core networking libraries I wrote for Guild Wars comprise about 50K lines of code and took several years to fully stabilize because of esoteric issues. To give you a brief idea of some of the scary bits:

• Setting TCP window size: enabling high throughput over long-distance connections without exposing servers to distributed denial-of-service (DDoS) attacks from clients connected to the same server.
• Message buffering: don’t drop a connection just because the other side stalled, but be careful not to allow DDoS attacks.
• Socket shutdown issues: close sockets “nicely” to avoid disconnect delay but don’t expose servers to DDoS attacks.
• Rate-limiting connections & throughput: ensure that servers can’t be abused by DDoS attacks – are you noticing a pattern here?!?

It is difficult to select a network library because there are so many parameters. Are you looking for a reliable message-pusher? Should it use TCP, UDP or both? Do you want object synchronization? Should object synchronization include client-side prediction? Do you want only client-to-server, or do you also need to handle server-to-server communications too? Cryptography? Network firewall traversal (aka UDP hole-punching)?

I’m going to start by eliminating libraries that don’t have a cross-platform story; let’s choose something that works for PC, console and mobile. While game projects don’t need to run on all of them, it is desirable for the programming team to be able to switch to different platforms without having to relearn everything.

Further, using a high-level API is a good choice for most game teams. Sending packets isn’t hard; sending reliable, compressed, scalable, encrypted, low-latency object-synchronization messages is hard. Let someone else do the work!

These high-level, cross-platform network libraries all look like they’re appropriate for game development:

• Raknet
• Ice
• PocoProject

Here are some other high-level libraries I looked at but don’t think are appropriate:

• Zoidcom – no pricing available on the site, which is a hassle up front.
• Torque Network Library (TNL) – no longer supported.
• TNL2 – also appears dead; GPL license prevents use in commercial game clients.
• ENet – lacking high-level documentation.
• ACE – looks dauntingly complex.

And here are some lower-level or more purpose-specific libraries:

• ZeroMQ – great for server-to-server but I wouldn’t use it for client->server: not robust against hacking.
• Boost ASIO – solid and complete, but (flame) using boost is a religious issue for some because it is especially hard-to-read C++ code, and can dramatically increase build-time (/flame).
• LibEvent – doesn’t support Windows well (only uses select, not IOCP).
• LibEv – doesn’t support Windows
• Google Protocol Buffers
• Apache Thrift

So that’s a start on the choosing anyway; I hope that helps reduce the scope of research required to get you started on your next game. Now can I join your beta?

I find that many slide decks tend are quite worthless because, while the slides outline up the presentation, what the speaker has to say is far more useful. Consequently I’ve included extensive commentary in the “speaker notes” section of each page of the presentation, which I hope you’ll find useful.

Writing reliable online game services [PDF]

Incidentally, the folks who run GDC also recorded a video, which you can download (with paid membership) from GDC Vault. You may note that there’s an odd jump in the video; the first half of the video was lost due to some sort of electrical issue (record button not pressed perhaps?) so they re-recorded it with me in front of an empty hall; consequently you’ll miss the scintillating display of me interacting with a live audience for the first bit.

Happy coding!

Of course I relied upon Google to find a solution, but boy did I have a tough time finding code to do the job. Many folks run external commands like “netsh” to configure firewall exceptions, but this sucks because the netsh.exe command has different syntax in different versions of Windows. Yuck!

Here’s my solution in visual basic (cscript.exe), which has the virtue of being installed on every modern (XP+) Windows computer in the world.

'configure-firewall.vbs
'Sets Windows firewall permissions for a specific application
'Grants full inbound/outbound access for TCP/UDP
'by Patrick Wyatt 12/22/2011
'MIT License - do with as you will; no warranty

option explicit

'**************************************
const NET_FW_IP_PROTOCOL_TCP  = 6
const NET_FW_IP_PROTOCOL_UDP  = 17

const NET_FW_RULE_DIR_IN  = 1
const NET_FW_RULE_DIR_OUT = 2

'**************************************
sub UpdateFirewallRule (addRules, policy, groupName, ruleName, exePath, netProtocol, direction)
  'Prettify the rule name
  dim name
  name = ruleName
  name = name & " - Allow"
  if netProtocol = NET_FW_IP_PROTOCOL_TCP then
    name = name & " TCP"
  elseif netProtocol = NET_FW_IP_PROTOCOL_UDP then
    name = name & " UDP"
  end if
  if direction = NET_FW_RULE_DIR_IN then
    name = name & " IN"
  elseif direction = NET_FW_RULE_DIR_OUT then
    name = name & " OUT"
  end if

  'Set rule parameters
  dim rule
  set rule              = CreateObject("HNetCfg.FwRule")
  rule.Enabled          = true
  rule.Grouping         = groupName
  rule.Name             = name
  rule.ApplicationName  = exePath
  rule.Protocol         = netProtocol
  rule.Direction        = direction

  'Remove old rule
  if addRules < 0 then
    Wscript.echo "  Removing rule '" & name
  end if

  'Always remove old rule to prevent duplicates
  policy.Rules.Remove rule.name

  if Err.Number <> 0 then
    Wscript.Echo "  Removing rule '" & name & "'failed with error '" & Err.Description & "' (" & Err.Number & ")"
    Wscript.Quit 1
  end if

  'Add new rule
  if addRules > 0 then
    Wscript.echo "  Adding rule '" & name
    policy.Rules.Add rule
    if Err.Number <> 0 then
      Wscript.Echo "  Adding rule '" & name & "'failed with error '" & Err.Description & "' (" & Err.Number & ")"
      Wscript.Echo "  did you remember to run this script as administrator?"
      Wscript.Quit 1
    end if
  end if

end sub

'**************************************
sub DisplayRules (policy, groupName)
  dim RulesObject
  set RulesObject = policy.Rules

  Wscript.echo "Displaying firewall rules"

  dim Rule
  for each Rule in RulesObject
    if Rule.Grouping = groupName then
      Wscript.Echo "  Rule: " & Rule.Name
    end if
  next

  Wscript.echo ""

end sub

'**************************************
'Main program
  dim command, groupName, ruleName, exePath
  command   = Wscript.Arguments.Named("Command")
  groupName = Wscript.Arguments.Named("GroupName")
  ruleName  = Wscript.Arguments.Named("RuleName")
  exePath   = Wscript.Arguments.Named("ExePath")

  Wscript.echo "Firewall rule update arguments:"
  Wscript.echo "  Command:  " & command
  Wscript.echo "  Grouping: " & groupName
  Wscript.echo "  RuleName: " & ruleName
  Wscript.echo "  ExePath:  " & exePath
  Wscript.echo ""

  'Parse command line
  dim addRules
  if command = "install" then
    Wscript.echo "Installing firewall rules"
    addRules = 1
  elseif command = "remove" then
    Wscript.echo "Removing firewall rules"
    addRules = -1
  elseif command = "display" then
    addRules = 0
  else
    Wscript.echo "Unknown command: " & command
    Wscript.Quit 1
  end if

  dim policy
  set policy = CreateObject("HNetCfg.FwPolicy2")

  if addRules <> 0 then
    UpdateFirewallRule addRules, policy, groupName, ruleName, exePath, NET_FW_IP_PROTOCOL_TCP, NET_FW_RULE_DIR_IN
    UpdateFirewallRule addRules, policy, groupName, ruleName, exePath, NET_FW_IP_PROTOCOL_TCP, NET_FW_RULE_DIR_OUT
    UpdateFirewallRule addRules, policy, groupName, ruleName, exePath, NET_FW_IP_PROTOCOL_UDP, NET_FW_RULE_DIR_IN
    UpdateFirewallRule addRules, policy, groupName, ruleName, exePath, NET_FW_IP_PROTOCOL_UDP, NET_FW_RULE_DIR_OUT
    Wscript.echo ""
  end if

  DisplayRules policy, groupName

  Wscript.echo "Success"
  Wscript.Quit 0

And here’s a batch file to call this script. Now in real life you wouldn’t use a batch file; you would call the configure-firewall.vbs script directly from your installer script. That’s one of the reasons that the command-line validation of my script doesn’t do much error checking — it’s not for end-users, it’s for you programmer-types.

::sample-firewall-configuration.bat
@echo off
SETLOCAL EnableExtensions

if "%1" == "" (
  echo Usage:
  echo   %0 display
  echo   %0 install
  echo   %0 remove
  exit /B 1
)

%SystemRoot%\System32\cscript.exe //nologo configure-firewall.vbs /GroupName:"!GroupNameHere" /RuleName:"!RuleNameHere" /ExePath:"C:\Windows\notepad.exe" /Command:%1

Use the code as you see fit!

Here’s an example class API defined in a header file:

// FoozleDll.h
class CFoozle { ... };
CFoozle * MakeFoozle (CBarzle * bar);

And the associated CPP file:

// FoozleDll.cpp
CFoozle * MakeFoozle (CBarzle * bar) {
    return new CFoozle(bar);
}

In the application code we’re supposed to create and later delete this object, so here goes:

// Main.cpp
void ProcessData (CBarzle * bar, iostream * outfile) {
    CFoozle * foo = MakeFoozle(bar);
    foo->WriteResults(outfile);
    delete foo;
}

That all looks pretty straightforward, but what isn’t immediately obvious is that the call to new() is made in the context of the Foozle DLL file, and the delete() call is done in the context of the application, which might be using an entirely different memory manager.

Since the DLL was compiled separately, it might link to the release version of the C runtime library, where new() calls malloc() behind the scenes. But the call to delete() occurs in the application code, which could link to the Debug library, which calls _free_dbg() instead of free(). When the application releases the memory via the call to delete it is calling the wrong memory manager, which leads to problems like the inability of the application to coalesce adjacent free memory blocks (memory leakage) or random memory corruption.

The correct solution is that a module which allocates an object should also free the object:

// FoozleDll.h
class CFoozle {
public:
    ...

    // This function must not be implemented in the header or the
    // linker will build the code to call the application delete()
    // function instead of the library delete() function.
    void DeleteThis ();

private:
    ~CFoozle (); // private so it can only be called by DeleteThis();
};

And the implementation:

// FoozleDll.cpp
void CFoozle::DeleteThis () {
    delete this;  // called from the right "delete context"
}

By calling delete() from within the same compilation unit, we can ensure that the compiler will generate a call to the correct delete function.

Incidentally, this same type of problem can occur in C, where a DLL function returns (for example), the result of strdup(), and the application is expected to call free() on the resulting string.

C++: so powerful, so easy to break things.

Y’see, hackers find ways to manipulate games by simulating the behavior of human players, but in ways that exploit bugs and misfeatures; for example, by performing tasks faster than humans could ever hope to do. One example is a “speed hack”, where a player is able to perform actions like swinging a sword more rapidly than should be possible according to the rules of the game. If you’ve ever seen a monster go from full health to instant death in a flurry of blows you’ve likely seen a speed-hack.

Now honestly a speed hack doesn’t sound like such a bad thing; before I was a game programmer I speed-hacked games myself, and thought it was more fun than playing! And I also had to “slow-hack” several older games (including one I developed — Warcraft) to make them playable on fast computers, because the programmers (me included) had forgotten that computers keep getting faster every year.

But then there are professional cyber-criminals, who steal accounts, use speed-hacks to generate lots of game-gold, and sell it for real money to other players, something known as Real Money Trading, or RMT for short. And with such hacks it becomes impossible for honest players to keep up. So what’s to be done?

The code I’ve shared below implements rate-limiting useful for preventing many forms of speed hacking. I’ve used similar code for login rate-limiting, to prevent hackers from brute-force cracking account passwords. It can be used to moderate online chat so that one person can’t “flood” a channel with messages. It’s great for transaction rate-limiting to ensure that no one person can overwhelm a server with requests. And in fact I’ve used it to successfully mitigate distributed denial of service (DDOS) attacks, which I’ll detail in a future article.

Here’s how to use the rate-limiter:

// Using these values a player can attempt to login once
// every 30 seconds, but with as many as ten login attempts
// in a burst. While this sounds like a lot many players
// forget their passwords and need a number of attempts to
// remember it, which I discovered by analyzing log files.
// They should try LastPass, which is an awesome solution
// to this problem.
const unsigned LOGIN_COST_MS      = 30 * 1000;
const unsigned MAX_LOGIN_COST_MS  = 10 * LOGIN_COST_MS;

ErrorCode CPlayer::PlayerLogin () {
    if (!m_rateLimiter.AddTime(LOGIN_COST_MS, MAX_LOGIN_COST_MS))
        return ERROR_LOGIN_RATE_LIMIT;
    ... login code here
}

In any event, here’s the code, which is surprisingly trivial considering how powerful it is. The algorithm is a modified form of the “Leaky Bucket Algorithm as a Meter“, but uses the passage of time instead of incrementing a counter to perform its magic.

RateLimiter.h definitions:

class CRateLimiter {
public:
    CRateLimiter ();
    bool AddTime (unsigned costMs, unsigned maxCostMs);

private:
    unsigned m_timeMs;
};

RateLimiter.cpp implementation:

CRateLimiter::CRateLimiter ()
:   m_timeMs(PlatformTimeMs())
{}

bool CRateLimiter::AddTime (unsigned costMs, unsigned maxCostMs) {
    ASSERT(costMs < maxCostMs);

    // Reset rate-limiter time value if it has expired
    // - handles integer overflow safely
    unsigned currTimeMs = PlatformTimeMs();
    if ((int) (currTimeMs - m_timeMs) > 0)
        m_timeMs = currTimeMs;

    // Has the user accrued too much time-cost?
    // - handles integer overflow safely
    unsigned newTimeMs = m_timeMs + costMs;
    if ((int) (newTimeMs - currTimeMs) >= (int) maxCostMs)
        return false;

    m_timeMs = newTimeMs;
    return true;
}

And somewhere you’ll have to define the PlatformTimeMs function:

unsigned PlatformTimeMs () {
#if defined(_WINDOWS_)
    return GetTickCount();
#else
    #error Your implementation here
   // something like clock_gettime(CLOCK_MONOTONIC, ...) for Unix/Linux
#endif
}

I hope you’ll find this code useful for your project, and would enjoy hearing about the novel purposes you find for it.

-- Delete the stored procedure if it already exists
if exists (
  select * from sys.objects where object_id = OBJECT_ID(N'p_MyProc')
  and type = N'P'
) then
  drop procedure p_MyProc
end

-- Now create it again
create procedure p_MyProc as begin
  -- awesome code here
end

-- And set permissions
grant execute on p_MyProc to SomeRole

There’s nothing wrong with this code. Just kick all your users off the server, switch to single user mode, execute the code above and it will work fine.

Oh, what’s that? You’re running a service that needs to be highly available, and you can’t take a maintenance period every time you want to change code?

Using the drop+create method there is obviously a small window of time when the stored procedure does not exist. And as we know from examples like the Seattle monorail crash (see notes at the end of this article), any system that is designed with a built-in flaw will eventually fail because of that flaw.

Here is a better solution:

-- if the stored procedure does not exist then create a placeholder
if not exists (
  select * from sys.objects where object_id = OBJECT_ID(N' p_MyProc')
  and type = N'P'
) then
  create procedure p_MyProc as RAISERROR ('MyProc not defined', 16, 1);
  grant execute on p_MyProc to SomeRole
end

-- update stored proc
alter procedure p_MyProc as begin
  -- awesome code here
end

This ensures that the stored procedure always exists, and (because SQL is transactional) it is possible to have one caller finishing a call to the old version of the stored procedure while the new version is added and called.

I’ve used this trick successfully for the development of Guild Wars using SQL Server 2000, and later SQL Server 2005, and have not encountered any problems using this technique, even on servers running sustained load of 3000+ transactions per second. In fact the standard operating procedure for our database updates was to update all stored procedures (several hundred of them) every time we performed a deployment.

I assume that similar tricks will work for MySQL and PostgreSQL, and would love to hear from users of those platforms about their experiences.

Update (11/5/2011):

It turns out that MySQL doesn’t support atomic updates of stored procedures; apparently this is a long-standing bug, first filed in 2005 and still not fixed (http://bugs.mysql.com/bug.php?id=9588). And PostgreSQL and Oracle both do properly support this feature with a different SQL syntax: “CREATE OR REPLACE PROCEDURE”.

Notes:

In case you’re wondering about the reference to the Seattle monorail crash, which is a great example of design failure, here’s a bit more information. The original Seattle monorail was built for 1962 World’s Fair, and had two sets of tracks so that two trains could operate side-by-side. The tracks were shortened in the 1988 to end at the newly constructed Westlake Center mall. The new design allowed for an automatic passenger-loading ramp to extend to the trains, but led to the tracks being too close together for two trains to be there at the same time. As you might imagine, this design eventually failed, though it took seventeen years (http://seattletimes.nwsource.com/html/localnews/2002650818_monorail28m.html). Of course, when you’re running several thousand SQL transactions per second, such failures are all the more likely.