A Guide to Networking, Matchmaking, and Host in Halo, version 2.1.1
Last Updated: 14th October 2009
by One One Seven
Originally posted on Bungie.net's Halo 3 Forum
Welcome to the second version of my networking guide. This thread is a revamped version of my previous guide from long ago, and aims to educate users on how Halo 3's networking and matchmaking work, how to improve the experience, and some of the improvements we can look forward to in Halo: Reach. This guide aims to be accessible to all users, but also to provide additional information for more technical users who are interested in further details.
There is some information on how Halo 3's Campaign networking works, which is fundamentally different to Multiplayer. Whilst quite a bit of the information here applies in both cases, this post is more focused on Multiplayer networking.
As featured by Microsoft Research!
Yes, believe it or not, my last thread on the subject was actually referenced by people at Microsoft Research in one of their papers on P2P Matchmaking. You never know who's reading.
Now at HBO!
Louis Wu has very kindly put up a local copy of this thread, and changed the formatting slightly to make it more easily navigable. So you might want to read it there instead, especially if you are only interested in specific pieces of information.
Bungie approved!
Thanks very much to Urk for mentioning this in the update!
Index
Before We Begin
A quick note on suggesting changes to networking. Networking is a complicated subject, and if you want to suggest anything sensible regarding changes, an understanding of the basics is essential. It also helps to understand Halo's model as best possible, and the strengths and weaknesses of it.
Networking Jargon
There are a lot of technical terms related to networking. You don't need to understand them all, but you may see them used throughout this thread.
Client
A computer that accesses a service on a server. Most players in Halo will be a client the majority of the time.
(Distributed) Denial of Service attack / (D)DoS attack
A DoS attack is where a network is attacked to overload it, or otherwise prevent access to it in some way. This can sometimes be achieved accidentally (such as when websites experience a massive surge in traffic), but in many cases it is a deliberate attack. Most attacks are distributed, and make use of other networks, especially hijacked computers (botnets), to perform the attack.
In Halo this is increasingly being used to remove any chosen player from the game, regardless of who is hosting. Whilst bridging players out is merely a Microsoft ToS violation, DoS attacks are illegal in most jurisdictions.
Unfortunately, it is almost impossible for regular users to effectively protect themselves against DoS attacks beyond extremely basic measures.
Internet Service Provider / ISP
A company that provides Internet access to customers. ISPs vary wildly on the quality and cost of the services they provide. Picking a good ISP is very important for a smooth online experience.
Jitter
How much your latency can fluctuate by, in milliseconds. Like the max pause, this should not be higher than a few milliseconds.
Lag
A vague term. Sometimes used as a synonym for latency, but can also refer to problems caused by packet loss, such as shots not counting, players teleporting, or other connection problems ("I lagged out").
Latency
The delay between information being sent and it arriving. The delay between cause and effect. Latency can be either 1-way or both ways (RTT), but most often refers to the RTT. This is one of the biggest causes of problems online.
Maximum Transmission Unit / MTU
The amount of information you can send per packet. Xbox Live requires a minimum MTU of 1364. Whilst larger sizes are more efficient, it's generally best not to change this setting if you're not experiencing any issues, especially if you're using a wireless connection.
Max Pause
How long your connection can be paused or frozen for. Anything higher than a few milliseconds is a sign of serious problems with a connection.
Network Address Translation / NAT
NAT is used to allow more than one computer to share the same IP address, and is done for a variety of reasons, including to increase security, and partly out of necessity (we don't have enough IP v4 addresses to give each device its own address). If your NAT is not configured properly and forwarding Xbox Live traffic to your console correctly, it creates connectivity problems between you and other players. If you need help solving this issue, please see the guides on configuring your router, or port forwarding at the bottom.
An "Open" NAT means you can connect to everyone with no problems. Conversely a "Moderate" or "Strict" NAT will mean you cannot connect to certain players (depending on their NAT setting), and means that you might not be able to join games they host, or hear them in a party.
Note: your NAT settings have no impact on any lag you experience, only whether you are able to connect to others or not.
Packet
A chunk of information. All traffic on the Internet is broken down in to smaller pieces in this way.
Packet Loss
The amount/rate of packets that fail to reach the destination. Packet loss is one of the biggest causes of problems when playing Halo online.
Ping
A test that measures the RTT, given in milliseconds. Often used as a verb ("I just pinged him") or as a synonym of latency ("I have a ping of 50ms").
Port
A data connection used to transmit data directly between computers or programs. Xbox Live uses specific ports to send data, and if these are blocked or not managed correctly, it will create problems.
Quality of Service / QoS
QoS can refer to checks on the quality of a connection, or it can actually refer to methods that ensure a quality connection, for example by prioritising certain types traffic. Most ISPs implement their own QoS system.
Round Trip Time / RTT
The time in milliseconds it takes a packet to get to a destination and for a response to be received. The latency "there and back".
Server
A server is a computer providing a service. When hosting a game of Halo, your console is acting as a server.
Online Gaming Jargon
A few terms relating to networking you might hear during the course of playing Halo or reading the forums.
Standby
Interrupting an Internet connection in order to cheat. This originally comes from players pressing the standby button on their modems, but has come to mean any way of interrupting a connection.
Bridging
Networking manipulation used to force a certain player to be host ("I bridged him host") or to selectively remove a player from the game when hosting ("I bridged him out"). Originally named because the easiest way of doing this is to create a Network Bridge and use a Firewall on it.
Host Advantage
The advantage the host of the game has over other players, due to shots always counting when they occur, and because things take effect instantly.
Halo's Networking - how does it work?
Campaign/Co-op
Campaign networking uses a completely different model to Multiplayer. It uses what is known as the synchronous lockstep model. In short, this model means every player has to agree on things before the game proceeds.
Whilst this has the advantage that everyone sees the same things in-game (essential for AI), the downside is that if a player is lagging behind, the others have to wait for them to catch up; if they didn't do this, the player will almost always get further and further behind until they go out of sync completely and get dropped.
Because of this, a single lagging player in Campaign will usually make the game lag for everyone else, and it's also why players notice lag in Co-op or Firefight that they don't notice in Multiplayer.
Multiplayer
Multiplayer uses an asynchronous distributed networking model where games are hosted on players' consoles. In any one game, one console will be the host, and all others will be clients. The host authorises essentially every action that takes place in game, including the important things like damage, and the creation and destruction of objects. Client players run a simulated version of the world that is based on the host's version of events, and which is constantly being updated. The game uses several tricks to make this less obvious.
As a client, when you take any action, you send that information to the host who then accepts or rejects it according to various criteria. In Halo 3, there is a 'latency window' of approximately 300ms. Providing your latency is less than this, and there is no problem in getting the information to the host, the shot should count. There is one exception when shooting a player close to geometry, shots may sometimes be rejected by the host if you don't maintain line-of-sight; this is an intentional design on Bungie's part.
Key Differences
Proneness to lag
Probably the biggest difference players will notice between the two is how much lag you experience. This is for several reasons:
In Campaign, the game is only as strong as the weakest connection. Because every console has to agree on things, if a player loses any information, it needs to be sent again. In Multiplayer, the game continues regardless, and any lost information a client sends is ignored as if it never happened.
Whilst in Multiplayer, you have to send information to the host, the game will nevertheless perform many actions instantly. If you jump, or fire a weapon for example (except for rockets or grenades), they occur instantly. If they don't get to the host on time, you get teleported back or you get bullets refunded. In Campaign, everything has to be sent and acknowledged by the other players which is why you may see delays before you jump, or your weapon fires.
Conflicting or incomplete accounts (or the lack of)
In Campaign, every player sees the same end result; you don't have conflicting or incomplete versions of what happened. Multiplayer on the other hand can have versions that conflict with the host, or with what other players see. Additionally, because the game prioritises information, and doesn't send any low priority data it can't afford to transmit, sometimes information is missing. An example are the player movements on the opposite side of the map which appear jerky in a saved film, or where a fusion coil may explode with no graphical effects.
Resilience
The other big difference between Campaign and Multiplayer is how resilient they are. Campaign has very little in the way of error recovery, and is particularly sensitive to packet loss. Multiplayer however can not only cope with packet loss far better, but it also is able to migrate the host to other players.
Why host games on players' consoles? Wouldn't dedicated servers be better?
With the limitations of current technology, a perfectly fair and problem-free experience over the Internet is not going to happen. Dedicated servers can reduce the severity of some of these problems, but they are not the magic bullet that many people seem to think. Bungie also now have a lot of experience with their current network model, and prefer to work on improving it, rather than ditching the system.
Advantages
So what's good about dedicated servers? The key benefits are:
- Stability - servers can offer higher stability and consistency; there's much less chance of bandwidth problems or connection fluctuations.
- Fairness - dedicated servers are a bit fairer because no individual player has host, and it reduces the difference in degrees of latency different players experience.
- Security - servers are also a bit more secure against certain networking attacks (though by no means immune).
Disadvantages
Unfortunately there are a lot of downsides to using dedicated servers as well:
- Cost - usually far more expensive than people realise, when you consider hardware, bandwidth, and support. This is also why servers for some games are shut down after a few years.
- Scaleability - this is a huge disadvantage with servers. You've got to have enough to meet demand, or the experience will be ruined for everyone. If you have too many at a given time, that money is wasted.
- Cost/Performance Ratio - the Internet continues to get faster and better. In a few years we've gone from broadband starting to take off to a position where most people can get high speeds. In Halo 2 only a handful of people could host a good quality 16 player game. These days there are many more who can do so, and plenty who can host decent 8 player games. As the Internet improves, the advantages gained by using dedicated servers are reduced, and you get a lot less value for the money and effort.
- Doesn't Solve Cheating - servers do not eliminate cheating, they simply make certain kinds of cheats harder. You might not get standbyed or bridged out of a game, but with DoS attacks becoming increasingly more common, you or even the server itself might be attacked. A few years ago in my original guide, some people were scoffing about the idea that cheaters would move over to DoS attacks, yet the situation has become so problematic that even the mass media has started to pick up on it.
- There is no way to stop every single cheat possible, and there will always be some who will be prepared to do it. Some are literally prepared to commit a crime to cheat at a video game nowadays! You really think that these people won't move over to the next way of cheating if you fix a particular one?
What are the advantages and disadvantages of the current Multiplayer model?
So why use the Halo model instead? And what are the problems with using it?
Advantages
- Cost - it costs the developer very little and allows a bigger budget to be used elsewhere.
- Scaleability - hosting the games on players' consoles is infinitely scaleable and works just as well with 1000 people as it does with 1 million.
- Experience - Bungie have many years of experience with this model now, and understand the problems far better.
Disadvantages
- Security - because players have so much control, security needs to be very good to prevent cheating.
- Fairness - whilst Halo 3 has done a lot to reduce this, the host still has a significant advantage over other players. This is particularly pronounced in smaller games (Head to Head, Double Team), fast paced games (SWAT), or at high levels of play when players are more skilled and have faster reflexes.
- Consistency - games can vary dramatically in the quality of the connection.
- International Experience - if a large portion of the playerbase comes from one area, it can create problems for those outside the region, unless managed properly.
How can I tell whether I am hosting the game?
In Halo 3, there are a few tell-tale signs when you're hosting the game in Multiplayer without using external tools:
Ragdoll physics
As a client, bodies tend to change position or teleport as you sync with the host. When hosting bodies move in a smooth arc and don't jump from position to position without going through intermediate stages
Grenade and rocket creation
Grenades and rockets are one of the few items that are created authoritatively by the server. In other words, the server has to acknowledge you've fired a rocket or thrown a grenade before it appears, compared to say, a sniper round which will appear instantly. If you are a client, you'll see a delay before they appear, usually of a few frames.
Saved films
When watching a saved film and viewing players on the other side of the map from you, they will be moving smoothly if you are watching the host's version, or moving jerkily if watching a client's version, due to the game prioritising information for performance reasons. Obviously this cannot be used during an individual game itself because if you die and watch a player on the other side of the map, the host updates that player's status more.
This is not intended to be a complete list, but these are some of the more obvious signs.
General FAQ
I have a 20 Mb/sec connection and get no lag, how can you say my connection is bad?
The ISP industry likes to advertise the nice big shiny number for maximum throughput. It's a large number, so the service has got to be good right? Well, that's exactly what they want you to think. Hell, even this number itself is a lie, because it's the theoretical maximum under perfect conditions, and ignores countless factors in the real world.
Firstly, the honest truth. That large number generally means nothing. Halo doesn't use 20 Mb/sec. If you're hosting a 16 player game, in most cases all you need is 756Kb/sec up and down. Under exceptionally rare conditions, the game can max this out by using up to 1Mb/sec, but this has only occasionally happened during the internal alpha/beta when the testers are in the same area. Increasing bandwidth past this stage will not increase the quality of the game in any way. The only thing it does do is give you a buffer if some of your bandwidth is used up by another computer, or if the speed drops.
Secondly, there's also a world of difference between a standard Internet download, and dealing with time-critical traffic. You might be able to download a movie really quickly, but that doesn't mean that it will handle your gaming traffic well.
Finally, many of the problems in Halo can be entirely transparent. As the host, things might look great to you, whilst in reality others are getting frustrated by having bullets not count.
Why could you not see any lag when you're hosting?
Several reasons:
- A lot of problems can be transparent. If I snipe you in the head, and your ISP loses that information, you don't see anything. How can you possibly tell the difference between me not shooting you, and me shooting you? You can't.
- The lag can sometimes be hard to notice; Halo is a very fast-paced game. Sometimes you actually need to go in to a saved film and slow things down to 1 frame per second to notice that your BR headshot didn't count. It's not as obvious as shooting a stationary player 4 times in the head to no effect, but this kind of stuff can still massively impact the game.
- Halo 3 does a very good job of compensating for lag. Predictive rendering, the improved client trust generally causing shots to count... the game has a variety of ways to mask problems. They are very useful don't get me wrong, but they can deceive people into thinking that there are no problems.
What improvements can we look forward to in Halo: Reach?
As before, I can't pretend to know all the details. Since Halo 3 however, I know that Bungie have been continuing to improve the netcode and have been working with Microsoft Research on a variety of ideas. Two particular improvements I am aware of at the moment are "significant" improvements to both bandwidth measurement, and bandwidth efficiency in game. So not only should the quality of hosts be determined more quickly and accurately, but by sending less data, it reduces the chances of problems.
How does the host record work?
The host record (or connection history) is simply a statistical record of how your connection has performed in the past, and is tied to an individual console ID. The record is updated at the end of each game, and measures the maximum and average throughput of the connection, as well as interruptions and other events. Whilst it is not the only factor used when determining who the host is, it is one of the most important since it generally decides who the game should consider to be the host.
Whilst the record gives a very good idea of a person's long term connection trend once established, it can take a long time to figure out how good your connection is, particularly if you don't get host. There was a case at one time where I played a hundred games or so without getting host, but once I did and the game figured it out, I got host over and over from after that point!
The host record due to this is also not very good at quickly adapting. If your connection drops, or you move to a place with a poorer connection, the game takes a while to realise this, and can constantly give you host in the meantime.
How does Halo's matchmaking work?
A full account of how matchmaking gathers players together is outside the scope of this article (see Chris' presentation at the bottom for a detailed version), but a brief summary is as follows:
Once you pick a hopper (playlist), your console contacts the Xbox Live matchmaking server and starts getting information on potential matches according to certain criteria (rank, user preferences, language etc). Once your console has potential matches, it does some very simple tests to determine connectivity and so on, but detailed checks on connection quality are not performed due to the time this would take.
Once all the players have joined the session, any settings such as Prefer Good Connection are then discarded, the host is selected, the gametype is chosen, and the countdown begins.
How can I improve my connection so I have the best experience possible?
The first thing to do is to make sure your own network is in good order. You could have the fastest Internet connection in the world, but if you have a badly configured network then you'll still get problems.
So, for your own network:
- Make sure your NAT is open and properly configured, and ports are forwarded correctly.
- Most routers these days allow you to prioritise certain types of traffic. So if you share an Internet connection, either giving your console a specific IP address and prioritising that, or by prioritising Xbox Live traffic specifically (port 3074 TCP & UDP and port 88 UDP), that does help a lot.
- Obviously also make sure any wired network cables are in good condition. If you use Wireless, try to ensure a good connection, either by using extra receivers, or by keeping the network away from interference (metal objects, microwaves, and so on).
That's really all most people can do for their own network. So what do you need in an ISP, if you want to play games online?
Most ISPs will do a decent job of handling gaming traffic, I should say that now. The problem however, is that gaming traffic is time-sensitive, and requires different treatment from general Internet traffic.
- Make sure your ISP prioritises gaming traffic. If it doesn't, you'll be competing against lots of P2P traffic, particularly during peak times, and even if you don't notice lag directly, there'll likely be quite a bit of packet loss
- Go for as low a contention ratio as realistically possible. This costs more, but it significantly reduces the chances of your connection slowing down. In practice, no one shares their connection with 50 other people because people don't use theirs all at the same time, but the more you could have to share it, the more likely this is to happen.
- Make sure your ISP has low levels of packet loss, jitter, and max pause. The fastest connection in the world will still be horrible for gaming if any of these are high
- If you want to host games in particular, having 1Mb upload is ideal, as that's enough to deal with 16 player games. 756Kb is fine, but doesn't give you much of a buffer if someone else uses the connection or the speed drops a bit.
Who should I party up with if I want to experience good games?
Firefight/Co-op - You want to party up with people who have a fast, reliable, consistent connection, and live as close together as possible. Make sure you have low levels of packet loss to avoid the game constantly pausing or seeming 'jerky'! Remember, a single lagging connection will have an impact on everyone else!
Multiplayer - it doesn't matter quite so much here, but if you have players all around the world in your party, there'll always be at least one who has a bad connection; there's no way to have a good host for everyone.
How does Prefer Good Connection (PGC) work?
Initially when the game shipped, PGC was soft filter that tried to limit players to 200ms latency at most, though this was not foolproof. This was later reduced to a much smaller figure (I believe it is around 100ms now, but haven't yet gotten official confirmation of that). Whilst this is also not foolproof, it does also appear to be a lot stricter than before, but if a player outside this range somehow 'gets through', then they can still be chosen as host, both before the game starts, or if the host migrates.
What about manual host selection in custom games?
At the moment, this is a low priority since in a lot of cases, the game has more information to go by than the average user, and the team would rather work on improving the ability to figure out who has a good connection. However, they do also recognise that a lack of control is frustrating, so it might appear at some point.
Why is Multiplayer fine but I get lag in...?
Co-op & Firefight - almost certainly due to packet loss, but really bad latency is the other big cause.
Just Firefight - the mostly likely cause is that the extra data being sent in Firefight (due to the more action-packed nature of it) is too much for at least one person's connection to handle, so the game needs to constantly be paused to allow everyone to sync up.
How much bandwidth does Halo 3 need?
You need for both upload and download, for a 16 player game (15 clients + 1 host) that is:
'Perfect' - 1Mb/sec total, ~67kb/sec or ~8KB/sec per player
Conditions are close to that of a LAN with the game providing updates on almost every event. Players move smoothly, shots constantly hit, and there is no weirdness whatsoever. This only occurs under very rare circumstances when players have very low latency to each other.
'Excellent' - 756Kb/sec total, ~50kb/sec or ~6KB/sec per player
Players should be moving smoothly, and the game suffers from no noticeable problems. Some information is prioritised, but the game provides updates on all relevant information. This is the best you will get for most players.
'Good' - 512 Kb/sec total, ~34kb/sec or ~4KB/sec per player
Players will generally be moving smoothly, but you might notice some lag or shots that don't count. Low priority events such as fusion coil explosions may not appear for a few seconds or don't appear at all.
These should only be taken as rough values and do not factor in the additional bandwidth used for voice. This does also not factor any other problems with the connection like packet loss. The quality can also vary per player due to latency, packet loss, and other factors.
What exactly do the bars on the scoreboard indicate?
The bars give an average indication of the connection quality between that particular player and the host. I'm not entirely sure on exactly how this is determined, but presumably it is based on bandwidth and latency. However, it is worth pointing out this is not an exact indicator; packet loss can still occur despite a seemingly good connection. Treat it as a rough approximation of the general trend.
What ports does Xbox Live use and what for?
3074 TCP & UDP - General console and gaming traffic
88 UDP - Xbox Live login and authorisation
Common Misconceptions
Bridging/Standby is illegal
This simply isn't true. It is against the MS ToS most definitely, but it is not at all illegal, since you are only modifying your own network, and you are not attacking anyone else's.
Monitoring your connection is illegal/against the ToS
I've also seen a few who seem to believe this. Again, providing you are only monitoring your own network, and are not interfering with it in any way, you are not breaking any laws, or any part of the ToS. To use the road analogy here, looking out the window at the traffic going by is not doing anything wrong!
This also means you can legitimately obtain users' IP addresses as well, providing you don't act on that information.
The party leader is the host
The host is not the same as the party leader. The party leader simply has control over aspects of the game, and has the authority to tell the host to end the game. If your party leader is also hosting the game, generally it's because they have the best connection (as determined by Halo).
The host record works perfectly
Some like to insist that if you don't get host, this is automatically because your connection is bad. Whilst this may very well be true, it is not always the case, as the record takes time to build up a profile of your connection.
Installing Halo 3: ODST to the HDD reduces lag
Installing the first disk of ODST to the hard drive only reduces the loading times I'm afraid.
Players can reset their host record by clearing their cache
Bad news for these players: no you can't! The connection history is stored on the Xbox Live servers, not on your console.
Using another account avoids damaging your host record on your main account
Unfortunately not the case. The host record is tied to the console ID, to help reduce the amount of people evading the banhammer by making multiple accounts. The only options here are to play on another console, refrain from playing, or accept damage to your host record.
The host is determined by who lives closer to Bungie's/Microsoft's servers
False, but understandable. The host is determined by connectivity to other players, the connection history, and a degree of randomness. That said, Americans are more likely to get host than those playing internationally for several reasons:
- A lot of Halo players are from the US, so any game you play is likely to have several American players in it.
- Because US players tend to have decent connections to each other, and there are more of them, they have a much greater chance of building up a good host record. This then has the side effect of causing them to get host even more in international games.
Acknowledgements
I'd like to thank Chris Butcher for continuing to maintain a dialogue on networking, and for providing very useful information on the workings of Halo's netcode and matchmaking, and the improvements being made for future projects.
Usage Limitations
I've had a few requests to use parts of my previous thread, and I've seen my old one copied around the Internet a bit. To pre-empt any requests this time:
Please feel free to quote from or host on other sites, or feature in works of your own providing
- it is for non-profit use
- that you don't pass it off as your own work
- you provide a link to the original thread on Bungie.net
Thanks!
About me
As a UK player, I didn't have the best of times playing Halo 2. Back then, I didn't really understand anything about networking but I was curious about the problems that occurred and the reasoning for them. As a result, I've learnt quite a lot on the subject over the past few years. I've also spent quite a bit of time learning about the various Halo networking models specifically, both by reading Bungie publications, as well as discussing the subject quite a bit with Chris via email, which actually scored me a direct invite from Bungie in to the Friends and Family part of the Halo 3 public beta.
During the course of Halo 3 I've continued to give feedback on how changes to matchmaking have been working out, as well as occasionally making my own suggestions for future improvements.
Further Reading
Papers, Publications and Articles
Guides and Tools
Port Forwarding Guide - Very useful if you need help configuring your router, as it provides instructions for most types.
VoIP Test - This allows you to get a rough idea of how your connection handles time-critical traffic. It isn't the same as testing XBL traffic, but gives you a better idea than a regular speed test.
Regular Speed Test - Good for giving a basic idea of how fast your connection should be under ideal conditions. Please note that a lot of ISPs like to prioritise such sites to make their services look better, so don't take as an exact indicator, just want your connection can do.
Final Thoughts
People like to bash Halo 3 when things don't work. It's understandable; we pay for the game, and to play it online, and we expect it to work. But we shouldn't be so quick to forget how much worse things were in Halo 2, and how much great progress has been made towards solving problems with networking, and with cheating. People also seem quick to gloss over the problems in other similar games.
Plenty like to insist these problems are simple to fix, and that Bungie are just lazy, or incompetent, or they don't care. Can Bungie make mistakes, or poor decisions? Of course they can, the people who work there are human just like you and me. Are they lazy? Do they not care? When you talk to people there, I don't honestly think anyone rational can accuse them of that.
How many other companies would have one of their head engineers discuss the problems one-on-one with a mere fan? How many other companies interact with the community to anywhere near the level Bungie do? Sure, there are companies out there that do this, but they are very rare. You can tell they love what they do.
Thanks very much for reading and for your feedback and support. It's always great to hear that people find this stuff useful.
