This forum is in permanent archive mode. Our new active community can be found here.

GeekNights Tuesday - Matchmaking in Games

Tonight on GeekNights, we discuss the high level difficulties of matchmaking in multiplayer games, and why it's so important. We've covered this in panels before, but never done a show on it, and now we're suffering mightily in Rocket League. In other news, Natural Selection 2 is queuing some big changes, Rym extols the virtues of Undertale, and Civ V continues to be the most dangerous "one more turn" game in the world as we play against humans

Download MP3
Source Link

Comments

  • edited November 2015
    I'm totally with you guys with regards to multiplayer Civ V. I remember a year or two ago you guys were organizing multiplayer games and I distinctly recall I was interested in playing with you guys, but I didn't have the balls to do it because I knew my Civ V skills were lacking.

    Since then I have gotten significantly better, and the main reason is that I actually bothered to play a few multiplayer games. To use a poker analogy, I think the key difference is that I shifted my play style from tight-passive to tight-aggressive.

    So yeah, I'm totally up for future games with you peoples.
    Post edited by lackofcheese on
  • edited November 2015
    You guys failed to mention one key way you can deal with skill disparities, which makes matchmaking even more versatile and powerful in chess (and go, and even golf) - handicapping.
    Post edited by lackofcheese on
  • In my experience, casual and semi-serious gamers are offended by handicaps. They get mad if you even broach the subject.
  • I only end up playing like 1 game of Rocket League and then I quit it every time because I'm sitting their waiting for ranked games for 10+ minutes then get matched in a game like Rym, in a game on another continent. All the match making handling for this game killed it.

    Rym's Glory to Rome board game analogy was really accurate, he and Scott taught me and another noob at PAX Aus how to play but neither of the 2 noobs were playing it was literally just Scott vs Rym. It wasn't a great experience and I didn't end up learning any intricacy to the game.

    The Starcraft 2 formula is kind of broken, I am still ranked as a Diamond player according to Blizzard but I haven't played the game for 4 years. There has to be time decay at high ranked play or the player base needs a soft rank reset every year.

    The only downside with ranked Counterstrike is that you can go into a ranked game with a group of people, less than 5. So if you have a 3 assholes who have had their prior Steam accounts banned, they let you win the game for them then kick you before the end of the game. Otherwise it's fine. For Counterstrike the max number of players to go into a 5v5 should be 1 or 2 and if you duo you should be put in a higher difficulty bracket.

    For players who want to grind, give them some superficial separate award that is clearly different from their skill ranking.

    Everyone hates MOBAs but Riot Games does all these things for their ranked player base. If you are particularly talented, the more ranked games you win in a row the faster your rank increases and then slows down when you start losing games and tries to keep you at a 50% win rate. I'm pretty sure Counterstrike has employed most of these things.
  • Every time I start feeling good about my life or my abilities in any field, I remember I've always been a year older than Roger Federer.
  • Every time I start feeling good about my life or my abilities in any field, I remember I've always been a year older than Roger Federer.

    I'm less than a year younger than he is.
  • For me, it's when I remember Neil Cicierega is the same age as me. So while I was laughing at Hyakugojyuuichi!! in ninth grade, he was making it.
  • pence said:

    For me, it's when I remember Neil Cicierega is the same age as me. So while I was laughing at Hyakugojyuuichi!! in ninth grade, he was making it.

    His sister is also ridiculously talented. The force is strong in that family. Rubin family has a lot of catching up to do.
  • I was interested when you said that you wanted to forfeit games and got annoyed when people wouldn't forfeit. I've literally never forfeited a game before. Reason one is that I've had games turn around in the last minutes even when the opposing team was like 4-5 goals ahead. The second is that if they are good enough to be stomping the yard so much then I want to learn what makes that happen and make myself better against it. I view the games against real players as valuable even if they are being completely dominant. There has only been a couple times when I've considered quitting when someone on my team is just so incompetent that it's frustrating. That's usually not the case though.

    In general though that is frustrating from a matchmaking perspective. The game doesn't do that great of a job of rewarding teamwork which is something that REALLY needs to happen. But there is a fine line between rewarding the player who is top on a losing team and at least acknowledging the players who were not. I think there is like +/- 1 point changes in point loss for top player on a losing team but that's not much. And they already have issues of people wanting to achieve that spot for their team. So you get a bunch of yahoos who all want to me MVP on their team but do dumb shit like crowd the ball into the corner and can't get it out or people who can't practice positioning on the pitch and are never in a position to save a runaway ball.

    I think one of the biggest issues with the game honestly is people just don't understand soccer. I remember hearing somewhere the players from one of the top teams talking about playing with randos and them saying that they're often just sitting in the back because they can't trust their teammates. They also talked about how new players are usually really bad with positioning and have a tendency to clump together. It may be partially a matchmaking issue trying to separate out serious people and non serious people. But at the same time there are issues with the playerbase itself who just don't understand how the inspiring game works. Imagine if everyone on a soccer team chased the ball into the corner in the same way. That would be disastrous and idiotic. The same thing happens in Rocket League.
  • The reason for forfeiting is because the ranking system is based only on wins and losses. If you are going to lose it is in your best interest to forfeit so you can save time and play another game with an actual chance of winning. If they fixed the ranking system to be based on individual advanced metrics, there would be no reason to ever forfeit, no matter the score.
  • MATATAT said:

    I think one of the biggest issues with the game honestly is people just don't understand soccer Rocket League.

    Good soccer players and Rocket League players will understand how presence on the pitch works and what can happen if the ball goes behind them. If they are bad the matchmaking system should demote them.
    I understand what you're trying to say but the matchmaking should account for those people who understand the basic concepts of what occurs on the pitch.
  • I'm a little behind on episodes, so I don't know if it was mentioned, but Titanfall was pretty much killed by bad Matchmaking.
  • Churba said:

    I'm a little behind on episodes, so I don't know if it was mentioned, but Titanfall was pretty much killed by bad Matchmaking.

    It wasn't mentioned but it was another of those games where not enough people played.
    I didn't experience a matchmaking system in that game.
    It felt as if it was just randoms every game.
  • Apreche said:

    The ranking system is based only on wins and losses.

    This is the right way to do it, yes? Why do you play?
    image
  • For teams, sure, but not for individual players.
  • Over many games your signal will emerge regardless. Also, if the only thing that matters is Ws and Ls, you get no benefit for selfish play. Unless it actually helps your team, in which case it's not really selfish then, is it?

    This does leave the issue of potentially requiring many games to get a signal out of the noise. Thankfully rocket games are short.
  • sK0pe said:

    Churba said:

    I'm a little behind on episodes, so I don't know if it was mentioned, but Titanfall was pretty much killed by bad Matchmaking.

    It wasn't mentioned but it was another of those games where not enough people played.
    I didn't experience a matchmaking system in that game.
    It felt as if it was just randoms every game.
    Sold double what Rocket Leauge has, and it had a matchmaking system - it just sucked real bad. It would try to skill-match people, but also tried to not break up successful teams. So what you ended up with was a few highest-rank players getting skill-matched in a game. Then one team would lose, some of the players would get swapped, etc - but since it didn't want to break up successful teams, you'd invariably end up with a team full of highest-rank players that it wouldn't break up, matched against a team of scrubs who would consistently get American history X style curb-stomped.

    Population rapidly dwindled as anyone who didn't have time to play constantly and get top rank(so, most people) left. It was already dying by the time they started fracturing the community with DLC, which was the final blow.
  • I think Halo 3 has had the best matchmaking system.

    Literally seperates game playlists into Social and Ranked. Rank is based on True skill ranking system (50 levels). If you lose 2 games in a row your level drops 1 , if you win 2 it goes up 1.

    The social playlists levels awards 1 exp per won game. Which represents how much of that gametype you've played. But not skill.

    Bungie tried something different for Halo Reach, which really wasn't good. They ended up abandonning it early on for something just as stupid. I can't even explain the system. It assumed too many things. Like, how many people were playing the game.

    Your rank would reset seasonally, to give you like a historical rank or something. Which was BS, because unlike in Halo 3 where everyone could visibly see what rank you were in skill and be intimidated, your skill rank was gidden in some dark alley where no one could find.

    Making it a pointless nymber you couldn't brag about.


    The Halo 3 system was dead simple. You win enough, you get a point, you lose enough, you lose a point. There was rarely a chance you would be playing someone out of your league.

    Even then, it was a fun challenge.

    Rage quitting penalised a whole XP if I remember correctly. That 1 measily xp took a whole game to earn, so you wouldn't quit unless there was some real BS going on. Like teamkills. Then you could report players anyhow.

    Those sweet sweet player bans. Satisfaction.

    I miss those Halo 3 days.
  • Starfox said:

    Over many games your signal will emerge regardless. Also, if the only thing that matters is Ws and Ls, you get no benefit for selfish play. Unless it actually helps your team, in which case it's not really selfish then, is it?

    This does leave the issue of potentially requiring many games to get a signal out of the noise. Thankfully rocket games are short.

    This would be true, under certain circumstances. But consider that Eli Manning and Odell Beckham Junior are some of the best in the NFL at their position, but their team has the same record as the Bucs. Consider that Bryce Harper was National League MVP this year, on a team that did not make the playoffs.

    Consider this hypothetical scenario. You have 10 players, with "true" skill levels of 1, 2,3,4,5,6,7,8,9, and 10 with 10 being the best player. But you don't know their true skills, because they all start at 0.

    So you get a game that is 10,7,2 vs. 3,4,5. 10,7,2 wins. Judging only by wins and losses you think that the player with 2 skill is as good as the player with 10, and you can't tell that the player with 5 skill is any better or worse than the player with 3.

    You are correct that if these 10 players keep playing a very large number of games against each other in lots and lots of combinations, and you judge only by wins and losses, eventually you will be able to determine their "true" ranks.

    But that is because it is a closed system with no new players entering and no players leaving. In a large online game you are constantly facing new opponents. Every single game you play will be determined by the single strongest player, and you will incorrectly adjust the ranks of the other players based on which team that player happened to be on.

    Yes, that very very strong player may be able to fly up through the ranks, but since it is a grinding system it will take them a very long time. In every game they play, that is one player who will correctly increase rank with 5 who just get dragged along for the ride.

    Because new players are constantly entering the game, at varying skill levels, you will be able to determine that a level 10 player is better than level 2, but you won't have enough data to determine who is 5 and who is 6.

    If you do want to stick with wins and losses as the only measure, at the very least you need to not have a grinding system. Someone who wins their very first game should probably immediately be matched up with the best players in the entire game. When they lose you can put them in a game with the median skill-level players and so on, like a quicksort.

    Obviously, that has problems. The pro players will not be happy with nubs showing up in their games. While that solution is somewhat good at figuring out player rankings, it's not good at maximizing player experience. You need to get people into games they belong in as quickly and consistently as possible. And to do that, you need you need to judge players based on more than just wins and losses.
  • How much of the math behind matchmaking systems do you know? Usually you get two numbers: a skill score and an uncertainty score.

    Starting out you get slotted somewhere in the middle, with high uncertainty. At the beginning, as you win or lose you shoot up or down dramatically. Once the system gains confidence about your skill level, your uncertainty goes down and you don't move as much. Enables quick convergence on your "true" skill, for whatever that means.
  • No matchmaking system will solve the problem of a highly stratified player base.
  • Yeah, finding a good match is predicated on a good match existing in the first place.
  • Starfox said:

    How much of the math behind matchmaking systems do you know? Usually you get two numbers: a skill score and an uncertainty score.

    Starting out you get slotted somewhere in the middle, with high uncertainty. At the beginning, as you win or lose you shoot up or down dramatically. Once the system gains confidence about your skill level, your uncertainty goes down and you don't move as much. Enables quick convergence on your "true" skill, for whatever that means.

    Yes, that is how the Microsoft Trueskill system I mentioned in the show works.

    http://research.microsoft.com/en-us/projects/trueskill/

    That is NOT how Rocket League works. In Rocket League you start at the bottom and have to grind your way to the top.

    The thing is, you can do even better than True Skill if you used advanced metrics to judge players on more than just wins/losses. That's just more difficult to code, and requires more computing resources, and programmers are lazy.
  • edited November 2015
    What metrics should they use, then? Even coming up with these things is super freaking hard. At least Psyonix has the benefit of having all the datas, so that should help.

    Edit: I forgot another piece! You can assume everyone entering the system is average, because on average, we are.
    Post edited by Starfox on
  • Starfox said:

    What metrics should they use, then? Even coming up with these things is super freaking hard. At least Psyonix has the benefit of having all the datas, so that should help.

    Edit: I forgot another piece! You can assume everyone entering the system is average, because on average, we are.

    You use individual statistics like we use for professional sports. Nowadays we have advanced metrics that are even better at judging how much an individual player contributes to winning. Using baseball as an example, that includes stats like wins above replacement, peripheral ERA, or defensive runs saved. There are also the old school statistics, which are as simple as things like hits, runs, batting average, strike outs.

    Neither Rocket League, nor any video game I know of, even tracks (at least not publicly) or uses the most basic statistics in their ranking, let alone advanced metrics. That's ridiculous considering that to track baseball statistics we have to have human beings watching every single player every single game and manually recording all the data. Occasionaly that recorded data is subjective (error or hit?). For a game like Rocket League we could have absolutely perfect objective recordings every single frame of every single game and digitally extract the statistics from that. If we develop more advanced metrics as time goes on, we can go back and reanalyze all the recording and re-rank players accordingly.

    Even if you just used the basic statistics you could drastically increase, as TrueSkill puts it, the number of interesting games. Someone loses a bunch of games because of teammates that score own goals? TrueSkill will rank you down because it will correct itself over time. Why bother? If you just tracked even the most basic statistics you could see that this player scored 3+ goals per game, but lost because of their shitty teammates. Rank them up even though they lost.
  • Starfox said:

    What metrics should they use, then? Even coming up with these things is super freaking hard. At least Psyonix has the benefit of having all the datas, so that should help.

    In the Rocket League thread I and others already contributed to a list of metrics.
  • edited November 2015

    Starfox said:

    What metrics should they use, then? Even coming up with these things is super freaking hard. At least Psyonix has the benefit of having all the datas, so that should help.

    In the Rocket League thread I and others already contributed to a list of metrics.
    Not only that, but you can simply start with a lot of the stats used in soccer.

    But that is actually part of the problem for Rocket League specifically. Historically soccer does not track many stats! It's the most unadvanced of all the major sports in this regard. If FIFA, which has a lot more resources than Psyonix, had developed soccer stats over the years, it would have been a lot less work for Rocket League to implement them.

    http://grantland.com/the-triangle/the-adolescence-of-soccer-stats/

    There is also the problem that there haven't been many sports games where each player is controlled by one individual human being. One person controls an entire team in Madden. Rocket League proves the concept is very viable. But also, it means that there isn't a history of tracking such metrics in video games. If there was a baseball video game where each human controlled a single virtual player, they could simply implement all the metrics that already exist.
    Post edited by Apreche on
  • I put 100 hours into Splatoon this Summer because it has a beautiful skill curve and a very effective matchmaking system. It accounts for both the relative strengths of the two teams and your individual performance and brings you up or down on a 100 point scale. If you hit 0 or 100, you slide into a different matchmaking category. You will almost never see anyone more than 1 category away from you.

    The skills required to win a Splatoon match are also very different from traditional FPS, but there was no "I learned this one skill, so now I am a god." You feel yourself slowly getting better at ink placement, stealth tactics, etc., along with traditional learning curve of map knowledge, weapon matchup strengths/weaknesses, etc.

    Mario Kart 8 doesn't use as complex of a matchmaking, but it seems very effective in dumping you, very quickly, into a race with equal skilled players.

    Has Nintendo come out of nowhere, from basically not supporting online play at all, to offering something few other companies can? I haven't played much Smash online at all, but can anyone weigh in on how the matchmaking works there?
Sign In or Register to comment.