Play Nice: the Science and Behavior of Online Games



so my name is Jeff my background is in neuroscience so I used to study stuff like visual attention using techniques like fMRI eye tracking stuff like that but ever eye games I'm currently a lead game designer with me it's called quo Karkos has had over ten years of experience as a producer and he's done games such as plants vs. zombies as many of you guys know content on League of Legends like spectator mode replays right now and he is the producer of the player behavior team at Riot Games so today we'll talk a little bit about the science behind player behavior in online games and some of the experiments we've been doing online and some of the lessons that we've been picking up in the past year so some of this data we've never presented outside of Riot Games so we really excited to see what's the feedback from you guys for those of you not familiar with Riot Games were a game studio and publisher located in Santa Monica our first game is League of Legends we're really focused on the player experience first and what that means is we're all hardcore gamers and we literally video games are a part of our lives our mission statement at the company is we aspire to be the most player focused game company in the world to give you a sense of what League of Legends is it's a game that launched in 2009 it's a team oriented competitive hardcore game usually played v e5 or 3v3 and we have over a hundred champions in the game these are characters that you can choose in any given game kind of like superheroes in the game and the game is set in a modern fantasy world so a few high-level stats every month over 30 million players log in and play League of Legends we see a lot of daily active players as well and we record over a billion play hours monthly to give you context on this data point YouTube for example if a record over 4 billion hours monthly so a lot of our players are really hardcore and they played the game with intensity they play the game a lot every single day and at peak time during today we have over 5 million players logging in at the same time playing League of Legends on the player behavior team we have a really simple mission statement we want League of Legends to be the most competitive most sportsmen like community and core competitive games and that's why he's the producer but some of you might be wondering why do we have a player behavior team at Riot Games and why do we need PhDs and statisticians and researchers why don't we why do we need modelers the reason is because we've all played online games and we've all experienced behavior that's less than polite a lot of psychologists sociologists they've tried to come up with theories on why behavior online is worse than real life but our favorite representation of this is a comic by our gamers fellow gamers at Penny Arcade so I see a couple smiles most of you have seen this particular comic but the insight we draw from this is that people players are not innately toxic right it's the behavior that can become toxic and it might be because there's a lack of consequences of lack of accountability or because they're anonymous online these are reasons that could make your behavior more toxic but why do we keep using this word toxic why is bad behavior toxic so let's do a thought exercise I'm gonna pull 10 random players and in real life they're all good people right in their own given context they all come from different backgrounds and they play a game together but one player gets pissed off so a teammate does a mistake maybe cost his team the entire game and he rages that that person he verbally abuses the person harasses him after the game and now there's two people in this game that are upset but actually just observing that one incident multiple people are actually fostered now they now have a negative experience from this incident so now we have four players in the game that are upset and one of these players might play one more game afterwards but he's starting the game already upset so the cycle continues and toxicity spreads and now many games seem toxic but the stem of the source might be just that original player having a bad day and raging out because of one incident so really early on we had to ask each other a question if we simply ban all the toxic players from the society or from the game do we solve the player behavior problem and the simple answer is no everybody's like I knew it so let's take a look at some data that might suggest that simply banning toxic players is not the right solution so in the game of League of Legends you can report or honor players on a variety of dimensions so if you had a great experience with a player you can honor them for being friendly or helpful if you had a negative experience you can report them for stuff like verbal abuse or negative attitude by aggregating all these metrics together we can basically segment our population to four distinct buckets and it looks something like this well you quickly see here is that 1% of our players are qualified as toxic what that means is they get reported often by many players that you know like every game they play and they are banned by our player behavior systems but what's more important is that over 92% of our players are neutral or positive let's take a look at a second metric let's look at all the toxicity in the system and how much of it stems from these four buckets we actually thought it would look something like this us even though it's a small population that is toxic they're responsible for the whole majority of toxicity in our system but that isn't the case actually looks something like this so even though the toxic population represents one percent that active player base they only represent 5% of the toxicity in the system the majority of toxicity still stems from the neutral and positive players in our game and we're going back to it it might just be because these players are having bad days right they're not innately toxic but if you have a bad day at work or a bad day at school maybe somebody yelled at you on the way home and you want to play league of legends and you play a terrible game and somebody's reaching at you there as well and you just flip you flip the tables you rage back at that person but you might not do that for another 100 games so on the players behavior team we are we do have a unique opportunity to shape online behavior and why do I mean that well we record over 20 million sets of data a day so what does that mean so let's break apart my own personal account League of Legends and show you some stats from that account so as just to mention I'm light online and we have game stats on me so how many total games have I ever played in League of Legends how many champions have I played in the game what were my favorite champions and apparently I like gangplank who's a pirate we have total wins total losses we can also get higher resolution we can get how many games I played today or yesterday or how many losses that I have yesterday we also have social stats so how many friends do I have online how many new friends did I make how many people did I honour today or yesterday we also have secondary Network stats for example if I honor these players how many people did they honor but beyond raw stats we can also quantify experiences for example what are the correlations in all the games where I report at least one player for verbal abuse or is there anything in common for all the games that I honored three people we even have data that extends beyond the game for some countries we even have their SES their education scores their age even their gender huh socioeconomic status yeah and we can correlate any of these data with in-game data as well so we have a team and we have this database that is seemingly infinite in potential but how do we tackle the problem as vague and complex as toxic player behavior online and that's what and that's really what we were faced with when we kicked off the player behavior no worries I learned behavior mission is you know we have all this research and data and we know that we want to make a difference but how do you break apart this problem you start to to understand how we can approach this and really pull it apart and we got into a room the entire group and we sort of said well what do we want to start to accomplish what are the things that we think we could work on you know we have the tribunal but what are other systems that we could do and we started brainstorming and discussing and we started to see themes really popping out and those themes became our strategies and really the foundation of how we approach the player behavior initiative and I'll take you through them so our first pillar is that we want to shield players from the impact of toxic behaviors this is really important because we recognize that the likelihood of us making a perfect system that stopped all toxicity forever and ever it is pretty small and so we need to be able to understand that when toxic events happen because they will eventually how we can stop the impact from being so heavy to our to our players and how we can stop that ripple effect that Jeff was talking about earlier our second core pillar is that we want to reform or remove toxic players and this that word choice and word order is really important so I want to call attention to it we want to reform first and remove second in many communities we jump right to the removal part we see bans as a way to solve the the player behavior problem in in most online communities and what we found that for our game because we are a free online game it's very easy to reenter the system with a new account you just come right back so what we really want to focus on is how do we reform those players who are exhibiting toxic behavior in to not exhibiting that anymore and being more neutral players or good players our third pillar is we want to create a culture of sportsmanship we recognize that this is not a problem that we're gonna solve on our own right we really need to involve the community and so the way we approach that is what we want to do is work with our community do do have discussions with them and create a real strong culture within that community that expects sportsmen like behavior and then as we address this player behavior problem you're getting a reinforcement not just from ranked games and laid in status quo in the forums but also from your peers the guys who play with in-game who you interact with every day our fourth pillar is reinforce positive behaviors we really want to make sure that even as we're trying to show how not to be bad how not to be toxic in the game we're also spotlighting really positive behaviors that the community sees as great examples so that we can build role models and create aspirational paths for players to follow who may previously have just known how not to be good or not to be bad they now know how to be good and then finally our fifth pillar is we want to create better match chemistry you know through through personal experience as well as looking at the data behind the game we recognize that when you do a what's called an in-house gaming are where you have ten friends all playing the game together those tend to be the most positive games you're all having fun you're familiar with each other there's a high degree of trust between all the players on each team and what we want to do is find a way to make every game that you potentially play whether it's with strangers or with friends feel like that experience where you know and have confidence with the teammates you have and have a really really great experience and what we wanted so this pillar is really about creating that situation and finding ways to improve trust within the game as early as to tamp select what I want to do and what the bulk of this presentation will be is we want to kind of take the data that Jeff talked about and the strategies that I just talked about and really apply them and show how it plays out for a given feature within the player behavior team and the the feature that the future that we're going to talk about is the tribunal which is one of our oldest we're going to talk about the tribunal which is one of our oldest features and has a really rich history that we can show a lot of a lot of how this has gone on we start out almost every feature we do with the very simple hypothesis and the tribunals was this that we want to engage with that we've believed that engaging our communities manage their own behavior can be an effective approach to finding toxicity and really this lines up with our core pillar of wanting to reform or remove toxic players for those of you who are not familiar with what the tribunal looks like or if you've never interacted it this this is a screen from the tribunal what you can see is that we're presenting back to the community yeah there's some pretty great chat there we're presenting back to the community information about toxic events that have happened in the game so in in this particular case you can see that the behavior you can see the number of times this player was reported in a number of games comments about their reports so details if they were shared as well as all the chat that happened in the game and actually if if we had scrolled down farther you would be able to see game states like what champions were played what the Katy A's were what items were built that sort of thing and what our community can do is take a look through this review this and say hey you know I think this this behavior is actually totally acceptable within our game so we should part in this case or this behavior is totally unacceptable for this community we should punish this case this is not this is a approach that we've taken that other people have taken fold it as an example is a University of Washington project that engages the community to solve protein folding problems and what we found is that in in both cases engaging the community in especially difficult problems that are hard for computers or algorithms to deal with is a really really effective way to solve the problem as well as a really great way to engage everyone in the understanding what the problem set is and bought into the approach today the tribunal we we've considered it fairly very successful we've logged 105 million votes and then we've also reformed over 280,000 players when I say reform what I mean is people who have gone through the tribunal been punished but are as of today currently in positive standing and we've also done a lot of analysis around who votes in the tribunal and specifically we've looked one of the sets we've looked at is percentage of games that they participate in that result in a report and we looked at three populations the first population is toxic players who look a little bit like this then we have the people who vote in the tribunal and then just a random sampling of players across the board and what you can see is the tribunal judges have about a 20% Delta from the toxic players but 3% from the random sampling our conclusions from this is that there aren't toxic players going in and trying to mess with the Tribunal for for the most part the judges are pretty close to your everyday player who trimmed a little bit more positive and this is really great because it shows that they're pretty representative of the community but just going down that path and trying to make a difference in the community around them to sort of evaluate the accuracy of the tribunal we pulled some cases and we're going to show them to you here and this sort of demonstrates how our players are able to pretty quickly pick up on what's going on in and have a high degree of accuracy so I encourage you to to read all the fun chat so this was an example that was flagged as moderate toxicity and qualified for an email warning and as you can see players pretty quickly picked up that our friend mastery here was using some homophobic language a mom joke you know nothing too bad but definitely some toxicity a more severe case is the cigs which was flagged for severe toxicity and qualified for a very long time ban and you can see that there are violence threats he's much more aggressive he's he's spamming to chat a lot and that was flagged by our players finally our players are also really good at noticing when things are not quite right and there are cases on the other side for example this guy was pardoned and received no punishment because they ass they saw that yeah it's some guy reported him for trash-talking but nothing was said in game so probably that's probably acceptable within the community and so the community part in him what we did is we also took a lot of cases and saw how players voted them voted on them and they took those same cases and put it through our riot player support to see how closely does our community line up with what are our internal riot player support system would flag and what we found actually is that between the two of them that there's a really high level of agreement between the community and the riders about 79 percent agreement and when we dug into this to sort of understand and really really dig into this data we found that the the reason why the agreement level is not higher is that our internal player support judges tend to be a little bit more strict and we'll punish more offenses even though the system is pretty accurate we haven't had a perfect history here we've learned a lot from our past mistakes and I'll give you one key example so over a year ago we used to ban flares with a really vague email message in a client pop-up the really good thing here is it tells players when they be banned until but they have no idea why but we know from psychology that's speed and clarity of feedback these play critical roles in shaping behavior so this leads to the next feature in the tribunal that we call reform cards so again we started with the simple hypothesis and we want to take a pretty big risk with this one we want to show players exactly why they were a band and we hope they didn't prove him for perform rates so what does a reform card look like looks very similar to a tribunal case but at the top center you can see there's the actual decision there's the community agreement and there's the eventual punishment but we send these cards to every single player banned by the tribunal so let's take a look at some data from this experiment what I'm going to show you is data from last year when we were still sending vague email messages and client messages and this is the percentage change in reports received after they received their warnings or their other punishments so on average after they got a warning they did improve their behavior they got fewer reports after their warning at their vague email but for three day seven day and 14 day bans they got worse so players came back to the game angry and Fuster dude they had no idea why they were being banned and they got banned even quicker after that after their first ban but after we launched reform cards we saw sharp improvements in every category of punishment and this makes sense players now knew what was okay or not okay in League of Legends and even if they didn't agree with that they knew that that was inappropriate and they could get them banned here we made a conscious a conscious decision to host all of our reform cards online so we actually sent URLs to every single player that they could share with their friends or their community so as you can imagine when we first launched this feature we were really scared about the forum post that would go up and this is the first thread that went up after the future went live perma banned because my team sucked and I called them now we saw that this player had posted his reform and he posted paragraphs about how riot sucks about how tribunals broken about how his bans simply unfair and we were kind of concerned how had players respond to this what were the responses be like we were shocked in the best way possible here's the first response in that thread and here's some more from that exact forum thread but the key here is that our own community was speaking up and they were loud and clear this behavior was not okay in online games the final experiment talked about today is the Justice Review so to give you a little bit of context when we first launched a tribunal we actually gave currency when people completed cases so there was a small perception that Guru overtime that players didn't really care about justice they didn't care about the cases or reviewing them accurately all they cared about was currency so we had to design an experiment around the motivations of our players and although recent recent models of motivation are quite complicated we're gonna focus on two really simple elements of this we're gonna focus on extrinsic motivation which is simply going to Tribunal and getting some extra currency and compare that against intrinsic motivation which is I just want to do good for the community and contribute to the greater good to do this we created a feature called justice reviews it's a simple profile page you log in and you see some stats about your contributions to the community and to the tribunal you can see there's total cases completed your accuracy some ratings your longest streak we added a fun stat in their toxic days prevention but quite a few players also enjoy players permabanned so let's take a look at some data from this particular experiment so this was a three-phase experiment first we collected 30 days of data when we were still giving currency for completed cases so there's our baseline and we're looking at the daily number of judges that logged into the system for 30 days we then removed the reward so no more currency when you complete a case correctly we lost about 10% of our players when we did this but we saw a huge increase when we introduced justice reviews so almost a hundred percent increase we can also take a look at a second statistic so the average number of completed cases when you do log into the tribunal so again our 30-day baseline before we removed currency we saw an increase actually in a number of completed cases when we remove currency from the system so the hypothesis here is that the players that were going to the system and just doing it for currency they left when that reward was gone but those players completed fewer cases on average and the rest of the judges anyways so this average went up and then we saw a second increase when we introduced justice reviews so a few takeaways from all the tribunal experiments a vast majority of the gaming community do find toxic behavior disgusting one of the coolest insights we saw was that if you look at a word like or the C word or the N word online in a tribunal card those are the most highly punished words in the entire system providing tools like the tribunal is how we change this online culture to by showing toxic players reform reform cards and peer feedback these players changed and this is really critical because this is no longer about Riot Games versus players this is players talking to their own peers and them saying that this behavior is not okay and finally highlighting personal contributions to the community and to the tribunal was far more engaging to my community judges than the externship currency so we're pretty happy with results we've seen so far and we've learned a lot from these experiments and that pretty much takes us up to whoa I'm very loud that pretty much takes us up to current day tribunal and sort of where we're at I want to take a few minutes and just kind of go over what does our future hold what are we looking at next with the player behavior team in our features one of the weaknesses that we've noticed with the tribunal is that the speed of feedback is not very fast where I like the turnaround time tends to be a few days maybe a week or two and again we know that having faster feedback is better so we're developing a feature that we call behavior alerts and what this is is really an add-on feature that works in conjunction with the tribunal but it can detect spikes in toxicity as they happen and can intervene to those players and say hey hey dude like it seems like you're ragin a lot right now maybe you want to chill out maybe you want to relax there's not really any punishment involved it's more of just a quick intervention a quick feedback point to say hey your toxicity is spiking that might not be so great for you in the long term so yeah the second feature is what we call the reformed system some of you if you follow us on the forums and follow our experiments online you may have seen we're strict to chat actually just go to the live servers with the the most recent patch this is the first step towards the reformed system the idea here is I mentioned before that in a free online community it's very easy to create a new account and come back in and so banning is not a very strong punishment and so what we actually want to do is remove banning from our toolkit of or not from our toolkit but remove banning from what we do to players as part of reforming them and instead use things like restrict a chat to slowly take away portions of their experience but still have them be able to work through a a punishment cycle that helps teach them and reform them towards neutral behavior and restrictive chat is really the first one of these and the idea there is initially when you're banned it will put a restriction on your account where you only have a certain number of messages to say inside the game which means that if you want to burn your messages for raging and flight and flaming other players there's only so many of those that you can spend and you have to make a conscious decision to give up being helpful to your team and calling em IAS and giving game information in order to spend it on raging instead the we're also building a lot more tools to understand the world around us and to gather more data one of the things that I wanted to highlight here is what we call the micro feedback tool and what this allows us to do is really target specific players on a bunch of different spectrums and really get in the moment feedback on almost any topic that we want a recent example of this is that we asked we saw a bunch of threads on the forums and direct information from our players saying hey you know we there's these ranked borders that we got from being a player in season two and we actually kind of don't like having them because we feel like we're getting bullied when we have these games people are saying oh that guy has a silver border in a gold game and that's no good and we were concerned by this all right we want our to be awesome for our players and so we really wanted to dig deep into this and so we actually sent out the survey to a bunch of our players and said hey you know a value tell us your level of agreement with the following statement I like that ranked players received loading screen borders in season two players were presented with this question immediately after playing a game and so they were in the moment they had just potentially experienced this and we got immediate feedback from our players very very quickly within a day we had 60,000 responses that said you know actually people kind of agree that they really like this wrinkle border loading screen and so while there are definitely outliers and people who feel this way as expressed on the forums it's not the overall sentiment of our population and so these things are not as toxic as those forums hoods are making them out to be and that's kind of it so those are some stuff that we're working on in the future that were we're happy to share with you guys we haven't talked a lot about them before and now I believe we're gonna do some QA so that'll be fun and Jeffrey is going to tape things so one thing I want to do just again to first of all this was fabulous and really neat to see you know I am a league of Legends player you know from the beta and so in addition to being a bad one as well but at least one who's played for a long time and and so it's been fun for me to sort of watch these things unfold and to see some of the really careful thought that's gone into to develop things of different pieces I want to make sure to though I mean I think I'm going to challenge you to do in a minute is help us think more broadly I've been thinking during this talk of this recent study that came out of the George Mason University Center which got quite a bit of report of of press I don't know if people saw this the study about toxicity in comments and people's reactions to writing so what they did was they posted a kind of research report and a discussion on it and they – and they randomly assign people to two conditions in one condition they there are a bunch of comments about the research report that were basically polite and civil that agreed or disagree with a report and then a bunch of comments about the research report that were not civil that were that were inappropriate what they found was when people read the research report with the toxic comments below it magnified whatever their opinions were so people who thought generally they agreed with the research report was felt more strongly that they agree the research report people who disagree with the research report disagreed more strongly so there's basically a way like public perception of scientific research or other kinds of things as its as its presented in the press which has a tremendous impact you know on all kinds of Simek dilemmas in our society can be infected by it could be affected by the toxic behavior of people in online forums and so I think a bunch of the things that you're talking about you know obviously are about like making people play nice with each other in games but also have a lot of resonance for for sort of broader institutions in our society so I mean let me let me take the first two questions and then I'll let you I'll let the audience jump in so the first one is how do you get your one of the things that you talked about is engaging the community around these issues but you also said that your community consists of 32 million people who play every month how do you how do you wrap your minds around engaging a community of people because your team how many how many people is on the player behavior team I believe 2020 okay so how does the 20 of you think about sort of engaging a community of thirty two million people and how do you think about that problem how do you think about that community of thirty two million people I can take the first crack at it it's not easy so ball back when we first launched tribunal and we had to do a global launch flush in several territories this is something we're still learning actually so when we launched tribunal we had like a war room that we call set up and we had several of the player behavior team and at different stations and we had moderators that would collect questions from every community around the world like China like Taiwan and they would feed the questions to a person they would answer a live team would translate it directed back up to the community so we were doing live Q&A with multiple countries at the same time that was not easy and and took several hours of a huge team coordinating across the world but we're constantly trying to find out better ways to do this right now yeah and what I would say is generally our the way we approach it is what we want to do is kind of kick off discussions with the community bring up issues that we're thinking about that the community has brought up recently we did one around toxicity and champ select and the idea there is really start the discussion and moderate it lightly and present our viewpoint and stuff but not with an agenda of like oh we want to – this is our approach and I hope you all agree but more like hey guys what do you think about this is this is this like you guys had this idea and this would be the costs of it right and having those discussions what we found it tends to spawn more of those discussions within the community like that one forum thread that we spent a lot of time in and answering questions spun off a bunch of other community threads that we didn't need to go into because there was healthy discussion happening there that was the community figuring out how it felt about these things and eventually it boiled up into riot please threads as most things do but you know it's true Demeter said it would be doing it next year yeah that was three years ago soon to him the the the idea there is like we I don't feel like we need to be part of every discussion or or a a giant hand in each of those discussions what's important is getting our community understanding that we're thinking about this problem and then having them discuss it that's more what I think what I think of as engagement is not oh I need to go and have a conversation with every one of our 32 million players because that would be my life for the next ever but having it be where those discussions can happen within our community and they're engaged with the problem so so here's my second question what if you guys were the heads of the player behavior team for the internet so so so what if we promoted you as you're thinking about sort of what you're learning for designing social systems within um right games you know as you guys are looking at youtube comment forums as you guys are looking at other kinds of interactive spaces online are there are the things that strike you you know the the White House petitioning system you know that that requires a certain you know if you get a certain number of people to petition they'll respond in some kind of way like like are there things that you look at that you say wow there are some design principles that we've come up with or some strategies that we've come up with within this space for gaming that seem like they'd have a lot of relevance in other places as other people are designing other sorts of public spheres I think the really interesting thing there is having peers do much of that changing so for example really early on the tribunal we had a lot of rioters go on the forums and they would say stuff like hey this language is bad or hey this is racist and we would try to enact change by just being pillars of the community but after tribunal launched after reform cards launch we got a letter from a lot from a boy and he said he was 10 years old and he writes like dr. light um I never knew you that anyone who was really bad and nobody told me I could not say this word online and he's like I'm really sorry like I will not say this again and I'm sitting there like taking this letter to my team and we're looking at this letter saying like really like nobody told you you couldn't say that word online okay but the fact was his his own peers his own friends his people he plays with are like hey this is not okay and that's when it clicked for him and that that's when he changed and we get letters like these every single week now yeah and I would say like from kind of the broad strategic producer II level I would say that there is absolutely a lot of overlap I think that the specific strategies that you that I would employ as product owner of the internet which sounds really weird would be different like I think that you know we we we choose very specific things because we are a competitive game and because our aim is sportsmanship if our if our aim was civil discussion like I think these specific strategies we would employ would be slightly different but along the same lines of spirit like he said like engaging the community having that culture of civic discussion rather than a culture of sportsmanship is really really important it's it's really important to not you know we product owner of the Internet will not be able to affect change on his own it you have to get everyone involved and so it's a lot of the same ideas but it would be a lot more work and I would want a vacation first that's great yeah let's take some questions I do we I don't know I think passing a mic to everyone will be difficult and we are recording and want this to be able to go up later so I think what I'm gonna do is call on you and then though it'll be pedantic cuz everyone in the room can hear I'll repeat your question so it gets recorded well and then and then go around so this gentleman right here had this hand up how do you recruit the judges for this system and who watches the Watchmen good question so actually tribunals available for any player in the game over level 20 and it used to be level 30 for a long time in the stretch but in our research we saw that the majority of players that self volunteered and self selected themselves they tend to be a really representative example of the community and there was very little abuse or cheating so uh we once did a sweep of all the cheaters that we thought were abusing the system or maybe they weren't judging cases and we're just pressing random buttons it was less than 1% of the population at that given time so I guess riot is the Watchers but we haven't had to intervene in the history of the tribunal Todd you ask because you get one that the people actually up because I feel like it's it's not not an economic imperative it's an investment imperative when people feel that they have an investment in the community at the same time some reality to the experience of law right like you play with your forest rangers and you have to whether you lose and then unless one of them makes such a good impression that you link up with them in a social networking sort of way like they don't have a friends list and you queue again you never see those people again so how how do your efforts for increasing investment how cutting merge those two things right where you want people to have a significant investment and the people they play with but the people they play with rotates so fast and the experience is so ephemeral that's great so how do how do you get people to have a commitment to improving the quality of the people they play with when in the way the way League of Legends is structured the people you play with are in a sense random and anonymous and you'll probably never play with them again I think that a lot of that is well first of all we're talking a lot about that problem more and more it sort of goes to the core pillar of we want to create better match chemistry a lot of that is is trying to take I don't want to take away that ephemerality but sort of reduce the impact that that has on how you approach things we've anecdotally you often have the experience of even in solo rank you you have that game where you're not gonna friend anyone after the game but you're like man I really enjoyed playing with that tarik he was he was really good at sending all the guys and we totally won that game because we were super positive and we we turned it around at the end even though we were down to inhibitors and that sort of thing and it really comes down to you can sort of see those games coming even a champs like because you're like oh I want to play this role and the response back is trust building and confidence building because other people are like oh okay cool well I'll play this or I'll play that or hey what you want me to pick a champion for you early because that's a role that you kind of want to pick earlier or whatever so building on that and really creating that better match chemistry so when you come in you're just more likely to the people in the game with you a little bit more seriously even if it's only for a game or two games I think is is what the problem that we're worrying about now and then layer it on top of that creating that culture of sportsmanship so that you know that your expectations of your peers are that they will be sportsmen even if you only see them once in your gaming life and there are expectations of you that you are very aware of as well is that you will also be sportsmanship sportsman like to them those are the kind of tutu strategies that I look at as how we address that problem grace did you have your hand up good so two questions there one is you know what's the right balance between punishment and reward in terms of changing player behavior and what kinds of concerns you have about over-regulation what concerns you especially with the idea that people come to the internet often times with the sense that it's a space for free expression and here there are all these kind of boundaries and regulations on expression you know how much concern do you have around those kinds of things so so just as a quick introduction we're bringing up another colleague of ours Dave and Pallas who is one of our player researchers and helps me out a bunch with sort of our he helps us he helps me out with the data's as as regards the over-regulation question I would suggest that we're definitely sensitive to imposing too much regulation but that the context within league of legends is not entirely the these sort of free speech context of the internet like all the players who are coming here are coming with a particular goal and that goal isn't necessarily self-expression it's to play a game and have fun and if that's their their primary goal we want to work towards encouraging that primary goal and removing things that detract from that in terms of reward and Punishment I think it comes down to different contexts we're shifting pretty heavily to focus on encouraging behaviors because as you soften a presentation of where the toxicity comes from it's not bad people who are who are causing the most of this it's normal people who have bad days or have a bad experience and what we want there is to figure out ways to stop that from happening like you don't want to smack them and be like hey yo chill do that nice thing you did right like that that is way cooler to us than just being a big blunt hammer that was a very Santa Monica based response to that sort of in the back corner there you how do you think differently about sort of first-order toxicity and second-order toxicity so how do you how do you think differently about people who sort of introduce toxic behavior into a system and then how do you feel alright how does the system both maybe how do you guys on the I think there's by two ways to adjust that one is how do you all on the player behavior team think about sort of first-order toxicity in second-order toxicity and then maybe you have some data from the tribunal where if there's if you've done any kind of research about how the player community in the tribunal responds to those two different kinds of conditions one of the most common player complaints by the tribunal is I want to punish somebody else in the case or I want to punish somebody who started the argument or a lot of ban players will come to us and say well somebody else was worse than me so why why isn't why are you banning that person instead why are you banning me but our common stance is something we've been constantly talking to our players about for the past a year is that retaliation is never okay and the main philosophy there is that we're simply adding fuel to the fire instead of being a part of the solution and trying to either blunt the toxicity or shielding others from it we're now adding fuel to it and getting more people involved so when I talk to a player and lay it really really simply it's if you jump in and you retaliate you're now creating a negative experience for eight other players in the game right so it's not just about you and him having an argument you're now creating a bad experience for everybody around you so we're still talking to players about it and we still see this complain a lot but it's gonna be a non ongoing discussion right here okay so what so it sounds like valve which game are you referring to hmm okay so dota 2 we're claiming has a system of sort of taking the the 1% of toxic players and like putting them in the same matchmaking system and just letting them have their own sort of cesspool what's your response to that see I'll talk a little bit to this and I'm sure I think daven has some words to say about this internally we we call that idea prisoner's Island and the main concern we have with prisoners Island is it's basically a one-way trip right when you throw a bunch of toxic players together they're not going to get better right and there's a lot of players when they go to prisoners and they they actually lack self-awareness they don't feel that they're toxic and they they feel like they shouldn't be there right so they're in prisoner's island they're looking around they're like whoa these guys are all jerks like what am I doing here like I don't deserve to be here so a lot of players and not making new accounts anyways and shifting their tops to city back to the low levels again and back to our new player base so for us and our game prisoner on prisoners island is not the ideal solution okay let me just we've mentioned levels a few time and I know there are some people who don't play League of Legends that it's a free-to-play game that's sort of the full suite of features of the game is open to you when you achieve level 30 which you get to over a period of time and so it's actually a fair amount of work to build a character and there are rewards that you can earn within the game that are associated with a character and so when we talk about people recreating accounts they're basically going back to level 1 under a new name and starting over and since it's the games model is free to play to begin with and then you purchase sort of in-game items as you go along or in-game items isn't the right word but so that's sort of an important dynamic here that you know just actually just like we would see in lots of other places like you know if YouTube or the New York Times sort of banned someone from their comment forums like it's your one email address away from jumping back in especially if you're in a place like MIT where you share you know a couple of IP addresses and like and to hit a little bit more on prisoner's island it's a really seductive idea it's it's extremely intuitively appealing to people and extremely intuitively appealing to our players we've actually done some research on sort of what ideas about punishment and about reward appeal to players but given that it's effectively just a time-delayed shift on to eventual ditching and then reintroducing that toxicity at lower levels even though it sounds pretty cool we're not a fan of that okay this actually may be have you ever thought about putting some kind of extremely trivial costs to starting anew like like 99 cents for an account or some kind of way um just like I mean I guess I guess I guess the appeal of the free account is that there's no barrier to entry for all the people you want to sort of hook in but and I mean I don't think that it's got an account to level 30 you know having even a trivial cost is not actually going to be a cost to getting back in and it's so it sort of serves as just a negative experience for a bunch of players that doesn't end up actually helping the problem at all yeah and so we haven't really considered that at all good lady right here good so so League of Legends has a pretty active kind of eSports community and they're players that play at a professional level for pots that are you know in the tens of thousands of dollars for winning tournament don't think there's an it because even a hundred thousand dollars to any team yet oh we gave away like we've given away millions of dollars buddy but like in one in one tournament can I win more than a hundred thousand dollars yes good alright so up to sort of six-figure six-figure for the world championships so the question there is to what what role are you playing and managing those professional players and and their behavior sure so the first thing I would say is that you know the player behavior team as a constructor is mostly a development team that really focuses and worries about this problem but we also see ourselves as ambassadors probably the wrong word but sort of mentors and leaders of conversation within Riot Games as a whole around player behavior issues right like it's very easy to say oh the player behavior team is working about all that but this is a company-wide goal and everyone at the company plays a part and we definitely work with the eSports guys we helped talk through sort of their sportsmen and conduct rules that they have in there their guidelines and all that stuff and we also help them with the sweeps of data around the pro players accounts and the activity they're in so we're absolutely a part of that conversation they're there they're the leaders there as far as eSports and following those guidelines but we're also taking just taking it taking a stand that you know a lot of other competitive organizations take of you have to be sportsman like and so it's something that the entire company worries about including eSports so right here how does the player behavior team work with right where I could see the scenario coming up where the design team maybe wants to implement voice check for example and you know that would have a huge impact on player behavior so I'm kind of curious as to how your team works within the whole riot and how decisions are made so the question is how does the player behavior team work with other development teams especially in circumstances where developments that might you know sort of accentuate one area of gameplay would have a big impact on player behavior in expected or unexpected ways sure and I'll take part of that I think that you know specifically to design Jeff can speak a little bit more like I said before you know the player behavior initiative is a company-wide goal and so a lot of a lot of my job actually is you know to talk to a lot of other teams a lot of our other departments sort of explain a lot of what we just just explained to you like hey here's here's the problem and here's how we're trying to solve it and how we're trying to approach it and in many cases other teams come to us and are like hey you know we're doing we're think about doing this thing right like we're we're thing about implementing this feature or approaching this problem and we're curious do you think this has any player behavior impact we don't feel like it does but maybe it doesn't our team in various circumstances talk to them about like well it might affect this or it might affect that you probably want to look out for this or that and actually we have this research that we happen to do that says that what you're the problem you're trying to solve is actually this problem over here not that problem over there so like maybe that should guide your approach and really we're sort of like internal consultants in a lot of ways around the subtleties of the problems but everyone is aware that we need to be conscious of that and tries to take a very measured approach to anything we add to the game so for for game design in particular we're extremely collaborative so on the player behavior team we have three game designers for example and we have our hands in almost every single feature that is being made for League of Legends so as soon as something spins up as soon as a new idea happens were already in that discussion about hey what are the player behavior concerns like Carl was mentioning good take a few more questions yeah really curious as to what you seen coming out of that so one of the things that we didn't talk about here is an honor system which allows players to sort of tag other players after at the end of the game with it with an honor badge which if you earn enough of them gets sort of permanently wrecked you get a permanent visual representation of your honor status throughout the game so maybe you can give us just like the highlights I think you guys are talking about this at PAX East a little bit okay it might come up yeah so one of the really cool stats we've been seeing is that really early on about 23% of players that were formerly in the bad demographic for example they sort of really shifting their ways and trying to earn honor and a lot of players came to us and well they're just faking it they're just pretending to be nice they're no longer swearing anymore but they just want to get some points right right right right right so we saw that which was pretty cool we're still investigating and following those players to see is the change lasting or is a temporary another really cool thing that we saw is in the game as Justin mentioned you can earn on our badges to display on the loading screen and it's this little ribbon that shows you know are you a great teammate are you a great leader what's really cool is if you if we define a good game as a game where no complaints were filed at all it turns out that if you play with one of these players with the badges you're highly likely to have one of those games so what we're finding is that people who do have badges they are linked to many games that have very very few complaints and very positive experiences I mean yeah it's fine you know as soon as I saw those badges I was like oh I want one of those and I definitely have the experience of like when you see a loading screen and it's got kind of five of those badges on there you're psyched you're like oh this is like I bet these are gonna be nice people who I'm playing with I can totally imagine that unless I'm like that guy but I'm not I'm totally the guy who like is really trying to be nice and can't always succeed it's Ryan you wanna go ahead and so you know and if so did that just come about spontaneously educate people and kind of shake that and if not are there efforts that you guys are taking to try to get people to take all kinds of activities so the question is people who are people can behave in various kinds of toxic ways and probably we've emphasized a bunch of the toxic behavior around language because it's most easily represented in like single slides although there is toxic behavior about how you play in the game if you go and you know intentionally kill yourself or do other kinds of griefing behaviors that we haven't talked about but further kinds of sort of typed verbal toxicity which might you know does it does the tribunal judge community respond differently to different kinds of toxicity do you instruct them or try to coach them around those different kinds of things or have you seen what sort of spontaneously responds so the only guideline in the tribunal is what we call the summoner's code right some of you may have seen it it's ambiguous pretty vague rule said about how to be sportsmen like what we found that was really interesting was very early on tribunal judges looked at offensive language by itself and that was not punishable to them so if I'm just like I missed a skill-shot or I missed that play players don't punish for that and we didn't do any coaching on that but as soon as you direct that harassment to somebody else or target somebody then it becomes punishable and that to us was really interesting to the community kind of molded that way all on their own and another example that we saw was a lot of gamers might use the word rape for example so like I I beat you really well or I did a really good job against you and they would say I raped you our community judges actually are really against that type of language even if it's not you know even it's in the sense of a skill based play so again like we didn't have to coach players on that that just bubbled up organically I love this idea that like people are basically good like wonderful sense of behavior it's like you wouldn't necessarily get in something they use it's about it's about getting people in the right context and sort of getting some of that out of it I think that's one of the more encouraging things about the tribunal is there's a lot of stereotypes about gamers that they are like they swear a lot and are hateful towards women and minorities that's I mean that's not really the case from what we're seeing about the community it's the community is expressing its standards and those standards are the same standards we might hold in this Hall here today to make sure that players kind of stay in the game even though they good so for nukes oh so it's a so there's a steep learning curve to play the game there's a lot of toxicity at sort of the low levels of the game and for new people coming in and in fact if you're new at the game and not very skilled at it you're like more likely to be a target of that toxicity then the standard player what kinds of strategies are you doing to deal with we have you know I guess you're a connects to your pillar number one sort of shielding players from toxic behavior especially at low levels so we can't speak too much to features on this yet but this is something that I have intense passion for is the onboarding process and connecting players with other players who they're gonna have a good time with a new conform of social support network so like I can't talk anything about what we're actually doing that is something we are super aware of and I would just add that you know I think that the reformed system that I talked to at the beginning we recognize that there is a large amount of toxic behavior that ends it ends up getting shifted into earlier levels just through that process of banning players and them coming back and creating new accounts and so that's one of the main reasons why we want to put such a big effort towards the reform system is if we stop banning people or stop making it so people are effectively encouraged to create new accounts then then hopefully that by itself will begin to reduce the toxicity in early levels and then we will continue to worry about the problem in other ways you know one one feature change that you made not long ago was so there are two teams and every and and there's two kinds of chat rooms basically while you're playing one that is open to the five people on the team and one that's open to ten people in the game basically a sort of open chat with your opponents and one of the things you did recently was changed the default settings and sort of a like in the from the Harvard Law School I'm required to reference the work of Casa Steen who talked a lot about the sort of the default settings is an important part of society but how you know how much does that kind of decision play into you know what was some of the thinking behind that decision related to this question a little bit so that was actually one of the first experiments we ever tried on the player behavior team so this was a year ago we were looking and thinking about the player behavior problem in the space and how large it was currently and we wanted to do a really really small experiment to see how much of an impact could we make on this so it was focused on the first pillar to shield our players from toxicity and we defaulted this off after the patch and the reason is because we know from psychology research that people opting out it's much more rare than people opting in so patch happened and we found that a majority of players actually still opted in to all chat and I think something in the 70% range just immediately after I'm going back into all chat but what was really cool was the environment completely changed so in the four months following the feature for example we saw 17% drop in 17% of offensive language we saw a drop I believe 12% in verbal abuse so there's a really good hypothesis that we have about this where if you're a toxic player and you're talking to all chat we feel like you're trying to get a response out of people and an intentional choice we made was not to show who was opted into all China who was opted out so to a toxic player he had no idea if there was an audience or everybody was opted out so those players just took their energies elsewhere so even though all chat was much more positive than before we were concerned that were they were they taking their toxicities somewhere else were they spending an energy against their own teammates now and it turns out that didn't happen either so we didn't see an increase in with the within team toxicity or within team reports so it's kind of like that behavior just just went away so we're still trying to figure out where did that energy go did this did it simply dissipate does making a feature that shields our players also reduce the frequency of a negative behavior we're still answering those questions so the question is thinking about this time lag between bad behavior having a decision to punish someone or maybe you're you talking about punishing them are just having them be reported okay so the initial act of you've done something wrong you know what's the time lag between finding out that you've been reported and being reported in finding out that you've been reported and what's your thinking about potentially closing that time or how you think about that time lag sure I can talk about this a little bit the general right now in the game you are not notified when you are reported the first notification you'll get of being reported is if you go to tribunal and are found guilty and you will get your email warning that's that's kind of your first notification that you were being reported all along that is a fairly intentional choice up to this point there is an incidental rate of reports that everyone is sort of subject to right like I have my account is considered a very positive standing and I still have a collection of fun reports of oh he didn't know how to play a lease so I reported him right and so we we need to be sensitive to that because there are lots of very very good or neutral players who if you said hey by the way you're reported we see this a lot on threads in the forum when we talk when people are talking about how like they I was banned and I'm so upset and you know Jeff will come on and posted like yeah you were kind of ported 160 times heading games we're not even sure how that's mathematically possible you'll see a lot of players jump onto that side and be like Oh tell me how many times I'm reported I'm so worried it might I feel like I'm going toxic and and you look up a few of those accounts and it's like you've never been reported in the lifetime of your account which is two years long and so there's a great deal of anxiety that comes without especially for good initial players and because people are reported occasionally and that's not a big deal we don't want to be too explicit about it behavior alerts is kind of our first foray into really giving that that faster feedback of hey dude you've been reported recently and it's much more about you've had a string of toxic reports in a row and your toxicity levels are climbing rapidly and we're kind of like keeping an eye on that Plus so then hey some dude reported you last game because you didn't like your name you know so we have to be careful with that because we don't want to create really negative experiences for good and neutral players but we do want to increase the speed of that feedback so oh great well I wanna thank all of you for joining us they I really want to thank the riot team for coming out here and spending a bunch of time with us you know we we increasingly live in a world in which in which private companies owned public spaces and it's incredibly important for us to be able to have conversations about what's happening and those privately owned public spaces and so for you guys to come out here and sort of role model the work that you're doing to you know make those public spaces as rich and welcoming and civil as possible I think is just really really admirable and really exemplary so I really appreciate that and you know and I hope that all of us you know all of us in one way or another our participants and managers and different kinds of communities and I hope that we can think about how we can take some of these design principles and some of these ideas and experiments and things that are working and continue to share them because I think you know this goal of increasing civility in a society is being rapidly transformed by online tools whether it's in our games whether it's in our public discourse whether it's in our political discourse you know these are real issues that affect all of our lives and the the health and sustenance of our democracy in in different kinds of ways I mean it's kind of amazing to think that like the ten-year-old boy that Jeff talks to like some of his fundamental ideas about the judicial system are gonna be shaped by the tribunal that he's playing with in League of Legends he's but I mean he's much more likely to encounter that judicial system than some of the other judicial systems that exist in the world so if we can give our so if we can give our three panelists a big round of applause for I think this is gonna be made of the slides in the video will be made available and if you guys are you should definitely fight if you're interested in getting that follow the game lab and get on their Twitter and Facebook feeds and check their email list but thank you all for coming and have a wonderful evening




Comments
  1. HOW CAN YOU SAY @ 15:40 that GAY comments aren't serious????!!! any form of homophobic behaviour IS ALWAYS a serious matter, we don't like to be referred to as a derogative aspect and always when we see GAY word used this way How the fuck do you think we feel?????!! It just shows how backward thinking you are and that if this is coming from the top then no wonder the game itself PROMOTES toxicity among players and therefore PROMOTES homophobia…. deleting the game now!!

  2. Lye is a fucking joke. "LET ME MENTION THAT I GOT A PhD" with weird numbers. 

    Doesn't get better when you know that Riot is a very questionable company when it comes to the community and marketing. 

    Also what the hell is with this word "toxic, toxicity"?

  3. How can you be the "most player focused company in the world" when your whole business model is anti-consumer?

  4. Or,you know, just play Dota where people who actually know what the fuck they are doing and care for the game are.

    Lyte is an idiot,he doesnt see the forest before the trees 99% of the time and pulls numbers out of his ass

  5. The only toxicity problem is pack mentality. It is most clearly seen in 5v5 with the peer pressure and back room planning out of game. Did the toxic player queue solo or with a friend? This should be noted and factored into the equation.

  6. turns out honoring is bogus and ambiguous. I mean, a op player might later be honored for "friendly"…. WTF

  7. the tribunal was created as a free way to "manage" the community because riot support was getting swamped with reports (you can only press ctrl + v a certain number of times per work day). i would like to know the sample size, categorization method and how they dealt with confounding factors such as smurfs when they put together this study.

  8. 36 mins in.

    This is the prboblem with tribunal, ITS DONE BY RANDOM PLAYERS. Trolls spam punishing trolls. Stupid system.

  9. I just started the game, on my first game with a friend. The rest of the players in my team was composed with pros. We were not very good and the other players flipped at us, called us feeders and insulted us.

  10. If there would be a system which conects players in position they want to play, there would be more than 50% less toxic players.
    Agree = Like

  11. "We want league of Legends to have the most sportsmanlike community" he will have a job for life then.

  12. "We want league of legends to have the most sportsmanlike community" well, you have a long way to go… xD 

  13. you are making it up .,. stop talking shite.. utter shite… you are fooling nobody.. NOBODY!! get a grip.. im falling asleeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee

  14. cool stories bros, toxic people, like myself just create new accounts and terrorize new players, keep on banning me plz

  15. The Tribunal is full of hypocrites who are just as "toxic" if not moreso than the ones they're judging. They DO NOT have standards. They go by their whims. And if you are polite on the forums, what do they do? Verbally abuse the [email protected]#$ out of you for no reason.

  16. Pretty sure that pie chart is a bit wrong. It's more like 77% of the community is toxic, 17% is bad, 5% is neutral, and 1% is good. Just try playing it, or the forums. Even amongst the moderators and those participating in the tribunal, the same stats apply.

Leave a Reply

Your email address will not be published. Required fields are marked *