I’m trying some experimental tiers on Patreon to see if I can get to substack-like levels of financial support for this blog without moving to substack!

---

95%-ile isn’t that good

Reaching 95%-ile isn’t very impressive because it’s not that hard to do. I think this is one of my most ridiculable ideas. It doesn’t help that, when stated nakedly, that sounds elitist. But I think it’s just the opposite: most people can become (relatively) good at most things.

Note that when I say 95%-ile, I mean 95%-ile among people who participate, not all people (for many activities, just doing it at all makes you 99%-ile or above across all people). I’m also not referring to 95%-ile among people who practice regularly. The “one weird trick” is that, for a lot of activities, being something like 10%-ile among people who practice can make you something like 90%-ile or 99%-ile among people who participate.

This post is going to refer to specifics since the discussions I’ve seen about this are all in the abstract, which turns them into Rorschach tests. For example, Scott Adams has a widely cited post claiming that it’s better to be a generalist than a specialist because, to become “extraordinary”, you have to either be “the best” at one thing or 75%-ile at two things. If that were strictly true, it would surely be better to be a generalist, but that’s of course exaggeration and it’s possible to get a lot of value out of a specialized skill without being “the best“; since the precise claim, as written, is obviously silly and the rest of the post is vague handwaving, discussions will inevitably devolve into people stating their prior beliefs and basically ignoring the content of the post.

Personally, in every activity I’ve participated in where it’s possible to get a rough percentile ranking, people who are 95%-ile constantly make mistakes that seem like they should be easy to observe and correct. “Real world” activities typically can’t be reduced to a percentile rating, but achieving what appears to be a similar level of proficiency seems similarly easy.

We’ll start by looking at Overwatch (a video game) in detail because it’s an activity I’m familiar with where it’s easy to get ranking information and observe what’s happening, and then we’ll look at some “real world” examples where we can observe the same phenomena, although we won’t be able to get ranking information for real world examples.

Overwatch

At 90%-ile and 95%-ile ranks in Overwatch, the vast majority of players will pretty much constantly make basic game losing mistakes. These are simple mistakes like standing next to the objective instead of on top of the objective while the match timer runs out, turning a probable victory into a certain defeat. See the attached footnote if you want enough detail about specific mistakes that you can decide for yourself if a mistake is “basic” or not.

Some reasons we might expect this to happen are:

1. People don’t want to win or don’t care about winning
2. People understand their mistakes but haven’t put in enough time to fix them
3. People are untalented
4. People don’t understand how to spot their mistakes and fix them

In Overwatch, you may see a lot of (1), people who don’t seem to care about winning, at lower ranks, but by the time you get to 30%-ile, it’s common to see people indicate their desire to win in various ways, such as yelling at players who are perceived as uncaring about victory or unskilled, complaining about people who they perceive to make mistakes that prevented their team from winning, etc.. Other than the occasional troll, it’s not unreasonable to think that people are generally trying to win when they’re severely angered by losing.

(2), not having put in time enough to fix their mistakes will, at some point, apply to all players who are improving, but if you look at the median time played at 50%-ile, people who are stably ranked there have put in hundreds of hours (and the median time played at higher ranks is higher). Given how simple the mistakes we’re discussing are, not having put in enough time cannot be the case for most players.

A common complaint among low-ranked Overwatch players in Overwatch forums is that they’re just not talented and can never get better. Most people probably don’t have the talent to play in a professional league regardless of their practice regimen, but when you can get to 95%-ile by fixing mistakes like “not realizing that you should stand on the objective”, you don’t really need a lot of talent to get to 95%-ile.

While (4), people not understanding how to spot and fix their mistakes, isn’t the only other possible explanation, I believe it’s the most likely explanation for most players. Most players who express frustration that they’re stuck at a rank up to maybe 95%-ile or 99%-ile don’t seem to realize that they could drastically improve by observing their own gameplay or having someone else look at their gameplay.

One thing that’s curious about this is that Overwatch makes it easy to spot basic mistakes (compared to most other activities). After you’re killed, the game shows you how you died from the viewpoint of the player who killed you, allowing you to see what led to your death. Overwatch also records the entire game and lets you watch a replay of the game, allowing you to figure out what happened and why the game was won or lost. In many other games, you’d have to set up recording software to be able to view a replay.

If you read Overwatch forums, you’ll see a regular stream of posts that are basically “I’m SOOOOOO FRUSTRATED! I’ve played this game for 1200 hours and I’m still ranked 10%-ile, [some Overwatch specific stuff that will vary from player to player]”. Another user will inevitably respond with something like “we can’t tell what’s wrong from your text, please post a video of your gameplay”. In the cases where the original poster responds with a recording of their play, people will post helpful feedback that will immediately make the player much better if they take it seriously. If you follow these people who ask for help, you’ll often see them ask for feedback at a much higher rank (e.g., moving from 10%-ile to 40%-ile) shortly afterwards. It’s nice to see that the advice works, but it’s unfortunate that so many players don’t realize that watching their own recordings or posting recordings for feedback could have saved 1198 hours of frustration.

It appears to be common for Overwatch players (well into 95%-ile and above) to:

• Want to improve
• Not get feedback
• Improve slowly when getting feedback would make improving quickly easy

Overwatch provides the tools to make it relatively easy to get feedback, but people who very strongly express a desire to improve don’t avail themselves of these tools.

Real life

My experience is that other games are similar and I think that “real life” activities aren’t so different, although there are some complications.

One complication is that real life activities tend not to have a single, one-dimensional, objective to optimize for. Another is that what makes someone good at a real life activity tends to be poorly understood (by comparison to games and sports) even in relation to a specific, well defined, goal.

Games with rating systems are easy to optimize for: your meta-goal can be to get a high rating, which can typically be achieved by increasing your win rate by fixing the kinds of mistakes described above, like not realizing that you should step onto the objective. For any particular mistake, you can even make a reasonable guess at the impact on your win rate and therefore the impact on your rating.

In real life, if you want to be (for example) “a good speaker”, that might mean that you want to give informative talks that help people learn or that you want to give entertaining talks that people enjoy or that you want to give keynotes at prestigious conferences or that you want to be asked to give talks for $50k an appearance. Those are all different objectives, with different strategies for achieving them and for some particular mistake (e.g., spending 8 minutes on introducing yourself during a 20 minute talk), it’s unclear what that means with respect to your goal.

Another thing that makes games, at least mainstream ones, easy to optimize for is that they tend to have a lot of aficionados who have obsessively tried to figure out what’s effective. This means that if you want to improve, unless you’re trying to be among the top in the world, you can simply figure out what resources have worked for other people, pick one up, read/watch it, and then practice. For example, if you want to be 99%-ile in a trick-taking card game like bridge or spades (among all players, not subgroups like “ACBL players with masterpoints” or “people who regularly attend North American Bridge Championships”), you can do this by:

• learning the basics of the game
• reading a beginner book on cardplay
• practicing applying the material

If you want to become a good speaker and you have a specific definition of “a good speaker” in mind, there still isn’t an obvious path forward. Great speakers will give directly contradictory advice (e.g., avoid focusing on presentation skills vs. practice presentation skills). Relatively few people obsessively try to improve and figure out what works, which results in a lack of rigorous curricula for improving. However, this also means that it’s easy to improve in percentile terms since relatively few people are trying to improve at all.

Despite all of the caveats above, my belief is that it’s easier to become relatively good at real life activities relative to games or sports because there’s so little delibrate practice put into most real life activities. Just for example, if you’re a local table tennis hotshot who can beat every rando at a local bar, when you challenge someone to a game and they say “sure, what’s your rating?” you know you’re in shellacking by someone who can probably beat you while playing with a shoe brush. You’re probably 99%-ile, but someone with no talent who’s put in the time to practice the basics is going to have a serve that you can’t return as well as be able to kill any shot a local bar expert is able to consitently hit. In most real life activities, there’s almost no one who puts in a level of delibrate practice equivalent to someone who goes down to their local table tennis club and practices two hours a week, let alone someone like a top pro, who might seriously train for four hours a day.

To give a couple of concrete examples, I helped Leah prepare for talks from 2013 to 2017. The first couple practice talks she gave were about the same as you’d expect if you walked into a random talk at a large tech conference. For the first couple years she was speaking, she did something like 30 or so practice runs for each public talk, of which I watched and gave feedback on half. Her first public talk was (IMO) well above average for a talk at a large, well regarded, tech conference and her talks got better from there until she stopped speaking in 2017.

As we discussed above, this is more subjective than game ratings and there’s no way to really determine a percentile, but if you look at how most people prepare for talks, it’s not too surprising that Leah was above average. At one of the first conferences she spoke at, the night before the conference, we talked to another speaker who mentioned that they hadn’t finished their talk yet and only had fifteen minutes of material (for a forty minute talk). They were trying to figure out how to fill the rest of the time. That kind of preparation isn’t unusual and the vast majority of talks prepared like that aren’t great.

Most people consider doing 30 practice runs for a talk to be absurd, a totally obsessive amount of practice, but I think Gary Bernhardt has it right when he says that, if you’re giving a 30-minute talk to a 300 person audience, that’s 150 person-hours watching your talk, so it’s not obviously unreasonable to spend 15 hours practicing (and 30 practice runs will probably be less than 15 hours since you can cut a number of the runs short and/or repeatedly practice problem sections). One thing to note that this level of practice, considered obessive when giving a talk, still pales in comparison to the amount of time a middling table tennis club player will spend practicing.

If you’ve studied pedagogy, you might say that the help I gave to Leah was incredibly lame. It’s known that having laypeople try to figure out how to improve among themselves is among the worst possible ways to learn something, good instruction is more effective and having a skilled coach or teacher give one-on-one instruction is more effective still. That’s 100% true, my help was incredibly lame. However, most people aren’t going to practice a talk more than a couple times and many won’t even practice a single time (I don’t have great data proving this, this is from informally polling speakers at conferences I’ve attended). This makes Leah’s 30 practice runs an extraordinary amount of practice compared to most speakers, which resulted in a relatively good outcome even though we were using one of the worst possible techniques for improvement.

My writing is another example. I’m not going to compare myself to anyone else, but my writing improved dramatically the first couple of years I wrote this blog just because I spent a little bit of effort on getting and taking feedback.

Leah read one or two drafts of almost every post and gave me feedback. On the first posts, since neither one of us knew anything about writing, we had a hard time identifying what was wrong. If I had some awkward prose or confusing narrative structure, we’d be able to point at it and say “that looks wrong” without being able to describe what was wrong or suggest a fix. It was like, in the era before spellcheck, when you misspelled a word and could tell that something was wrong, but every permutation you came up with was just as wrong.

My fix for that was to hire a professional editor whose writing I respected with the instructions “I don’t care about spelling and grammar fixes, there are fundamental problems with my writing that I don’t understand, please explain to me what they are”. I think this was more effective than my helping Leah with talks because we got someone who’s basically a professional coach involved. An example of something my editor helped us with was giving us a vocabulary we could use to discuss structural problems, the way design patterns gave people a vocabulary to talk about OO design.

Back to this blog’s regularly scheduled topic: programming

Programming is similar to the real life examples above in that it’s impossible to assign a rating or calculate percentiles or anything like that, but it is still possible to make significant improvements relative to your former self without too much effort by getting feedback on what you’re doing.

For example, here’s one thing Michael Malis did:

One incredibly useful exercise I’ve found is to watch myself program. Throughout the week, I have a program running in the background that records my screen. At the end of the week, I’ll watch a few segments from the previous week. Usually I will watch the times that felt like it took a lot longer to complete some task than it should have. While watching them, I’ll pay attention to specifically where the time went and figure out what I could have done better. When I first did this, I was really surprised at where all of my time was going.

For example, previously when writing code, I would write all my code for a new feature up front and then test all of the code collectively. When testing code this way, I would have to isolate which function the bug was in and then debug that individual function. After watching a recording of myself writing code, I realized I was spending about a quarter of the total time implementing the feature tracking down which functions the bugs were in! This was completely non-obvious to me and I wouldn’t have found it out without recording myself. Now that I’m aware that I spent so much time isolating which function a bugs are in, I now test each function as I write it to make sure they work. This allows me to write code a lot faster as it dramatically reduces the amount of time it takes to debug my code.

In the past, I’ve spent time figuring out where time is going when I code and basically saw the same thing as in Overwatch, except instead of constantly making game-losing mistakes, I was constantly doing pointlessly time-losing things. Just getting rid of some of those bad habits has probably been at least a 2x productivity increase for me, pretty easy to measure since fixing these is basically just clawing back wasted time. For example, I noticed how I’d get distracted for N minutes if I read something on the internet when I needed to wait for two minutes, so I made sure to keep a queue of useful work to fill dead time (and if I was working on something very latency sensitive where I didn’t want to task switch, I’d do nothing until I was done waiting).

One thing to note here is that it’s important to actually track what you’re doing and not just guess at what you’re doing. When I’ve recorded what people do and compare it to what they think they’re doing, these are often quite different. It would generally be considered absurd to operate a complex software system without metrics or tracing, but it’s normal to operate yourself without metrics or tracing, even though you’re much more complex and harder to understand than the software you work on.

Jonathan Tang has noted that choosing the right problem dominates execution speed. I don’t disagree with that, but doubling execution speed is still decent win that’s independent of selecting the right problem to work on and I don’t think that discussing how to choose the right problem can be effectively described in the abstract and the context necessary to give examples would be much longer than the already too long Overwatch examples in this post, maybe I’ll write another post that’s just about that.

Anyway, this is sort of an odd post for me to write since I think that culturally, we care a bit too much about productivity in the U.S., especially in places I’ve lived recently (NYC & SF). But at a personal level, higher productivity doing work or chores doesn’t have to be converted into more work or chores, it can also be converted into more vacation time or more time doing whatever you value.

And for games like Overwatch, I don’t think improving is a moral imperative; there’s nothing wrong with having fun at 50%-ile or 10%-ile or any rank. But in every game I’ve played with a rating and/or league/tournament system, a lot of people get really upset and unhappy when they lose even when they haven’t put much effort into improving. If that’s the case, why not put a little bit of effort into improving and spend a little bit less time being upset?

Some meta-techniques for improving

• Get feedback and practice
  • Ideally from an expert coach but, if not, this can be from a layperson or even yourself (if you have some way of recording/tracing what you’re doing)
• Guided exercises or exercises with solutions
  • This is very easy to find in books for “old” games, like chess or Bridge.
  • For particular areas, you can often find series of books that have these, e.g., in math, books in the Springer Undergraduate Mathematics Series (SUMS) tend to have problems with solutions

Appendix: other most ridiculable ideas

Here are the ideas I’ve posted about that were the most widely ridiculed at the time of the post:

• It’s not uncommon for programmers at trendy tech companies to make $350k/yr or more (2015, stated number was $250k/yr at the time)
• Monorepos can be reasonable (2015)
• We should expect to see a lot more CPU bugs (2016)
• Markets are not incompatible with discrimination (2014)
• Computers are getting slower in some ways (2017)
• Empirical evidence on the benefit of types is almost non-existent (2014)
• It’s reasonable to write technical posts on a subject that avoid domain-specific terminology

My posts on compensation have the dubious distinction of being the posts most frequently called out both for being so obvious that they’re pointless as well as for being laughably wrong. I suspect they’re also the posts that have had the largest aggregate impact on people — I’ve had a double digit number of people tell me one of the compensation posts changed their life and they now make $x00,000/yr more than they used to because they know it’s possible to get a much higher paying job and I doubt that I even hear from 10% of the people who make a big change as a result of learning that it’s possible to make a lot more money.

When I wrote my first post on compensation, in 2015, I got ridiculed more for writing something obviously wrong than for writing something obvious, but the last few years things have flipped around. I still get the occasional bit of ridicule for being wrong when some corner of Twitter or a web forum that’s well outside the HN/reddit bubble runs across my post, but the ratio of “obviously wrong” to “obvious” has probably gone from 20:1 to 1:5.

Opinions on monorepos have also seen a similar change since 2015. Outside of some folks at big companies, monorepos used to be considered obviously stupid among people who keep up with trends, but this has really changed. Not as much as opinions on compensation, but enough that I’m now a little surprised when I meet a hardline anti-monorepo-er.

Although it’s taken longer for opinions to come around on CPU bugs, that’s probably the post that now gets the least ridicule from the list above.

That markets don’t eliminate all discrimination is the one where opinions have come around the least. Hardline “all markets are efficient” folks aren’t really convinced by academic work like Becker’s The Economics of Discrimination or summaries like the evidence laid out in the post.

The posts on computers having higher latency and the lack of empirical evidence of the benefit of types are the posts I’ve seen pointed to the most often to defend a ridiculable opinion. I didn’t know when I started doing the work for either post and they both happen to have turned up evidence that’s the opposite of the most common loud claims (there’s very good evidence that advanced type systems improve safety in practice and of course computers are faster in every way, people who think they’re slower are just indulging in nostalgia). I don’t know if this has changed many opinion. However, I haven’t gotten much direct ridicule for either post even though both posts directly state a position I see commonly ridiculed online. I suspect that’s partially because both posts are empirical, so there’s not much to dispute (though the post on discrimnation is also empirical, but it still gets its share of ridicule).

The last idea in the list is more meta: no one directly tells me that I should use more obscure terminology. Instead, I get comments that I must not know much about X because I’m not using terms of art. Using terms of art is a common way to establish credibility or authority, but that’s something I don’t really believe in. Arguing from authority doesn’t tell you anything; adding needless terminology just makes things more difficult for readers who aren’t in the field and are reading because they’re interested in the topic but don’t want to actually get into the field.

This is a pretty fundamental disagreement that I have with a lot of people. Just for example, I recently got into a discussion with an authority who insisted that it wasn’t possible for me to reasonably disagree with them (I suggested we agree to disagree) because they’re an authority on the topic and I’m not. It happens that I worked on the formal verification of a system very similar to the system we were discussing, but I didn’t mention that because I don’t believe that my status as an authority on the topic matters. If someone has such a weak argument that they have to fall back on an infallible authority, that’s usually a sign that they don’t have a well-reasoned defense of their position. This goes double when they point to themselves as the infallible authority.

I have about 20 other posts on stupid sounding ideas queued up in my head, but I mostly try to avoid writing things that are controversial, so I don’t know that I’ll write many of those up. If I were to write one post a month (much more frequently than my recent rate) and limit myself to 10% posts on ridiculable ideas, it would take 16 years to write up all of the ridiculable ideas I currently have.

Thanks to Leah Hanson, Hillel Wayne, Robert Schuessler, Michael Malis, Kevin Burke, Jeremie Jost, Pierre-Yves Baccou, Veit Heller, Jeff Fowler, Malte Skarupe, David Turner, Akiva Leffert, Lifan Zeng, John Hergenroder, Wesley Aptekar-Cassels, Chris Lample, Julia Evans, Anja Boskovic, Vaibhav Sagar, Sean Talts, Valentin Hartmann, Sean Barrett, Kevin Shannon, Enzo Ferey, Yuri Vishnevsky, and an anonymous commenter for comments/corrections/discussion.>