Six Habits of Highly Effective Testers

09/06/2024

The past year has been a phenomenal one for me, in terms of competitive Magic results. I qualified for my first PT playing limited, and then proceeded to do very well for my first three PTs in a row - one 14th and two top 8s!

There are many things I could contribute my success to - perhaps a strong background in limited, or perhaps the privilege of having a lot of free time from being unemployed with a cushion of savings from a year of working in tech. But, in my opinion, the most important factor by far was having a good testing team to work with.

I cannot stress enough how amazing it has been to work with Sanctum of All. Throughout the past year, I've been getting more and more involved with Sanctum, and have started to take a more active leadership role to boot. Being able to share ideas with, learn from, and work alongside other Magic players who are also looking to learn and improve at the game has been essential to my growth in Magic.

So, as Regional Championship DC is coming up in a month, I figured it would be a good time to talk about the ins and outs of effecive testing. How do you make the most out of the limited time that you have? How do you make sure that the conclusions you come to are good ones? How do you separate the signal from the noise?

---

Habit #1: Make a Plan

Before you start testing, it can be extremely helpful to make a concrete plan. Figure out what your goals are, and what you want to focus on in the time you have before the tournament. Do you have a specific deck you're locked on that you want to refine to be the best that it can? Do you want to explore the metagame and figure out which deck(s) are best positioned? Do you want to brew something new to attack the metagame?

Make sure that the goals of your testing line up with the scope of how much time you have to spend. If you only have 10 hours total of testing before a tournament, you probably can't use that to build and refine a completely new brew from scratch. But if you're planning on spending hundreds of hours testing for a big event, you probably should expand your focus at least a bit further than a single stock deck.

When considering your plan for testing, it's also very important to keep in mind when various cut-off dates are. If you're playing in the physical tournament, you probably can't wait until two days before the event to lock in which deck you're playing. If you want to be able to get physical cards, you probably need to at least have some idea of what you're playing a week before the event.

So, you should naturally expect to spend the earlier stages of testing figuring out what deck(s) you're interested; and focus the later stages of your testing on tuning the deck(s) that you've settled on.

Regardless, the important part is to have some idea of you want to work on before jumping into testing. Having a sense of direction is key to not getting sidetracked.

---

Habit #2: Fail Forward

One of the easiest ways to get bad testing is to let your ego get in the way.

The entire purpose of testing is to learn. It doesn't matter who's right, or who's the best Magic player. The point is to find the truth, and to improve. And in order to do that, you must be willing to accept that some of the preconceived notions you have might be wrong.

Likewise, it's also extremely important to treat those you're testing with with proper respect. Even if you think you're a "better Magic player" than someone (and boy is that phrase extremely fraught and meaningless on its own), they almost certainly bring a new and valuable perspective to the table that you could not replicate on your own.

This is, in fact, one of the most valuable parts of being on a testing team. No one has the time or energy to test everything by themselves. No one is going to have a complete perspective on all of Magic. As it turns out, Magic is a absolutely enormous game - even a single draft format, limited to a single set, has more moving parts and interactions than you can expect a single person to fully explore and master by themselves over the course of a couple of months.

So, don't be ashamed about being wrong! Everyone is going to be wrong in testing, and being wrong in testing is completely fine - if anything, it's encouraged. Testing is the perfect place to try out wild ideas that might fail spectacularly, in order to find the few that actually pay off.

In order to learn new things, you must be willing to try new things. And trying new things will always come with some risk of failure. So if you want to be effective at testing, you must be prepared to fail, over and over again, in order to give yourself a better chance at success when it really matters.

---

Habit #3: Use Nuance, Not Hyperbole

Magic players are notorious for hyperbole. "This card is busted!", "This card is strictly better!", "This matchup is free!" - these are all statements that you'll commonly hear out of the mouths of Magic players.

And these statements are usually harmless. When you say that a matchup is free, it's generally understood that it's not a literal 100% winrate matchup - just very favored. When you say that the card is strictly better, it's generally understood that there might be some niche situations where it isn't better.

But what these hyperbolic statements always lack is nuance - and nuance is key in testing! So when you're trying to communicate during testing, you should really try your best to avoid hyperbole as much as possible. Hyperbole is useful for communicating exaggerated emotion, and a serving as shorthand. It has its place in normal, casual conversation - but in testing, it is well worth the extra effort to add nuance and context to your statements.

So instead of describing a matchup as "free", perhaps try putting a number on it. Is it an 80% matchup, or is it more like 60%? Even better, describe why you came to that conclusion. What makes the matchup so favored? What are the important play patterns and cards in the matchup that lead to one deck winning more than the other? What experiences did you have that led you to make this claim?

---

Habit #4: Avoid Garbage Time

Testing time is always limited and always quite precious. Even speaking as someone who is unemployed and spends most of my time on Magic, there are not nearly enough hours in the day to do everything I want in testing. There will always be more brews to explore, more matchups to test, more sideboard plans to experiment with. As such, it is often quite important to use testing time efficiently.

One very common way of doing this is to cut down on "garbage time" during testing. In this context, "garbage time" refers to time during a game of magic where the game technically hasn't ended yet, but there are very few remaining interesting decisions. For instance, many control decks often end up extending the game into garbage time - they essentially win the game after they establish a nigh-unassailable controlling position, but then take quite a few turns to actually end the game.

A common refrain in competitive magic is to always play to your outs, no matter how unlikely they are. And this is extremely good advice for when you're actually playing in a competitive setting! But the incentives in testing are simply different. You have nothing to lose and everything to gain by playing out garbage time in an actual tournament; but during testing, everything costs time. So in testing, if you feel that you are no longer getting anything productive out of playing a game, feel free to simply concede. The goal is not to win, but to learn.

I will caution, though, that it is important to not over-apply this concept. It can still be useful to play out games that look mostly determined. The important question is whether you are learning anything from the experience. For example, sometimes you might want to go through the motions of a combo in testing, in order to practice going through and explaining lines. If you feel that that's an area that you need to improve, then it's very reasonable to spend testing time doing so - you just shouldn't feel obligated to.

---

Habit #5: Bend the Rules

As I keep stressing, a testing environment is not a tournament. You are practicing for a tournament, but practice does not need to completely match reality to be useful. And in fact, the most effective methods of deliberate practice often involve breaking things down, and focusing on individual parts rather than the whole.

So do whatever you need to do in order to get the testing that you need. Don't be afraid to bend and break the rules of Magic to better suit your purposes.

If you make a simple mistake - such as missing some obvious line, or accidentally doing the wrong thing when you meant to do something else - don't be afraid to back up and rewind as much as you want. Obviously, you might make such mistakes at the tournament as well, and often won't be able to backup then - but often it's more useful testing to simulate what happens if both players are playing tightly.

Similarly, after a game is over, consider going back and simulating what would've happened if you had taken different lines, or had different draws. This can be a very useful tool for the purpose of exploring many different options and comparing them to each other.

If there is a specific card you want to get more data on in testing, one thing you can do is to intentionally manipulate where it is in your library to make you more likely to draw it. If you want to see how good a card is in your opening hand, consider simply forcibly putting it in your opening hand for a few games to try it out. If you want to see what happens when you draw it later, consider shuffling it into the top 20 cards of your library at the start of the game. If you want to see how the card performs when you draw multiple copies, consider shuffling multiple copies into your top 20.

Incidentally, testing tricks like these are why I am such a big fan of looser, non-engine-based platforms like Cockatrice (or paper testing!). Arena and MTGO are great tools, but they are very rigid with how they implement the magic rules engine. This can be both a benefit and a detriment – on one hand, they both handle the rules engine for you, so you don't have to think about it; but on the other hand, if someone accidentally passes through their turn, you have no way to fix that.

So don't be afraid to get creative! Do whatever you need to, in order to improve what you want to. Don't let the bounds of Magic's rules limit you.

---

Habit #6: Prioritize Play Patterns

Finally, keep in mind that the most useful conclusions you can come to from testing are always about play patterns, and not raw data.

Knowing that a certain matchup is a "60% matchup" can be useful, but at the end of the day, it doesn't actually help you play games. Instead, a description of how games play out is much more valuable. Which cards are important? How are certain cards used? What is your sideboard plan? Why is that your sideboard plan?`

And again, this comes back to the concept of nuance from earlier. Give a player a good deck, and they'll probably do okay. Teach a player how to play that deck, and they'll do much much better.

So, when communicating results from testing sessions, be sure to focus on the "Why?", not just on the "What?". Don't just report what your record was – write about how those games played out, and what conclusions those games led you to make.

And don't be afraid to express uncertainty! Magic is a game full of uncertainties that we can never truly fully resolve. It's important to recognize that you very rarely will find fully concrete and solid claims in testing - so expressing not-fully-certain conclusions is still valuable!

The best we can do is develop and refine strategic theories, testing those theories and changing and improving them based on the results. We can never fully reach perfection, but we can strive towards it - and doing so requires nuance.