I’ve owed a guy this study since last summer, so let’s get right into it.
[EDIT: See Andrew Bucholtz's article based on this study here.]
In the NFL, coaches are notorious for conservative play-calling on fourth down. Brian Burke at Advanced NFL Stats had a detailed study on that topic in the fall of 2009, and a few months later the Patriots’ Bill Belichick rather famously chose to gamble on a 4th-and-2 situation (and lost). There have been similar calls for years (with similar debate after they worked, or didn’t), and Burke points to other studies, one of which was published in 1971, so this is hardly a new area for analysis.
What hasn’t been studied, at least to my knowledge, is the equivalent situation in Canadian football. Even if you know when to go for it on fourth down in the NFL, that knowledge doesn’t really apply to the CFL game — yes, our balls are bigger, but we also have larger fields, different rules on offensive motion, more space between the offensive and defensive line, the ability to earn a point on punts or missed field goals, and (most critically) only three downs. You pretty much have to do an entirely new study using CFL data.
To that end, one thing the CFL thankfully shares with the NFL is published play-by-play data. So I took a few warm evenings last summer, sat out on the balcony, and coded it all up.
Part 1: What’s a first down worth?
Before we can answer whether a team should try to get the first down, we need to know what that first down is worth to them. To do that, we use Expected Points (EP) — to quote Brian Burke, “the average potential points a team can expect given a certain situation.”
Imagine a first and goal at the 1. Intuitively, the EP value there is almost 7, because in most situations you’re going to score and convert a touchdown. However, teams don’t always do so — whether due to turnovers, large losses on sacks leading to field goals, or being coached by Charlie Taaffe. And even if they do score, they have to give the ball back to their opponents, which is worth something positive for them.
And the farther away you are from your opponent’s end zone, the lower your Expected Points will be. Here is a chart with the expected number of points based on first-down field position:
The points are observed averages (the average of the next score following that first-down position); however, we’ll use the linear best-fit for the rest of this article. Though we should note, even if just in passing, that starting inside your own 10 might be worse than that best-fit implies.
The breakeven point (the point at which each team is equally likely to score next) is the offensive team’s 23 yard line. That makes perfect sense: after all, teams start from the 25 after a single point.
Let’s take a play from the 2010 Grey Cup as an example. In the first quarter, on their first drive, the Als had a 3rd-and-1 from the Riders’ 47 (play #5). What are their possible outcomes here?
- First down at their opponents’ 48: +2.8 points
- Turning the ball over on the 47: -2.3 points (that is, 2.3 points to Saskatchewan for starting at their 47)
- A made field goal: +2.2 points (3 minus the Riders’ expected field position after the FG, at their own 35)
- Punting: based on net punting averages (punt distance minus return distance, by field position), the Riders would get the ball on their own 8, so that’s +1.0 for Montreal
There are other possibilities, too, but those are enough for an illustration.
The best decision, assuming it works out, is the first down. But “assuming it works” is the key phrase there: we have to consider the success rates for each of these scenarios. Making a 55-yard field goal is unlikely, so that +2.2 is unlikely to occur.
So let’s look closer at the three main decisions: whether to go for it, punt the ball away, or attempt a field goal.
Part 2a: Converting 3rd-and-something
This table shouldn’t need much explanation:
|Yards to go on 3rd down||2009 CFL success rate (%)|
“Wait a second, Rob. Are you saying teams convert 3rd-and-10 more than one-third of the time? That’s impossible.”
That’s what I thought too. Technically, those are smoothed values, much like our EP chart above (though the empirical results are similar). And it’s based on only one year of data, something I hope to change soon. And there’s probably quite a bit of bias here: aside from teams who are desperate or don’t care if they fail, who goes for it on 3rd-and-10? Teams that think they can succeed. So these values might be entirely useless (though, as the guy who spent hours figuring them out, I’m happy to assume they’re in fact perfect).
It might be too high, but 36.7% is still not a great success rate. Not to give away the ending, but eventually we’ll find that going for it on 3rd-and-10 is almost never a good idea. So it shouldn’t hurt too much for us to take those numbers on faith for now.
Back to our Grey Cup example. Given the success rate above, if Montreal chose to go for it, there was an 80.4% chance that they would gain those 2.8 points based on field position, and consequently a 19.6% chance they’d fail and end up “giving” 2.3 points to the Riders. So the EP value would come out as +1.8. That’s quite high, which means, if I were standing near Marc Trestman at the time, I’d be dropping hints about how good Avon Cobourne is looking today and how I’m sure he can gain three feet.
Part 2b: Field-goal success rates
Just a quick note first: like with any “average” rate, the field-goal success rate will be higher if you’re a better team (say, you have a significantly better kicker). Not to mention the effects of favourable wind and weather, the quality of the field you’re playing on, or the ability of the other team to block your kick or cause your holder to rush and set up incorrectly. So all of this analysis assumes average players on an average field on an average day.
Field goals in the CFL are pretty successful as long as the line of scrimmage is inside the 30. I have 181 field goals attempted from inside that distance, 169 of which were good. It starts to get hairy around the 40 (where the odds fall below 4 in 5), and the dropoff is steep after that: the 50% point is around the 55 yard line. So kicking a field goal is really only a realistic option when you’re inside the 45 — again, depending on the factors mentioned above.
Another reason that field goals are better to attempt the closer you get is the single point, that quirky Canadian rule (which probably doubled the length of time I spent on this study…totally not a fan of the rouge right now). Miss a 15-yarder? No problem: your kick probably went out the back of the end zone anyway, and you get a point.
Any single point, by the way, is technically worth 0.8 points: 1 point (duh) minus possession at the 25-yard line for the other team, which is a fifth of a point for them.
So how would I have advised Mr. Trestman in this case? He had the ball on the Riders’ 47, where the success rate is about 57% (once again, maybe higher with Damon Duval, maybe lower based on the wind, etc.). If he makes it, that’s +2.2 (from above). If he misses it, the probability of a rouge is pretty small, so let’s assume the Riders return it to their own 21. (That return average is based on punt return averages, adjusted for the fact that FG teams aren’t great at going downfield to tackle the kick returner.)
57% times +2.2 plus 43% times -0.2 works out to +1.2 points, lower than the expected value of going for it. I would have told Trestman not to kick a 50+ yard field goal, and he would have wondered who let me on the sideline.
Part 2c: Punting your chances away
I’ll admit I don’t like punts. Especially in the CFL. “Rush for no gain, incomplete pass, weak punt, no yards penalty” seems to be the standard drive sometimes. So I am generally going to take any evidence I can get to “prove” that punting is not worth it.
In 2009, the average punt went 43.3 yards. The average return was about 5.4 yards, for a net punt distance of 37.9. This distance varies with field position: if you’re deep in your own end, you obviously have less space to execute a punt (and some teams don’t even bother: they just give up the safety). Though punting from your own end is not as damaging as you’d expect: the average net distance is 36.5 on all punts from inside the team’s 15. Again we probably have some self-selection bias: teams select themselves as a punting team on that play because they think they can gain enough yards on the punt, rather than avoiding the risk and just conceding two points. (The fact that giving up two points, and possession, is very damaging unless those points don’t matter seems to be lost on most coaches…)
There is obviously a lot of variability on any one punt, but we’ll use expected averages here for the decision-making. Once again going back to the first drive of the Grey Cup, the Als chose to punt from their opponent’s 47. The expected field position for the Riders is their own 8: 47 plus the 39 expected net punt yards. Pinning a team at their own 8 is worth, on average, +1.0 points for Montreal.
Part 2d: Putting it all together
If you’ve been following along, we have the following EP values for each of the Als’ potential decisions in that game:
- Go for it: +1.8
- Attempt a field goal: +1.2
- Punt: +1.0
They’re all positive because it’s generally a good thing to be in your opponent’s territory. But the best decision appears to be going for it. Punting would cost Montreal nearly a point in the long run.
Of course they punted. Wrong choice (by the numbers). I would have told Trestman that, and he’d wave to an offensive lineman, point to me, and indicate that I should be removed, forcibly, from Commonwealth Stadium, and deposited in one of Edmonton’s darker alleys.
But what actually happened? Saskatchewan took over on their own 14 after the punt, and I’d love to say that they then went down and scored, showing Montreal what’s what, but that’s not the case, and it would be too clever by half for me to cherry-pick a poor decision. (I just picked the first third-down play in the first quarter.)
No, the Riders punted away (giving up 15 extra yards on a no-yards call), and then Montreal had just 38 yards to go before they scored. And they did.
Was that first Montreal punt really the wrong decision, then?
Well, not if you’re favouring process over outcome. The idea of this third-down analysis is that it works in the long run, because it’s about field position. Coaches are always talking about the importance of field position; yet many of them prefer to punt and give up position rather than take a small risk and maintain possession.
This analysis does not hold near the end of the half, or the end of the game: it’s not that important to worry about the long-term drawbacks to your decisions, when there isn’t a long run because the clock is running out.
For most situations, though, this should work as a quick guide as to which decision, based on the data available, has worked out the best in the recent past.
Stepping back from that one specific play, we can put all the numbers together and figure out when it’s best to go for it in general, based on how far you are from the end zone and how many yards you need to gain to move the sticks. Here’s a chart of the breakeven points (based on Burke’s chart for the NFL).
The thick line is the breakeven point between going for it and bringing out the special teams. Anything below it, you should go for it on third down. Anything above it (and within field-goal range), you should kick a field goal. The rest of the time, punt.
Obviously there is no such thing as a 3rd-and-10 from the 8, but the breakeven point being higher than that means you should always go for it when you’re inside your opponent’s 10. (As long as it’s not the end of a half or you need the three points.) Other sound strategies appear to be always risking it on 3rd-and-4, regardless of field position, and even going for it on 3rd-and-8 from the 25.
If we assume those decisions above are the “right” ones then couldn’t we go through all the third-down situations and figure out which teams made the right call most often? Yes. Yes we could. But that’s a post for another day.