When Draft Pundits Are Right, and N.F.L. Teams Wrong

It’s every N.F.L. fan’s nightmare. After months of anticipation, your team’s first-round pick in the N.F.L. draft (the 2022 edition starts tonight) is deemed a “reach” — a player picked well before where the experts thought he should be selected.

From the draft guru Mel Kiper Jr. on ESPN to the local bloggers on Twitter, everyone seems to think your team blew it. There were at least a dozen better players on the board, they say. The player might have even been around in the next round, they say.

But are they right? It turns out they might be.

In this case, it’s worth taking the pundits and the blowhards seriously, based on an analysis of ESPN predraft grades over nearly 20 years.

The outcomes have not been great for teams using a top draft pick on a reach — defined here as a nonquarterback whom ESPN ranks at least 10 spots lower than the team’s draft slot in the top half of the first round. Overall, these reaches have yielded below-average picks in 17 of 19 cases — or 89 percent of the time — since 2005.

The draft gurus do not always have all the answers, of course. Other analysis has found that their post-draft grades are essentially worthless. Many of the players the experts consider “steals” — the highly regarded players who unexpectedly fall to a seemingly lucky team — are not what they are cracked up to be. And teams do a better job of finding hidden gems later in the draft. As a result, the teams ultimately beat the analysts over the course of the draft, if only modestly. If the analysts don’t love your team’s draft, it very well might still turn OK — or even great.

But despite superior resources — and despite being paid the big bucks to get it right — the teams often err when they defy media experts in the draft’s highest-stakes moment: the top half of the first round.

The stakes are so high because the draft is not quite the crapshoot that it can sometimes seem. Despite the occasional Tom Brady unearthed in the mid-to-late rounds, a high first-round pick is much likelier to end up as a good football player than later selections. Even when teams are convinced they have found a diamond in the rough, they could probably trade back to a later selection, and receive additional draft picks while obtaining the same player.

For fans, a first-round reach is so noteworthy because it can spark a media firestorm. A seeming “steal” may not turn out to be a steal, but the pundits and your team appear to be in agreement. There will be no controversy; there probably will not be too much second-guessing if the pick does not pan out.

Yet teams rarely beat the conventional wisdom when they reach for a low-ranked player in the first round, despite all of the incentives to select a reach only when the team is absolutely sure.

The experts have a worse record in a seemingly similar case: when several teams pass on a player talked up by analysts in the days and weeks ahead of the draft. These so-called steals can seem like a tremendous value for the lucky team that lands them. Your team might be praised on talk radio. But these picks don’t tend to turn out unusually well. On average, the steals offer no greater value to teams than one would expect for their draft slot.

Why do the experts offer value over the teams in some cases but not others? Timo Riske, a data scientist for Pro Football Focus who reached similar conclusions in an analysis of drafts since 2013, observed that “when a player is a steal, it means that several teams — maybe even all 32 teams — passed on him,” while a reach could mean that only a single team liked a player more than experts.

“Betting against the N.F.L.’s overall assessment of players is essentially saying your evaluation of a player is better than everyone else” said Ben Robinson, the founder of a website that analyzes the draft. “It’s not sustainable.”

The analysis relies on Pro Football Reference’s Approximate Value metric, an imperfect but useful measure of a player’s contribution to a team’s success. For the remainder of this article, a reach or a steal is a player selected in a slot with an expected approximate value that’s fivepointshigher or lower than one might guess from ESPN’s ranking. This clunky definition helps account for the greater differentiation of players at the top of the draft: Selecting ESPN’s 10th-best player with the first overall pick is considered a reach by this measure, but a player would have to be ranked 170th or lower by ESPN to be considered a reach with the 100th pick.

The analysis also excludes quarterbacks, a position that poses unusual challenges for draft analysis. On one hand, many teams will pass on a highly regarded quarterback if they already have what they view as a good quarterback. On the other hand, teams rank quarterbacks with the position’s unusually high value in mind while many pundit draft boards do not.

A player projected as merely a league-average starting quarterback may be far more valuable than a player projected as, say, an above-average cornerback. But the cornerback could be ranked higher on a draft board. And a team that “reaches” on a quarterback could appear to be vindicated by the data, even if the player does not pan out, simply because of the position’s high value.

One example is Daniel Jones, who was ranked 59th by ESPN before he was selected by the Giants with the sixth overall pick in 2019. By this measure, it’s the second-biggest reach in the top half of the first round since 2005. The pundits declared the selection a “mistake,” and they might be right. Many analysts have already labeled him a bust, as he has posted below-average numbers every year he has been in the league. But he’s a starting quarterback; by the approximate value measure and others, he’s on track to post good enough value for the selection.

As the draft goes on, the outside analysis gradually loses its value. As soon as the second round — and for the remainder of the draft — you can largely expect pundits to be of little service. The steals still aren’t steals, but the purported reaches aren’t reaches anymore: A player taken 50th overall but ranked 100th is likely to be about as valuable as he ought to be given his selection. That doesn’t mean the teams are brilliant; the draft really does become more like a crapshoot by the time you get into the middle rounds. But you don’t need to pay very much attention to your team’s critics — or praisers.

At this late stage, the superior resources of the teams may give them a better chance than the public analysts to identify value. And teams may select players with a specific idea of how they might capitalize on a player’s strengths and minimize weaknesses.

The teams also control playing time. A third-round reach is likely to be given the ample playing time of a third-rounder; a seventh-round steal might be treated like a fringe player, even if the analysts were right about that player’s upside.

In the most extreme case, a possible win for the experts might never get a chance to prove himself.Imagine if Brady had ultimately gone undrafted — he went 199th out of 254 picks — even though the experts had ranked him as a middle-round pick, and then imagine he never played a game.

By the data, the experts would have seemed wrong to rank him so high.

Josh Katz contributed reporting.