DawgTalkers.net Forums DawgTalk Pure Football Forum Total Quarterback Rating

Quote:

---

The late Don Smith never claimed his passer-rating formula was perfect.

Quite the opposite, in fact.

"Some people call it a quarterback rating system, but that really is not what it is," Smith told me during a 2002 interview. "It’s simply a passing statistic."

I've actually defended Smith's rating system because the quarterbacks with the highest ratings -- Tom Brady, Philip Rivers and Aaron Rodgers led the way last season -- usually are the best quarterbacks. But there's so much more to quarterbacking than passing stats for touchdowns, interceptions, attempts, completions and yardage.

Game situations should count for something, and now they do.

With input from football people, including ESPN analyst Trent Dilfer, our statistical analysts have developed a 100-point ratings scale for quarterbacks taking into account advanced stats, game situations and relevant non-passing stats, including fumbles and sacks, to evaluate quarterbacks far more thoroughly. The methodology is complex -- one of the formula's key algorithms spans some 10,000 lines -- but the resulting "Total Quarterback Rating" (QBR for short) beats the old passer rating in every conceivable fashion. The ratings scale will debut this season.


[+] Enlarge
Al Bello/Getty Images
According to an outline for the rating system, Tom Brady would fall in the "top tier" category.I've been bugging the Stats & Information team for a sneak peak ever since learning former NBA statistical analyst Dean Oliver had joined our production analytics unit and was playing a prominent role in QBR development. Oliver, a Caltech grad with a Ph.D. in statistical applications, revolutionized how NBA teams use advanced statistics. Menlo College professor Ben Alamar, who has consulted with the San Francisco 49ers, is also part of the team.

Our stats team has been using game video to track stats relating to pressure, personnel, formation, game situation and more since 2008. The QBR stat represents a significant leap in harnessing those statistics for something more.

The old formula Smith created treated stats the same regardless of circumstance. A touchdown pass thrown against a prevent defense during a blowout defeat equals one thrown against pressure to win the game. A 5-yard completion on third-and-4 counts the same as a 5-yarder on third-and-15. A critical quarterback scramble, sack or fumble doesn't even factor.

"There is no way to statistically say how effective a guy is under fire," Smith lamented during our 2002 conversation. "None of that can be put into something like this."

Now it can, along with a whole lot more.

The QBR formula takes into account down, distance, field position, time remaining, rushing, passing sacks, fumbles, interceptions, how far each pass travels in the air, from where on the field the ball was thrown, yards after the catch, dropped balls, defensed balls, whether the quarterback was hit, whether he threw away the ball to avoid a sack, whether the pass was thrown accurately, etc. Each play carries "clutch weight" based on its importance to game outcome, as determined by analyzing those 60,000 plays since 2008. The stats adjust for quarterbacks facing an unusually high number of these situations.


Year of the Quarterback ESPN has dedicated 2011 to examining one of the most crucial positions in all of sports -- the quarterback.
Year of the QB »
"If it is a running clock late in the game, maybe you only get a few yards here or there, that is the right football play to make," Jeff Bennett, senior director of ESPN's production analytics team, said Sunday. "We spent a month learning about ratings to make sure quarterbacks couldn’t game the system, so they're not afraid to throw that deep pass at the end of the first half and risk an interception."

I've seen an outline for the rating system breaking down 2010 quarterbacks into six general categories, from top tier to poor. Precise rating numbers were not yet available. The quarterbacks under consideration broke down as follows:


•Top tier: Brady, Peyton Manning, Matt Ryan, Michael Vick, Rodgers and Drew Brees.


•Well above average: Josh Freeman, Eli Manning and Philip Rivers.


•Above average: Ben Roethlisberger, Tony Romo, Joe Flacco, Matt Schaub, David Garrard and Kerry Collins.


•Around average: Matt Cassel, Ryan Fitzpatrick, Mark Sanchez, Carson Palmer, Colt McCoy, Kyle Orton and Jon Kitna.


•Below average: Shaun Hill, Jason Campbell, Jay Cutler, Matt Hasselbeck, Chad Henne, Donovan McNabb, Sam Bradford and Alex Smith.


•Poor: Derek Anderon, Brett Favre and Jimmy Clausen.

ESPN plans to enlist several quarterbacks when introducing the stat during an hour-long "SportsCenter" special Aug. 5 at 8 p.m. ET. We'll be referencing the stat on the blogs and elsewhere. Bennett said he's allocating enough manpower to produce ratings on game days, a huge help for those of us analyzing player performances shortly after games.

"We want to reward a good football play," Bennett said.

http://espn.go.com/blog/nfcwest/post/_/id/41811/how-to-identify-nfls-best-quarterbacks

---

Quote:

---

Stat (News) of the Week
ESPN will release this week a proposal for a new passer rating, called the Total QBR, or Total Quarterback Rating. It has been developed by several quarterbackmeisters at the network -- most notably Trent Dilfer -- and by some stat heads in the network's production analytics department. The point: Passer rating, developed in 1973 to measure a passer's efficiency, does that, but it doesn't necessarily measure what makes a quarterback great.

So the analysts at ESPN have taken every game played in the NFL since 2008 and measured the quarterback's contribution to the result on every play except handoffs. They say they've divined a system to rate quarterback performance in every game, and for full seasons, on a scale of 1 to 100 (no more 158.3 rating).

"This is a game-changer,'' Dilfer said. "Mark my words: This is the number scouts and coaches and the media will use to quantitatively discuss and judge the ability of quarterbacks going forward.''

If it sticks, of course. You know how the sporting public (and the larger American public) is with new ideas. But judge for yourself. ESPN will explain the proposed Total QBR in a special Friday night show at 8 Eastern with Dilfer and the Monday night crew -- Mike Tirico, Ron Jaworski and Jon Gruden.

Passer rating has probably declined in significance, with some of the smarter analysts all but ignoring it. But it still has juice. The problem, of course, is that in passer rating, so much is ignored. A series of short completions with big yards after the catch mean the same thing as a series of harder-to-complete long passes. The ESPN formula weighs out the yards after the catch, and weighs in things like lost fumbles in the pocket and sacks taken. And timing. A 17-yard completion in a tie game with two minutes left gets a quarterback more credit than the same completion in the middle of the first quarter -- as it should.

"We've included every play that a quarterback has direct control over,'' said Jeff Bennett, the senior director of production analytics at ESPN. "We think a rating system should evaluate all the quarterback's contributions in the context of the game.''

It's be interesting to see if the Total QBR gets traction and usurps passer rating. I think we're ready for a system that scores on a scale of 1 to 100, takes more factors into account than passer rating, and involves the measurement of so-called clutch play. Like OPS (on-base plus slugging) and WAR (wins above replacement value) in baseball, it's time for a more thoughtful number to judge how quarterbacks play. Is this the one? I don't know. But I like rethinking passer rating.



Read more: http://sportsillustrated.cnn.com/2011/writers/peter_king/08/01/camps/index.html#ixzz1Tn9jwg1i

---

Quote:

---

•Poor: Derek Anderon, Brett Favre and Jimmy Clausen.

---

But they're GUNSLINGERS!!!

Quote:

---

Quote:

---

•Poor: Derek Anderon, Brett Favre and Jimmy Clausen.

---

But they're GUNSLINGERS!!!

---

Quote:

---

So... Trent Dilfer made this?

Do you get points for handoffs?

---

Quote:

---

So... Trent Dilfer made this?

Do you get points for handoffs?

---

Quote:

---

Early in a scoreless game, a quarterback throws a 20-yard pass just by the reaching arms of a defender and into the hands of his intended receiver, who holds on despite the distraction, then scampers the remaining 15 yards for a touchdown.



Another quarterback, down 30-10 with five minutes left in the fourth quarter, throws a 3-yard screen pass to a running back, who maneuvers another 32 yards through prevent defense to pick up a first down deep in opponent territory.



Year of the Quarterback ESPN has dedicated 2011 to examining one of the most crucial positions in all of sports -- the quarterback.
Year of the QB »


Both are called good plays, but labeling them as "good" isn't enough. Each play has a different level of contribution to winning, and each play illustrates a different level of quarterback contribution. What is the quarterback's contribution to winning in each situation? Coaches want to know this, players want to know this and fans want to know this.



The Total Quarterback Rating is a statistical measure that incorporates the contexts and details of those throws and what they mean for wins. It's built from the team level down to the quarterback, where we understand first what each play means to the team, then give credit to the quarterback for what happened on that play based on what he contributed.



At the team level, identifying what wins games is not revolutionary: scoring points and not allowing points. Back in the 1980s, "The Hidden Game of Football" did some pioneering work on that topic and how yardage relates to points. We went back and updated what that book did … then we went further. At the individual level, more detailed information about what quarterbacks do is really necessary. Brian Burke at AdvancedNFLStats.com has done very good work in advancing that effort, and FootballOutsiders.com has done some of this by charting data, but over the past three years ESPN has charted football games in immense detail. By putting all these ideas together and incorporating division of credit, we have built a metric of quarterback value, the Total Quarterback Rating, either Total QBR or QBR for short.



What follows is a summary of what goes into QBR. It took several thousand lines of code to implement, but we'll keep this shorter.



Win Probability and Expected Points


The goal behind any player rating should be determining how much a player contributes to a win. We went back through ten years of NFL play-by-play data to look at game situation (down, distance, yard line, clock time, timeouts, home field, field surface and score), along with the ultimate outcome of the game, to develop a win probability function.



Total QBR BasicsA quick primer on the fundamentals of Total Quarterback Rating:

Scoring: 0-100, from low to high. An average QB would be at 50.
Win Probability: All QB plays are scored based on how much they contribute to a win. By determining expected point totals for almost any situation, Total QBR is able to apply points to a quarterback based on every type of play he would be involved in.
Dividing Credit: Total QBR factors in such things as overthrows, underthrows, yards after the catch and more to accurately determine how much a QB contributes to each play.
Clutch Index: How critical a certain play is based on when it happens in a game is factored into the score.


This function treats every win the same, regardless of whether it was 45-3 or 24-23, though there is clearly a difference between such games. The first game represents total domination, whereas the other represents two fairly evenly matched teams. Because win probability treats every win the same, it misses some of what goes into the win, specifically many of the points that represent domination or the points that lead up to a last-second victory. So, although QBR uses win probability to assess how "clutch" a situation is, it uses expected points as the basis of evaluating quarterbacks. It has more of the details, and understands the difference between wins, but still strongly relates to wins in general.



The concept of expected points was discussed as early as the mid-1980s with Pete Palmer and "The Hidden Game of Football," where they talk about "point potential." Their idea was that, as you move closer to the opponents' end zone, you are actually gaining points. Brian Burke took it further to note that third-and-10 from midfield, for instance, has fewer expected points than first-and-10 from midfield. In other words, down and distance also matter in terms of points. We took this even further to look at clock time, home field, timeouts and field surface to generate the expected points for any team given its situation in a drive. One particular situation to note is that, at the end of the half, a team is less likely to score any points than at most times of the game, just because the half is going to expire.



It's useful to mention here that expected points are expected net points. It's possible that a team has expected points less than 0. This simply implies that the other team is generally more likely to score. This usually happens when a team is backed up deep in its own side of the field, especially if it is third or fourth down.



What then happens is an evaluation of expected points added. How does a team go from 1.1 expected points to 2.1? However they do it, that is 1.0 expected points to be distributed to the offensive players on the field. But how they do it is what determines how credit is given to a quarterback.



Dividing Credit


Division of credit is the next step. Dividing credit among teammates is one of the most difficult but important aspects of sports. Teammates rely upon each other and, as the cliché goes, a team may not be the sum of its parts. By dividing credit, we are forcing the parts to sum up to the team, understanding the limitations, but knowing that it is the best way statistically for the rating.



On a pass play, for instance, there are a few basic components:
• The pass protection
• The throw
• The catch
• The run after the catch



In the first segment, the blockers and the quarterback have responsibility for keeping the play alive, and the receivers have to get open for a QB to avoid a sack or throw the ball away. On the throw itself, a quarterback has to throw an accurate ball to the intended receiver. Certain receivers may run better or worse routes, so the ability of a QB to be on target also relates somewhat to the receivers. For the catch, it may be a very easy one where the QB laid it in right in stride and no defenders were there to distract the receiver. Or it could be that the QB threaded a needle and defenders absolutely hammered the receiver as he caught the ball, making it difficult to hold on. So even the catch is about both the receiver and the QB. Finally, the run after the catch depends on whether a QB hit the receiver in stride beyond the defense, and the ability of a receiver to be elusive. Whatever credit we give to the blockers, receivers and quarterback in these situations is designed to sum to the team expected points added.



The ESPN video tracking has been useful in helping to separate credit in plays like these. We track over-throws, under-throws, dropped passes, defended passes and yards after the catch. The big part was taking this information and analyzing how much of it was related to the QB, the receivers and the blockers. Not surprisingly, pass protection is related mostly to the QB and the offensive line, but yards after the catch is more about what the receiver does. Statistical analysis was able to show this and we divided credit based on those things.



As a relevant side note, statistical analysis showed that what we call a dropped pass was not all a receiver's fault either. A receiver may drop a ball because he wanted to run before catching it, because the defense distracted him, because it was a little bit behind him or because he was about to get hit by a defender. If the defender was there a half second before, the defender would have knocked the ball free and it would have been called a "defended pass", not a "dropped pass." There are shades of gray even on a dropped pass and analysis showed that. Drops are less a QB's fault than defended passes or underthrows, but the QB does share some blame.



On most other plays, quarterbacks receive some portion of credit for the result of the play, including defensive pass interference, intentional grounding, scrambles, sacks, fumbles, fumble recoveries (Carson Palmer once recovered a teammate's fumble that saved the game for the Bengals) and throwaways. On plays where the QB just hands off to a running back, we didn't assign any credit to the QB. Our NFL experts did suggest that some QBs are very good at interpreting defenses pre-snap and identifying better holes for their backs.



However, they also told us that it would be nearly impossible to incorporate. Because they suggested this, we built in the ability to give credit for QBs when they just handed off, but we couldn't find the right analysis to do it in 2011.



Clutch Index


The final major step is to look at how "clutch" the situation was when creating expected points. A normal play has a clutch index of 1.0. For instance, first-and-goal from the 10-yard line in a tie game at the start of the second quarter has a clutch index of almost exactly 1.0. A more clutch situation, one late in the game where the game is close -- the same situation as above but midway through the fourth quarter, for example -- has a clutch index of about 2.0. Maximum clutch indices are about 3.0 and minimum indices are about 0.3.

These clutch index values came from an analysis of how different situations affect a game's win probability on average. One way to think of it is in terms of ressure. A clutch play is defined before the play by how close the game appears to be. Down four points with three seconds to go and facing third-and-goal from the 3-yard line -- that is a high-pressure and high-clutch index situation because the play can realistically raise the odds of winning to almost 100 percent or bring them down from around 40 percent to almost 0 percent. The same situation from midfield isn't as high pressure because it's very unlikely that the team will pull out the victory. Sure, a Hail Mary can pull the game out, but if it doesn't work, the team didn't fail on that play so much as they failed before then. On third-and-goal from the 3-yard-line, failure means that people will be talking about that final play and what went wrong.



The clutch indices are multiplied by the quarterback's expected points on plays where they had a significant contribution, then divided by the sum of the clutch indices and multiplied by 100 to get a clutch-valued expected points added per 100 plays.



A Rating from 0 to 100


The final step is transforming the clutch-valued expected points rate to a number from 0 to 100. This is just a mathematical formula with no significance other than to make it easier to communicate. A value of 90 and above sounds good whether you're talking about a season, a game or just third-and-long situations; a value of four or 14 doesn't sound very good; a value of 50 is average and that is what QBR generates for an average performance.



That being said, the top values in a season tend to be around 75 and above, whereas the top values in a game are in the upper 90s. Aaron Rodgers may have gone 31 of 36 for 366 yards, with three passing TDs, another TD running, 19 first down conversions and eight conversions on third or fourth down in one game -- for a single game Total QBR of 97.2 -- but he can't keep that up all year long. Pro Bowl-level performance for a season usually means a QBR of at least 65 or 70. We don't expect to see a season with a QBR in the 90s.



Defensive Adjustment


With this rating, we have intentionally not adjusted for opponents. This doesn't mean that we won't adjust for opponents as we use it, but that we want QBR to be flexible for many purposes, and keeping opponents' strength out gives us that flexibility. As it stands, QBR can be broken down for all sorts of situations -- red zone, third-and-long, throwing to a certain receiver, in bad weather, against different defensive formations. We didn't want to muddy it up with opponent adjustments that aren't as useful for those situations. How to implement a defensive adjustment for third-and-long may also be different than one for the whole season. Beyond this, a defensive adjustment is often not a constant factor. A defense that looks good in Week 4 may actually not be so good after a few more weeks. Because it isn't a constant thing, it makes sense to leave that for analysis, rather than constant incorporation into QBR.



There will be analyses that we do on ESPN that will suggest the use of an opponent adjustment, but we will do that when needed, not up front.



Concluding Thoughts

What underlies QBR is an understanding of how football works and a lot of detailed situational data. What it yields are results that should reflect that. It illustrates that converting on third-and-long is important to a quarterback. It shows that a pass that is in the air for 40 yards is more reflective of a quarterback than a pass that is in the air for five yards, where the receiver has 35 yards of run after the catch. These premises should sound reasonable to football fans. They come out of a lot of statistical analysis, but they are also consistent with what coaches and players understand.


As we neared the end of the development of QBR, we talked to Ron Jaworski and Greg Cosell at NFL Films about its evolution. Cosell said at one point, "Football is not complex, but it is very detailed." I realized then that QBR is like that. It is very detailed, accounting for a lot of different situations, but it is not particularly complex. It really does try to see the game the way we have gotten used to seeing it in its elegant simplicity. We hope you, the fan, appreciate it as well.

---

Quote:

---

Early in a scoreless game, a quarterback throws a 20-yard pass just by the reaching arms of a defender and into the hands of his intended receiver, who holds on despite the distraction,

---

Quote:

---

Defensive Adjustment


With this rating, we have intentionally not adjusted for opponents.

"As it stands, QBR can be broken down for all sorts of situations -- red zone, third-and-long, throwing to a certain receiver, in bad weather, against different defensive formations. We didn't want to muddy it up with opponent adjustments that aren't as useful for those situations. "

---