Pro Football's All-Time Best Passers

Now that I've got a method for evaluating a quarterback as a passer and can use it compare his career to that of others, I thought it might be illustrative to compare the Top 50 on my list to how they rank on the all-time passer rating list provided by the Pro Football Hall of Fame.  The list provided by the Hall only includes those passers who've attempted 1,500 or more passes in their career.  Also, as of today (April 27, 2010), the Hall of Fame list does not include stats from the 2009 season.  As I've mentioned in the past, the HOF list unfairly penalizes those players who played in an era where passing was secondary.  Why?  Because of the restriction of limiting it to only those passers attempting 1,500 or more passes.  So, to supplement the list from the HOF, I'll show you their passer rating ranking based on my list of 416 passers, which is a broader and more inclusive list, especially of those who played in the 1930's - 1970's.  My list includes any quarterback who qualified in at least one season.

Here's the top 50 list of All-Time Best Passers:

My list of 50 not only covers the entire spectrum of pro football history - from the beginning in the 1930's to current players, but it also presents the players who played in earlier years fairly, and in context.  The question isn't whether Joe Montana was a better passer than Sammy Baugh, but, whether Joe Montana relative to Montana's peers, was a better passer than Sammy Baugh was relative to Baugh's peers.  The HOF list can be used to answer that question, since a measure relative to a fixed standard (in other words the passer rating system) can be used to do that.  However, the HOF list cannot be used to answer the next question - how does one compare Sammy Baugh to Joe Montana given that they both played an entirely different game, with different rules and circumstances.  A fixed standard cannot solve that particular puzzle.  The use of a mean and standard deviation by year (z scores) which in turn we convert to percentile ranks can be used.  C7 is simply a quarterback's 7 best years as a passer in terms of percentile rank summed up.

Better Than Jay Cutler

A lot of noise this past month about Jay Cutler. First, the Denver Broncos' new head coach, former New England Patriots offensive coordinator Josh McDaniels, apparently tried to work a trade to get Matt Cassel from the New England Patriots. This apparently didn't it well with the Pro-Bowl quarterback. After a lot of posturing from both sides, Cutler was eventually traded to the Chicago Bears for Kyle Orton and draft picks. This post isn't about who was right or wrong in the fracas, or which team got the better end of the deal in the trade. This post is about answering the following two questions:

1) How good is Jay Cutler?

2) Are there quarterbacks with similar experience that are better than Jay Cutler, and, if so, who are they?

The table below ranks each of the 30 quarterbacks that threw enough passes last year to qualify to be ranked. The ranking is based on CMI, and not the NFL's quarterback rating system. In numerous posts prior to this, I have discussed why CMI is a better measure of a quarterback's passing performance than the NFL passer rating system.

Just looking at last year, the year that Jay Cutler got named to his first Pro-Bowl, he was, by this measure, a below-average quarterback. His CMI of 0.536 was (0.14) standard deviations below the mean. It should be noted that even though he was below average relative to all quarterbacks in 2008, it wasn't unexpected. Previously, I discussed the notion of performance improving with experience. So, if you were to look only at quarterbacks and their second year performance, their average is (0.21) standard deviations relative to the mean. Relative to that measure, he actually did better than the typical second year quarterback.

Let's take a closer look at the table above. The last 2 columns indicate the # of years the quarterback has played since he first qualified, and the total # of years he has qualified. So, Cutler has played 2 seasons where he threw enough passes to qualify, and this is second consecutive year of being qualified. There are six other quarterbacks in this table who have similar experience.

They are (rank in parenthesis):

Jason Campbell - WAS (5)

Matt Schaub - HOU (7)

Trent Edwards - BUF (8)

David Garrard - JAX (12)

Jay Cutler - DEN (16)

Derek Anderson - CLE (30)

Of the six quarterbacks who were in their second year (Garrard has the most experience of all the quarterbacks, even though he has only qualified in two years), Cutler only outperformed Derek Anderson of Cleveland in 2008. Ok, so 1 year does not make a quarterback. Let's take a look at these six quarterbacks and see how they've done in each of their first two (qualifying) years.

According to this table, both Garrard and Schaub have outperformed Cutler in each of 2007 and 2008. And, arguably, over the two-year period, so did Campbell. The only quarterback in this class that has clearly underperformed the rest of the group is Derek Anderson. He will not last long in the NFL.

Admittedly, it's difficult to judge a quarterback after his first two years. But, so far at least, there appear to be several quarterbacks that have outperformed Cutler. It will be interesting to watch this group over the next several years, and especially Cutler, as he will be with a new team.

Does Experience Improve Performance?

In my earlier post, "First Year Performance - An Evaluation Method", I discussed how, over the history of the NFL, from 1940-2008, a quarterback's passing performance, as measured using CMI, and more specifically, the standard deviation from the mean of CMI, changes over time (in that post, see in particular the third graphical illustration labeled "CMI - SDs from Mean - By Qualifying Year").  That illustration pointed out that it takes about 5 years for a quarterback to become "average" (or, better than average).  That's obviously only true for quarterbacks who qualify in at least that many years.  Meaning that since the graphical illustration includes all quarterbacks (who have thrown enough passes to qualify in at least 1 year), that the earlier years could be biased by quarterbacks who only qualify in  those earlier years, and not the later ones.  

This is the issue pointed out by a commenter regarding that post.  He (she) was suggesting that I have a biased sample, in that the later years are only populated by quarterbacks who throw for that many years, and hence, performance should be expected to improve, as the NFL weeds out the under-performers of the earlier years.

Here's the comment:

"This effect could be due to sampling bias.  It may be that there are a large pool of first year QBs.  Some are good and some are bad.  The good ones make it to year four, and the bad ones are cut, released, or benched before they can play four years...."

He (she) then went on to ask the following question:

"...if you restrict the analysis to only QBs that have played >= 4 years, do you get the same result?"

Before I answer the question, let's address the comment.  

First, the commenter does make a very valid (and what would seem to be an obvious) point, in that those QBs that make it to year 5 and beyond, must necessarily be the better ones, as the under-performers drop-off.  This point has several implications though.  The first is that the NFL does not look at quarterbacks (or, more accurately, their passing performance) the way I do.  So we can't simply assume that the point is in fact that obvious.  Second, perhaps the way I look at quarterbacks passing performance, CMI, is in fact, a decent (good?) measure of a their true passing ability, and, the graph validates that fact (this would be rather presumptuous of course, but that is what I am trying to prove after all - that I have, in fact, developed a method that appropriately evaluates a QBs passing performance, and, perhaps even more importantly, that I have developed a method by which one can identify a good passer versus a poor passer early on).

Second, the commenter suggests that my sample is biased because of this fact (i.e. it is biased towards the better players in the later years).  There are two ways we can look at that.  The first is that the sample isn't biased because it uses the entire population.  In other words it is what it is (and perhaps this again validates my measurement technique).  The second point is that the sample is biased (remember, however, the context in which the graphical illustration was shown - in that it was trying to look at all quarterbacks, and how their performances changed over time), and that we can, if we choose to do so, look at it differently.  This is, in fact, what the commenter was suggesting with their question.

So let's take a look.  As a matter of fact, let's take the question a bit further and look at it in four different groupings:  

QBs qualifying in at least 1 year (the original graph) - 345 Quarterbacks

QBs qualifying in at least 3 or more years - 180 Quarterbacks

QBs qualifying in at least 5 or more years - 121 Quarterbacks

QBs qualifying in at least 7 or more years - 84 Quarterbacks

The graphical illustration is followed by a tabular display of the same data.

There's no question about it.  Performance improves as a quarterback gains experience.  All groupings follow the same basic pattern:  Sub-par performance in year 1, followed by several years of gradually improving performance.  When the improvement takes place, and by how much, does vary by group.  For example, after three years, teams still haven't been able to markedly separate the good quarterbacks from the bad ones.  By the fifth year, it's now quite apparent who has "cut it", and who hasn't.  For all quarterbacks qualifying in at least 7 or more years, they appear to show their skill, or differentiate themselves, as early as year two.  So, to prove the obvious point that the commenter in the previous post making, those that don't turn in positive performances early do tend to get weeded out.  The process, however, happens to take a few years.

The question this prompts however, is, why does it take so long to weed out the poor performers?  

Well, there are at least two different answers.  

The first, something I alluded to earlier, is that the NFL doesn't evaluate passers the way I do.  I am a statistical analyst, and what I have created is a an objective methodology to evaluate a player's performance.  Teams in the NFL, aren't inclined to do the type of statistical analysis that I do.  In addition, it's just simply not clear, even using my statistical methods, that you can be sure that you can identify the good and bad future quarterbacks after year 1.  By year 2, however, year 3 the latest, my method does show who the keepers should be.

The second is, that teams pay a lot of supposedly smart people (scouts, GMs, etc) a lot of money to evaluate, and draft good talent.  Subsequently, these teams draft these "talented" players early in the draft, and pay them lofty sums of money, a lot of which is guaranteed.  Now what do you do?  You've got a poor-performing team (generally, a team's draft order is determined by their won-loss record, with the teams with poorer records drafting earlier, and vice-versa) that just drafted the next "Peyton Manning", and have paid them a lot of money, and they have a lousy first year.  You're not likely to blame the young quarterback for that year's performance.  The team still has a lot of weaknesses elsewhere, so you pick early again the next year (perhaps shoring up the defense).  Second year passing performance is better than the first, but still not better than average.  You even ignore the statistical evidence (he can't be that bad).  You're still paying him lots of money, you've seen improvement, team has won perhaps a couple of more games, and this gives you hope that, the third year is THE YEAR.  Then, much to your dismay, the quarterback that you paid millions of dollars for, doesn't pan out as you have another miserable year (the teams is "close" to 0.500).  You get impatient,  fire the head coach, try an aging veteran as your quarterback, he has a decent year, a quarterback controversy brews, ya da ya da ya da....and, you continue to ignore the statistical evidence....

Just off the top of my head, I can think of several recent obvious examples - Alex Smith of the Forty Niners (and, proving the corollary, the approach taken by the Green Bay Packers with Aaron Rodgers), and JaMarcus Russell of the Oakland Raiders.

Here are three quarterbacks that the NFL, teams, and fans, have fallen in love with, but, statistically have yet to prove themselves.

Eli Manning (I can hear the argument - but they won the Superbowl)

Tony Romo

Michael Vick

So there you have it.  My method is the Moneyball approach.  What actually happens in the NFL is quite different.  NFL teams could save themselves a lot of money (and perhaps even win a few more games) if they were to adopt a statistical approach to their business (I think a lot of NFL teams actually believe they take a statistical approach, but I don't think that most actually go through rigorous analyses the way I do).  

My next post will be my preseason rankings for 2009.

The Best and Worst of 2008

This is a simple post.  Really simple.  The list should pretty much speak for itself.  The best and worst of 2008.  I use the same criteria I've been using the past few posts.  CMI, and, more specifically, standard deviation of a player's CMI relative to the mean.  30 players attempted enough passes in 2008 to make the list.  

The Best 3:

3.  Jeff Garcia, Tampa Bay Buccaneers - Why no team has picked him up is a complete mystery to me - the man can still play, even though his durability is certainly a question.  Completed nearly 65% of his passes, and less than 1.6% of his passes were intercepted.

2.  Peyton Manning, Indianapolis Colts - Do we even need to say anything? Coming off off-season knee surgery, Mr. Consistency now holds the highest current active streak of games started consecutively - 176 games. He completed almost 67% of his passes.

1.  Chad Pennington, Miami Dolphins - Let go by the Jets so that they could welcome Brett Favre (that worked out well for the Jets, didn't it?) and subsequently picked up by the Dolphins. Had a terrific season, completing more than 67% of his passes, and less than 1.5% of his passes were intercepted.

In previous posts on my blog, I have spent time discussing each of the above 3 players.

The Worst 3:

28.  Tyler Thigpen, Kansas City Chiefs - Won't show up at the bottom of the list in 2009.  Why?  Matt Cassel (#10 in 2008).  The Chiefs made a deal with the New England Patriots to get Cassel (and Mike Vrabel) for a 2nd round pick in the upcoming draft.  In 2008, Thigpen, replacing Damon Huard, completed fewer than 55% of his passes.

29.  Gus Frerotte, Minnesota Vikings - Replaced an ineffective Tarvaris Jackson in the 3rd game of the season, started 11 games, and then was himself replaced by Jackson against the Lions.  Completed less than 60% of his passes, and at almost 5.0%, was one of only three quarterbacks whose interception percentage exceeded 3.0% - the other two being Ben Roethlisberger of the Steelers (3.2%), and Brett Favre of the Jets (4.2%).

30.  Derek Anderson, Cleveland Browns - Barely completed 50% of his passes, then got hurt.  Was replaced by Brady Quinn.  Is it Quinn's time in 2009?

Here's the entire list.

My next post - a pre-season ranking for 2009

First Year Quarterback Performance - An Evaluation Method

This past weekend, the New England Patriots traded Matt Cassel to the Kansas City Chiefs.  In fact, they traded linebacker Mike Vrabel and Cassel for a 2nd round pick (pick #34) in the upcoming April draft (see here for the details).  Depending on what your news source is, this was labeled as either a good deal for Kansas City, or a bad deal for New England.  The experts discussing this issue don't seem to support their arguments by any objective measures - they just opine.  I am not going to jump into the debate here.  I will offer this however - we will probably not know who will end up on the better end of this deal until many years from now, and even then, we may not know.  

What about the objective evidence?

Matt Cassel is not the only quarterback in 2008 that had a good year as a first-year quarterback.  There was Aaron Rodgers of the Packers, Shaun Hill of the 49'ers, and Matt Ryan of the Falcons.  There were four other quarterbacks who threw enough passes this year to make the list of first-year quarterbacks - Ryan Fitzpatrick of the Bengals, Joe Flacco of the Ravens, JaMarcus Russell of the Raiders and Tyler Thigpen of the Chiefs.  

Notice here that I didn't use the term rookie quarterback, but instead used the term first year.  The way I look at a quarterback, is that I only consider a quarterback who has thrown "enough" passes to qualify in a given year.  The # of passes has varied by year (as opposed to the fixed standard used by NFL.com).  For the 69-year period from 1940-2008, my database contains 1,451 passing seasons, 336 of which were by a quarterback qualifying for the first time.

How did the aforementioned 8 quarterbacks do?  

I use a statistical measure I have previously developed and discussed, called CMI (Completions Minus Interceptions, calculated as Completions/Attempts - 3*Interceptions/Attempts) instead of the NFL Passer Rating system (using the NFL passer rating system gets you essentially the same answers), and then use the standard deviation from the mean as the measuring stick (for the pool of qualifying passers, I calculate the the mean and standard deviations for each year, and relate a particular performance in that year to the mean using that year's standard deviation).  Using the mean and standard deviation this way not only allows me to compare these 8 quarterbacks relative to each other, but also relative to all quarterbacks this year, or any other year.

In a post not too long ago, I showed how, over the past 69 years from 1940-2008, when the standard deviation is used as the measuring stick, the accumulated data form a near-perfect theoretical standard normal distribution.  And I also showed how, this is true regardless of whether one used the NFL passer rating system or CMI as the statistical base.

I can now show you how these first year quarterbacks did compare to the rest of the QBs in 2008, using this measure.  But this picture may not tell us much, if anything.  Here's the graphical illustration:

Perhaps the only thing that may be apparent from this is that first year quarterbacks don't do nearly as well as quarterbacks with more experience.  Even drawing that conclusion from this is a little iffy.  However, it is clear (at least by this measure) that Cassel had the best year of the 8 first year QBs.  Rodgers was a close second.  JaMarcus Russell and Tyler Thigpen had disastrous first years.  In an earlier post regarding Jeff Garcia, I had suggested that Kansas City might be a good place for Garcia.  Scott Pioli, the new Chiefs' GM apparently was thinking the same thing - upgrade at quarterback, and he did - with Cassel.

Going back to the standard normal curve, let's look at just the first year performances.  Recall that, when all 1,451 seasons are aggregated in a histogram, it looks very much like a standard normal curve.

This doesn't quite look "normal", does it?  The first year performances appear skewed to the left, or, compared to all quarterback performances, appear to under-perform.  Since the total database is normally distributed, then performances in years 2 and beyond must obviously be skewed to the right.  This prompts the question: does experience improve performance, and, in particular, how and when?

One way to look at this is to take the entire database, and look at the average standard deviation from the mean for each additional year a player qualified.  Here's what that graph looks like.

Wow!  I didn't realize that it takes about 5 qualifying years to turn into an average quarterback.  And that's just for those playing that long!  Even though this graph goes out to 18 years, there is hardly any data beyond the 15th year (there are only 8 quarterbacks who have had qualifying seasons 16 different years).  According to this illustration, basically, first year quarterbacks, as a group, struggle mightily.  They then gradually improve each year until about year nine.  Quarterbacks, on average, must still endure sub-par years in years 2-4, and for those that survive, their reward is about four more years of improving performance, assuming they stay healthy for that long.

Note: This analysis - the method of breaking down a quarterback's expected career in terms of passing performance as it relates to experience and how it is expected to change over time, is the first I've seen published publicly.  But I digress.

Ok, so we've taken a look at these 8 QBs relative to all quarterbacks in 2008, and we've taken a look at first year quarterbacks in general.  So the next step is to take a look at the best first year seasons.  In my database, there are 336 first year seasons in all (in other words, in the 69-year period from 1940-2008, 336 different quarterbacks threw enough passes in a season to qualify at least once).  The table below shows the 75 best.

Cassel (#48) and Rodgers (#60) are the only 2 of the 8 in 2008 that crack the top 75 all-time.  On the other end of the scale, we get JaMarcus Russell (#264) and Tyler Thigpen (#290).

When Matt Ryan (#128) got off to a solid start, and the Falcons surprised many by getting into the playoffs, there were many debates as to whether his season was one of the best ever by a first year player.  Not according to this measure.  Often, during those discussions, Kurt Warner's first year was brought up.  In looking at the table above, well, he did have an outstanding first year in 1999.  It ranks #6 all-time in terms of first year seasons.  The best ever - Roger Staubach's 1971 season (Staubach was a rookie in 1966, and played in both 1969 and 1970.  However, he didn't attempt enough passes in either year to qualify).  

Notable quarterbacks high on this list: Joe Montana in 1980 (#4), Tom Brady in 2001 (#14) - (see my post earlier comparing Brady's season to Cassel's here), Brett Favre in 1992 (#15), Dan Marino in 1983 (#21), and Johnny Unitas in 1956 (#22).  

Notable absentees:  Steve Young in 1986 (#154), Peyton Manning in 1998 (#208), Joe Namath in 1970 (#202)*, and Troy Aikman in 1989 (#311).  The all-time worst first year performance - Terry Bradshaw in 1970 (#337)*.

* Keep in mind that 1970 was the first year of the NFL following the merger, and 15 quarterbacks showed up in the database as having that year as their first year, although Bradshaw's 1970 season was his actual rookie season.

Are we done?  Hardly.  So after I went through this list, and still trying to objectively value Matt Cassel, I was interested in answering the following question(s):  what does the first-year tell us about a quarterback's future potential?  Do quarterbacks who have good first years, have good careers?  What about quarterbacks who have sub-par first years?  In other words, is there a correlation between year 1 performance and subsequent years?  When in doubt, it's always a good idea to take a look at the data.

Whoa!  The correlation between year 1 performance, and career performance, as measured by CMI standard deviations relative to the mean, is 0.75.  What that really means is that 57% of the variation in a quarterback's career performance can be explained by his year 1 performance.  I find that incredible.  So I looked a little deeper.  It turns out that 112 of the 336 quarterbacks only have 1 qualifying year.  And, for that group, the correlation coefficient is, well, 100%!  If I only looked at those quarterbacks whose career included at least 2 different years in which they threw enough passes to qualify, then the correlation coefficient drops to 64%.  Still quite remarkable.  

The table below breaks down the 336 first year seasons in terms of CMI standard deviations to see if we can glean any additional insight into whether a good first year translates into a good career and vice versa.


Well, the evidence is pretty clear.  Taken as two groups, players that have better than average (remember, the mean and standard deviation is relative to all quarterbacks, not just first year quarterbacks) first years tend to have longer careers and more productive careers than those that have sub-par first years.  And, breaking it down even further (in other words, just looking at the group whose first year was better than average, or the group whose first year was worse than average), you can clearly see that even within those sub-groups, the better the first year, the longer the career (although this correlation isn't nearly as strong), and the more productive the career. 

I haven't spent too much time on my blog discussing individual careers, except in the posts discussing Brett Favre and Jeff Garcia.  If I do this en masse, it would be such a powerful post - I'll basically be giving you my list of greatest passers to ever play the game, that I need to put a lot more thought into it than I have to-date.  In any case, some of the obvious names will show up on that list.  However, you'll find some exceptions as well (and I'm sure many of you will take exceptions to the list).  The point I really want to make here is that, when combining the table above, and the discussion following the list of the top 75 first years, it is quite remarkable how Troy Aikman "escaped" the trend, and turned into a great quarterback (or, more accurately, a great passer) - he truly turned out to be an exception (as for Young and Manning, even though they didn't rank well on the first year list, their first years weren't "that bad" compared to other first year seasons - as a matter of fact, Young's was "above average" compared to a typical first year).  Notice how I don't mention Bradshaw and Namath - it is because, statistically, their careers were sub-par.  When I publish my list of all-time greatest (and worst) passers, we'll get into it in more detail.

So, in trying to look at Matt Cassel's first year performance objectively, we've looked at how quarterbacks perform as their careers progress, how quarterbacks perform in their first year, who had great first years, where Cassel's season ranks on that list, and, how a first year translates into (or, is predictive of) a career.  Now perhaps, we can view Matt Cassel's 2008 year in an objective manner, and, if so inclined, you can at least evaluate the trade armed with some data.

Naturally, while I was putting this post together a question popped into my mind, and that is:  is a quarterback's second year a good indicator of their career?  What about looking at their first and second years?  Check back in a week or two to find out.

A Brief History of the NFL Passer Rating System

Philip Rivers of the San Diego Chargers had a phenomenal season in 2008.  Or did he?  According to the NFL's passer rating system, Rivers' season, where he completed 312 passes out of 478 attempted, threw for 4,009 yards, and 34 touchdowns against 11 interceptions resulted in a passer rating of 105.5.  That's great, right?  Before we answer that question, we'll need to answer a few more questions.

How do the statistics 312 for 478, with 4,009 yards, 34 touchdowns and 11 interceptions translate to a rating of 105.5?

What does having a rating of 105.5 mean?

Where does Rivers' season rank all-time?

How has the average NFL passer rating changed over time, and how does that affect how we look at Rivers' season?

The NFL began keeping statistics in 1932.  The NFL has used various methods to determine the "best" passer in the years since.  According to the Pro Football Hall of Fame, the following is a chronology and description of the different methods used:

1932-1937: Total yards passing

1938-1940: Percentage of completions

1941-1948: Inverse ranking system of the following categories: completions, percentage of completions, total yards, total TD passes, number of interceptions, and percentage of interceptions

1949: The same formula used from 1941-1948 except the number of interceptions were dropped from the equation

1950-1959: Average yards gained per pass with a minimum of 100 attempts needed to qualify

1960-1961: Inverse ranking system based on six categories: total completions, total yards, total TD passes, percentage of completions, percentage of interceptions, average gain per attempt with the principle established of at least 10 attempts per game to qualify

1962-1971: Inverse ranking system based on four categories: percentage of completions, total touchdown passes, percentage of interceptions, average gain per attempt

1972: Same system used from 1962-1971 except that the percentage of touchdown passes was substituted for total touchdown passes

1973 to present:  See below.

Current NFL Passer Rating System:

The passer rating system used by the NFL has been in place since 1973.  In 1971, after the merger, then-commissioner Pete Rozelle wanted to implement a standardized set of statistics, including a standard measure of passing performance.  As the description on the Hall of Fame site suggests, this is a measure of a quarterback's passing effectiveness, not a measure of how good a quarterback is.  Rozelle called upon Don Smith, then an executive with the Hall of Fame, to work with the league's official statistician, the Elias Sports Bureau, to develop a new standard.  

One of the problems with the standard that existed at the time, was that you didn't know where you stood until all the teams' quarterbacks had finished playing, as it was a relative ranking system.  In addition, there wasn't a convenient way to compare a performance in a given year to that of another.

Don liked the use of the combination of statistics - in other words, the completion percentage, the yardage per attempt, the touchdown percentage and interception percentage.  He had to figure out how to use all four in some manner that would make sense.

Working with the Elias Sports Bureau, he studied passing statistics for each of the four categories from the decade before - the sixties.  After much study and thought, he devised a system whereby, for each of the four statistical measures, he would convert that particular measure to a "score" between 0 and 2.  A score of 0 would indicate poor performance, a score of 1.00 would indicate "average" performance, and a score of 2.00 would indicate "superior" performance.  With truly exceptional performance, it was possible to exceed a score of 2.00.  He arbitrarily chose the maximum score to be 2.375.  

He decided that an "average" performance across all four categories, that is, a score of 1.00 for each of the four measures, should be a rating of 66.7.  Under this new rating system, it would be possible for a rating to exceed 100.0, but, he theorized that those instances would be rare.

(Note:  I took a lot of this history from an article written by Don Steinberg, published in Slate magazine in 2001 - see here for the full article)

In summary then, this is how we end up with the current NFL passer rating system:  we take four components of passing, convert each component, using league averages from the 1960's, to a score with a minimum of 0.000 and a maximum of 2.375, combine these scores by adding them, and convert to a rating system that has 66.7 as the "average".

Here's the math that does this:

Q = [ ( J + K + L + M ) * 100 ] / 6

where,

Q = Passer Rating

J = max [ min ( C, 2.375 ), 0 ]

K = max [ min ( Y, 2.375 ), 0 ]

L = max [ min ( T, 2.375 ), 0 ]

M = max [ min ( I, 2.375 ), 0 ]

and where,

C = [ ( Completions / Attempts ) * 100 - k1 ] / 20

Y = [ ( Yards / Attempts ) - k2 ] / 4

T = [ ( Touchdowns / Attempts ) * k3 ] * 20

I = 2.375 - [ ( Interceptions / Attempts ) * k4 ] * 25

and where,

k1 = 30

k2 = 3

k3 = 1

k4 = 1

In the first formula, you can easily see where if J = K = L = M = 1.000, the passer rating formula would yield Q = 66.7.  In order to get J = K = L = M = 1.000, certain "transformations" were needed to each of the passing statistics to convert the averages to a score of 1.000.  The transformations for each are shown as C, Y, T, and I, and, more specifically, k1, k2, k3 and k4.  The figures for k1-k4 were derived using league statistics from the 1960's, and to make it somewhat easier, rounded.  If you looked at data from 1960-1969, you get the following actual values for k1-k4:

k1 = 31.70

k2 = 3.24

k3 = 0.96

k4 = 0.99

The table below shows, for each NFL season from 1940-2008, how the averages, or, more precisely, the "scores" for those averages have changed from year to year (for a graphical illustration see my previous post on the subject here).  For example, you can see that the average score for "J", in 2008 was 1.550, reflecting the fact that the "C" component, or, completions per attempt, for the average NFL quarterback was 61.0%.  Compare this to the average score for "J" in 1968 of 1.079, reflecting the "C" component, or completions per attempt of that time of 51.6%.  This is what happens when the transformations do not change over time, even when the actual game itself has undergone many transformations.  If one wanted to keep everything in balance from year to year in the passer rating formula, in other words, to keep J = K = L = M = 1.000 for every year, then one would have to change the values of k1 - k4 every year.  I have done that in the table below.

You can use these different k values every year to "adjust" the passer rating formula, so that you can make meaningful comparisons from year to year.  If you didn't do that, and simply compared a quarterback rating from one year to the next, you wouldn't get an appropriate comparison, for the simple reason that the four components would be "out of balance".  

While it was never Don Smith's intention to have a relative measure of performance - he wanted performance as measured relative to a fixed standard, that "fixed" standard has changed, and will continue to change over time.  One way to minimize the effect of these changes is to look at a given performance in a given year to that season's average, using the standard deviation as a measuring stick.  The other way, is to simply "adjust" the standard to reflect the averages for that year.  In a previous post, I discussed the former method.  In this, I am obviously discussing the latter.

This discussion is only to put Rivers' 2008 season in perspective.  It is not intended as passing judgment on the NFL passer rating system.  I have, in previous posts (see here and here) discussed why I do not like the system.  In a post in the future, I'll elaborate further on why I think the NFL's passer rating system should be revised.

The table below shows the 75 greatest seasons in the NFL, both in terms of the current NFL passer rating system, and using an "adjusted" passer rating system, going through the transformations as I have described above.  You can see that, using the current system, that Rivers' 2008 ranks as 13th best all-time.  Using an "adjusted" passer rating system, however, results in a ranking of 60th best all-time.  If we used the previously mentioned standard deviations from mean measurement, it would rank as the 16th best all-time.

We still have some unfinished business:

How does Rivers' stats translate to a rating of 105.5?

He completed 312 of 478 passes, or, a completion percentage of 65.3% - using k1 = 30, and Comp/Att = 0.653 in the formulas above, yields a J value of 1.764

He threw for 4,009 yards, or, yards per attempt of 8.4 - using k2 = 2, and Yards/Att of 8.4, we get a value for K of 1.347

He threw 34 TDs, or, TD/Att of 7.1% - using k3 = 1, and TD/Att = 0.071, we get a value for L of 1.423

He threw 11 INTs, or, INT/Att of 2.3% - using k4 = 1, and INT/Att - 0.023, we get a value for M of 1.800

Combining all, we get

Q = ( J + K + L + M ) * 100 / 6, or

Q = ( 1.764 + 1.347 + 1.423 + 1.800 ) * 100 / 6 = 105.5

There is no doubt that, whether we use the current NFL passer rating system, an "adjusted" passer rating system with different transformations for different years, or, the current system, but adjusted by looking at standard deviations from the mean, that Rivers' 2008 season was quite remarkable.  

Whether the passer rating system is an accurate reflection of a passer's ability is a completely different question altogether.  We shall, over time, address this question.

Brett Favre - Best Ever?

Brett Favre retired earlier this week.

He played in the NFL for 18 years. He set numerous NFL records during the course of his career. The most impressive has to be the fact that, as a quarterback, he started a record 269 consecutive regular season games. Peyton Manning is second on the list, his streak currently standing at 176 games. Third on the list is Ron Jaworski, his streak having ended at 116 games. Tom Brady is next on the list, but his streak ended in the first game of last season at 110 games. Peyton Manning needs another 94 games - he would have to start every single game through the 14th game of the 2014 season to break Favre's record. In addition to the four mentioned above, only two other quarterbacks in history - Joe Ferguson (107 games), and Dan Marino (95 games) have started that many games consecutively. I suspect Favre's record will never be broken.



People are fascinated with Favre. Many consider him to be among the all-time greatest to ever play the game. Some consider him the greatest ever. I do not belong to either group.


The media loved Favre. And vice versa. He never met a camera he didn't like, always seeking an opportunity to get into the limelight (of course, the media loved that). He also seemed intensely competitive, and the combination is what I believe led him to his unfortunate decision to come back for one more season in 2008.


Let's start from the beginning.


In 1991, the Atlanta Falcons took Favre in the second round of the draft. With Chris Miller as their starting QB, in 1992, the Falcons traded Favre to Green Bay. Favre started the season as the back-up to the 'Magic Man', Don Majkowski. In the third game of the season, against the Cincinnati Bengals, he replaced the injured Majkowski in the first quarter. With 13 seconds left, he threw a 35-yard touchdown that helped the Packers come-from-behind to win 24-23. He started the next game against the Steelers, a game the Packers won. The rest, as they say, is history, as he started every game for the Packers until his trade to the Jets following the 2007 season (for a detailed history of, or, more aptly, a tribute to Favre, see here).


The table below shows how Favre did each year.



Before we go on, some explanations.


In order to be able to truly compare Brett Favre to other quarterbacks so that we can properly gauge his place in history, we can't simply look at the NFL's passer rating system. First, we know that the average NFL passer rating in the NFL has been increasing over time. So, by simply using the NFL's passer rating system, the more recent quarterbacks would appear to have done better than quarterbacks from earlier years (see here and here for evidence). Second, we have to agree that the NFL's passer rating system is an accurate, if not adequate measure of a quarterback's performance. I have previously introduced a metric, CMI, which I believe is a better, if not more understandable measure of performance. CMI, however, also has the same bias as the passer rating system. One way to adjust for, and remove the bias favoring more recent quarterbacks is to look at each year's performance for each quarterback, and relate that to the average performance that year. The measure that we use is the standard deviation. I have discussed previously how using this measure tends to "normalize" performances across years. In other words, when a given year's performance by a particular quarterback is "normalized" to the average for that year, and we do that for every year, we can very easily see how that particular performance compares to other performances across years. Since CMI is very similar to the passer rating system, when using standard deviations, it shouldn't really matter whether we use CMI or the NFL passer rating system (there will be differences, and, I often include both so that you can see the differences). The two graphs below show how using either the "normalized" NFL's passer rating system or "normalized" CMI as measures compare to a standard normal curve with 1,451 observations (that's how many qualified quarterback seasons there have been over the past 69 years).



Ok, so back to the table.


Favre had a "pretty good", or, "above average" first year in 1992. Using the NFL passer rating, he finished 0.57 standard deviations above the average that year. Using a standard normal curve, that would mean between the 71st and 72nd percentiles. If we used CMI as the standard, then his performance that year, 1.24 standard deviations above the mean, would place him between the 89th and 90th percentiles.


He followed that with a sub-par year in 1993, finishing between the 30th and 31st percentiles according to the NFL passer rating system, or between the 35th and 36th percentiles using CMI. Then followed several "above average" to "very good" years (1994-1998), depending on your perspective, followed by a couple of mediocre years (1999-2000), followed by four above average years (2001-2004), followed by his two worst years (2005-2006). He finished his Green Bay tenure with an "above average" year in 2007.


He should have taken the opportunity to retire then.


His last year, his first full season not with Green Bay, was "below average". So he finished his career with 3 "below average" years out of 4. Perhaps the Packers brain trust was on to something when they decided that Aaron Rodgers was their future at the end of the 2007 season.


The two graphical illustrations below show each of Favre's seasons in terms of standard deviations from the mean. In case you haven't noticed, as much as I love statistics, I find that graphs are better way to illustrate a point than a table of statistical measures. I show the CMI graph first and the graph based on NFL passer rating next, for reasons that will become plainly apparent when you read on.





Ok, so until now, we've been looking at Brett Favre's career in a bit of a vacuum. Yes, we have compared his performances to an average, and yes, we have tried to put it into historical perspective, but I haven't given you any 'perspective'.


First, let's compare Brett Favre's career to some individual careers. The biggest problem we face in comparing Favre's career to any other individual career, is obviously the length of his career. There have been only 5 individuals who have thrown enough passes to qualify in at least 15 different seasons. Those are Fran Tarkenton (18 years), Brett Favre (17 years), Dan Marino (16 years), John Elway (16 years), and Johnny Unitas (15 years). So how does Favre's "picture" compare to these quarterbacks, all of whom are in the Hall of Fame?


Fran Tarkenton - a picture of "excellence". With a couple of exceptions, Tarkenton exceeded his peers consistently for a very long time. Quite remarkable. Without question, Favre's picture does not resemble that of Tarkenton.


Dan Marino - Marino was "above average" for many of his years, tailing off towards the end of his career. He was "very good" very early in his career. He really didn't have a "bad" year until his last, and wisely decided to hang it up after that. Although closer in resemblance, Marino's picture still looks better Favre's overall.


John Elway - Elway underperformed his peers much of his early career, exceeding the average in only two of his first ten years. He then was "above average", or even "very good" his last six years. He went out on top, having retired following the Broncos having won back-to-back Super Bowls (Super Bowl XXXII and Super Bowl XXXIII). I think it's safe to suggest that Favre's picture looks better than Elway's.


Johnny Unitas - In my opinion, Favre's picture most closely resembles that of Johnny Unitas. They both started out well, had a couple of mediocre years, then had several years where they outperformed their peers, and ended their careers with a few sub-par years.



What if we compared Favre's average performance (in other words, average each of his year's performance in terms of standard deviations), to other players? Well, over his career, in terms of NFL passer rating, Favre averaged 0.42 standard deviations above the mean (0.26 when looking at CMI). Tarkenton averaged 0.65, Marino averaged 0.63, Elway averaged 0.06, and Unitas averaged 0.51 standard deviations above the mean.


Let's expand the comparison list to players who qualified in at least 10 different seasons. 41 different quarterbacks make that list. The two graphs below illustrate where Favre ranks on those lists.



The same graph as above, except looking at it Using CMI as the measure of performance.




Looking at his career statistically, it is hard to make the argument that he belongs in the truly elite group of all-time quarterbacks - those that would be ranked in the top 3 or top 5 all time. I think it is fair to suggest that Favre has had a very good career. It was certainly a lengthy one, and, in that regard, it was exceptional. But as far as his actual performance on the field, there isn't a measure, or set of measures, that suggest that it was exceptional.


People will suggest that he holds the all-time record for yards, or touchdowns, for example. But that is merely a function of his longevity, just as much as he holds the record for interceptions thrown, all-time.


I also don't discuss measures frequently used as evidence of greatness. MVP awards, for example - these are popularity contests more than a reflection of outstanding statistics or an individual's value to a team (the NFL MVP award is given by the Associated Press). Pro-bowl selections for example are another popularity contest - just look at who got selected this past year in the AFC. Another measure frequently used is Super Bowls. Dan Marino didn't win any Super Bowls. Trent Dilfer won a Super Bowl. Winning percentage is also frequently used - last time I looked, a quarterback doesn't play defense, return kicks or punts, or kick field goals. These pieces of evidence would never enter any serious discussion in terms of statistical evaluation.


My eventual goal is to identify the greatest passer of all time. I will use several statistical measures, many of which are mentioned in this post, to identify the most worthy candidates for consideration. As far as my research is concerned so far, Favre would not make that list. There are several other quarterbacks, not mentioned in this post, nor even displayed in either of the last two graphs that would make more worthy candidates, but that is for a different article.

Happy New Year!

Happy 2009 Everyone!

It's been awhile since I last posted.  I was on vacation.  It made me realize how much I missed my computer, and my EXCEL database.  Anyways, I'm back, and I'll have a post tonight.  I was going to post something before the Jets-Dolphins game the last week of the season, showing how I had 'projected' a bad game for Brett Favre and a good game for Chad Pennington.  Well, my 'prediction' did come true, although my posting tonight on the subject will seem a bit hollow.  That's ok, it's the analysis that matters anyway.

Cheers.

Debating the 2009 Pro-Bowl Selections

Earlier this week, the NFL announced the rosters for the 2009 Pro Bowl.  Representing the AFC are QBs Peyton Manning, making his ninth appearance, Jay Cutler, making his first appearance, and Brett Favre, making his tenth appearance, but first time representing the AFC.  Representing the NFC are QBs Kurt Warner, making his fourth appearance, and Drew Brees and Eli Manning, who is making his first appearance.  This will be the first time that brothers playing quarterback have been selected to the Pro Bowl.

I am really not sure what criteria the voters (the players) use to select these quarterbacks.  Some probably use statistics, such as the quarterback rating, even though no one actually understands the QB rating or how it is derived.  Some probably look at the team's performance, even though the quarterback may have little to do with the team's actual performance, or won-loss record.  And then some may simply be basing it on some other factors, such as like-ability, or popularity.  Based on the selections, it appears that voters reward players for a good team record and # of touchdowns thrown (to the average fan, these two factors seem like reasonable criteria, I'm sure).  I of course disagree, but then again, that is the reason for this post.  

The Pro Bowl is meant to be a reward for performing exceptionally well on the field, and, since it is voted on by the players, a recognition from one's teammates as well as competitors.  So how do these selections stack up?

Before I get into the details, a quick refresher on the NFL passer rating calculation.  I'll keep the math simple.  The NFL passer rating is made up of four different components (see here for the details), each with a maximum possible value of 2.375, and a minimum possible value of 0.  The four components are given equal weight in the calculation of the rating.  The four components are:

Pass Completion percentage

Yards per Completion

TDs per Attempt (Touchdown percentage)

INTs per Attempt (Interception percentage)

In the table below, I show how each of the quarterbacks scored on each component, how that translated to their QB rating, and their respective ranks (amongst each of the four components as well as the final QB rating).  I also include CMI for each quarterback, and I have previously discussed my preference for CMI versus the current NFL passer rating formula.  The players shaded in green were the ones voted in.  The players shaded in blue are my obvious snubs, and the players shaded in orange one could argue deserve to go.

Ok, now on to the #s.

AFC

Peyton Manning - As I mentioned in a previous post, he is having another great year.  He has a passer rating of 90.3 (through 14 games).  In the AFC, he ranks 5th, 7th, 4th and 6th in each of the components, good enough for 4th in terms of passer rating.  He is 3rd in the AFC in terms of # of touchdowns passes.  He has led the Colts to a 10-4 record.  He is 2nd in the AFC in terms of CMI.  Very difficult to argue against the selection.

Brett Favre - For the four components, he is ranked 1st, 9th, 2nd and 16th, good enough for 7th overall in passer rating in the AFC.  He has taken the Jets to a 9-5 record, and he has thrown 21 TDs, fourth amongst AFC quarterbacks.  He has also thrown 17 INTs, worst in the NFL.  He ranks 7th in the AFC in CMI.  Based on statistics, should not have made it.  The only explanation is that he is probably perceived to have helped turn around the Jets, and he has thrown a lot of touchdowns.

Jay Cutler - Jay Cutler got off to a red-hot start, and has since cooled off.  He ranks 9th, 4th, 3rd and 11th on each of the four components, placing him 5th in the AFC in Passer Rating.  The Broncos are 8-6, and his 24 touchdowns are second only to Philip Rivers' 28.  His CMI places him 9th in the AFC.  Based on statistics, again hard to see why he got voted into the Pro Bowl.

NFC

Kurt Warner - He led the Cardinals to a 8-6 record and the NFC West Division title.  He ranks 1st, 4th, 4th and 7th in each of the components, good enough for 2nd in the NFC in passer rating.  He has thrown 26 touchdowns, 2nd best in the NFC.  He ranks 2nd in CMI in the NFC.  Based on the #s, good enough to go.

Drew Brees - Started out blazing, but has had a couple of mediocre games recently.  Has led the Saints to a 7-7 record, and has thrown a league-high 28 touchdowns.  He has also thrown an NFC-high 16 interceptions.  He ranks 3rd, 3rd, 2nd and 10th in each of the components, good enough for 4th in the NFC in terms of passer rating.  He ranks 4th in CMI.  An acceptable choice for the Pro Bowl.

Eli Manning - He has led the NY Giants to an 11-3 record, coming off that Giant upset in last year's Super Bowl against the previously undefeated Pats.  He has thrown 20 touchdown passes, only 6th-best in the NFC.  His ranks in the four components are 10th, 11th, 5th and 5th respectively - making his passer rating only 9th-best in the NFC.  His CMI rank is 10th.  Clearly a sentimental pick here.  I would not have voted him in.

In summary,

Peyton Manning - In

Brett Favre - Out

Jay Cutler - Out

Kurt Warner - In

Drew Brees - In

Eli Manning - Out

Obvious Snubs

There are two, in my mind - one each in the AFC and NFC, respectively.  

In the AFC, Chad Pennington is an obvious snub.  Clearly, he doesn't get attention amongst his peers, because he doesn't throw a lot of touchdowns.  I've never understood why throwing a touchdown is so important to determining how good a passer someone is.  CMI, which has a 97% correlation with the current NFL passer rating, doesn't include touchdowns.  Pennington has thrown for only 14 touchdowns.  He has led the Miami Dolphins to a 9-5 record (keep in mind that Miami only won 1 game last season).  Based on the four components, he ranks 3rd, 3rd, 10th (the component associated with TDs), and 1st, good enough for second-best in the AFC in passer rating.  He ranks 1st in CMI.

In the NFC, Jeff Garcia is an obvious snub.  Although he has led the Bucs to a 9-5 record, he suffers from the same 'disease' that Pennington has.  He doesn't get a lot of recognition around the league for how good he is because he doesn't throw a lot of touchdowns.  He has 'only' thrown 10 touchdowns.  That hurts him in terms of passer rating, as he ranks, 2nd, 7th, 12th and 1st in each of the components, good for 3rd in the NFC in terms of passer rating.  He ranks #1 in the NFC in CMI.

Possibly Overlooked

In the AFC, both Matt Schaub and Philip Rivers have possibly been overlooked.  Matt Schaub has led the Texans to a 7-7 record, and has a passer rating of 92.7, 3rd-best in the AFC.  In terms of the four components, he ranks 2nd, 2nd, 6th and 13th, respectively.  He is 3rd in the AFC in CMI.  Philip Rivers, although having a good year (in terms of the traditional touchdowns thrown), suffers from the Chargers being 6-8.  He leads the NFL in passer rating at 101.4, and leads the NFL in touchdowns thrown with 28.  He ranks 6th, 1st, 1st and 7th in each of the components, and ranks 4th in the AFC in CMI.

In the NFC, Jason Campbell and Aaron Rodgers have both been overlooked in my opinion.  Campbell's Redskins are 7-7 on the season, and he has thrown 'only' 12 touchdowns.  He ranks 7th, 12th, 13th, and 2nd in each of the four components, good enough for only 10th in terms of passer rating.  He ranks 3rd in CMI.  What hurts Rodgers most is the fact that the Packers are 5-9.  He ranks 4th in the NFC in touchdowns thrown, with 23.  He ranks 4th, 6th, 3rd and 9th in the four components, good for 6th in the NFC in passer rating.  His CMI rank is also 6th.

In looking at all these quarterbacks, something struck me, and that is the exceptional seasons that Garcia, Campbell and Pennington are having in terms of the infrequency of interceptions thrown (see here for their stats).  I'll have a post on this later.

It appears to me that what gets attention (and votes) is winning and throwing touchdowns.  At some point I will do an extensive analysis showing the relationship between a quarterback and winning.  What doesn't get attention (but should) is interceptions thrown.  

The table below shows the stats for the NFL's quarterbacks through 14 games this season (only those who 'qualified' (i.e. those throwing at least 224 passes) are shown.

Is Peyton Manning Having a Good Year?

Peyton Manning had a great game last night against the Jaguars, leading the Colts to a 31-24 victory, and a spot in the playoffs.  That makes it three games in a row now that he hasn't thrown an interception.  During that span, he has completed over 80% of his passes.

However, it seems to me that he hasn't received his proper due this year.  Perhaps it is because he has been rather mediocre until this recent stretch.  He did have three terrible games (week 3 at home against these same Jaguars, where he threw 2 picks and the Colts lost 23-21, week 7 on the road against the the Packers, where he threw two picks, and the Colts lost 34-14, and week 13, on the road against the Browns, where he again threw 2 picks, yet the Colts won 10-6).  He has thrown 2 picks in two other games this season, making it a total of 5 games this year where he has thrown at least two picks.  To truly appreciate how good Manning has been, consider this: Since 2002, he has never had a season where he threw 2 or more picks in a game in more than 2 games.

The table below compares each of Manning's years in the NFL.  For sake of comparison, I have included the league average CMI for qualified passers only - i.e. those who attempted at least 256 passes in a season (224 for the 14 games completed so far in the 2008 season).  The chart illustrates that the league average CMI has been steadily increasing during these years, and Manning continues to perform better than the league (he has done this every year, except for his rookie season - 1998).

From this data, I would conclude that he is indeed having a good year, at least as compared to the league.  He is having a typical year for him.

In the table below, the column "Manning CMI rel to League Avg" is calculated as follows:

Manning CMI Rel = [Manning CMI - League Avg CMI] / Std Dev of League Avg CMI

For example, for 2008,

Manning CPI Rel = [0.599 - 0.542] / 0.052 = 1.08