C Scores - A New Way to Evaluate Pro Football's Best Passers of All-Time

It has taken me two years, many, many iterations, and countless hours of mental anguish, but I think I've finally settled on a reliable method by which I can evaluate any quarterback's passing ability versus any other quarterback's passing ability regardless of when they played, or for how long they played.  So, for example, I can compare Peyton Manning versus Sammy Baugh, or, Otto Graham versus Brett Favre, or, Sonny Jurgensen versus Dan Marino.  You get the point.  It all started coming together for me as I wrote my post on Manning versus Brees versus Favre.  While I had already figured out how to effectively compare one quarterback's given season to that of another's given season, I had as yet to figure out an effective method by which we evaluated careers.

The inspiration came to me as I read Michael J. Schell's book, Baseball's All-Time Best Hitters, where he went about solving a different problem altogether, but had essentially the same characteristics of the problem I was trying to solve.  He was trying to account for the fact that for many hitters with long careers, their batting averages declined after they reached their mid-thirties.  And, he didn't think it was fair to compare a hitter who had played into their late thirties or early forties to a player that had not.  He chose to account for this by using a "late career adjustment".  What he did was to look only at a player's first 8,000 at-bats.  


I adopted this notion, and, instead of using a player's first x years, came up with the idea of using his best x years, given that a player's passing ability, as measured by CMTI, and compared to that of his peers tends to be a little volatile from year to year.


Deciding what x was going to be was an issue.  I wanted to balance outstanding achievement with duration.  In other words, I wanted to see who had performed at an exceedingly high level for a long time.  Remember, from my previous post, the average number of years a quarterback qualifies is 4.125.  The median is 3.  


I thought that a good starting point was to use 4 years.  This measure would include 169 players, or a little more than 40% of my database of 416 players going back to when the NFL officially started keeping statistics, and including both the AAFC in the 1940's and the AFL in the 1960's.


If I used 7 years, then I'd get 95 players, or almost 25% of the total.  That seemed reasonable for a couple of reasons.  First, 25% seemed like a good cut-off, albeit arbitrary.  Second, it would allow me to discard the worst years for those quarterbacks who had 8 or more years, which seemed like an appropriate way to reward longevity.  


I could have looked at 10 years - in which case, we would be limiting ourselves to 50 quarterbacks, or about 12% of the database, a rather small group, and, as we shall see, excluding many superb passers who didn't quite make it to 10 years.  Keep in mind that the median number of years a player qualifies, is 3 years.


My basic method for evaluating a player's performance in a given year is to look at his CMTI, and then, based on the mean and standard deviation for that given year, relate his performance to that of his peers.  Since the aggregate accumulation of these measures over the past 78 years looks very much like a normal (bell-shaped) curve, we can use the bell-curve to convert these measures to percentile ranks, giving a numerical value ranging from 1 to 99 for each of these performances.  The sum of a given quarterback's 4 best such performances are then used as my C4 calculation, and the sum of his 7 best performances are my C7 calculation.  Obviously, then the maximum possible value for C4 is 99 * 4 = 396, and, commensurately, the maximum C7 value is 99 * 7 = 693.  As I mentioned earlier, I've also calculated a C10 value for those players with an exceptionally long career.  Obviously, if we did this for every year a player qualified, one could create a career C score for each player.


Let's take a look at the lists.  We will start with C4.

Wow!  What a list.  Not bad for starters.  Remember, this is a ranking based on a quarterback's 4 best years as a passer.  Look at the top 14 on this list.  12 of them are in the Pro Football Hall of Fame!  Peyton Manning surely will be when he is done.  That's 13 out of 14!  Ken Anderson ranks 3rd all-time as a passer according to this list.  It is important to keep in mind that this is a list of the top-ranked passers, not top-ranked quarterbacks.  It is hard enought to determine a ranking of passers, so it would stand to reason then that it is much harder to determine what makes a top-ranked quarterback.  I don't know how the NFL goes about determining who gets in to the Hall of Fame and who deos not, but it sure seems odd that Ken Anderson is the only player in the top 14 on this list that isn't (assuming that Peyton is in, of course). 

One of the most aesthetically pleasing aspects of this list, especially looking at the top 15, is that every era and every decade is represented.  There does not seem to be a bias.  I would think that is an admirable quality of any top-ranked list in that it shows that the measure used to determine who is better, isn't necessarily biased towards a particular era - as opposed to say, the list of top ranked passers (see here and here) in terms of the NFL Passer Rating system at the Pro-Football Hall of Fame.  


Look at Steve Young's C4.  That's almost unbelievable!  The highest C4 possible is 396.  In other words, that is being in the 99th percentile four years.  Well, Young comes close.  His 1996 and 1997 seasons were in the 99th percentile, and his 1992 and 1994 seasons were in the 98th percentile.


In my previous post, we discussed Kurt Warner's merits as a passer.  We ignored any discussion about whether he will be enshrined in Canton.  Based on this list, it would appear that he is right on the fence.


There are a total of 27 quarterbacks on this list that are members of the Hall of Fame.  There are a total of 31 quarterbacks that have been inducted into the Hall of Fame.  The four that are in the Hall of Fame, but are not on this list are omitted for good reason.  They all played in the 'early years' of the NFL - prior to the keeping of official statistics.  They are, Jimmy Conzelman, who played from 1921-1929, John (Paddy) Driscoll, who played from 1920-1929, Earl (Dutch) Clark, who played from 1931-1938 and actually had two very good years in 1934 and 1936, and Benny Friedman, who played from 1927-1934, and who, many during his time argued was the finest passer of his time.  It is most unfortunate that we don't have official statistics that would show that.


OK, what about C7?

Woah!  An even better list than C4!  First, while the ordering changed, the top 15 stayed exactly the same as the C4 list, which probably suggests more of the power of the C4 list than anything else (in other words, a quarterback's best 4 years is a pretty good indicator of their passing ability).  However, now, we not only have 13 of the top 15 in the HOF (I am considering it a foregone conclusion that Peyton Manning will be inducted into the HOF), but 16 of the top 20 are either in the Hall of Fame or will be (I am also considering it a foregone conclusion that Brett Favre will also be inducted into the HOF).  That is a phenomenal list.  


It seems obvious then, that, in order to be inducted into the Hall of Fame as a quarterback, one must also have been a great passer.  Having said that, it's even more shocking to me now the omission of Ken Anderson from the HOF.  I simply do not understand why he is not in.  I will have to post an article on Ken Anderson in the near future.


What does C7 say about Kurt Warner?  Again, right on the fence.  

From my vantage point, it certainly appears that we should be discussing putting Ken Anderson in the Hall of Fame ahead of Kurt Warner.


Finally, we'll take a look at C10.  This list only includes 50 quarterbacks, and many exceptional passers are not on this list, simply because they didn't qualify in at least 10 years.

The story is similar to what we've already seen with the C4 and C7 lists.  14 of the top 15 (if you include Peyton Manning and Brett Favre as in the Hall of Fame) are or will be in the Hall of Fame.  The lone exception is - you guessed it - Ken Anderson.


There's several conclusions that I can draw from all of this.


I feel pretty good that I've come up with a method by which I can determine the all-time best passers in the history of professional football.  I also feel that this method will stand the test of time.  It will work regardless of how the game changes over time.  It is a method that allows me to conveniently compare players who played in different eras, and it allows me to compare players who have had different career lengths.  


Although being inducted into the Hall of Fame as a quarterback shouldn't be a criterion by which we evaluate a quarterback's passing ability, it does provide a reasonableness check of the method I've employed.  Given the likelihood of being in the Hall of Fame and showing up at the top of these lists, one could theoretically use it as a measure of whether a quarterback will be inducted into the Hall of Fame.  Obviously, there are other factors that enter that equation, but, I can draw some conclusions as to what might be necessary as a passer to be considered a good candidate for the Hall.


A C4 score of 370 or better is a must to be a shoo-in for the Hall.  The caveat there being the quarterback would have to had at least 8 seasons where they qualified.  Looking at C7, I'd say  a C7 score of 625 or better is a requirement, although a score of 600 or better conceivably also works.  A C10 score of 775 or better would be considered a must to have any chance of being inducted into the Hall of Fame (in my opinion anyway).


As I mentioned earlier in the article, I think C4 doesn't consider enough years.  Conversely, I think C10 is too exclusive.  C7 is a good compromise.  I could have created a C8 or C6, but C7 feels like it strikes the right balance between excellence and longevity.  From now on, I will use C7 as the standard measure of a player's career passing prowess, although I may use C4 and C10 where appropriate.  For example, if a player does not have a C7 score, I may use C4 as an early gauge, or, I may use C10 to compare to passers who have had lengthy careers.


Well, now that we've developed the method by which we can fairly present any quarterback's career as a passer, the natural question is, who is the greatest passer of all-time?  My method (C7) would suggest it is Joe Montana.  Steve Young would come in a close second.  You could also look at C4, and you'd see that they are the top 2, although in reverse order.  How good Joe Montana was, and for how long he performed at an incredibly high level, is evident by looking at C10 scores.  He is the only passer in the history of professional football to have a C10 score exceeding 900, and, at 939, he is 50 ahead of Peyton Manning.


A quick digression here, in case it wasn't obvious.  C scores can never decrease, only increase.  So for example, Peyton Manning's scores can only increase from here on out.  Since we consider a player's best years, if a subsequent year for a player isn't one of the best that already was included in the score, then it wouldn't count towards that score.  It would only count if it exceeded, and thereby replaced, one of the years already included.  Since Peyton's 10th best season so far is a 79 (his performance in 2007 was in the 79th percentile), he would have to have a performance better than that in the future to increase his C10 score (I would consider that quite likely).  On the other hand, his 7th best season is an 85 (his 2009 season), and I'd say that he's only about 50/50 to improve upon that, and thereby increase his C7 score (and if he did improve upon that, he would also improve his C10 score).


So, of the current crop of players, who might eventually knock on the door of the Hall of Fame?  Drew Brees and Tom Brady are currently ranked 21 and 22 on the C7 list with scores of 570 and 567, respectively.  A few more good years, and they would move up the ladder.  For example, in the case of Drew Brees, his score of 570 is made up of scores of 93, 93, 89, 88, 77, 75 and 55.  It's almost a guarantee that he would end up dropping the 55.  I think it's a foregone conclusion that his C7 score will eventually exceed 600 by a comfortable margin.  What about Brady?  Well, he is probably already a shoo-in for the Hall on account of his three Super Bowl victories.  But, as a passer, he appears a lot weaker than Brees.  Brady's C7 score of 567 is made up of scores of 98, 89, 80, 76, 75, 75, and 74.  Given his history, I don't see him improving too much from the 567.  Perhaps when it is all said and done, he might end up in the 580-590 range.  One never knows though.


What about the next generation of passers?  Who might be good candidates to have great careers as passers?  Chad Pennington, whose current C4 score is 366, one less than that of Kurt Warner, is an interesting case.  However, given his history of injuries, he may not be able to accumulate a solid C7 score.  If he has one decent year, say, in the 70th percentile, then his C7 score would be 575.  There are two players that I think could potentially have great careers based on their current C4 scores - Carson Palmer and Philip Rivers.  Based on their yearly scores, I think Philip Rivers might be the better bet, even though Palmer currently has the higher C4 score.  Interestingly, Philip Rivers currently ranks 2nd to Steve Young on the all-time passer rating list.  Of those who don't even have a C4 score, Matt Schaub has shown excellent early potential.

Quarterbacks and the NFL Draft - Part 1

In this post, I want to take a trip back in time (back to 1979, specifically), to see which quarterbacks were picked first overall in the regular NFL Draft.  

Why 1979?  Two reasons.  First, it gives us exactly 30 years of data.  Second, it was the year where Joe Montana was drafted out of Notre Dame.  Since 1979, there have been 13 quarterbacks selected as the first overall pick in the regular NFL draft.  This does not include Steve Young, who was taken first overall in the 1984 Supplemental Draft.

A couple of observations.  In the 1980's, 3 quarterbacks were taken as the first overall pick.  In the 1990's there were 4.  In the 2000's, there have been 6 taken, through 2008, and, with the Detroit Lions taking Matthew Stafford first overall in 2009 , that would make 7.

Another way to look at it:  In the 19-year period from 1979 to 1997, there were 5 quarterbacks taken first overall.  Since then, 12 years if you include the 2009 draft, 9 quarterbacks have been taken first overall.  2000, 2006 and 2008 were the exceptions.

Here's the full list.

Anybody Looking for a Good QB?

Available:  Experienced quarterback with an accurate arm with ability to make immediate contributions to team; Ideally, would like to start, but not a requirement; may be perfect as back-up; durability maybe an issue.

I am referring of course to Jeff Garcia, released today by the Tampa Bay Buccaneers.  I am personally a big fan of Garcia, because I believe he is underrated, and his achievements on the field have gone relatively unnoticed.

Let's take a look.

He began his NFL career in San Francisco in 1999, replacing not one, but two back-to-back Hall of Famers, and arguably the two greatest passers of all-time, Joe Montana and Steve Young.  There was no way Jeff could live up to those lofty expectations.  However, he did perform very well, as the table and chart below shows.  While he may not have performed at Joe Montana and Steve Young levels, his years in San Francisco were very productive.  After a sub-par year in 2003, he bounced around for three years, first as a starter in Cleveland (2004), then as a back-up in Detroit (2005) and Philly (2006).  He finished the 2006 season as the starter in Philadelphia for the last 6 games after Donovan McNabb got injured.  He showed that he could still play, and Tampa Bay, in need of a quarterback following an injury to Chris Simms, picked him up.  He did well above average each of 2007 and 2008, as you can see from the table and charts (he doesn't show up in the charts in the years 2004-2006 since he didn't attempt enough passes to qualify).  

Now, Tampa Bay, deciding to turn to the youth movement, have invested in Luke McCown, and, have deemed Garcia expendable.  

Garcia has been a very accurate as a passer throughout his career.  In a previous post, I discussed interceptions at length.  His 2007 season ranks 18th all-time (see Exhibit 3 in this post) in interceptions per attempt, in terms of standard deviations from the mean (his 2008 season ranks 118th, out of the 1,451 seasons in the database since 1940).  As a career passer, he ranks 12th all-time (see Exhibit 6 in this post).  By not throwing interceptions, he will help your team win.

I believe that Garcia has a couple of good years still left in him, if not as a starter, as a back-up.  His durability has been an issue the past two seasons, and hence, might be a liability as the starter.  

So who might be worthwhile candidates?

The following charts show, for each of 2007 and 2008, how a team's passing game stacked up in terms of standard deviations from the mean.  It's no coincidence that teams that were above average won more games than they lost, and teams that were below average lost more games than they won.  The charts show how the teams did both in terms of NFL passer rating and CMI.  Note that Garcia averaged 0.53 standard deviations above average in terms of NFL passer rating for the past two seasons, and averaged 1.23 standard deviations above average in terms of CMI.

To see which teams should be looking to get Garcia, let's start at the bottom, and move our way up.

Cleveland Browns - They have two young quarterbacks, Derek Anderson and Brady Quinn, both coming off season-ending injuries, that will compete for the starting job.  Not a candidate.  Look for another miserable year in Cleveland.

St Louis Rams - Marc Bulger is the starter, and for now at least, another soon-to-be 39 year-old - Trent Green, is the back-up.  Not a candidate.  By the way, what's happened to Bulger?  Will 2009 be more like his first few years, or his last two?  I suspect the latter.  Another long year in St Louis.

Oakland Raiders - Paying a lot of money for JaMarcus Russell.  They've also got Andrew Walter, Marques Tuiasosopo, and they just picked up Bruce Gradkowski off waivers.  I haven't a clue what the Raiders are doing, and I don't think the Raiders do either.  Not a candidate.  Perhaps we'll start paying attention when Al Davis passes away.

San Francisco Forty Niners - They're not sure who their starting quarterback is.  Shaun Hill?  J.T. O'Sullivan?  Alex Smith?  The niners maybe a candidate.  Garcia played here, and is originally from close-by Gilroy.

Detroit Lions - Daunte Culpepper is the starter.  Or is it Dan Orlovsky?  Didn't seem to matter in 2008, as they went 0-16.  If I was GM, I'd replace Culpepper with Garcia, and draft a quarterback.  But that's just me.  Garcia may not want to go here, but that's a different matter altogeher.

Chicago Bears - Kyle Orton is the starter, and Rex Grossman is the back-up.  Neither of whom are very good.  I would think that Garcia would be an excellent choice to be a back-up here.

Kansas City Chiefs - Tyler Thigpen is the starter.  Previous starter Damon Huard is the back-up.  Garcia could be a potential back-up here.

Cincinnati Bengals - Ryan Fitzpatrick finished the year as the starter, having replaced the injured Carson Palmer.  Not a candidate.

Of the remaining teams, perhaps the Minnesota Vikings might be interested.  Gus Frerotte is simply not a good option.  Maybe they'll lure Brett Favre out of retirement!  The NY Jets might be an option for Garcia as well.  Most likely, Kellen Clemens will be the starter, but Garcia could be a viable back-up here.  

So these are teams that should be interested in Garcia.  We'll wait and see where he actually ends up.  

The Importance of Interceptions (or lack thereof)

I have spent quite a bit of time lately talking about interceptions.  In case you have any doubt that an interception can make a difference you got that answer on Super Bowl Sunday.  In probably one of the greatest plays in Super Bowl history (until the catch by Santonio Holmes with 35 seconds left that gave the Steelers a come-from behind victory), with Arizona on the Pittsburgh 1 yard line, first and goal, and 18 seconds left in the first half, the Steelers' James Harrison picked off Arizona's Kurt Warner and returned it all the way for a touchdown as time expired.  This was, in effect, a 13-point play, as Arizona's expected points at the Pittsburgh 1 was about 6 points.  Brian Burke, who I've highlighted before, has an excellent post on the subject on his site at advancednflstats.com.   

Take a look at the graph below.  The blue line (on the left scale) shows the league average QB passer rating (for those QBs who thew enough passes during the season to qualify) by year since 1940.  As you know, the NFL's QB passer rating formula has four components - completion percentage, yards per attempt, touchdown percentage, and interception percentage (see my previous posts on the subject here and here).  When it was designed in 1973, the formula used the 1972 season as a "base", and hence created adjustments to each component, such that the average would be a score of 1.00 for each component, resulting in a passer rating for 66.7 for a quarterback who had average statistics in each of the four categories.  The actual calculations for each of the four components in 1972 yielded the following four figures - 1.085, 0.954, 0.897, and 1.043, respectively, which in turn yielded the average quarterback passer rating of 66.3 (the figure for the qualified leaders turns out only slightly higher - 67.9).  Back then, each of the four components were essentially balanced.

Today, it's a different story.  If the system was balanced, then we would expect the interceptions component to make up about 25% of the quarterback passer rating score.  Looking at the graph again, and this time looking at the red line (on the right scale) shows how much the interceptions component influences the league's quarterback rating system.  It hasn't been below 30% since 1983, and the last time it was "around" 25% was actually 1971.  The point is that the NFL passer rating system is not what it used to be, and interceptions are the leading weight in today's NFL passer rating system.

Let's take a look at another example.  Tom Brady's record-breaking 2007 season.  That year, Brady completed 398 of 578 passes, for 4,806 yards, and threw for 50 touchdowns while only throwing 8 interceptions.  His passer rating that year - 117.2, is second only to Peyton Manning's all-time best 121.1 in 2004.  Most people will remember the 50 touchdowns that Brady threw.  Impressive as it was, it wasn't that impressive (I'll have a post later on this subject).  Most people will not remember that he only threw 8 interceptions in 578 attempts.  That ranks as 14th best all-time in terms of single-season interception percentage (see Exhibit 2), and, it accounted for 29% of his quarterback passer rating that year (his touchdowns accounted for 25%).

There are 7 exhibits on interceptions attached to this post.   Note that for all exhibits, I only considered those passers in any given year that threw enough passes to qualify - in other words, if a quarterback threw 20 passes during a season, that would not be enough to qualify him.  On the nfl.com site, the minimum qualification standard is 14 passes per game (so, for the 2008 season, the standard would be 14 * 16 = 224 passes).  The NFL uses this standard EVERY year.  

I think this universal standard is inappropriate.  Using this method for example, only three passers qualified in 1940.  The 14 passes comes from the fact that during the 50's, 60's and 70's, the average number of passes attempted by a team during a game was about 28.  Of course, we all know that this has changed over time.  The average number of passes attempted in a game by a team was 32.3 in 2008.  It has been below 30.0 once since 1979.  

The standards that I use varies by year, and reflects the average # of passes attempted during these years.  Here are the standards:

1940-1946 - 6.5

1947-1960 - 11.0

1961-1969 - 14.0

1970-1977 - 12.0

1978-1994 - 15.0

1995-2008 - 16.0

Using these standards, I get 1,451 quarterbacks in my database, with a low of 9 quarterbacks qualifying in each of 1941 and 1943, and a high of 32 quarterbacks qualifying in each of 1999 and 2005.  For 2008, I had 30 quarterbacks in my qualified database, whereas the NFL.com has 32.  So, not a big difference in recent years.  I just think that applying a universal standard across all these years is silly, especially when the game has changed so much.

In any case, now that we got the some of the technical stuff out of the way, here are the 7 exhibits:

Exhibit 1 - Chronological list of NFL leader (lowest) in interception percentage

Exhibit 2 - Best seasons in terms of interception percentage

Exhibit 3 - Best seasons in terms of standard deviations from the mean

Exhibit 4 - Worst seasons in terms of standard deviations from the mean

Exhibit 5 - Best (lowest) career interception rate (minimum of 1,000 passes attempted)

Exhibit 6 - Best career interception rate relative to league average

Exhibit 7 - Worst (highest) career interception rate relative to league average

Exhibit 1 - Chronological list:

Nothing spectacular here, it's simply each year's best.  I observe a few things: 

Clearly, the average interception rate has been decreasing every decade.  

The best quarterbacks in a given year seem to be around 1.8 standard deviations better than the average.

Slinging Sammy Baugh led the league 4 out of 6 years during the period from 1942 to 1947 - the only quarterback to have led the league on 4 separate occasions.  

6 quarterbacks - Bart Starr, Bobby Thomason, Charlie Conerly, Ken Anderson, Ken O'Brien, and Roger Staubach have led the league on 3 different occasions.

5 quarterbacks led the league in consecutive seasons - Sammy Baugh, Bobby Thomason, Milt Plum, Ken Anderson and Ken O'Brien.

Only 2 quarterbacks in history, Steve DeBerg in 1990 (0.90%), and David Garrard in 2007 (0.92%) completed a season where less than 1 percentage of their attempted passes were intercepted (this is not entirely evident by looking at Exhibit 1, but can be confirmed by Exhibit 2).  What's most unusual about DeBerg's performance that year was the fact that of the four interceptions he threw during the season, three of them were in one game!  In other words, he threw 1 INT the rest of the season.  Let's take a look at Exhibit 2.

Exhibit 2 - Best seasons in terms of interception percentage:

So indeed, only 2 quarterbacks have had seasons with fewer than 1% interceptions.  The list below is the kind of list that would show up in a record book.  

You'll see, for example, that Jason Campbell's 2008 season ranks in the top 5 all-time, and 2 other quarterbacks in 2008, Chad Pennington and Jeff Garcia, also had noteworthy seasons, with both finishing in the Top 30 all-time.  

If you look carefully, you'll notice that the list is dominated by quarterbacks in the past 2 decades.  Of the Top 50, 45 have occurred since 1990.  21 last decade and 24 this decade.  The 5 seasons in the top 50 not to have occurred in the last 19 years are Steve Bartkowski's 1983 season (#3), Bart Starr's 1966 and 1964 seasons, respectively (#7 and #18), and Ken O'Brien's 1985 and 1988 season's respectively (#36 and #38).  

So, what are we saying?  Are we saying that quarterbacks prior to 1990 were not very good?  No, not at all.  The league has changed.  From Exhibit 1, you can see that the average interception rate has been decreasing every decade.  So, we simply can't just compare a quarterback from one decade to another.  That's where Exhibit 3 comes in.  Let's take a look at that.

Exhibit 3 - Best seasons - Interceptions percentage, ranked by how different the particular season was compared to the mean, using the standard deviation as the measuring stick:

Wow, what a difference.  You can quickly see that this is a much better representation of the past 7 decades.

1940's - 3

1950's - 2

1960's - 10

1970's - 9

1980's - 7

1990's - 6

2000's - 13

Look at Bart Starr!  Ranked twice in the Top 5, and 3 times in the Top 25.  

And, this measure doesn't discount Steve DeBerg's and David Garrard's great seasons - they're both still in the Top 5.  But it does give one a bit more perspective.  In other words, this suggests that Bart Starr's 1962 season (which ranks #430 in absolute terms), when compared to everyone else's performance during that season, was better than David Garrard's performance, when compared to how all the other quarterbacks did in 2007.

Now for a little math.  Why are we using standard deviation as a measure of separation?  And why does using it make comparing quarterbacks across years more meaningful?  

A non-technical definition of the standard deviation is that it is a measure of the dispersion of a set of data around the average.  By dispersion we mean spread.  Knowing the average of the data, and knowing how spread the data is, we can try to determine how likely a given observed value is.  We can use this data to compare different data sets, and relate them to one another.   So for example, in a data set where the average is 5, and the standard deviation is 2, an observation of 8, would mean 1.5 standard deviations ((8-5)/2 = 1.5) above the the mean.  In a data set where the average was 7, and the standard deviation was 4, a value of 13 would also be 1.5 standard deviations from the mean.   You can now see why the use of a standard deviation could be useful in comparing different sets of data.  

As you can see from Exhibit 1, the league average interception rate has been changing over time.  In addition, while I have not shown it explicitly, the standard deviation around the mean has also been changing.  As a matter of fact, in the early 1940's, the standard deviations were quite high because there were fewer players involved, the number of attempted passes were fewer, and arguably the talent pool was not as great (i.e. passing was a fairly new concept).  

By relating a given observation of an interception rate in 1943 to the mean interception rate that year, and the standard deviation of interception rate that year, one can then compare that particular observation to an observation of an interception rate in 2003, by its relationship to the mean and standard deviation of interception rates in 2003.  If one assumes, in particular,  that in any given year, that all observations about the mean are distributed normally (i.e. a "bell-shaped" curve), then the comparisons become that much more meaningful.  For example we know that in a standard bell-curve, that approximately 68% of the observations will fall into a band +/- 1 standard deviation from the mean, and approximately 95% of observations will fall into a band +/- 2 standard deviations from the mean.  Also, if, after relating the observations in each year to each year's mean and standard deviation, one aggregates the data across all years (since the data has been "normalized" to the same scale - a number in relation to a mean and standard deviation; in the example earlier, the observation of 8 in the first data set has the same value on a normalized basis as the observation of 13 in the second data set - 1.5), then the aggregated data should look like a standard normal curve, with a mean of 0, and a standard deviation of 1.  

Well, I went through the trouble of doing that, and guess what - that is exactly what the looks like.  All 1,451 qualified passers over the 69 years from 1940 to 2008 were analyzed in this manner, and the result is in the graphical illustration below.  The average for the entire data set is -0.06 (close to 0), and the standard deviation is 1.00!  Also, it turns out that 69% of the observations are within +/- 1 standard deviation, and 96% are within +/- 2 standard deviations.  Quite a remarkable achievement!  I've graphed a standard normal distribution as well, so that you can see for yourself how close the actual data is to the theoretical curve.

Exhibit 4 - Same as Exhibit 3, except ranks the worst seasons:

Terry Bradshaw's 1970 rookie season ranks as the all-time worst.  And Vinny Testaverde's 1998 campaign is not that far behind.  Although Testaverde's rookie season was 1987, he didn't "get exposed" until 1988.  Most recently, Gus Frerotte's 2008 campaign was an absolute disaster.  There's some names on the list that I would not have expected to see (let's face it, these are the 50 worst performances over the past 69 years - 1,451 quarterback seasons) - names like Favre and Aikman, along with the aforementioned Bradshaw.  What was even more surprising to me was that they each showed up not once, but twice!

So now you've seen the best and the worst seasons.  How about careers?  Let's take a look at Exhibit 5.

Exhibit 5 - Lowest career interception rate - absolute figures - with a minimum of 1,000 passes attempted:

For reasons I discussed above, I don't like this list that much, as it is biased towards the more recent years.  But, since people like looking at these types of lists, I have included it.  Exhibits 6 (best) and 7 (worst) reorder the data after a player's career has been compared to the average during their career.

Exhibits 6 (best/lowest) and 7 (worst/highest) - career interception rate relative to league average during the same time:

First question is: why, after the long dissertation about standard deviations, am I not using that as the measuring stick?  Simple answer - it's difficult!  It was a relatively simply exercise to calculate standard deviations for each year.  And it's not that difficult to do it for a given set of consecutive years.  Where it becomes increasingly difficult is to do it for every combination of multiple years, over a 69-year period.  At some point before the 2009 NFL season, I'll have it done.  That will be a better reflection of who had good or bad careers, but, in the meantime, this will have to suffice.  It's the next best thing.  It is most certainly better than the absolute comparison shown in Exhibit 5.

Ok, on to a few observations.

Exhibit 6 is a who's who of quarterbacks in football history.  Taking out the 13 players who are either currently playing, or who have retired in the past five years (i.e. not Hall of Fame eligible), 14 of the remaining 37 are in the Hall of Fame.  Let's look at it another way.  There are 27 quarterbacks who are in the Hall of Fame, who played football post 1940.  And two of them, Arnie Herber, and Clarence (Ace) Parker, played much of their careers before 1940.  Of the remaining 25, 14 show up in Exhibit 6.  

Who are the 14?

Joe Montana (#3)

Roger Staubach (#9)

Steve Young (#14)

Sammy Baugh (#19)

Fran Tarkenton (#20)

Bart Starr (#22)

Sonny Jurgensen (#23)

Otto Graham (#32)

Dan Marino (#33)

Sid Luckman (#37)

John Elway (#39)

Len Dawson (#42)

Johnny Unitas (#46)

Norm Van Brocklin (#50)

And 2 more, Y.A. Tittle (#53), and Troy Aikman (#55) just missed being in the Top 50.  Not a bad list.

What about Exhibit 7?  3 Hall of Famers, Joe Namath, George Blanda, and Terry Bradshaw are in the worst 50 all-time, in terms of their career interception percentage as it relates to the league average during the time that they played.  

Super Bowl Quarterbacks

Super Bowl XLIII is tomorrow.  So, I thought it might be a good exercise to review quarterback performances in each of the past 42 Super Bowls.  The first table below shows how each of the starting quarterbacks did.  Here are a few observations.

Quarterbacks in the Super Bowl make lots of mistakes.  Look at the quarterbacks on the losing team.  Until Super Bowl XXV between the Giants and Bills, every starting quarterback on a losing team had thrown at least one interception.  As a matter of fact, there have been only 4 times - Super Bowl XXV between the NY Giants and Buffalo Bills, Super Bowl XXXIV between the St Louis Rams and Tennessee Titans, Super Bowl XXXVIII between the New England Patriots and Carolina Panthers and last year's Super Bowl XLII between the NY Giants and New England Patriots, where the losing team's quarterback did not throw an interception.  When you think about each of these four games, one could make the argument that they all rank as the best played, most exciting and closest Super Bowls of all time.  

Starting QBs on the losing teams have thrown a total of 83 interceptions in the 42 games.  This compares to only 22 for starting QBs on the winning teams.  Don't throw an interception, and you give your team an excellent chance of winning the game.  Conversely, throw a pick, and you significantly reduce your team's chances of winning.  

Nine times the losing team's QB threw 3 picks; 4 times they threw 4 picks, and in Super Bowl XXXVII between the Tampa Bay Buccaneers and Oakland Raiders, Rich Gannon threw 5 picks.  That's 14 times in 42 games where a QB threw 3 or more INTs.  There's a 1/3 chance that tomorrow one of the two QBs will throw at least 3 picks!  

Three times the winning team's QB has thrown for more than 2 INTs.  In Super Bowl XIV, Pittsburgh's Terry Bradshaw threw 3, in Super Bowl XVII, Washington's Joe Theismann threw 2, and in Super Bowl XL, Ben Roethlisberger threw 2 INTs.  Note that Johnny Unitas threw 2 INTs in Super Bowl V as a back-up in a win over Dallas.

There have been 9 occasions where the winning team's QB threw for 3 or more TDs.  4 times they have thrown 3, 3 times they have thrown 4, and in Super Bowl XXIV, San Francisco's Joe Montana threw 5 TDs and in Super Bowl XXIX, San Francisco's Steve Young threw for 6 TDs against the San Diego Chargers.  It appears that not throwing INTs is probably more important than throwing TDs.  

The classic example of this is Super Bowl III, where NY Jets' Joe Namath guaranteed a victory against the Baltimore Colts.  He delivered.  He was named the game's MVP.  Most people don't realize that he didn't threw a single TD in that game.  Most importantly, he didn't throw an INT either.  

The table below shows (in my opinion) the 10 best and 10 worst performances by a starting QB over the past 42 Super Bowls.  It's difficult to compare QBs over time.  This is because the most commonly used measure to evaluate a QBs passing performance, the NFL Passer Rating system, has shown that the league average has been increasing over time (see here for the details).  

I have come up with a measure, CMI (Completions Minus Interceptions, calculated as Completion % - 3 * Interception Percentage), which I believe is a better measure of a QBs passing performance.  However, this measure also has the same problem that the Passer Rating formula has, in that it also shows that the league average has been increasing over time (see here).  One way to adjust for this trend is to relate each QBs performance in a Super Bowl to the overall league average for that year.  Then, one can compare that particular measure across Super Bowls.  If I had game-by-game data, I would not only relate the measure to the mean, but would also take into account the standard deviation.  However, in the absence of game-by-game data going back 42 years, the measure relative to the mean shall suffice.  Also, I eliminated from consideration any QB who attempted fewer than 15 passes during the game.

With the technical stuff all out of the way, the single best performance by a QB in a Super Bowl is Phil Simms for the NY Giants in Super Bowl XXI, when he went a near-perfect 22 of 25 with no INTs.  Interestingly, his QB Rating for that game was 150.9, close to the perfect rating of 158.3.  Steve Young's 6 TD effort doesn't show up in the Top 10, although Joe Montana's 5 TD effort does.

At the other end of the Spectrum, Craig Morton's 4 for 15 effort with 4 INTs in Super Bowl XII is the absolute worst performance by a QB in a Super Bowl by any measure.  He also has the dubious distinction of showing up twice on the 10 Worst Performances list.  Ben Roethlisberger, Pittsburgh's QB in tomorrow's Super Bowl XLIII, is the only QB who shows up on the 10 Worst list, whose team actually won the game - Super Bowl XL against Seattle.

Crazy, Debatable and Lucky

Today, the Green Bay Packers lost at home to the Houston Texans, 24-21 on a last-second field goal.  This dropped the Packers to 5-8 for the season, and, for all practical purposes, knocked them out of the playoffs.  I'm sure the Packer faithful will start to think what might have been....

The Packers were coming off a 13-3 season, having won their division handily, and were one win away from the Super Bowl, a loss to the eventual winner, the New York Giants at home.  

Then the soap opera started.  First, Brett Favre, the Packers QB for the past 16 seasons, decided it was time to retire.  The Packers, who had drafted Aaron Rodgers in 2005, were prepared.  During the offseason, the Packers named Aaron Rodgers the starting QB for the 2008 season.  Second, Brett Favre decided he wanted to play in 2008.  Nothing wrong with that.  Only that he had "lost" his starting job to Aaron Rodgers.  No way a future hall of famer is going to be carrying a clipboard for an untested, essentially rookie QB.  Not after the season Favre just had had.  After much ballyhoo, the Packers traded Favre to the New York Jets.  In return, the Packers received a conditional draft pick.  For now, it's a fairly safe bet that it will be at least a second rounder.  The Jets, who finished a lousy 4-12 season, now having found their "saviour", then released Chad Pennington.  Of course, the woeful Miami Dolphins, having just completed a miserable 1-15 season, and desperately needing some talent, promptly picked him up.

So there we have it.  The Packers make a "crazy" move, going with untested Aaron Rodgers, over Hall of Famer Favre.  The Jets, in need of a new direction, trade for Favre and release Pennington in a debatable move.  And the Dolphins, well, got lucky.  Pennington is a dramatic improvement over any Dolphin QB.

Ok, so that's how we all thought about it back in the beginning of the season.  So, let's take a look at how the decisions panned out, in terms of their actual performances.

The table at the top of this article shows each player's stats for the 2008 season, the 2007 season, and their career #s.

First, let's start with Rodgers.

He is having a very good year, despite the team's troubles.  His QB Rating is a robust 92.1 (league average is 85.3), and his CMI is 0.560, about 0.4 standard deviations above the mean of 0.538.  Not bad for being a first-year starter.  Compare him to three other first-year starters here.

Now, let's take a look at Favre.

He had a career year in 2007.  His QB Rating of 95.7 was only exceeded in his MVP years of 1995 (99.5) and 1996 (95.8).  His third MVP year, 1997, his QB rating was 92.6.  Favre's career QB Rating (leading up to the 2007 season) was 85.0.  It has since improved to 85.9 through 13 games of the 2008 season.  In terms of his CMI, 2007 was by far and away, his best year (0.581), eclipsing his previous best of 0.561 in 1995.  Given that it had been a long time since he had put up that kind of #s, it would be reasonable to assume that it wouldn't necessarily continue into 2008.  Well, he's having a (surprisingly) outstanding year in 2008.  His QB Rating is a healthy 88.2, and his CMI is 0.577 (almost as good as last year).  Is he why the Jets have improved?  Hard to say.  Especially, when you take a look at what Pennington has done this year.

Pennington's #s:

Chad is easily having his best year since his first full year in the league, in 1992.  Back then, he produced a QB Rating of 104.2 and a CMI of 0.644.  Only Tom Brady in 2007 (0.647), and Hall of Famer Troy Aikman, back in 2003 (0.645), have had better seasons.  This year, Chad's #s are a QB Rating of 93.7, and a CMI of 0.618.  Compare these to his career #s of 89.7 and 0.578.  By the way, Pennington's career CMI of 0.578 is the all-time career NFL mark.  Next is Steve Young at 0.565.  Pennington's #s are also, needless to say, a dramatic improvement compared to those of the 2007 Dolphins (QB Rating of 67.4, and CMI of 0.484).

So, what does this all mean?  It means that the Packers weren't crazy when they decided to put their faith in Rodgers, the Jets decision is debatable in giving up a 2nd round pick in 2009, and exchanging a QB who, while having an outstanding year, is not doing as well as the QB they gave up on, and the Dolphins got awfully lucky to end up with Pennington.

The 25 Best Seasons by a Quarterback (using current QB Formula)

Here are the 25 best seasons that a quarterback has had in the NFL (since 1940) using the current QB Rating system.  I have previously mentioned my dislike for the system.  However, for now, this is the list.

Note:  This list is slightly different than that shown at The Pro Football Hall of Fame.  The HOF has Bart Starr's 1968 season at #19 with a Rating of 104.3.  I did not include him here as he didn't "qualify", having only attempted 171 passes.  For 1968, I used 14 attempts per game (or, 196 for the season) as the qualifier.

Best Quarterbacks in History (in terms of current QB Rating Formula)

Here are the Top 50 quarterbacks all-time, based on QB Rating.  I only considered those with at least 1,500 pass attempts (and there were 142 in all over the course of history that had at least 1,500 attempts).  Data is only through the full 2007 season.  I will update this list following the end of the 2008 season.  One can clearly see the bias favoring the modern quarterback.

Here's the next 50.  One needs to look at this list if you wanted to see QB's from yesteryear.

So, it begs the question, does the QB Rating system need to be changed so that fair comparisons can be made over time?  After all, the methodology behind it indexes the statistics to the 1972 season.