Was Steve McNair A Good Passer?

Earlier this month, Steve McNair was shot and killed. It was a story that caught everyone by surprise, and there was much said about his contributions to the game, and how he played the game. I thought it might be worthwhile to digress briefly from my current topic - the economics of the NFL Draft to take a look back at McNair's career.

Steve McNair was drafted in 1995 out of little-known Alcorn State. The 3rd overall pick in the draft, he was the 1st quarterback taken that year. The then Houston Oilers (now Tennessee Titans) drafted him ahead of current Titans quarterback Kerry Collins.

He started out his career backing up Chris Chandler in both 1995 and 1996 while the team was still the Houston Oilers. The first year in Tennessee, he took over the starting job. He had a sub-par first year (as most quarterbacks tend to do), and then went on to have a fine career. He is of course, most notably remembered for coming one yard short of leading the Titans to a victory against the St Louis Rams in Superbowl XXXIV. The table and chart below illustrate his career in terms of the standard deviations from the mean CMI.

As you can see, from 1998 until his retirement after the 2007 season, he finished each year at above the mean CMI, with the exceptions of the 2004 and 2007 seasons which were cut short due to injury. His best year was 2003, where he finished 1.40 standard deviations above the mean CMI. His statistics that year: 62.5% completion rate, and a 1.75% interception rate. For his performance on the field that year, he was named Co-NFL MVP along with Peyton Manning. He was traded to the Baltimore Ravens following the 2005 season.

All told, Steve McNair had 9 qualifying years in his 13-year career. Of the 51 quarterbacks since 1940 to throw enough passes to qualify in 9 or more seasons, he ranks 17th. His career average standard deviation from the mean CMI during his 9 years was 0.46, putting him in the company of quarterbacks such as Norm Van Brocklin, John Brodie and Sid Luckman. The chart below illustrates the 51 quarterbacks, and Steve McNair's standing among them.

The Importance of Interceptions (or lack thereof)

I have spent quite a bit of time lately talking about interceptions.  In case you have any doubt that an interception can make a difference you got that answer on Super Bowl Sunday.  In probably one of the greatest plays in Super Bowl history (until the catch by Santonio Holmes with 35 seconds left that gave the Steelers a come-from behind victory), with Arizona on the Pittsburgh 1 yard line, first and goal, and 18 seconds left in the first half, the Steelers' James Harrison picked off Arizona's Kurt Warner and returned it all the way for a touchdown as time expired.  This was, in effect, a 13-point play, as Arizona's expected points at the Pittsburgh 1 was about 6 points.  Brian Burke, who I've highlighted before, has an excellent post on the subject on his site at advancednflstats.com.   

Take a look at the graph below.  The blue line (on the left scale) shows the league average QB passer rating (for those QBs who thew enough passes during the season to qualify) by year since 1940.  As you know, the NFL's QB passer rating formula has four components - completion percentage, yards per attempt, touchdown percentage, and interception percentage (see my previous posts on the subject here and here).  When it was designed in 1973, the formula used the 1972 season as a "base", and hence created adjustments to each component, such that the average would be a score of 1.00 for each component, resulting in a passer rating for 66.7 for a quarterback who had average statistics in each of the four categories.  The actual calculations for each of the four components in 1972 yielded the following four figures - 1.085, 0.954, 0.897, and 1.043, respectively, which in turn yielded the average quarterback passer rating of 66.3 (the figure for the qualified leaders turns out only slightly higher - 67.9).  Back then, each of the four components were essentially balanced.

Today, it's a different story.  If the system was balanced, then we would expect the interceptions component to make up about 25% of the quarterback passer rating score.  Looking at the graph again, and this time looking at the red line (on the right scale) shows how much the interceptions component influences the league's quarterback rating system.  It hasn't been below 30% since 1983, and the last time it was "around" 25% was actually 1971.  The point is that the NFL passer rating system is not what it used to be, and interceptions are the leading weight in today's NFL passer rating system.

Let's take a look at another example.  Tom Brady's record-breaking 2007 season.  That year, Brady completed 398 of 578 passes, for 4,806 yards, and threw for 50 touchdowns while only throwing 8 interceptions.  His passer rating that year - 117.2, is second only to Peyton Manning's all-time best 121.1 in 2004.  Most people will remember the 50 touchdowns that Brady threw.  Impressive as it was, it wasn't that impressive (I'll have a post later on this subject).  Most people will not remember that he only threw 8 interceptions in 578 attempts.  That ranks as 14th best all-time in terms of single-season interception percentage (see Exhibit 2), and, it accounted for 29% of his quarterback passer rating that year (his touchdowns accounted for 25%).

There are 7 exhibits on interceptions attached to this post.   Note that for all exhibits, I only considered those passers in any given year that threw enough passes to qualify - in other words, if a quarterback threw 20 passes during a season, that would not be enough to qualify him.  On the nfl.com site, the minimum qualification standard is 14 passes per game (so, for the 2008 season, the standard would be 14 * 16 = 224 passes).  The NFL uses this standard EVERY year.  

I think this universal standard is inappropriate.  Using this method for example, only three passers qualified in 1940.  The 14 passes comes from the fact that during the 50's, 60's and 70's, the average number of passes attempted by a team during a game was about 28.  Of course, we all know that this has changed over time.  The average number of passes attempted in a game by a team was 32.3 in 2008.  It has been below 30.0 once since 1979.  

The standards that I use varies by year, and reflects the average # of passes attempted during these years.  Here are the standards:

1940-1946 - 6.5

1947-1960 - 11.0

1961-1969 - 14.0

1970-1977 - 12.0

1978-1994 - 15.0

1995-2008 - 16.0

Using these standards, I get 1,451 quarterbacks in my database, with a low of 9 quarterbacks qualifying in each of 1941 and 1943, and a high of 32 quarterbacks qualifying in each of 1999 and 2005.  For 2008, I had 30 quarterbacks in my qualified database, whereas the NFL.com has 32.  So, not a big difference in recent years.  I just think that applying a universal standard across all these years is silly, especially when the game has changed so much.

In any case, now that we got the some of the technical stuff out of the way, here are the 7 exhibits:

Exhibit 1 - Chronological list of NFL leader (lowest) in interception percentage

Exhibit 2 - Best seasons in terms of interception percentage

Exhibit 3 - Best seasons in terms of standard deviations from the mean

Exhibit 4 - Worst seasons in terms of standard deviations from the mean

Exhibit 5 - Best (lowest) career interception rate (minimum of 1,000 passes attempted)

Exhibit 6 - Best career interception rate relative to league average

Exhibit 7 - Worst (highest) career interception rate relative to league average

Exhibit 1 - Chronological list:

Nothing spectacular here, it's simply each year's best.  I observe a few things: 

Clearly, the average interception rate has been decreasing every decade.  

The best quarterbacks in a given year seem to be around 1.8 standard deviations better than the average.

Slinging Sammy Baugh led the league 4 out of 6 years during the period from 1942 to 1947 - the only quarterback to have led the league on 4 separate occasions.  

6 quarterbacks - Bart Starr, Bobby Thomason, Charlie Conerly, Ken Anderson, Ken O'Brien, and Roger Staubach have led the league on 3 different occasions.

5 quarterbacks led the league in consecutive seasons - Sammy Baugh, Bobby Thomason, Milt Plum, Ken Anderson and Ken O'Brien.

Only 2 quarterbacks in history, Steve DeBerg in 1990 (0.90%), and David Garrard in 2007 (0.92%) completed a season where less than 1 percentage of their attempted passes were intercepted (this is not entirely evident by looking at Exhibit 1, but can be confirmed by Exhibit 2).  What's most unusual about DeBerg's performance that year was the fact that of the four interceptions he threw during the season, three of them were in one game!  In other words, he threw 1 INT the rest of the season.  Let's take a look at Exhibit 2.

Exhibit 2 - Best seasons in terms of interception percentage:

So indeed, only 2 quarterbacks have had seasons with fewer than 1% interceptions.  The list below is the kind of list that would show up in a record book.  

You'll see, for example, that Jason Campbell's 2008 season ranks in the top 5 all-time, and 2 other quarterbacks in 2008, Chad Pennington and Jeff Garcia, also had noteworthy seasons, with both finishing in the Top 30 all-time.  

If you look carefully, you'll notice that the list is dominated by quarterbacks in the past 2 decades.  Of the Top 50, 45 have occurred since 1990.  21 last decade and 24 this decade.  The 5 seasons in the top 50 not to have occurred in the last 19 years are Steve Bartkowski's 1983 season (#3), Bart Starr's 1966 and 1964 seasons, respectively (#7 and #18), and Ken O'Brien's 1985 and 1988 season's respectively (#36 and #38).  

So, what are we saying?  Are we saying that quarterbacks prior to 1990 were not very good?  No, not at all.  The league has changed.  From Exhibit 1, you can see that the average interception rate has been decreasing every decade.  So, we simply can't just compare a quarterback from one decade to another.  That's where Exhibit 3 comes in.  Let's take a look at that.

Exhibit 3 - Best seasons - Interceptions percentage, ranked by how different the particular season was compared to the mean, using the standard deviation as the measuring stick:

Wow, what a difference.  You can quickly see that this is a much better representation of the past 7 decades.

1940's - 3

1950's - 2

1960's - 10

1970's - 9

1980's - 7

1990's - 6

2000's - 13

Look at Bart Starr!  Ranked twice in the Top 5, and 3 times in the Top 25.  

And, this measure doesn't discount Steve DeBerg's and David Garrard's great seasons - they're both still in the Top 5.  But it does give one a bit more perspective.  In other words, this suggests that Bart Starr's 1962 season (which ranks #430 in absolute terms), when compared to everyone else's performance during that season, was better than David Garrard's performance, when compared to how all the other quarterbacks did in 2007.

Now for a little math.  Why are we using standard deviation as a measure of separation?  And why does using it make comparing quarterbacks across years more meaningful?  

A non-technical definition of the standard deviation is that it is a measure of the dispersion of a set of data around the average.  By dispersion we mean spread.  Knowing the average of the data, and knowing how spread the data is, we can try to determine how likely a given observed value is.  We can use this data to compare different data sets, and relate them to one another.   So for example, in a data set where the average is 5, and the standard deviation is 2, an observation of 8, would mean 1.5 standard deviations ((8-5)/2 = 1.5) above the the mean.  In a data set where the average was 7, and the standard deviation was 4, a value of 13 would also be 1.5 standard deviations from the mean.   You can now see why the use of a standard deviation could be useful in comparing different sets of data.  

As you can see from Exhibit 1, the league average interception rate has been changing over time.  In addition, while I have not shown it explicitly, the standard deviation around the mean has also been changing.  As a matter of fact, in the early 1940's, the standard deviations were quite high because there were fewer players involved, the number of attempted passes were fewer, and arguably the talent pool was not as great (i.e. passing was a fairly new concept).  

By relating a given observation of an interception rate in 1943 to the mean interception rate that year, and the standard deviation of interception rate that year, one can then compare that particular observation to an observation of an interception rate in 2003, by its relationship to the mean and standard deviation of interception rates in 2003.  If one assumes, in particular,  that in any given year, that all observations about the mean are distributed normally (i.e. a "bell-shaped" curve), then the comparisons become that much more meaningful.  For example we know that in a standard bell-curve, that approximately 68% of the observations will fall into a band +/- 1 standard deviation from the mean, and approximately 95% of observations will fall into a band +/- 2 standard deviations from the mean.  Also, if, after relating the observations in each year to each year's mean and standard deviation, one aggregates the data across all years (since the data has been "normalized" to the same scale - a number in relation to a mean and standard deviation; in the example earlier, the observation of 8 in the first data set has the same value on a normalized basis as the observation of 13 in the second data set - 1.5), then the aggregated data should look like a standard normal curve, with a mean of 0, and a standard deviation of 1.  

Well, I went through the trouble of doing that, and guess what - that is exactly what the looks like.  All 1,451 qualified passers over the 69 years from 1940 to 2008 were analyzed in this manner, and the result is in the graphical illustration below.  The average for the entire data set is -0.06 (close to 0), and the standard deviation is 1.00!  Also, it turns out that 69% of the observations are within +/- 1 standard deviation, and 96% are within +/- 2 standard deviations.  Quite a remarkable achievement!  I've graphed a standard normal distribution as well, so that you can see for yourself how close the actual data is to the theoretical curve.

Exhibit 4 - Same as Exhibit 3, except ranks the worst seasons:

Terry Bradshaw's 1970 rookie season ranks as the all-time worst.  And Vinny Testaverde's 1998 campaign is not that far behind.  Although Testaverde's rookie season was 1987, he didn't "get exposed" until 1988.  Most recently, Gus Frerotte's 2008 campaign was an absolute disaster.  There's some names on the list that I would not have expected to see (let's face it, these are the 50 worst performances over the past 69 years - 1,451 quarterback seasons) - names like Favre and Aikman, along with the aforementioned Bradshaw.  What was even more surprising to me was that they each showed up not once, but twice!

So now you've seen the best and the worst seasons.  How about careers?  Let's take a look at Exhibit 5.

Exhibit 5 - Lowest career interception rate - absolute figures - with a minimum of 1,000 passes attempted:

For reasons I discussed above, I don't like this list that much, as it is biased towards the more recent years.  But, since people like looking at these types of lists, I have included it.  Exhibits 6 (best) and 7 (worst) reorder the data after a player's career has been compared to the average during their career.

Exhibits 6 (best/lowest) and 7 (worst/highest) - career interception rate relative to league average during the same time:

First question is: why, after the long dissertation about standard deviations, am I not using that as the measuring stick?  Simple answer - it's difficult!  It was a relatively simply exercise to calculate standard deviations for each year.  And it's not that difficult to do it for a given set of consecutive years.  Where it becomes increasingly difficult is to do it for every combination of multiple years, over a 69-year period.  At some point before the 2009 NFL season, I'll have it done.  That will be a better reflection of who had good or bad careers, but, in the meantime, this will have to suffice.  It's the next best thing.  It is most certainly better than the absolute comparison shown in Exhibit 5.

Ok, on to a few observations.

Exhibit 6 is a who's who of quarterbacks in football history.  Taking out the 13 players who are either currently playing, or who have retired in the past five years (i.e. not Hall of Fame eligible), 14 of the remaining 37 are in the Hall of Fame.  Let's look at it another way.  There are 27 quarterbacks who are in the Hall of Fame, who played football post 1940.  And two of them, Arnie Herber, and Clarence (Ace) Parker, played much of their careers before 1940.  Of the remaining 25, 14 show up in Exhibit 6.  

Who are the 14?

Joe Montana (#3)

Roger Staubach (#9)

Steve Young (#14)

Sammy Baugh (#19)

Fran Tarkenton (#20)

Bart Starr (#22)

Sonny Jurgensen (#23)

Otto Graham (#32)

Dan Marino (#33)

Sid Luckman (#37)

John Elway (#39)

Len Dawson (#42)

Johnny Unitas (#46)

Norm Van Brocklin (#50)

And 2 more, Y.A. Tittle (#53), and Troy Aikman (#55) just missed being in the Top 50.  Not a bad list.

What about Exhibit 7?  3 Hall of Famers, Joe Namath, George Blanda, and Terry Bradshaw are in the worst 50 all-time, in terms of their career interception percentage as it relates to the league average during the time that they played.  

Super Bowl Quarterbacks

Super Bowl XLIII is tomorrow.  So, I thought it might be a good exercise to review quarterback performances in each of the past 42 Super Bowls.  The first table below shows how each of the starting quarterbacks did.  Here are a few observations.

Quarterbacks in the Super Bowl make lots of mistakes.  Look at the quarterbacks on the losing team.  Until Super Bowl XXV between the Giants and Bills, every starting quarterback on a losing team had thrown at least one interception.  As a matter of fact, there have been only 4 times - Super Bowl XXV between the NY Giants and Buffalo Bills, Super Bowl XXXIV between the St Louis Rams and Tennessee Titans, Super Bowl XXXVIII between the New England Patriots and Carolina Panthers and last year's Super Bowl XLII between the NY Giants and New England Patriots, where the losing team's quarterback did not throw an interception.  When you think about each of these four games, one could make the argument that they all rank as the best played, most exciting and closest Super Bowls of all time.  

Starting QBs on the losing teams have thrown a total of 83 interceptions in the 42 games.  This compares to only 22 for starting QBs on the winning teams.  Don't throw an interception, and you give your team an excellent chance of winning the game.  Conversely, throw a pick, and you significantly reduce your team's chances of winning.  

Nine times the losing team's QB threw 3 picks; 4 times they threw 4 picks, and in Super Bowl XXXVII between the Tampa Bay Buccaneers and Oakland Raiders, Rich Gannon threw 5 picks.  That's 14 times in 42 games where a QB threw 3 or more INTs.  There's a 1/3 chance that tomorrow one of the two QBs will throw at least 3 picks!  

Three times the winning team's QB has thrown for more than 2 INTs.  In Super Bowl XIV, Pittsburgh's Terry Bradshaw threw 3, in Super Bowl XVII, Washington's Joe Theismann threw 2, and in Super Bowl XL, Ben Roethlisberger threw 2 INTs.  Note that Johnny Unitas threw 2 INTs in Super Bowl V as a back-up in a win over Dallas.

There have been 9 occasions where the winning team's QB threw for 3 or more TDs.  4 times they have thrown 3, 3 times they have thrown 4, and in Super Bowl XXIV, San Francisco's Joe Montana threw 5 TDs and in Super Bowl XXIX, San Francisco's Steve Young threw for 6 TDs against the San Diego Chargers.  It appears that not throwing INTs is probably more important than throwing TDs.  

The classic example of this is Super Bowl III, where NY Jets' Joe Namath guaranteed a victory against the Baltimore Colts.  He delivered.  He was named the game's MVP.  Most people don't realize that he didn't threw a single TD in that game.  Most importantly, he didn't throw an INT either.  

The table below shows (in my opinion) the 10 best and 10 worst performances by a starting QB over the past 42 Super Bowls.  It's difficult to compare QBs over time.  This is because the most commonly used measure to evaluate a QBs passing performance, the NFL Passer Rating system, has shown that the league average has been increasing over time (see here for the details).  

I have come up with a measure, CMI (Completions Minus Interceptions, calculated as Completion % - 3 * Interception Percentage), which I believe is a better measure of a QBs passing performance.  However, this measure also has the same problem that the Passer Rating formula has, in that it also shows that the league average has been increasing over time (see here).  One way to adjust for this trend is to relate each QBs performance in a Super Bowl to the overall league average for that year.  Then, one can compare that particular measure across Super Bowls.  If I had game-by-game data, I would not only relate the measure to the mean, but would also take into account the standard deviation.  However, in the absence of game-by-game data going back 42 years, the measure relative to the mean shall suffice.  Also, I eliminated from consideration any QB who attempted fewer than 15 passes during the game.

With the technical stuff all out of the way, the single best performance by a QB in a Super Bowl is Phil Simms for the NY Giants in Super Bowl XXI, when he went a near-perfect 22 of 25 with no INTs.  Interestingly, his QB Rating for that game was 150.9, close to the perfect rating of 158.3.  Steve Young's 6 TD effort doesn't show up in the Top 10, although Joe Montana's 5 TD effort does.

At the other end of the Spectrum, Craig Morton's 4 for 15 effort with 4 INTs in Super Bowl XII is the absolute worst performance by a QB in a Super Bowl by any measure.  He also has the dubious distinction of showing up twice on the 10 Worst Performances list.  Ben Roethlisberger, Pittsburgh's QB in tomorrow's Super Bowl XLIII, is the only QB who shows up on the 10 Worst list, whose team actually won the game - Super Bowl XL against Seattle.

Super Bowl Squares

It's that time of the year, when seemingly uninterested people run around feigning interest in what is now a true American celebration - the Super Bowl.  This year's Super Bowl features the Pittsburgh Steelers and the Arizona Cardinals.  Not that anyone outside of Phoenix and Pittsburgh actually cares who is in it.  That however won't prevent Super Bowl parties from cropping up across the nation on Sunday, February 1st.  And, at most of those parties, someone will usually run around trying to get the last few stragglers to participate in the second greatest tradition of the Super Bowl - the "Super Bowl Squares" (the first has to be watching the Super Bowl commercials).  

Basically, for a small donation, you get to place your name in a 10x10 square grid.  For multiple donations, you may be able to place your name on multiple squares.  Before the actual game starts, #s from 0-9 are drawn randomly and placed across each of the 10 columns.  The same process is repeated to the left of each row.  Then the name of team 1 is drawn and placed at the top, and the second team is placed on the left.  Now you have a grid that has each of the possible last-digits of the scores of each team.  When the game ends, you look at the score, then look at the last digits of each team's score, and see whose name corresponds to that permutation and that individual wins a pre-determined amount of the accumulated donations.  This process doesn't have to be limited to the game-ending score.  Many variations exist.  For example, frequently some smaller amounts could be won based on the digit permutations at the end of each quarter.

This year, as I almost always do, I participated in one of these Super Bowl Squares.  I donated for 2 squares.  After all the squares were filled out, the #s were randomly drawn, the team's assigned, the coordinator of the game handed me my sheet.  I had drawn the following 2 permutations:

Arizona 2 - Pittsburgh 5

Arizona 8 - Pittsburgh 2

I promptly tossed the sheet in the recycling bin.

Later on, I decided to see for myself the likelihood of my winning.  

The analysis below shows the aggregated game-ending digit permutations and combinations for every NFL game played in the Super Bowl era, including playoff games.  That's 9,509 games!  That also means that there's a reasonable likelihood that the probabilities shown are close to the true probabilities.  As a matter of fact, every single permutation has been "hit" at least once.  A game ending in the 2 2 permutation has only happened once - on Sunday, December 5th, 2004 the Buffalo Bills beat the Miami Dolphins in Miami 42-32.

First, a little math.  I refer to both permutations and combinations.  There is a difference between the two.  A combination refers to a sequence or collection without regard to order.  A permutation is a combination with a specific order.  Here's an example.  Take what we commonly (and mistakenly) refer to as a "combination" lock.  We say to unlock the lock, "use combination 472".  Well, that's actually only mildly helpful.  Knowing those three #s alone we wouldn't be able to open the lock.  We need to know the specific order of that combination of #s.  In other words, is it 274, 247, 427, 472, 724, or 742.  So in this example, there is one combination.  There are six permutations.

Ok, now on to the tables and charts below.

In TABLE 1, I show all 100 permutations of game-ending scores.  So for example, one can see that the likelihood of the game ending with the winning team's score ending in a 4, and the losing team's score ending in a 3 is 2.94% (to see this, in TABLE 1, go down to the row with the digit 4, then across to the column with the digit 3).  So, this specific permutation has a 2.94% likelihood.  

If you didn't care about whether it was the winning or losing team that had the 3 or the 4 in the last digit, as long as there was a 3 and a 4, then look to TABLE 3.  TABLE 3 shows the probabilities of each combination.  As such, the 3 4 or 4 3 combination has about a 3.67% likelihood of occurring.  CHART 2 graphically illustrates what's in TABLE 3.

TABLE 2 is not meaningful in and of itself, but simply shows the probability of any given digit occurring (note that in this table, the percentages add up to greater than 100.00% since "any 7" will include for example, the "1 7" combination, that will also show up under "any 1").  CHART 1 simply illustrates what's in TABLE 2.

Let's take a look at what my chances are.  It's a little complicated so bear with me.  Remember, I have 2 specific permutations.  Ari 2/Pit 5 and Ari 8/Pit 2.  However, since I don't know ahead of time who will win the game, I need to average the 2 permutations that yield the 2 5 combination for the first scenario and the two that yield the 8 2 combination for the second.  I can do this by either going to TABLE 1, and adding the respective likelihoods of each of those permutations, or I can simply go to TABLE 3.  From TABLE 3 I can easily see that the 2 5 combination shows a likelihood of 0.36% (this is made up by the 5 2 permutation likelihood of 0.23% and adding it to the 2 5 permutation likelihood of 0.13%).  Therefore, the average expectation for the specific 2 5 permutation is 0.18%*.  For the second scenario, from TABLE 3, I can see that the likelihood of this combination is 0.28%, and hence the average expectation for the specific permutation is 0.14%.

* (Technically, I shouldn't be averaging the permutations expectations.  What I should be doing is weighing each permutation by the likelihood of Arizona (or Pittsburgh) winning or losing the game.  So for example, if the likelihood of Pittsburgh winning the game is estimated to be 70%, then a truer expectation for my specific 2 5 permutation might be 0.7*0.23% +0.3*0.13% = 0.20%.  However, if you assume that each team's likelihood of winning the game is close to 50%, then averaging is fine).

So there you have it, the combined likelihood that I would win ANYTHING is about 0.32% (0.18%+0.14%).  

Hence why I tossed the sheet.  Hope you have better permutations!  Good Luck!