Creating a New Passer Rating System for the NFL

C Scores for Current Quarterbacks

2010-05-09T19:39:00.000-07:00

I thought it might be illustrative to show you C scores for all the quarterbacks who qualified in 2009. With Kurt Warner retiring at the end of last season, it is certain that not all of them will qualify again in 2010, and, almost assuredly some of the remaining won't qualify in 2010. The difficulty in looking at these players is their body of work. Not all of them will have a C7 score, our standard measure. That being said however, it is still worth a look. Instead of simply limiting the list to those that have a C7 score, we will look at all of their C scores (C1, C4, C7 and C10).

Here's what the list looks like:

Not too many surprises here.

Did you expect someone other than Peyton Manning to be the best passer in the game today? If you did, I'd love to hear from you, and I'd like to see the analysis. Peyton Manning is one of the all-time best (see here and here), and most certainly the greatest passer of his era. Based on C7, I currently have him 7th all-time (C7 of 644), right behind both Fran Tarkenton and Len Dawson who are tied for sixth at 646. He has an outside chance (I'd say about 25%) of catching Sammy Baugh, who is 5th with a C7 of 654.

In the article I wrote on Kurt Warner (see here), I showed that Warner compares favorably to players like Drew Brees, and Tom Brady. In my recent post formally introducing C scores I suggested that as far as the Hall of Fame, that Warner appears to be right on the fence. I certainly would make the argument that Ken Anderson should be in before Warner.

In that same article I also discussed Tom Brady and Drew Brees. I think if they both played long enough that they would both be in the HOF, and I'm fairly certain that Brees would end up as the better passer.

As far as the next group of players, certainly Carson Palmer and Philip Rivers appear to be on their way to becoming good passers, although they've still got a few years ahead of them before we can make any definitive statements.

And finally, looking further out in to the future, I'd say that Matt Schaub is an up-and-comer, and, really taking a flyer, perhaps Aaron Rodgers can parlay his early success to be considered a great passer. Only time will tell.

Pro Football's All-Time Best Passers

2010-04-27T18:41:00.000-07:00

Now that I've got a method for evaluating a quarterback as a passer and can use it compare his career to that of others, I thought it might be illustrative to compare the Top 50 on my list to how they rank on the all-time passer rating list provided by the Pro Football Hall of Fame. The list provided by the Hall only includes those passers who've attempted 1,500 or more passes in their career. Also, as of today (April 27, 2010), the Hall of Fame list does not include stats from the 2009 season. As I've mentioned in the past, the HOF list unfairly penalizes those players who played in an era where passing was secondary. Why? Because of the restriction of limiting it to only those passers attempting 1,500 or more passes. So, to supplement the list from the HOF, I'll show you their passer rating ranking based on my list of 416 passers, which is a broader and more inclusive list, especially of those who played in the 1930's - 1970's. My list includes any quarterback who qualified in at least one season.

Here's the top 50 list of All-Time Best Passers:

My list of 50 not only covers the entire spectrum of pro football history - from the beginning in the 1930's to current players, but it also presents the players who played in earlier years fairly, and in context. The question isn't whether Joe Montana was a better passer than Sammy Baugh, but, whether Joe Montana relative to Montana's peers, was a better passer than Sammy Baugh was relative to Baugh's peers. The HOF list can be used to answer that question, since a measure relative to a fixed standard (in other words the passer rating system) can be used to do that. However, the HOF list cannot be used to answer the next question - how does one compare Sammy Baugh to Joe Montana given that they both played an entirely different game, with different rules and circumstances. A fixed standard cannot solve that particular puzzle. The use of a mean and standard deviation by year (z scores) which in turn we convert to percentile ranks can be used. C7 is simply a quarterback's 7 best years as a passer in terms of percentile rank summed up.

C Scores - A New Way to Evaluate Pro Football's Best Passers of All-Time

2010-04-26T18:19:00.000-07:00

It has taken me two years, many, many iterations, and countless hours of mental anguish, but I think I've finally settled on a reliable method by which I can evaluate any quarterback's passing ability versus any other quarterback's passing ability regardless of when they played, or for how long they played. So, for example, I can compare Peyton Manning versus Sammy Baugh, or, Otto Graham versus Brett Favre, or, Sonny Jurgensen versus Dan Marino. You get the point. It all started coming together for me as I wrote my post on Manning versus Brees versus Favre. While I had already figured out how to effectively compare one quarterback's given season to that of another's given season, I had as yet to figure out an effective method by which we evaluated careers.

The inspiration came to me as I read Michael J. Schell's book, Baseball's All-Time Best Hitters, where he went about solving a different problem altogether, but had essentially the same characteristics of the problem I was trying to solve. He was trying to account for the fact that for many hitters with long careers, their batting averages declined after they reached their mid-thirties. And, he didn't think it was fair to compare a hitter who had played into their late thirties or early forties to a player that had not. He chose to account for this by using a "late career adjustment". What he did was to look only at a player's first 8,000 at-bats.

I adopted this notion, and, instead of using a player's first x years, came up with the idea of using his best x years, given that a player's passing ability, as measured by CMTI, and compared to that of his peers tends to be a little volatile from year to year.

Deciding what x was going to be was an issue. I wanted to balance outstanding achievement with duration. In other words, I wanted to see who had performed at an exceedingly high level for a long time. Remember, from my previous post, the average number of years a quarterback qualifies is 4.125. The median is 3.

I thought that a good starting point was to use 4 years. This measure would include 169 players, or a little more than 40% of my database of 416 players going back to when the NFL officially started keeping statistics, and including both the AAFC in the 1940's and the AFL in the 1960's.

If I used 7 years, then I'd get 95 players, or almost 25% of the total. That seemed reasonable for a couple of reasons. First, 25% seemed like a good cut-off, albeit arbitrary. Second, it would allow me to discard the worst years for those quarterbacks who had 8 or more years, which seemed like an appropriate way to reward longevity.

I could have looked at 10 years - in which case, we would be limiting ourselves to 50 quarterbacks, or about 12% of the database, a rather small group, and, as we shall see, excluding many superb passers who didn't quite make it to 10 years. Keep in mind that the median number of years a player qualifies, is 3 years.

My basic method for evaluating a player's performance in a given year is to look at his CMTI, and then, based on the mean and standard deviation for that given year, relate his performance to that of his peers. Since the aggregate accumulation of these measures over the past 78 years looks very much like a normal (bell-shaped) curve, we can use the bell-curve to convert these measures to percentile ranks, giving a numerical value ranging from 1 to 99 for each of these performances. The sum of a given quarterback's 4 best such performances are then used as my C4 calculation, and the sum of his 7 best performances are my C7 calculation. Obviously, then the maximum possible value for C4 is 99 * 4 = 396, and, commensurately, the maximum C7 value is 99 * 7 = 693. As I mentioned earlier, I've also calculated a C10 value for those players with an exceptionally long career. Obviously, if we did this for every year a player qualified, one could create a career C score for each player.

Let's take a look at the lists. We will start with C4.

Wow! What a list. Not bad for starters. Remember, this is a ranking based on a quarterback's 4 best years as a passer. Look at the top 14 on this list. 12 of them are in the Pro Football Hall of Fame! Peyton Manning surely will be when he is done. That's 13 out of 14! Ken Anderson ranks 3rd all-time as a passer according to this list. It is important to keep in mind that this is a list of the top-ranked passers, not top-ranked quarterbacks. It is hard enought to determine a ranking of passers, so it would stand to reason then that it is much harder to determine what makes a top-ranked quarterback. I don't know how the NFL goes about determining who gets in to the Hall of Fame and who deos not, but it sure seems odd that Ken Anderson is the only player in the top 14 on this list that isn't (assuming that Peyton is in, of course).

One of the most aesthetically pleasing aspects of this list, especially looking at the top 15, is that every era and every decade is represented. There does not seem to be a bias. I would think that is an admirable quality of any top-ranked list in that it shows that the measure used to determine who is better, isn't necessarily biased towards a particular era - as opposed to say, the list of top ranked passers (see here and here) in terms of the NFL Passer Rating system at the Pro-Football Hall of Fame.

Look at Steve Young's C4. That's almost unbelievable! The highest C4 possible is 396. In other words, that is being in the 99th percentile four years. Well, Young comes close. His 1996 and 1997 seasons were in the 99th percentile, and his 1992 and 1994 seasons were in the 98th percentile.

In my previous post, we discussed Kurt Warner's merits as a passer. We ignored any discussion about whether he will be enshrined in Canton. Based on this list, it would appear that he is right on the fence.

There are a total of 27 quarterbacks on this list that are members of the Hall of Fame. There are a total of 31 quarterbacks that have been inducted into the Hall of Fame. The four that are in the Hall of Fame, but are not on this list are omitted for good reason. They all played in the 'early years' of the NFL - prior to the keeping of official statistics. They are, Jimmy Conzelman, who played from 1921-1929, John (Paddy) Driscoll, who played from 1920-1929, Earl (Dutch) Clark, who played from 1931-1938 and actually had two very good years in 1934 and 1936, and Benny Friedman, who played from 1927-1934, and who, many during his time argued was the finest passer of his time. It is most unfortunate that we don't have official statistics that would show that.

OK, what about C7?

Woah! An even better list than C4! First, while the ordering changed, the top 15 stayed exactly the same as the C4 list, which probably suggests more of the power of the C4 list than anything else (in other words, a quarterback's best 4 years is a pretty good indicator of their passing ability). However, now, we not only have 13 of the top 15 in the HOF (I am considering it a foregone conclusion that Peyton Manning will be inducted into the HOF), but 16 of the top 20 are either in the Hall of Fame or will be (I am also considering it a foregone conclusion that Brett Favre will also be inducted into the HOF). That is a phenomenal list.

It seems obvious then, that, in order to be inducted into the Hall of Fame as a quarterback, one must also have been a great passer. Having said that, it's even more shocking to me now the omission of Ken Anderson from the HOF. I simply do not understand why he is not in. I will have to post an article on Ken Anderson in the near future.

What does C7 say about Kurt Warner? Again, right on the fence.

From my vantage point, it certainly appears that we should be discussing putting Ken Anderson in the Hall of Fame ahead of Kurt Warner.

Finally, we'll take a look at C10. This list only includes 50 quarterbacks, and many exceptional passers are not on this list, simply because they didn't qualify in at least 10 years.

The story is similar to what we've already seen with the C4 and C7 lists. 14 of the top 15 (if you include Peyton Manning and Brett Favre as in the Hall of Fame) are or will be in the Hall of Fame. The lone exception is - you guessed it - Ken Anderson.

There's several conclusions that I can draw from all of this.

I feel pretty good that I've come up with a method by which I can determine the all-time best passers in the history of professional football. I also feel that this method will stand the test of time. It will work regardless of how the game changes over time. It is a method that allows me to conveniently compare players who played in different eras, and it allows me to compare players who have had different career lengths.

Although being inducted into the Hall of Fame as a quarterback shouldn't be a criterion by which we evaluate a quarterback's passing ability, it does provide a reasonableness check of the method I've employed. Given the likelihood of being in the Hall of Fame and showing up at the top of these lists, one could theoretically use it as a measure of whether a quarterback will be inducted into the Hall of Fame. Obviously, there are other factors that enter that equation, but, I can draw some conclusions as to what might be necessary as a passer to be considered a good candidate for the Hall.

A C4 score of 370 or better is a must to be a shoo-in for the Hall. The caveat there being the quarterback would have to had at least 8 seasons where they qualified. Looking at C7, I'd say a C7 score of 625 or better is a requirement, although a score of 600 or better conceivably also works. A C10 score of 775 or better would be considered a must to have any chance of being inducted into the Hall of Fame (in my opinion anyway).

As I mentioned earlier in the article, I think C4 doesn't consider enough years. Conversely, I think C10 is too exclusive. C7 is a good compromise. I could have created a C8 or C6, but C7 feels like it strikes the right balance between excellence and longevity. From now on, I will use C7 as the standard measure of a player's career passing prowess, although I may use C4 and C10 where appropriate. For example, if a player does not have a C7 score, I may use C4 as an early gauge, or, I may use C10 to compare to passers who have had lengthy careers.

Well, now that we've developed the method by which we can fairly present any quarterback's career as a passer, the natural question is, who is the greatest passer of all-time? My method (C7) would suggest it is Joe Montana. Steve Young would come in a close second. You could also look at C4, and you'd see that they are the top 2, although in reverse order. How good Joe Montana was, and for how long he performed at an incredibly high level, is evident by looking at C10 scores. He is the only passer in the history of professional football to have a C10 score exceeding 900, and, at 939, he is 50 ahead of Peyton Manning.

A quick digression here, in case it wasn't obvious. C scores can never decrease, only increase. So for example, Peyton Manning's scores can only increase from here on out. Since we consider a player's best years, if a subsequent year for a player isn't one of the best that already was included in the score, then it wouldn't count towards that score. It would only count if it exceeded, and thereby replaced, one of the years already included. Since Peyton's 10th best season so far is a 79 (his performance in 2007 was in the 79th percentile), he would have to have a performance better than that in the future to increase his C10 score (I would consider that quite likely). On the other hand, his 7th best season is an 85 (his 2009 season), and I'd say that he's only about 50/50 to improve upon that, and thereby increase his C7 score (and if he did improve upon that, he would also improve his C10 score).

So, of the current crop of players, who might eventually knock on the door of the Hall of Fame? Drew Brees and Tom Brady are currently ranked 21 and 22 on the C7 list with scores of 570 and 567, respectively. A few more good years, and they would move up the ladder. For example, in the case of Drew Brees, his score of 570 is made up of scores of 93, 93, 89, 88, 77, 75 and 55. It's almost a guarantee that he would end up dropping the 55. I think it's a foregone conclusion that his C7 score will eventually exceed 600 by a comfortable margin. What about Brady? Well, he is probably already a shoo-in for the Hall on account of his three Super Bowl victories. But, as a passer, he appears a lot weaker than Brees. Brady's C7 score of 567 is made up of scores of 98, 89, 80, 76, 75, 75, and 74. Given his history, I don't see him improving too much from the 567. Perhaps when it is all said and done, he might end up in the 580-590 range. One never knows though.

What about the next generation of passers? Who might be good candidates to have great careers as passers? Chad Pennington, whose current C4 score is 366, one less than that of Kurt Warner, is an interesting case. However, given his history of injuries, he may not be able to accumulate a solid C7 score. If he has one decent year, say, in the 70th percentile, then his C7 score would be 575. There are two players that I think could potentially have great careers based on their current C4 scores - Carson Palmer and Philip Rivers. Based on their yearly scores, I think Philip Rivers might be the better bet, even though Palmer currently has the higher C4 score. Interestingly, Philip Rivers currently ranks 2nd to Steve Young on the all-time passer rating list. Of those who don't even have a C4 score, Matt Schaub has shown excellent early potential.

A Look Back At Kurt Warner's Career As a Passer

2010-03-09T22:32:00.000-08:00

On the Friday before Superbowl XLIV, Kurt Warner announced that he was retiring fro m the NFL.

Of course, the conversations now turn to whether he will be enshrined in Canton. There are those who say yes, some who say maybe, and those who say no. The fact is, we don't know. There is no formula to get in.

In many of the posts on the topic, the usual comparisons are made. The easiest comparisons are to his peers - Manning, Brady, Favre, Brees, and others. The other comparisons I've seen are to players like Dan Fouts, Jim Kelly, Troy Aikman, and Warren Moon, all four of whom are in Canton.

My post is not about whether he was a good quarterback or not, and whether he should or should not be in the Hall of Fame. This post is about whether or not he was a good passer. And how did he compare to his peers? And where would he rank in terms of recent quarterbacks who have retired? And how would he compare to all quarterbacks? These are the questions I will explore. All of these questions will be answered in the context of his passing ability in relation to the others.

Ok, let's start with a look back at his performance over the years.

Several things jump out at me.

His career passer rating, for one. It's very high. According to the Pro Football Hall of Fame, Warner's career passer rating of 93.7 ranks fifth best all-time, behind Steve Young, Philip Rivers, Tony Romo and Peyton Manning. As I have discussed before, the NFL passer rating formula isn't a very meaningful way to compare quarterbacks over time. I'll discuss how we adjust the passer rating, and even simplify it, and then compare that figure to his peers, in order to get a more meaningful picture. The average passer rating of 95.1 shown above differs from that shown by the Pro Football Hall of Fame simply because mine is just a simple average of each year.

Another fact that jumps out at me is that he got a late start. He didn't start really playing until he was 28. In my database of 416 quarterbacks going all the way back to 1932, including players from both the AAFC in the late 1940's and the AFL in the 1960's, and only looking at quarterbacks who have qualified in 8 or more years, only 7 quarterbacks had their first qualifying season at age 28 or later. For all quarterbacks who qualified in at least 8 seasons, both the median and mean age at which they qualified in their first season is 24. There are a total of 77 quarterbacks who qualified in at least 8 seasons. Keep in mind that the average number of years in which a quarterback qualifies (given that he qualifies in at least 1 year) is 4.125, and median number of years is 3. So to be one of only 7 quarterbacks who began at age 28 or later and qualify in 8 different seasons is pretty rare company. The other six players are Jeff Garcia and Roger Staubach at age 29, and Warren Moon, Billy Kilmer, Ken Stabler, and Brad Johnson at age 28. Note that Staubach and Moon are both in the Hall of Fame.

Next thing I notice is that there are many gaps in his career. He got injured, then was signed by the NY Giants as an insurance policy against Eli Manning. Following that, he ended up getting an opportunity to play in Arizona, until the Cardinals drafted Matt Leinart. When Leinart got hurt in the fifth game of the 2007 season, Kurt stepped in, and went on to lead the Cardinals into Superbowl XLIII, and into the playoffs this past season, where the Cardinals lost to eventual Superbowl Champs New Orleans.

It is extremely difficult to use the NFL Passer Rating system, especially the career passer rating for a given quarterback to see how they compare to other quarterbacks. Just take a look at the list provided by the Pro Football Hall of Fame. This is simply not a list of the best passers over time. It merely reflects the fact that the passer rating calculation favors current passers. I have previously shown (here) why this is not an accurate reflection of passing ability.

Perhaps one way to adjust the passer rating is to reflect the standards such that the average in each year turns out to be 66.7 (as the intended application of the passer rating). See my post on the subject here. In that regard, Warner's (adjusted) career passer rating is around 83. Of the 416 quarterbacks, that would rank him 23rd. Of the quarterbacks qualifying in 8 or more years, it would rank him 10th.

While the Adjusted Passer Rating is better than the NFL Passer Rating, in that at least one can compare quarterbacks from one era with those of another, it's still not an ideal (or simple) measure of a passer's effectiveness. I've used CMTI in the past, and I'll use it here. More specifically, I'll use the CMTIPR, or percentile rank of CMTI.

In my previous post, I compared Manning, Favre and Brees using CMTIPR. We had to adjust CMTIPR for length of career, and I used their best seasons as the measure. At the end of that post, a graphical illustration showed the effect. I've since improved upon that concept. Using the concept of "Best Seasons", I've come up with 4 statistical measures, C1, C4, C7 and C10, that basically shows a given quarterback's best season (C1), 4 best seasons (C4), 7 best seasons (C7) and 10 best seasons (C10). C1 is simply the quarterback's best season in terms of percentile rank of CMTI. C4 is similarly the sum of the percentile ranks of his 4 best seasons, and so on.

So let's take a look at Warner, and how he compares to Manning, Favre, and Brees.

Obviously, neither Brees nor Warner will have a C10 score as they've not had 10 seasons in which they qualified. I don't think one can evaluate a career based on 1 year, so let's not use C1. It is shown simply for illustration purposes to highlight their best year. So that leaves us with C4 or C7. In order to truly define greatness, one must exemplify superior performance, but also perform at that level for a number of years. The more years performing at a high level, the greater the player. I think that C7 strikes a good balance. In my database, there are 50 quarterbacks who have a C10 score, 95 quarterbacks who have a C7 score, and 169 quarterbacks who have a C4 score. So, at least based on a C7 score, Warner compares very favorably to Manning, Favre and Brees.

We will go into much more details regarding the C scores for quarterbacks in my next post.

3 Things I Would Change About The NFL If I was NFL Commissioner

2010-02-28T20:18:00.000-08:00

This past season, we saw a great football game between the New Orleans Saints and the Minnesota Vikings in the NFC Championship game. In overtime, the Saints defeated the Vikings, and, as a result, headed to Superbowl XLIV, where they eventually defeated the Indianapolis Colts. It was a story-book ending to their season.

For Minnesota Vikings fans it was a bitter conclusion to what they were hoping was also going to be a story-book ending. There they were, driving down the field in the waning moments of the fourth quarter when Brett Favre threw a fateful interception. The Vikings never touched the ball after that. At the start of the overtime, the Vikings lost the coin toss, and the Saints took the ball, drove down the field, and little known Garrett Hartley kicked the winning field goal.

This made me wonder about the merits of the overtime rules as they are, as well as other musings I've had on the game.

In this post, I will discuss 3 things that I would change about the game that would (in my opinion) make the game better. Different, but better.

Points After Touchdown
Field Goals
Overtime

Point After Touchdown

The point after attempt in the NFL is worth one point, if the team chooses to kick the ball, and the ball goes between the uprights. The kick is placed on the 2-yard line, in effect making it a 19-yard field goal. The team does have an option of either passing the ball into the end zone or running it in, and, if successful, the point after attempt is then worth two points. Here are the success rates of the one point point-after attempts in the NFL in each of the last three seasons:

2007 - 1,165/1,177 = 98.98%
2008 - 1,170/1,176 = 99.49%
2009 - 1,165/1,185 = 98.31%

2007 - 2009 - 3,500/3,538 = 98.93%

But that's not all. I mentioned earlier that a point after attempt (PAT) is the same as kicking a 19-yard field goal. Over the past three seasons, the success rate on field goals 0-19 yards are 100% (35/35). As a matter of fact, the last kicker to miss a field goal less than 20 yards was Kris Brown, then with the Houston Texans, all the way back in 2002.

My Proposed Change(s)

Eliminate the point after kick, as well as any field goal attempt less than 20 yards.

I offer two alternatives in lieu of this:

First, after each touchdown, you could place the ball on the 2-yard line, or maybe even the 3-yard line, and you must either run it in, or throw it into the end zone. A successful attempt would be worth 2 points. I have read that the success today on a two-point conversion is somewhere between 50%-55%. If the ball was placed on the 3-yard line, presumably this would decrease to about 45%.

Second, if you choose, you could place the ball on the 17-yard line, and kick the point-after attempt. This would be equivalent to kicking a 34-yard field goal. A successful conversion would be worth 1 point. Here are the success rates of field goals 30-39 yards for each of the past three seasons.

2007 - 253/279 = 90.68%
2008 - 286/321 = 89.10%
2009 - 240/287 = 83.62%

2007 - 2009 - 779/887 = 87.82%

Both of these scenarios would have about the same expected point value - about 0.9 points.

One of the biggest reasons sports is exciting is the uncertainty of the outcome. Moving away from a basically meaningless play (the current PAT) and converting it into both a strategic decision and introducing additional elements of uncertainty would create tremendous excitement. In my opinion, it would also reduce the likelihood of there being a tie game at the end of regulation.

Field Goals

Two rules that changed the game of basketball for the better and made it more exciting for the fans are the introduction of the shot clock in 1954, and the introduction of the 3-point shot in the 1979-1980 season. It is hard to argue that these two elements make the game less exciting today than it was before these inventions. A given team always has the chance of "going for the three". The success rate is lower, but the reward is better. The basic premise of the 3-point shot is that the further out you are from the basket, the tougher the shot, and hence the higher the point value. There is a single line, basically 23 feet 9 inches from the basket (the reason it's not exactly 23 feet 9 inches, and other facts about the three point shot are discussed here).

The following table shows field goal accuracy in the NFL, by distance, over each of the past three seasons.

Makes perfect sense, doesn't it? The further out you are, the more difficult the kick. Also, the further out you are in attempting a kick, the more likely it is to get blocked.

My Proposed Change(s)

Vary the value of the field goal depending on the distance. Remember I said earlier that no field goal less than 20 yards should be allowed. I also placed the point-after attempt on the 17-yard line, and gave it 1 point. Here's how I would break down the values assigned to field goals.

0-19 yards = Cannot Attempt; must go for the touchdown
20-39 yards = 1 point
40-49 yards = 2 points
50 or more yards = 3 points

I know what many of you are probably thinking. This would decrease the number of points scored in a game. I disagree. The notion that this change would decrease the number of points scored by a team in a game is based on the presumption that the coaches will call the plays the same way as they do today. They won't. I think that two things will happen.

First, inside the 2-yard line, since the team has to go for it, the expected outcome should be greater than 3 points (6 points * 55% plus either 2 points * 55% or 1 point * 90% - see PAT discussion above). In addition, if the team fails to make it, the other team will be in a fairly bad spot, and there's a significant chance that the team that failed to make the touchdown gets the ball back with great field position.

Second, when the team is further out, say on the 10-yard line, I think that more teams will "go for it" on fourth down when they're faced with the possibility of only getting 1 point if they kick the field goal. In other words, the lure of the extra 6 (or 7) points would be worth the risk. Today, the lure of an extra 3 points (versus a 3-point field goal) isn't worth the risk in coaches minds. David Romer, in an excellent paper, suggested that coaches today are making a big mistake by not going for it on fourth down more often than they do (he was referring to the general scenario, and not specifically being inside the opponent's 17 yard line). If Romer is right (and I think that he is), then the new scenario ought to give the coaches that extra incentive. The fans would prefer it, and it would make the game more exciting.

Overtime

There have been numerous blog posts since the Vikings-Saints game discussing the overtime rules, and suggestions on how to change the rules such that the outcomes are "fairer". Even the NFL is contemplating changing the current format. The basic premise being that the winner of the coin toss (who invariably elects to receive the ball) wins approximately 60% of the time, and 2/3 times in that scenario, the loser of the coin toss doesn't even get a chance to touch the ball on offense.

Brian Burke at Advanced NFL Stats in particular has done an analysis that suggest that today's overtime rules are tremendously unfair to the loser of the coin toss. Instead of rehashing his analysis (you can read it here), I'll merely point out that I agree with his assessment. In his article, he (and commenters) offers some suggestions. Although these suggested improvements are quite valid, I believe that they are mostly theoretical and appeal only to those who have an intellectual bent. These alternatives are not practical, and therefore, I believe, unlikely to be implemented by the NFL.

The NFL's proposed changes make it slightly better for the loser of the coin toss, but it still isn't a 50/50 proposition. As such, I do not agree with the proposed changes by the NFL.

My Proposed Change(s)

First, no overtime games during the regular season. What's wrong with a tie? Especially, if the current overtime rules dictate that the way in which the winner is determined is basically unfair. Might as well flip a coin (actually, given today's rules, it would be better to flip a coin).

Second, winner in overtime must score at least 7 points in the overtime, win by at least 4 points, and, if the team receiving the ball first scores on its opening possession, then the other team must be given a chance to respond.

Many have suggested that receiving the ball second is advantageous since they know what they need to do win the game, but since the requirements are that a team scores at least 7 and wins by at least 4, the second team has to score at least a touchdown to win on its first possession. The win by 4 requirement makes it fairer than it is today for both teams. If the team that receives possession first scores a field goal, it cannot win by that score alone. The proposed change is a little bit like the tie-breaker rule in tennis, where one player gets one serve to start the tie-breaker, but then each player alternates two serves at a time, with the requirement that the winner score at least 7 points, and win by at least 2 points.

Conclusions

If all three of my proposed changes are adopted, the game would be a lot different than the game played today. There wouldn't be any meaningless extra points kicked after a touchdown, teams would "go for it" more often than they do today, points scored would be quite different, as each decision becomes strategic, more points would be scored, there would be less likelihood of a tie at the end of regulation, in the event of a tie, there would be no overtime games during the regular season, and the overtime rules in playoffs would be fairer to the team that loses the coin toss.

Who's Better, Who's Best - Manning, Favre and Brees

2010-01-31T23:31:00.000-08:00

Last Sunday we were all treated to some good football. The AFC Championship featured the surprising NY Jets, against the Peyton Manning-led Indianapolis Colts, and the NFC Championship showcased the Drew Brees-led New Orleans Saints who played host to Brett Favre and his Minnesota Vikings.

Not only are these outstanding football teams, but three of the four teams featured three of the game's best quarterbacks in Manning, Favre and Brees. I thought it would be illustrative to take a look at each of the three quarterbacks individually, and then compare the three side by side to get a sense who maybe better, and perhaps even, who may be best. In evaluating their relative merits, I will only consider their ability to throw the football, not win games, or how well they run, or whether they are clutch performers, or how many Super Bowl victories they have.

Here are the career NFL Passer Ratings for these quarterbacks:

Manning is clearly the best. Well, at least that's what looking at the career NFL Passer Rating will tell you. These figures differ from what you would find at the Pro Football Hall of Fame (as of January 23rd, 2010). The reason being that the figures listed on the Pro Football Hall of Fame site were as of the beginning of the 2009 season, and mine are as of the end of the 2009 season just completed.

For reasons that I have chronicled in the past (see here, here, and here), the NFL Passer Rating formula is in and of itself is of little use in determining who among the three might be best. The three quarterbacks very well might end up being ranked this way when it is all said and done, but that doesn't justify the rating system.

In order to complete our analysis, we have to make several adjustments to the NFL's passer rating system.

The first, is that we need to take each player's passer rating, and relate it to some baseline, so that we may account for players playing in different eras (yeah, I know, it's a little bit odd to be talking about players in different eras when comparing three current quarterbacks, but Brett Favre has been throwing since 1992, and Drew Brees only since 2002. And since the average NFL passer rating has been increasing over time (see here), we need to make an adjustment for that. The method that I have chosen is to relate a given quarterback's performance relative to the mean for that year, and use the standard deviation in that year as the measuring stick. In other words, the difference between the quarterback's actual performance compared to the mean, as a function of the standard deviation. As I described here, we then convert each of these resulting measures, also called z-values, into percentile ranks, using the standard normal distribution.

The second, is that we need to account for the fact that they've each played a different number of seasons. While I used a rather simplistic approach last year, I will present a more sophisticated approach starting with this blog. This method will need further refinement over time, since I am early in my thought process regarding this particular topic. However, it is an improvement over last year's simplistic approach. Actually, the method used last year (average deviation) really doesn't take into account the number of years at all. When I evaluated Brett Favre's career, I simply compared him to a group of quarterbacks who had played a like number of years, and simply ignored any comparisons to players who played far fewer seasons. That method won't work in this case, as we need to compare three quarterbacks with three vastly different career lengths. All else being equal, and they rarely, if ever, are, a player who has a 10 year career versus a player who has a 7 year career should get a higher rating (even if, and especially if, they both had the exact same ratings in every year they played). But the question becomes, "how much higher"?

The third, is that we need to somehow account for the fact that each of the three quarterbacks has played not only a different number of seasons, but that those seasons represent different parts of their careers. In some respects, since the last 8 seasons have been common to all three, we could simply look at the last 8 seasons. But, that would mean that we would be comparing Brett Favre in his 11th through 18th seasons, Peyton Manning through his 5th through 12th seasons, and Drew Brees his first 8 seasons. As I have shown in the past, the early years for a quarterback tend to be the "learning years", and hence this comparison would be inappropriate without the proper adjustments. Brett Favre is also 40 years old, and we have seen declining performances as quarterbacks age (actually, their performances increase at first as they gain experience, and then subsequently decline as the physical toll takes effect). One method is to look at each player's first 8 years, since in that case the comparisons would be over the identical portions of their careers. And perhaps we could compare Favre and Manning for the first 12 years.

Last, we need not assume that the NFL Passer Rating is the measure to be used to evaluate who might be the better passer. The NFL Passer Rating system has a lot of flaws. I will use two measurement methods. The first is the concept of Adjusted Passer Rating. As I have previously mentioned, the average NFL Passer Rating has been steadily increasing every year. The Adjusted Passer Rating normalizes the passer rating so that the average for each year is 66.7. This is then very helpful in evaluating players who played in different years (such as the case here). Last year, I introduced a simple measure of passing efficiency - CMI. I now call this CMTI (or, simplistically, Completions Minus Thrice Interceptions). It is not only simple, but it has a 86% correlation with the NFL Passer Rating. And, because it is so powerful in explaining any variances in the NFL Passer Rating, and because it is so easy to compute, use and explain, I will use it here.

So, to recap, we will make the following adjustments:

Relate performance in given year to the mean, and then in relation to standard deviation for that year - then convert to percentile rank
Use a 'better' measure than NFL Passer Rating. We will show that Adjusted NFL Passer Rating is 'better', and CMTI is simpler
Take into account differing number of seasons played by each quarterback
Take into account quarterbacks who are at different stages of their career

Here are the respective #s for each quarterback by season:

Brett Favre

Drew Brees

Peyton Manning

What do these performance measures say? Well, there's a lot of information here, so let's take these one at a time.

The first is that the career averages shown (actually all of the averages shown) are simple arithmetic averages of each of the years, not a true weighted average. You can see that these simple averages are close enough to the true average by looking at the career average NFL passer ratings shown in these three exhibits and compare them to the career average for each of the quarterbacks shown earlier.

The second is that regardless of what measure we use, whether it is the NFL Passer Rating, Adjusted NFL Passer Rating, CMTI, first 8 years, last 12 years or what not, it should be very clear that Peyton Manning has performed at a truly superior level compared to the other 2 players (or anyone else for that matter). Just look at CMTI over the past 8 years, for example. He has averaged being in the 90th percentile over the past 8 years. Basically, that suggests that he is, on average, in the Top 3 every year. Truly remarkable. In Peyton Manning, we are watching one of the all-time best. A once-in-a-generation quarterback. In case you didn't notice, the rows highlighted in green indicate a performance that is in the 85th percentile or higher. The areas in blue are 95th percentile or higher.

Ok, Manning is best. That was easy. Now comes the difficult part. Who's better, Favre or Brees?

The case for Favre:

On an Adjusted Passer Rating basis, Brett Favre's first 8 years are comparable to, if not better than Drew Brees' first 8 years (70th percentile versus 68th percentile). We can't directly compare their two careers since one has had an eighteen year career while the other only eight. It is very evident that, but for the last year, Favre's performance has been quite pedestrian over the past 10 years. Well, we can't necessarily penalize him for that, because we don't know how Brees might do in his next 10 years. We don't even know if he is likely to play another 10 years. Given history, however, it is fairly safe to assume that he won't play another 10 years. In my database, going back to 1932, and including both the AAFC in the 1940's as well as the AFL in the 1960's, I have a total of 416 quarterbacks who qualified in at least 1 year. 206 of these quarterbacks qualified in only 1 or 2 years. The average is 4.125 years. There are only 6 quarterbacks that qualified in 15 or more years. Y.A. Tittle qualified in fifteen different seasons. John Elway, Dan Marino and Johnny Unitas each qualified in sixteen different years. And then there are the ultimate marathon men, Fran Tarkenton and Brett Favre, who both qualified on 18 different occasions.

Michael J. Schell, in his book Baseball's All-Time Best Hitters, makes an adjustment to player batting averages, called a late-career adjustment. In essence, he recognizes that a player's performance declines as he ages, and so he makes an adjustment for it. Schell basically caps the at-bats of a player at 8,000. All at-bats beyond 8,000 are ignored. This way, he can compare each player through their first 8,000 at-bats. Doing that type of adjustment would be extremely difficult in the case of football. We know that the average number of passes attempted by a player per game has increased over time (see here and here), so limiting to a fixed number of attempts would be difficult and meaningless at best.

We could limit the player performances to a given number of seasons. Look at the three graphs below.

The first is the average percentile rank of CMTI for all 416 quarterbacks by age.

And the second is the same as the first, except with the x-axis denoting the number of seasons in which they qualified instead of their age.

It appears that there is a learning curve at first (and an elimination process, as presumably the worst players end up having shorter careers), followed by a decline after about 12 years. Keep in mind that this first graph shows the career percentile rank. What we're really interested in is does performance decline as a player ages? So, what we really want to see is the average percentile rank for a given year of a player's career no matter how long they have played.

This is what the third graph shows. It's actually a bit surprising. There doesn't appear to be a significant drop-off. As a matter of fact, out in the 15th-18th years, their performance seems to have got better! Ok, so in the outer years, we're not talking about a lot of players (for years 17 and 18 its just 2 players - Tarkenton and Favre), so anomalies should be expected. So what do we do?

I mentioned earlier that the average number of years in which a quarterback qualifies is 4.125 for all 416 quarterbacks. The table below shows the data in more detail.

There are a total of 77 quarterbacks who have played (qualified) in 8 or more years. That would be 77/416 = 18.5% of the eligible list. The average number of years for this group is 10.8, and the median number of years for this group is 11. It would seem reasonable then to look at a minimum of 8 years as a cut-off. But how do we recognize players who have played a lot longer? The method that I have chosen is to look at a given player's best 8 years. This means several things. First, the obvious - it eliminates from consideration any player who has qualified in fewer than 8 years. Second, it accounts for the fact that players have played longer, and, that fact doesn't hurt them. As a matter of fact, it's the opposite. The less a player has played, the more the "penalty". Third, given the small sample of players playing beyond 14 years, it is quite possible that one or more of them had good years in those later years, so why exclude those years?

So, going back to the tables where I showed each of the three quarterbacks and their individual seasons, you can see that Favre's best 8 years are better than Brees' 8 years. This gives credit to Favre's lengthy career. Remember, less than 20% of the qualified quarterbacks even get to 8 years total, let alone be fortunate enough to have 8 good years. Perhaps over time, Brees might very well exceed Favre. And, if and when he does, we'll give him credit for it at that time.

The case for Brees:

Although comparable to Favre in his first 8 years based on Adjusted Passer Rating, using CMTI gives Brees the nod. Also notice that Brees has had four years where he was better than 85th percentile, while Favre only had one. Brees' last four years in New Orleans have been especially strong. In each of years 5-8, Brees outperforms Favre, while in the first four, Favre outdoes Brees 3 out of four years. Favre, it appears, built his legend largely on the success of his first year, and perhaps even starting with his first game. Check out the following chart.

It shows the average percentile rank for all quarterbacks who played longer than 8 years, for each year they played. Actually, it's not quite the number of years they played in as much as it is the number of years in which they threw enough passes to qualify to have a rating. It's the blue line, and it includes data from the previously mentioned group of 77 quarterbacks. The blue line in this graph is different than the line shown on the third graph earlier, in that this only includes data for those quarterbacks qualifying in 8 or more years. The red line is the data for Drew Brees. The green line is the data for Favre. Clearly, outside of his first year, and then his 3 MVP years (years 3-5 for him), Favre hugged the average for this group of quarterbacks. Some years he did better, and some he did worse.

We're almost done. One more graph.

This one puts a slight twist on the notion of the best years. Instead of just looking at the best 8 years, why don't we look at the best n years, for each year n that the quarterback has played. So this way, we can look at the best year for each quarterback, the best two years, the best three years, and so on. For good measure, I also threw in Manning's #s.

So what do we conclude from all of this?

Again, if there, was any doubt before, this should put it to rest. Manning is clearly the best.

Between Favre and Brees, an argument could be made for either. In terms of a total career, Favre gets the nod, simply because of the number of years he's played. Brees has put up some impressive numbers so far. If he continues this for a few more years, he will most likely finish ahead of Favre, but that remains to be seen. Based on the evidence so far however, he is doing at least as well as, if not better than Favre did in his first 8 years. Perhaps we'll check-in in a few years when Brees retires. And, speaking of retirement, in my next post, I'll take a look at Kurt Warner's career.

The Adjusted NFL Passer Rating Revisited

2010-01-17T17:43:00.000-08:00

This posting is an update and an enhancement to a previous posting that discussed the concept of the Adjusted NFL Passer Rating. Back in 1971, when Don Smith, an executive with the Pro Football Hall of Fame, and his team developed the current NFL Passer Rating formula, he suggested that the intent of the formula was such that an 'average' Passer Rating would be equal to 66.7. In essence, he decided that an average performance for each of the four components that make up the passer rating system should be equal to 1.0, with an exceptional performance defined as a score greater than 2.0, and a poor performance getting a 0. Since the formula adds the scores for each of the four components and divides by 6, an average performance would be equal to 66.7.

Since we have seen that the average NFL passer rating has increased over time (see my previous post on the topic here), I was curious as to what the average in 2009 looked like, and, more importantly, what would the actual NFL passer ratings have looked like if we adjusted each of the components such that the averages for 2009 were in fact 1.0.

Recall that in the formula , based on data from the 1960's, adjustments are made to each of the four components such that the average for each component worked out to be 1.0. The adjustments, call them k1, k2, k3 and k4, are:

k1 = 30
k2 = 3
k3 = 1
k4 = 1

Again, these factors were arrived at by looking at data from the 1960's. And, these same factors are applied to the formula regardless of when a quarterback actually played. This has the benefit of course of creating a formula that is the same for all quarterbacks against a fixed standard. The problem of course is when the overall standard changes over time, and this is what has happened over the past 80 years or so.

Well, we don't have to use data from the 1960's. We will use each year as a stand-alone year. So, for 2009, the factors that make each component equal to 1.0 for an average performance turn out to be:

2009k1 = 40.90
2009k2 = 2.98
2009k3 = 1.20
2009k4 = 1.78

Likewise, the factors for 2008 were:

2008k1 = 41.00
2008k2 = 2.94
2008k3 = 1.28
2008k4 = 1.95

And, for good measure, the actual factors for the NFL for 1969 were:

1969k1 = 32.64
1969k2 = 2.99
1969k3 = 0.99
1969k4 = 1.12

These 1969 factors are very close to the k1, k2, k3 and k4 values used in the formula.

The table below shows the 2009 Adjusted Passer Ratings for the 32 quarterbacks that threw enough passes this past season to qualify. The table also shows their actual 2009 NFL Passer Ratings. You can see that the Adjusted Passer Ratings are clearly lower (as a group) than the actual NFL Passer Ratings. That's because the actual NFL average passer rating in 2009 was 83.0, and, obviously, the average Adjusted Passer Rating for the NFL for 2009 was 66.7 (because we forced it to be so by adjusting the formula such that the average value for each component is equal to 1).

Although the Adjusted Passer Ratings themselves are a lot lower as a group than the actual NFL Passer Ratings, you'll notice that the rankings for each are essentially the same. Why is that? Very simply, since both calculations use the same essential formula with the values changed, all this really does is 'normalize' the actual ratings around a value of 66.7 as opposed to 83.0. And yes, since the relative value of each of the four components now change, there should be some movement in the rankings. Interestingly, I also included my CMTI calculations (and their respective ranks), and it also highly correlates with both the Adjusted Passer Rating rankings and the NFL Passer Rating Rankings.

Just so that you can see for yourself, I've included the table (shown below) that shows the average NFL passer rating by year from 1932 to 2009 using the current NFL passer rating formula (where k1 = 30, k2 = 3 and k3=k4=1 for all years), as well as the adjusted k1-k4 factors for each year. There is no need to show the Adjusted Passer Rating for the NFL in each year, since it is the same in every year - 66.7. If you look at the bottom of the table, for years 1960-1969, you'll see where Don Smith derived his k values from.

Looking at this table you should be able to understand why using the current NFL passer rating system is of little to no use when you need to compare quarterbacks from one era to another. It can truly only be used to compare one passer versus another in any given year, but not across years. It stands to reason then, that the current NFL passer rating formula isn't much use when evaluating a quarterback's career, especially ones who've had a long career (you can see the career NFL passer rating leaders here). In future posts, I will discuss this in more detail.

2009 NFL Season - Passer Ratings

2010-01-10T19:18:00.000-08:00

We now have data for the entire 2009 season, and I can calculate the best and worst performers for the season, using my standard measure, CMI - or more appropriately, Standard Deviations from mean CMI. CMI is easily calculated, as it is simply [pass completion percentage] - 3 * [interception percentage]. I can calculate that at any point during the season. In order to calculate the standard deviations from the mean CMI, I like to wait until the season is over (actually, even this can be calculated at any point during the season, but it's a lot simpler for me to just do it at the end of the season - perhaps next year, I can do this after every week during the regular NFL season). For those of you familiar with statistics, the measure of standard deviation from the mean is also called a z-score (or z-value). And, furthermore, z-score can be very easily converted to a percentile rank (assuming that the population is normally distributed. In a later post, I will show that z scores for CMTI or NFL passer rating over the years is distributed normally). The beautiful thing about a percentile rank is that it is always a number between 0 and 100. What's not to like about that!

A quick couple of notes here regarding CMI, and many of the calculations that I will be using from here on out. While I will go through the exercise of creating arduous, complex and sometimes arbitrary (and even unnecessary) calculations, I will strive for simplicity whenever possible when I present my ideas, as I believe they are more likely to be accepted. The use of percentile ranks is an example of this.

Last year, when I first developed CMI, and especially as I looked back in time, I had to figure out how many players to use each year to determine the number of qualifiers. I went through some elaborate mechanisms to calculate the number of passes attempted during the season (per team per game), and some qualifying standard. As you are aware, the NFL uses 14 passes per game as the standard. And yes, while it is simple to use, I am not a big fan of that standard since the average # of passes attempted per game has changed (increased) over time, as you can see from the table below.

The standard today should be closer to 16.

Last year, I arbitrarily assigned a factor close to 50% (with adjustments for the earlier years) to the total # of passes attempted by each team per game to determine the # of qualifiers. I realized this year, that if I simply used the # of teams in the league as the cutoff for determining the # of qualifiers, I get fairly close to my original method. And because it is simple, easy to explain, does a better job (in my opinion, anyway) than the current NFL standard, I will use that as the cutoff. The table below shows the # of qualifiers by season using the NFL standard, my original standard, and my current standard.

As you can see, the # of qualifiers, regardless of which method you select is fairly close in each year, with the exceptions being the earliest years, where the NFL method simply excludes too many passers (in my opinion anyway). Also note that the the AAFC is excluded by the NFL altogether, while the AFL and NFL have been combined in the years 1960-1969 by the NFL. This doesn't make sense to me, since each league played a different # of games (and had a different # of teams). Regardless, if you combine the AFL and NFL results from my method, you get close to the #s used by the NFL. While I think that my method works, I also recognize that this standard may not be the most appropriate for future NFL seasons.

OK, back to the passer ratings.

Because I will discuss other passer rating systems/calculations that are related to CMI in later posts, I will take this moment to re-classify CMI as CMTI (acronym stands for Completions Minus Three times Interceptions). The definition itself hasn't changed - just the acronym.

The table below shows each of the qualifying quarterbacks with their 2009 season's statistics, their NFL passer rating, the percentile rank of their 2009 NFL passer rating (in other words, the converted z-scores), their 2009 rank in terms of NFL passer rating, their 2008 NFL passer rating, the percentile rank of their 2008 NFL passer rating, their 2008 NFL passer rating rank, their 2009 CMTI, 2009 CMTI percentile rank, their 2008 CMTI and 2008 CMTI percentile rank, and their 2008 CMTI ranking, and finally, their average 2-year CMTI percentile rank (again a # between 0 and 100), together with that ranking.

I don't think there's any question as to who the elite quarterbacks are. Also notice the struggles of first-year quarterbacks in each year.

Happy New Year!

2010-01-10T12:25:00.000-08:00

Hi Everyone.

Happy New Year to you all.

It's been awhile since I've posted. The reasons are many, so I shall not go into them. Regardless, I am back posting.

I have been able to do some additional research in the meantime. I have added data from the 1932 - 1939 NFL seasons to my database. I have also added the 1946-1949 AAFC seasons to the database as well as the 1960-1969 AFL seasons. And of course, updated for the just concluded 2009 NFL season.

I think that adding the data from the AAFC in the late 1940's as well as the AFL during the 1960's makes for a much richer database. There are those however, who do not believe that including that data makes sense. I disagree.

Having spent some time thinking about my pursuit here (creating a new passer rating system for the NFL), I've created new statistical measures in addition to those I created last year that I will discuss in upcoming posts. I've also improved upon my original concept of CMI.

Anyhow, I'm glad to be back posting and look forward to a terrific 2010.

Cheers,
Kiran

Was Steve McNair A Good Passer?

2009-07-26T18:53:00.000-07:00

Earlier this month, Steve McNair was shot and killed. It was a story that caught everyone by surprise, and there was much said about his contributions to the game, and how he played the game. I thought it might be worthwhile to digress briefly from my current topic - the economics of the NFL Draft to take a look back at McNair's career.

Steve McNair was drafted in 1995 out of little-known Alcorn State. The 3rd overall pick in the draft, he was the 1st quarterback taken that year. The then Houston Oilers (now Tennessee Titans) drafted him ahead of current Titans quarterback Kerry Collins.

He started out his career backing up Chris Chandler in both 1995 and 1996 while the team was still the Houston Oilers. The first year in Tennessee, he took over the starting job. He had a sub-par first year (as most quarterbacks tend to do), and then went on to have a fine career. He is of course, most notably remembered for coming one yard short of leading the Titans to a victory against the St Louis Rams in Superbowl XXXIV. The table and chart below illustrate his career in terms of the standard deviations from the mean CMI.

As you can see, from 1998 until his retirement after the 2007 season, he finished each year at above the mean CMI, with the exceptions of the 2004 and 2007 seasons which were cut short due to injury. His best year was 2003, where he finished 1.40 standard deviations above the mean CMI. His statistics that year: 62.5% completion rate, and a 1.75% interception rate. For his performance on the field that year, he was named Co-NFL MVP along with Peyton Manning. He was traded to the Baltimore Ravens following the 2005 season.

All told, Steve McNair had 9 qualifying years in his 13-year career. Of the 51 quarterbacks since 1940 to throw enough passes to qualify in 9 or more seasons, he ranks 17th. His career average standard deviation from the mean CMI during his 9 years was 0.46, putting him in the company of quarterbacks such as Norm Van Brocklin, John Brodie and Sid Luckman. The chart below illustrates the 51 quarterbacks, and Steve McNair's standing among them.

NFl Draft Economics - Part 1 - Introduction

2009-06-30T22:07:00.000-07:00

Here I will begin a series of posts discussing the economics of the NFL Draft. The NFL Draft puzzles me. Every year. There are many economists (those who are both smarter and more qualified than I am) who've tried to explain it, and I'll cover their analyses in the posts to come.

Naturally, this is a rather difficult topic to address in one simple post, so I'll do it over a few. It's a very broad topic - it covers the evolution of the game, identifying and measuring talent, economics, labor issues, psychology, and human behavior, among others. I won't go into all these topics, but will address some of them either directly or indirectly.

Through all of this, the fundamental question I'll be trying to answer is why did the Detroit Lions take Matthew Stafford as the 1st pick in 2009 Draft, and pay him $78 million (actually, he has the potential to earn as much as $78 million over six years, and about $42 million is "guaranteed" - the reason the "guarantee" is in quotes is that while that is the quoted number in the press, his actual true guaranteed money is about $17 million).

And why did they do this after they took Joey Harrington with the 3rd pick overall in 2002? And why did they do that after they took Andre Ware with the 7th overall pick in 1990? And why did they do that after they took Chuck Long with the 12th overall pick in 1986? That is the set of questions I'll attempt to answer. There must be a reason why. All the evidence suggests that they shouldn't have done this. So why did they do this?

Quarterbacks and the NFL Draft - Part 5

2009-06-14T23:03:00.000-07:00

This post is the corollary to the previous post. That is, we will attempt to answer the following question: which quarterbacks drafted in the past 30 years turned out to be the biggests busts? Naturally, these would be the quarterbacks drafted with really high expectations - those that were drafted in the first round, or better yet, drafted in the Top 10, or even better, #1 overall, perhaps. These cases of buyers remorse are further exacerbated when quarterbacks taken later (much later in some cases) turn out to have far superior careers.

The list you see below, along with the list in my previous post, should illustrate the difficulty of identifying who might be the best NFL quarterbacks before their first game. It is certainly an inexact science. It is why I question the merits of a) picking a quarterback so early in the draft, and b) paying them such large sums of money before they ever throw a pass. I will explore these questions in my next post.

In the meantime, here's my list of the 10 quarterbacks who were the biggest busts of the NFL Draft over the past 30 years. Unlike the biggest surprises, this list is a bit more difficult to assess. For the surprises, one merely looks at the relevant statistics, and then finds an appropriate weight (for example, how much weight to assign to longevity). For the busts, how does one quantify those that don't play (or don't play enough). In my system of quantifying quarterback passing statistics, one must throw enough passes in a given season to qualify to be ranked (in other words, I don't rank every player that threw a pass as that would be meaningless). For that reason, I shall merely list the players, and not rank them, starting with the earliest drafts.

The List

Jack Thompson (1979, 1st Round, 3rd Overall Pick, 1st Quarterback Drafted) - Thompson was drafted out of Washington State by the Cincinnati Bengals. He was taken ahead of Phil Simms (7), and Joe Montana (82). He qualified in 1 year. He makes the list not because of his statistics as much as he was drafted so far in advance of the the quarterback who arguably was the greatest passer of all time.

Rich Campbell (1981, 1st Round, 6th Overall Pick, 1st Quarterback Drafted) - Campbell was drafted out of The University of California by the Green Bay Packers. He was taken ahead of Neil Lomax (33). Campbell didn't attempt enough passes in any year to qualify, and threw a total of 68 passes in two different seasons.

Art Schlichter (1982, 1st Round, 4th Overall Pick, 1st Quarterback Drafted) - Schlichter was drafted out of Ohio State by the Baltimore Colts. The Chicago Bears selected Jim McMahon (5) out of Brigham Young one pick later. What bust list would be complete without Schlichter, known more famously for his gambling addictions.

Todd Blackledge (1983, 1st Round, 7th Overall Pick, 2nd Quarterback Drafted) - Blackledge was drafted out of Penn State by the Kansas City Chiefs. Quarterbacks taken after Blackledge that year include Jim Kelly (14), Tony Eason (15), Ken O'Brien (24) and Dan Marino (27). Todd qualified in 1984, a year when he completed 50.0% of his passes and had 3.7% of his attempts picked off. John Elway was drafted 1st Overall that year. Elway, Kelly and Marino are all in the Pro Football Hall of Fame.

Kelly Stouffer (1987, 1st Round, 6th Overall Pick, 2nd Quarterback Drafted) - Stouffer was drafted out of Colorado State by the St. Louis Cardinals. Notable quarterbacks taken after Stouffer in that year's draft includes Jim Harbaugh (26), Rich Gannon (98) and Steve Beuerlein (110). Stouffer did not attempt enough passes in any given year to qualify, and threw a total of 437 passes over the course of four different seasons.

Heath Shuler (1994, 1st Round, 3rd Overall Pick, 1st Quarterback Drafted) - Shuler was drafted out of Tennessee by the Washington Redskins. He qualified in 1 year (in 1994), when he completed 45.3% of his passes, and had 4.5% of his passes intercepted, a poor enough performance to be -2.54 standard deviations below the mean.

Ryan Leaf (1998, 1st Round, 2nd Overall Pick, 2nd Quarterback Drafted) - Leaf was almost picked by the Indianapolis Colts. Instead, they went with some fellow named Peyton Manning. The San Diego Chargers drafted Leaf out of Washington State. Successful quarterbacks drafted following Leaf that year are Brian Griese (91) and Matt Hasselbeck (187). In 2000, Leaf completed 50.0% of his passes, and had a 5.6% interception rate, a performance so poor it was -2.81 standard deviations below the mean.

Tim Couch (1999, 1st Round, 1st Overall Pick, 1st Quarterback Drafted) - Couch was drafted out of Kentucky by the Cleveland Browns. That year, the following quarterbacks were also drafted. Donovan McNabb (2), and Daunte Culpepper (11). Couch played enough to qualify in 3 seasons with an average standard deviation of -0.58. His best year was in 1999, when he finished when he completed 55.9% of his passes, and had 3.3% intercepted, good for -0.42 standard deviations below the mean.

Akili Smith (1999, 1st Round, 3rd Overall Pick, 3rd Quarterback Drafted) - Smith was drafted out of Oregon by the Cincinnati Bengals. Donovan McNabb was taken 1 pick ahead of him, and Daunte Culpepper was taken 11th that year. Smith qualified in 1 year (in 2000) when he completed just 44.2% of his passes and 2.2% picked off, sufficient to be -2.10 standard deviations below the mean.

Alex Smith (2005, 1st Round, 1st Overall Pick, 1st Quarterback Drafted) - Smith was drafted out of Utah by the San Francisco 49ers. Although still a little early to be evaluating this draft, a couple of quarterbacks that were taken subsequent to Smith are Aaron Rodgers (24), and Jason Campbell (25). Smith has qualified in 1 year (2006), when he completed 58.1% of his passes and had 3.6% of his attempts intercepted. He finished that year -0.61 standard deviations below the mean CMI.

That's a who's who of busts and disappointments. I could name another dozen or so who arguably could make this list, but we must draw the line somewhere!

Quarterbacks and the NFL Draft - Part 4

2009-06-04T21:14:00.000-07:00

In my last post, we looked at quarterbacks and their expected performance as compared to where they were drafted. We concluded that their on-field performance cannot be attributed to their draft position, and perhaps those drafted earlier play a bit longer. There definitely was a correlation between draft order, and their likelihood of actually playing in the NFL (playing as defined by throwing enough passes in a given year to qualify statistically).

All we really looked at in the previous post were averages over the past 30 years. We didn't spend any time looking at individual quarterbacks. In this post, we'll do so. We'll take a stab at answering the following question: Which quarterbacks have been the surprises, or the diamonds in the rough? These would be the quarterbacks that, when drafted, either had little likelihood of actually playing in the NFL, or, even when thought that they would play, surprised us as to how well they played and/or how long they played. These would also be quarterbacks that were drafted early in the draft (perhaps even in the first round), but worked out much better than the earlier picks.

Here is my list of the 10 best surprises of the NFL Draft over the past 30 years. Admittedly, it is somewhat subjective, but I do use objective criteria to determine the list.

10. Matt Hasselbeck (1998, 6th Round, 187th Overall Pick, 6th Quarterback Drafted) - Hasselbeck was drafted by the Green Bay Packers out of Boston College behind Peyton Manning (1), Ryan Leaf (2), Charlie Batch (60), Jonathan Quinn (86), and Brian Griese (91). Hasselbeck was traded by the Packers to the Seattle Seahawks in 2001. He has thrown enough passes to have qualified in 7 seasons, and his CMI has averaged 0.14 standard deviations above the mean over that period. His best year was in 2005, where he completed 65.5% of his passes and only had 2.0% of his pass attempts intercepted, a performance good enough to be 1.50 standard deviations above the mean.

9. Brian Griese (1998, 3rd Round, 91st Overall Pick, 5th Quarterback Drafted) - Griese was drafted by the Denver Broncos out of Michigan behind Peyton Manning (1), Ryan Leaf (2), Charlie Batch (60), and Jonathan Quinn (86). Griese became the starter in 1999 following the retirement of John Elway, and in 6 qualified seasons his CMI has averaged 0.42 standard deviations above the mean. His best season was 2000, where he completed 64.3% of his passes and only had 1.2% of his passes intercepted, worthy of being 2.04 standard deviations above the mean.

8. Neil O'Donnell (1990, 3rd Round, 70th Overall Pick, 5th Quarterback Drafted) - O'Donnell was drafted by the Pittsburgh Steelers out of the University of Maryland behind Jeff George (1), Andre Ware (7), Tom Hodson (59), and Peter Tom Willis (63). He started in 1991, qualified in 5 seasons for the Steelers, and threw enough passes in 2 more seasons - one for the Jets in 1997 and one for Cincinnati in 1998. He averaged 0.47 standard deviations better than the mean over those 7 years, and his best year was 1998, when he completed 61.8% of his passes, and was intercepted 1.2% of the time, good enough to finish 1.52 standard deviations above the mean.

7. Trent Green (1993, 8th Round, 222nd Overall Pick, 8th Quarterback Drafted) - Green was drafted by the San Diego Chargers out of Indiana University behind Drew Bledsoe (1), Rick Mirer (2), Billy Joe Hobert (58), Mark Brunell (118), Gino Torretta (192), Alex Van Pelt (216), and Elvis Grbac (219). He was the last quarterback drafted in 1993. He first threw enough passes to qualify in 1998 for the Washington Redskins. He qualified in 6 seasons altogether, the last 5 with the Kansas City Chiefs from 2001-2005. Over those 6 seasons, his CMI averaged 0.26 standard deviations above the mean. His best year came in 2003, when he completed 63.1% of his passes, and had 2.3% intercepted, good enough to finish 1.18 standard deviations better than the mean.

6. Mark Brunell (1993, 5th Round, 118th Overall Pick, 4th Quarterback Drafted) - Brunell was drafted by the Green Bay Packers out of the University of Washington behind Drew Bledsoe (1), Rick Mirer (2), and Billy Joe Hobert (58). He was traded to the Jacksonville Jaguars in 1995. He qualified in 8 straight seasons there, and then 2 more in 2005 and 2006 for the Redskins. Over those 10 seasons, his CMI averaged 0.64 standard deviations above the mean. His best year was 1997, where he completed 60.7% of his passes, and had 1.6% of his passes intercepted - good enough to be 1.23 standard deviations better than the mean.

5. Rich Gannon (1987, 4th Round, 98th Overall Pick, 7th Quarterback Drafted) - Gannon was drafted by the New England Patriots from the University of Delaware behind Vinny Testaverde (1), Kelly Stouffer (6), Chris Miller (13), Jim Harbaugh (26), Cody Carlson (64), and Mark Vlasic (88). He was traded to the Minnesota Vikings and qualified in three years (1990-1992) while there. He threw enough passes to qualify again in 1998 with Kansas City, and then from 1999-2002 with the Oakland Raiders. Over the 10 qualifying seasons, he averaged 0.64 standard deviations better than the mean. His best year was in 2001, when he completed 65.8% of his passes, and only had 1.6% of his attempts intercepted. This was good enough to be 2.54 standard deviations better than the mean that year.

4. Dan Marino (1983, 1st Round, 27th Overall Pick, 6th Quarterback Drafted) - This was the "Year of the Quarterback". Marino was drafted by the Miami Dolphins out of the University of Pittsburgh behind John Elway (1), Todd Blackledge (7), Jim Kelly (14), Tony Eason (15), and Ken O'Brien (24). He started immediately, and went on to throw enough passes in 16 different seasons, averaging 0.51 standard deviations above the mean CMI. His best year was 1984, where he completed 64.2% of his passes, and had 3.0% of his passes intercepted. This was good to finish 1.28 standard deviations above the mean CMI that year. Dan Marino was elected to the Pro Football Hall of Fame in 1995.

3. Brad Johnson (1992, 9th Round, 227th Overall Pick, 13th Quarterback Drafted) - Johnson was drafted by the Minnesota Vikings out of Florida State behind David Klingler (6), Tommy Maddox (25), Matt Blundin (40), Tony Sacca (46), Craig Erickson (86), Casey Weldon (102), Will Furer (107), Chris Hakel (112), Jeff Blake (166), Kent Graham (211), Bucky Richardson (220), and Mike Pawlawski (222). He qualified in 9 different seasons - first with Minnesota (1996-1997), then the Redskins (1999-2000), then Tampa Bay (2001-2003), and finally with the Vikings again (2005-2006). Over those 9 seasons, his CMI averaged 0.73 standard deviations better than the mean. 2005 was his best year, where he completed 62.6% of his passes and only 1.4% of his attempts were intercepted, a performance good enough to be 1.32 standard deviations better than the mean CMI that year.

2. Tom Brady (2000, 6th Round, 199th Overall Pick, 7th Quarterback Drafted) - Brady was selected by the New England Patriots from the University of Michigan behind Chad Pennington (18), Giovanni Carmazzi (65), Chris Redman (75), Tee Martin (163), Marc Bulger (168), and Spergon Wynn (183). After replacing an injured Drew Bledsoe in the second game of the 2001 season, he started every game until he got injured during the first game of the 2008 season. Over the 7 year period from 2001-2007, his CMI was good enough to average 0.82 standard deviations above the mean. His best year came in 2007, when he completed 68.9% of his passes, and was intercepted 1.4% of the time, good enough for 2.14 standard deviations above the mean. Tom Brady will likely get into the Hall of Fame.

1. Joe Montana (1979, 3rd Round, 82nd Overall Pick, 4th Quarterback Drafted) - Montana was drafted by the San Francisco Forty Niners from the University of Notre Dame behind Jack Thompson (3), Phil Simms (7), and Steve Fuller (23). Although he played every single game in the 1979 season, he only threw 23 passes that year. Beginning in 1980, he qualified every year until 1990, when injuries slowed him down. He was eventually traded to the Kansas City Chiefs, where he played for 2 more seasons. Over those 13 years, his performances on the field, he averaged 1.54 standard deviations above the mean, including the remarkable feat of never having a season below the mean. His best season was 1989, when he completed 70.2% of his passes and had 2.1% of his pass attempts intercepted, a performance so good relative to his peers, that it ranked 2.96 standard deviations above the mean. Montana was elected to the Pro Football Hall of Fame in 2000.

Now that's a pretty good list!

Quarterbacks and the NFL Draft - Part 3

2009-06-03T21:02:00.000-07:00

OK, so we have taken a look at those QBs that were taken first overall over the past 30 years. There have been 13 of them. And we've discussed how they've done, compared to each other. Of course, there have been many quarterbacks taken since 1979 - a total of 340 quarterbacks drafted by NFL teams, not including the 8 quarterbacks that have been taken in supplemental drafts. In this and subsequent posts that follow, we'll take a look at these quarterbacks, and we'll try to answer some questions. Questions such as:

> Do quarterbacks drafted earlier do better than quarterbacks drafted later in a draft?
> Who have been the biggest surprises (performance relative to draft position)?
> Who have been the biggest busts?
> Does it pay to draft a quarterback #1?

Some of these questions are easier to answer than others. In the course of trying to answer these questions, we will encounter some themes we will have to address along the way. Themes such as:

> The economics of the draft
> The goal(s) of a team owner
> The importance of winning

Let's start with a picture of where quarterbacks have been generally drafted over the years.

Selection	QBs Taken	% of Total
1	13	4%
2-5	15	4%
6-15	18	5%
16-25	13	4%
26-50	25	7%
51-75	22	6%
76-100	40	12%
101-125	26	8%
126-150	25	7%
151-175	31	9%
176-200	27	8%
201-225	31	9%
226-250	24	7%
251-300	19	6%
301-350	11	3%
Total	340	100%

Determining whether quarterbacks drafted earlier perform better than quarterbacks drafted later is a little tricky. Here's how I will go about trying to answer this question. First, I will look at how many of these quarterbacks that got drafted "played" in the NFL. I define "played", as the number of seasons a quarterback threw a sufficient amount of passers to have qualified statistically. The statistic that I use (and have used in the past) is the standard deviation relative to the mean as measured by CMI. Next, I will look at how many years, did these quarterbacks play? Certainly, length of playing career could be construed as a measure of success. Third, I will look at the average standard deviation of CMI for these quarterbacks. The higher the average standard deviation, the better the performance.

Let's take a look at the first measure - the likelihood of "playing". Some definitions: "Play" = # of years qualifying as a quarterback. Likelihood of playing is defined as # of players qualifying in at least 1 year relative to # of players drafted.

Here's the data:

Selection	QBs Taken	QBs "Played"	% "Playing"
1	13	13	100%
2-5	15	14	93%
6-15	18	15	83%
16-25	13	10	77%
26-50	25	15	60%
51-75	22	12	55%
76-100	40	16	40%
101-125	26	8	31%
126-150	25	10	40%
151-175	31	5	16%
176-200	27	8	30%
201-225	31	9	29%
226-250	24	5	21%
251-300	19	2	11%
301-350	11		0%
Total	340	142	42%

At least according to just this measure, the evidence is fairly compelling. The results in this table show that players drafted in the earlier rounds are more likely to see significant playing time. It could suggest that the players drafted earlier are in fact more talented, or, if you are skeptical, it might suggest that team management influences coaches to give more playing time to the players drafted earlier to justify their investment (I find this latter suggestion a little hard to believe).

OK, so they are more likely to see more playing time. But what happens when they do play? How do these players perform on the field? I will look at two measures. The first, is the number of years that they play in the NFL. Presumably, the better you are, the longer you get to play (all else being equal). I have actually shown this to be true in previous posts (see here and here). So, if the players drafted earlier are in fact better players, we should see the length of their careers exceeding those of players drafted later. Let's see what the data shows:

Selection	QBs Taken	QBs "Played"	Yrs "Played"
1	13	13	7.2
2-5	15	14	4.7
6-15	18	15	4.2
16-25	13	10	2.6
26-50	25	15	6.9
51-75	22	12	2.8
76-100	40	16	4.0
101-125	26	8	3.1
126-150	25	10	2.5
151-175	31	5	4.6
176-200	27	8	3.3
201-225	31	9	2.7
226-250	24	5	2.6
251-300	19	2	3.0
301-350	11
Total	340	142	4.2

Other than the quarterbacks taken first overall, there does not appear to be a appreciable difference in the # of years played by draft position. I am not suggesting that there isn't. There is. It's just not as much as I would have thought prior to looking at the data.

There are two possible explanations. The first is that, by the time the coaches and scouts have determined those worthy of getting playing time, there should only be minor differences in talent levels as it relates to the draft position. The previous table showed that there is a weeding out process that is more severe at the lower end of the draft, so the ones that do end up getting playing time, are probably on par with quarterbacks taken higher (i.e. pre-season training camp could be used as an additional, more rigorous screening process than merely watching college football games, and the NFL combine). The second reason is that again, there maybe a slight inherent bias to play the early draftees more, simply to give them more opportunities to continue to "prove" themselves.

We're still not done. I said earlier that we'll look at two measures. The first was the length of their careers. The second is their actual statistical performance, as measured by the standard deviation from the mean, using CMI as the measuring statistic. A number greater than zero indicating better than average performance, and a number less than zero indicating worse than average. Let's take a look.

Selection	QBs Taken	QBs "Played"	Yrs "Played"	Avg Dev
1	13	13	7.2	-0.06
2-5	15	14	4.7	-0.34
6-15	18	15	4.2	-0.15
16-25	13	10	2.6	0.36
26-50	25	15	6.9	-0.01
51-75	22	12	2.8	-0.32
76-100	40	16	4.0	0.16
101-125	26	8	3.1	-0.01
126-150	25	10	2.5	-0.70
151-175	31	5	4.6	-0.35
176-200	27	8	3.3	-0.16
201-225	31	9	2.7	-0.33
226-250	24	5	2.6	0.42
251-300	19	2	3.0	-0.32
301-350	11
Total	340	142	4.2	-0.11

Looking at this table, it's really difficult to see that the players drafted earlier, who actually ended up playing, have done better than those players drafted later. This suggests to me that once it is determined who actually plays, then the only difference that can be associated with draft order, if anything, is the number of years the player plays, and that probably is somewhat influenced by the team's investment in the players drafted earlier.

So what does this all mean for Matthew Stafford and the Detroit Lions, or, Mark Sanchez and the New York Jets. About the only thing we can be certain of at this point, is that they will play. They may end up playing 5 years, maybe more. As for their actual performance on the field, our best guess is that they'll be about average. Maybe I'm wrong on this. Maybe Matthew Stafford is the next Peyton Manning. Or, he could be the next Tim Couch. Or, he could be the next Eli Manning. Only time will tell.

Quarterbacks and the NFL Draft - Part 2

2009-05-31T17:19:00.001-07:00

In my previous post, I pointed out the quarterbacks that have been taken first overall in the NFL draft. With expectations sky-high for Matthew Stafford to lead the Lions out of the doldrums, let's take a look back and see how these previous #1 picks did.

A quarterback taken first overall is hardly a sure thing. Of course, it is still too early to figure where the most recent #1 picks will end up, as they have a few years ahead of them. 2 of the 13 are in the Hall of Fame - John Elway and Troy Aikman. Peyton Manning will be. Although Elway was elected to the Pro Football Hall of Fame, his statistics (as measured by standard deviation from the mean CMI) do not suggest that he was worthy (but that is for another discussion). Without a doubt, both Manning (Peyton, not Eli), and Aikman have had outstanding careers, and, in looking back, fully deserving of being the #1 pick. Of the rest, Carson Palmer appears the only quarterback currently justifying his lofty draft status. Only time will tell if the others turn out to have great, or even good careers (I wouldn't bet on it).

What does this all mean for Matthew Stafford and the Lions? Well, it could mean absolutely nothing. Or, if you believe in statistics, and you believe that NFL scouts' abilities to pick out talent isn't better today than it has been in the past, then the odds are against Stafford. He is, in all likelihood, not going to be the savior that the Lions seek. I will have more on this in the next few weeks.

Quarterbacks and the NFL Draft - Part 1

2009-05-30T19:30:00.000-07:00

In this post, I want to take a trip back in time (back to 1979, specifically), to see which quarterbacks were picked first overall in the regular NFL Draft.

Why 1979? Two reasons. First, it gives us exactly 30 years of data. Second, it was the year where Joe Montana was drafted out of Notre Dame. Since 1979, there have been 13 quarterbacks selected as the first overall pick in the regular NFL draft. This does not include Steve Young, who was taken first overall in the 1984 Supplemental Draft.

A couple of observations. In the 1980's, 3 quarterbacks were taken as the first overall pick. In the 1990's there were 4. In the 2000's, there have been 6 taken, through 2008, and, with the Detroit Lions taking Matthew Stafford first overall in 2009 , that would make 7.

Another way to look at it: In the 19-year period from 1979 to 1997, there were 5 quarterbacks taken first overall. Since then, 12 years if you include the 2009 draft, 9 quarterbacks have been taken first overall. 2000, 2006 and 2008 were the exceptions.

Here's the full list.

Better Than Jay Cutler

2009-04-30T19:56:00.000-07:00

A lot of noise this past month about Jay Cutler. First, the Denver Broncos' new head coach, former New England Patriots offensive coordinator Josh McDaniels, apparently tried to work a trade to get Matt Cassel from the New England Patriots. This apparently didn't it well with the Pro-Bowl quarterback. After a lot of posturing from both sides, Cutler was eventually traded to the Chicago Bears for Kyle Orton and draft picks. This post isn't about who was right or wrong in the fracas, or which team got the better end of the deal in the trade. This post is about answering the following two questions:

1) How good is Jay Cutler?

2) Are there quarterbacks with similar experience that are better than Jay Cutler, and, if so, who are they?

The table below ranks each of the 30 quarterbacks that threw enough passes last year to qualify to be ranked. The ranking is based on CMI, and not the NFL's quarterback rating system. In numerous posts prior to this, I have discussed why CMI is a better measure of a quarterback's passing performance than the NFL passer rating system.

Just looking at last year, the year that Jay Cutler got named to his first Pro-Bowl, he was, by this measure, a below-average quarterback. His CMI of 0.536 was (0.14) standard deviations below the mean. It should be noted that even though he was below average relative to all quarterbacks in 2008, it wasn't unexpected. Previously, I discussed the notion of performance improving with experience. So, if you were to look only at quarterbacks and their second year performance, their average is (0.21) standard deviations relative to the mean. Relative to that measure, he actually did better than the typical second year quarterback.

Let's take a closer look at the table above. The last 2 columns indicate the # of years the quarterback has played since he first qualified, and the total # of years he has qualified. So, Cutler has played 2 seasons where he threw enough passes to qualify, and this is second consecutive year of being qualified. There are six other quarterbacks in this table who have similar experience.

They are (rank in parenthesis):

Jason Campbell - WAS (5)

Matt Schaub - HOU (7)

Trent Edwards - BUF (8)

David Garrard - JAX (12)

Jay Cutler - DEN (16)

Derek Anderson - CLE (30)

Of the six quarterbacks who were in their second year (Garrard has the most experience of all the quarterbacks, even though he has only qualified in two years), Cutler only outperformed Derek Anderson of Cleveland in 2008. Ok, so 1 year does not make a quarterback. Let's take a look at these six quarterbacks and see how they've done in each of their first two (qualifying) years.

According to this table, both Garrard and Schaub have outperformed Cutler in each of 2007 and 2008. And, arguably, over the two-year period, so did Campbell. The only quarterback in this class that has clearly underperformed the rest of the group is Derek Anderson. He will not last long in the NFL.

Admittedly, it's difficult to judge a quarterback after his first two years. But, so far at least, there appear to be several quarterbacks that have outperformed Cutler. It will be interesting to watch this group over the next several years, and especially Cutler, as he will be with a new team.

A Few Weeks Off

2009-04-28T21:28:00.000-07:00

I took a few weeks off from posting. It was necessary for several reasons.

First, I do have a day job as the Chief Financial Officer of InsWeb Corporation, a publicly traded company (NASDAQ CM: INSW). The months of March and April, I get particularly absorbed in my work as we have to officially close the books for the prior fiscal year (in this case the year ending December 31, 2008), and we have to get prepared for closing the books for the first quarter of 2009. This is followed by a meeting with our Board of Directors. It is an extremely busy time, and a very important time for our business.

Second, posting a blog (the way I do it), takes a lot of time, as I spend considerable time before posting doing a lot of research, as I like to support my arguments with facts and analyses.

Third, I got distracted by a couple of things. The "event" known as March Madness - the NCAA basketball tournament, and the 2009 Major League Baseball season. Well, not the baseball season per se, but the fantasy baseball leagues that I participate in. Preparing for these takes up a lot of time.

Fourth, I needed a break.

Well, I'm back at it. I have two immediate topics that I'd like to cover.

The first is the trade of Jay Cutler from the Denver Broncos to the Chicago Bears. Quick, name three quarterbacks who, like Cutler, finished their second year in the NFL (technically, 2008 was Cutler's third season, but the way I count it, it was his second season throwing enough passes to qualify for a rating) last year that are arguably better than Jay Cutler. Didn't think you could do it. I'll have the answers in my next post. The answers might surprise you.

The second is the NFL draft. I have a lot of observations about the draft. I won't be discussing whether Matthew Stafford will be a better NFL quarterback than Mark Sanchez. Why? Because I don't know. And neither do you. And nor does anyone else. Only time will tell. And even then, we may never know.

Does Experience Improve Performance?

2009-03-14T13:34:00.000-07:00

In my earlier post, "First Year Performance - An Evaluation Method", I discussed how, over the history of the NFL, from 1940-2008, a quarterback's passing performance, as measured using CMI, and more specifically, the standard deviation from the mean of CMI, changes over time (in that post, see in particular the third graphical illustration labeled "CMI - SDs from Mean - By Qualifying Year"). That illustration pointed out that it takes about 5 years for a quarterback to become "average" (or, better than average). That's obviously only true for quarterbacks who qualify in at least that many years. Meaning that since the graphical illustration includes all quarterbacks (who have thrown enough passes to qualify in at least 1 year), that the earlier years could be biased by quarterbacks who only qualify in those earlier years, and not the later ones.

This is the issue pointed out by a commenter regarding that post. He (she) was suggesting that I have a biased sample, in that the later years are only populated by quarterbacks who throw for that many years, and hence, performance should be expected to improve, as the NFL weeds out the under-performers of the earlier years.

Here's the comment:

"This effect could be due to sampling bias. It may be that there are a large pool of first year QBs. Some are good and some are bad. The good ones make it to year four, and the bad ones are cut, released, or benched before they can play four years...."

He (she) then went on to ask the following question:

"...if you restrict the analysis to only QBs that have played >= 4 years, do you get the same result?"

Before I answer the question, let's address the comment.

First, the commenter does make a very valid (and what would seem to be an obvious) point, in that those QBs that make it to year 5 and beyond, must necessarily be the better ones, as the under-performers drop-off. This point has several implications though. The first is that the NFL does not look at quarterbacks (or, more accurately, their passing performance) the way I do. So we can't simply assume that the point is in fact that obvious. Second, perhaps the way I look at quarterbacks passing performance, CMI, is in fact, a decent (good?) measure of a their true passing ability, and, the graph validates that fact (this would be rather presumptuous of course, but that is what I am trying to prove after all - that I have, in fact, developed a method that appropriately evaluates a QBs passing performance, and, perhaps even more importantly, that I have developed a method by which one can identify a good passer versus a poor passer early on).

Second, the commenter suggests that my sample is biased because of this fact (i.e. it is biased towards the better players in the later years). There are two ways we can look at that. The first is that the sample isn't biased because it uses the entire population. In other words it is what it is (and perhaps this again validates my measurement technique). The second point is that the sample is biased (remember, however, the context in which the graphical illustration was shown - in that it was trying to look at all quarterbacks, and how their performances changed over time), and that we can, if we choose to do so, look at it differently. This is, in fact, what the commenter was suggesting with their question.

So let's take a look. As a matter of fact, let's take the question a bit further and look at it in four different groupings:

QBs qualifying in at least 1 year (the original graph) - 345 Quarterbacks

QBs qualifying in at least 3 or more years - 180 Quarterbacks

QBs qualifying in at least 5 or more years - 121 Quarterbacks

QBs qualifying in at least 7 or more years - 84 Quarterbacks

The graphical illustration is followed by a tabular display of the same data.

There's no question about it. Performance improves as a quarterback gains experience. All groupings follow the same basic pattern: Sub-par performance in year 1, followed by several years of gradually improving performance. When the improvement takes place, and by how much, does vary by group. For example, after three years, teams still haven't been able to markedly separate the good quarterbacks from the bad ones. By the fifth year, it's now quite apparent who has "cut it", and who hasn't. For all quarterbacks qualifying in at least 7 or more years, they appear to show their skill, or differentiate themselves, as early as year two. So, to prove the obvious point that the commenter in the previous post making, those that don't turn in positive performances early do tend to get weeded out. The process, however, happens to take a few years.

The question this prompts however, is, why does it take so long to weed out the poor performers?

Well, there are at least two different answers.

The first, something I alluded to earlier, is that the NFL doesn't evaluate passers the way I do. I am a statistical analyst, and what I have created is a an objective methodology to evaluate a player's performance. Teams in the NFL, aren't inclined to do the type of statistical analysis that I do. In addition, it's just simply not clear, even using my statistical methods, that you can be sure that you can identify the good and bad future quarterbacks after year 1. By year 2, however, year 3 the latest, my method does show who the keepers should be.

The second is, that teams pay a lot of supposedly smart people (scouts, GMs, etc) a lot of money to evaluate, and draft good talent. Subsequently, these teams draft these "talented" players early in the draft, and pay them lofty sums of money, a lot of which is guaranteed. Now what do you do? You've got a poor-performing team (generally, a team's draft order is determined by their won-loss record, with the teams with poorer records drafting earlier, and vice-versa) that just drafted the next "Peyton Manning", and have paid them a lot of money, and they have a lousy first year. You're not likely to blame the young quarterback for that year's performance. The team still has a lot of weaknesses elsewhere, so you pick early again the next year (perhaps shoring up the defense). Second year passing performance is better than the first, but still not better than average. You even ignore the statistical evidence (he can't be that bad). You're still paying him lots of money, you've seen improvement, team has won perhaps a couple of more games, and this gives you hope that, the third year is THE YEAR. Then, much to your dismay, the quarterback that you paid millions of dollars for, doesn't pan out as you have another miserable year (the teams is "close" to 0.500). You get impatient, fire the head coach, try an aging veteran as your quarterback, he has a decent year, a quarterback controversy brews, ya da ya da ya da....and, you continue to ignore the statistical evidence....

Just off the top of my head, I can think of several recent obvious examples - Alex Smith of the Forty Niners (and, proving the corollary, the approach taken by the Green Bay Packers with Aaron Rodgers), and JaMarcus Russell of the Oakland Raiders.

Here are three quarterbacks that the NFL, teams, and fans, have fallen in love with, but, statistically have yet to prove themselves.

Eli Manning (I can hear the argument - but they won the Superbowl)

Tony Romo

Michael Vick

So there you have it. My method is the Moneyball approach. What actually happens in the NFL is quite different. NFL teams could save themselves a lot of money (and perhaps even win a few more games) if they were to adopt a statistical approach to their business (I think a lot of NFL teams actually believe they take a statistical approach, but I don't think that most actually go through rigorous analyses the way I do).

My next post will be my preseason rankings for 2009.

The Best and Worst of 2008

2009-03-11T21:10:00.000-07:00

This is a simple post. Really simple. The list should pretty much speak for itself. The best and worst of 2008. I use the same criteria I've been using the past few posts. CMI, and, more specifically, standard deviation of a player's CMI relative to the mean. 30 players attempted enough passes in 2008 to make the list.

The Best 3:

3. Jeff Garcia, Tampa Bay Buccaneers - Why no team has picked him up is a complete mystery to me - the man can still play, even though his durability is certainly a question. Completed nearly 65% of his passes, and less than 1.6% of his passes were intercepted.

2. Peyton Manning, Indianapolis Colts - Do we even need to say anything? Coming off off-season knee surgery, Mr. Consistency now holds the highest current active streak of games started consecutively - 176 games. He completed almost 67% of his passes.

1. Chad Pennington, Miami Dolphins - Let go by the Jets so that they could welcome Brett Favre (that worked out well for the Jets, didn't it?) and subsequently picked up by the Dolphins. Had a terrific season, completing more than 67% of his passes, and less than 1.5% of his passes were intercepted.

In previous posts on my blog, I have spent time discussing each of the above 3 players.

The Worst 3:

28. Tyler Thigpen, Kansas City Chiefs - Won't show up at the bottom of the list in 2009. Why? Matt Cassel (#10 in 2008). The Chiefs made a deal with the New England Patriots to get Cassel (and Mike Vrabel) for a 2nd round pick in the upcoming draft. In 2008, Thigpen, replacing Damon Huard, completed fewer than 55% of his passes.

29. Gus Frerotte, Minnesota Vikings - Replaced an ineffective Tarvaris Jackson in the 3rd game of the season, started 11 games, and then was himself replaced by Jackson against the Lions. Completed less than 60% of his passes, and at almost 5.0%, was one of only three quarterbacks whose interception percentage exceeded 3.0% - the other two being Ben Roethlisberger of the Steelers (3.2%), and Brett Favre of the Jets (4.2%).

30. Derek Anderson, Cleveland Browns - Barely completed 50% of his passes, then got hurt. Was replaced by Brady Quinn. Is it Quinn's time in 2009?

Here's the entire list.

My next post - a pre-season ranking for 2009

First Year Quarterback Performance - An Evaluation Method

2009-03-01T18:03:00.000-08:00

This past weekend, the New England Patriots traded Matt Cassel to the Kansas City Chiefs. In fact, they traded linebacker Mike Vrabel and Cassel for a 2nd round pick (pick #34) in the upcoming April draft (see here for the details). Depending on what your news source is, this was labeled as either a good deal for Kansas City, or a bad deal for New England. The experts discussing this issue don't seem to support their arguments by any objective measures - they just opine. I am not going to jump into the debate here. I will offer this however - we will probably not know who will end up on the better end of this deal until many years from now, and even then, we may not know.

What about the objective evidence?

Matt Cassel is not the only quarterback in 2008 that had a good year as a first-year quarterback. There was Aaron Rodgers of the Packers, Shaun Hill of the 49'ers, and Matt Ryan of the Falcons. There were four other quarterbacks who threw enough passes this year to make the list of first-year quarterbacks - Ryan Fitzpatrick of the Bengals, Joe Flacco of the Ravens, JaMarcus Russell of the Raiders and Tyler Thigpen of the Chiefs.

Notice here that I didn't use the term rookie quarterback, but instead used the term first year. The way I look at a quarterback, is that I only consider a quarterback who has thrown "enough" passes to qualify in a given year. The # of passes has varied by year (as opposed to the fixed standard used by NFL.com). For the 69-year period from 1940-2008, my database contains 1,451 passing seasons, 336 of which were by a quarterback qualifying for the first time.

How did the aforementioned 8 quarterbacks do?

I use a statistical measure I have previously developed and discussed, called CMI (Completions Minus Interceptions, calculated as Completions/Attempts - 3*Interceptions/Attempts) instead of the NFL Passer Rating system (using the NFL passer rating system gets you essentially the same answers), and then use the standard deviation from the mean as the measuring stick (for the pool of qualifying passers, I calculate the the mean and standard deviations for each year, and relate a particular performance in that year to the mean using that year's standard deviation). Using the mean and standard deviation this way not only allows me to compare these 8 quarterbacks relative to each other, but also relative to all quarterbacks this year, or any other year.

In a post not too long ago, I showed how, over the past 69 years from 1940-2008, when the standard deviation is used as the measuring stick, the accumulated data form a near-perfect theoretical standard normal distribution. And I also showed how, this is true regardless of whether one used the NFL passer rating system or CMI as the statistical base.

I can now show you how these first year quarterbacks did compare to the rest of the QBs in 2008, using this measure. But this picture may not tell us much, if anything. Here's the graphical illustration:

Perhaps the only thing that may be apparent from this is that first year quarterbacks don't do nearly as well as quarterbacks with more experience. Even drawing that conclusion from this is a little iffy. However, it is clear (at least by this measure) that Cassel had the best year of the 8 first year QBs. Rodgers was a close second. JaMarcus Russell and Tyler Thigpen had disastrous first years. In an earlier post regarding Jeff Garcia, I had suggested that Kansas City might be a good place for Garcia. Scott Pioli, the new Chiefs' GM apparently was thinking the same thing - upgrade at quarterback, and he did - with Cassel.

Going back to the standard normal curve, let's look at just the first year performances. Recall that, when all 1,451 seasons are aggregated in a histogram, it looks very much like a standard normal curve.

This doesn't quite look "normal", does it? The first year performances appear skewed to the left, or, compared to all quarterback performances, appear to under-perform. Since the total database is normally distributed, then performances in years 2 and beyond must obviously be skewed to the right. This prompts the question: does experience improve performance, and, in particular, how and when?

One way to look at this is to take the entire database, and look at the average standard deviation from the mean for each additional year a player qualified. Here's what that graph looks like.

Wow! I didn't realize that it takes about 5 qualifying years to turn into an average quarterback. And that's just for those playing that long! Even though this graph goes out to 18 years, there is hardly any data beyond the 15th year (there are only 8 quarterbacks who have had qualifying seasons 16 different years). According to this illustration, basically, first year quarterbacks, as a group, struggle mightily. They then gradually improve each year until about year nine. Quarterbacks, on average, must still endure sub-par years in years 2-4, and for those that survive, their reward is about four more years of improving performance, assuming they stay healthy for that long.

Note: This analysis - the method of breaking down a quarterback's expected career in terms of passing performance as it relates to experience and how it is expected to change over time, is the first I've seen published publicly. But I digress.

Ok, so we've taken a look at these 8 QBs relative to all quarterbacks in 2008, and we've taken a look at first year quarterbacks in general. So the next step is to take a look at the best first year seasons. In my database, there are 336 first year seasons in all (in other words, in the 69-year period from 1940-2008, 336 different quarterbacks threw enough passes in a season to qualify at least once). The table below shows the 75 best.

Cassel (#48) and Rodgers (#60) are the only 2 of the 8 in 2008 that crack the top 75 all-time. On the other end of the scale, we get JaMarcus Russell (#264) and Tyler Thigpen (#290).

When Matt Ryan (#128) got off to a solid start, and the Falcons surprised many by getting into the playoffs, there were many debates as to whether his season was one of the best ever by a first year player. Not according to this measure. Often, during those discussions, Kurt Warner's first year was brought up. In looking at the table above, well, he did have an outstanding first year in 1999. It ranks #6 all-time in terms of first year seasons. The best ever - Roger Staubach's 1971 season (Staubach was a rookie in 1966, and played in both 1969 and 1970. However, he didn't attempt enough passes in either year to qualify).

Notable quarterbacks high on this list: Joe Montana in 1980 (#4), Tom Brady in 2001 (#14) - (see my post earlier comparing Brady's season to Cassel's here), Brett Favre in 1992 (#15), Dan Marino in 1983 (#21), and Johnny Unitas in 1956 (#22).

Notable absentees: Steve Young in 1986 (#154), Peyton Manning in 1998 (#208), Joe Namath in 1970 (#202)*, and Troy Aikman in 1989 (#311). The all-time worst first year performance - Terry Bradshaw in 1970 (#337)*.

* Keep in mind that 1970 was the first year of the NFL following the merger, and 15 quarterbacks showed up in the database as having that year as their first year, although Bradshaw's 1970 season was his actual rookie season.

Are we done? Hardly. So after I went through this list, and still trying to objectively value Matt Cassel, I was interested in answering the following question(s): what does the first-year tell us about a quarterback's future potential? Do quarterbacks who have good first years, have good careers? What about quarterbacks who have sub-par first years? In other words, is there a correlation between year 1 performance and subsequent years? When in doubt, it's always a good idea to take a look at the data.

Whoa! The correlation between year 1 performance, and career performance, as measured by CMI standard deviations relative to the mean, is 0.75. What that really means is that 57% of the variation in a quarterback's career performance can be explained by his year 1 performance. I find that incredible. So I looked a little deeper. It turns out that 112 of the 336 quarterbacks only have 1 qualifying year. And, for that group, the correlation coefficient is, well, 100%! If I only looked at those quarterbacks whose career included at least 2 different years in which they threw enough passes to qualify, then the correlation coefficient drops to 64%. Still quite remarkable.

The table below breaks down the 336 first year seasons in terms of CMI standard deviations to see if we can glean any additional insight into whether a good first year translates into a good career and vice versa.

Well, the evidence is pretty clear. Taken as two groups, players that have better than average (remember, the mean and standard deviation is relative to all quarterbacks, not just first year quarterbacks) first years tend to have longer careers and more productive careers than those that have sub-par first years. And, breaking it down even further (in other words, just looking at the group whose first year was better than average, or the group whose first year was worse than average), you can clearly see that even within those sub-groups, the better the first year, the longer the career (although this correlation isn't nearly as strong), and the more productive the career.

I haven't spent too much time on my blog discussing individual careers, except in the posts discussing Brett Favre and Jeff Garcia. If I do this en masse, it would be such a powerful post - I'll basically be giving you my list of greatest passers to ever play the game, that I need to put a lot more thought into it than I have to-date. In any case, some of the obvious names will show up on that list. However, you'll find some exceptions as well (and I'm sure many of you will take exceptions to the list). The point I really want to make here is that, when combining the table above, and the discussion following the list of the top 75 first years, it is quite remarkable how Troy Aikman "escaped" the trend, and turned into a great quarterback (or, more accurately, a great passer) - he truly turned out to be an exception (as for Young and Manning, even though they didn't rank well on the first year list, their first years weren't "that bad" compared to other first year seasons - as a matter of fact, Young's was "above average" compared to a typical first year). Notice how I don't mention Bradshaw and Namath - it is because, statistically, their careers were sub-par. When I publish my list of all-time greatest (and worst) passers, we'll get into it in more detail.

So, in trying to look at Matt Cassel's first year performance objectively, we've looked at how quarterbacks perform as their careers progress, how quarterbacks perform in their first year, who had great first years, where Cassel's season ranks on that list, and, how a first year translates into (or, is predictive of) a career. Now perhaps, we can view Matt Cassel's 2008 year in an objective manner, and, if so inclined, you can at least evaluate the trade armed with some data.

Naturally, while I was putting this post together a question popped into my mind, and that is: is a quarterback's second year a good indicator of their career? What about looking at their first and second years? Check back in a week or two to find out.

A Brief History of the NFL Passer Rating System

2009-02-21T17:23:00.000-08:00

Philip Rivers of the San Diego Chargers had a phenomenal season in 2008. Or did he? According to the NFL's passer rating system, Rivers' season, where he completed 312 passes out of 478 attempted, threw for 4,009 yards, and 34 touchdowns against 11 interceptions resulted in a passer rating of 105.5. That's great, right? Before we answer that question, we'll need to answer a few more questions.

How do the statistics 312 for 478, with 4,009 yards, 34 touchdowns and 11 interceptions translate to a rating of 105.5?

What does having a rating of 105.5 mean?

Where does Rivers' season rank all-time?

How has the average NFL passer rating changed over time, and how does that affect how we look at Rivers' season?

The NFL began keeping statistics in 1932. The NFL has used various methods to determine the "best" passer in the years since. According to the Pro Football Hall of Fame, the following is a chronology and description of the different methods used:

1932-1937: Total yards passing

1938-1940: Percentage of completions

1941-1948: Inverse ranking system of the following categories: completions, percentage of completions, total yards, total TD passes, number of interceptions, and percentage of interceptions

1949: The same formula used from 1941-1948 except the number of interceptions were dropped from the equation

1950-1959: Average yards gained per pass with a minimum of 100 attempts needed to qualify

1960-1961: Inverse ranking system based on six categories: total completions, total yards, total TD passes, percentage of completions, percentage of interceptions, average gain per attempt with the principle established of at least 10 attempts per game to qualify

1962-1971: Inverse ranking system based on four categories: percentage of completions, total touchdown passes, percentage of interceptions, average gain per attempt

1972: Same system used from 1962-1971 except that the percentage of touchdown passes was substituted for total touchdown passes

1973 to present: See below.

Current NFL Passer Rating System:

The passer rating system used by the NFL has been in place since 1973. In 1971, after the merger, then-commissioner Pete Rozelle wanted to implement a standardized set of statistics, including a standard measure of passing performance. As the description on the Hall of Fame site suggests, this is a measure of a quarterback's passing effectiveness, not a measure of how good a quarterback is. Rozelle called upon Don Smith, then an executive with the Hall of Fame, to work with the league's official statistician, the Elias Sports Bureau, to develop a new standard.

One of the problems with the standard that existed at the time, was that you didn't know where you stood until all the teams' quarterbacks had finished playing, as it was a relative ranking system. In addition, there wasn't a convenient way to compare a performance in a given year to that of another.

Don liked the use of the combination of statistics - in other words, the completion percentage, the yardage per attempt, the touchdown percentage and interception percentage. He had to figure out how to use all four in some manner that would make sense.

Working with the Elias Sports Bureau, he studied passing statistics for each of the four categories from the decade before - the sixties. After much study and thought, he devised a system whereby, for each of the four statistical measures, he would convert that particular measure to a "score" between 0 and 2. A score of 0 would indicate poor performance, a score of 1.00 would indicate "average" performance, and a score of 2.00 would indicate "superior" performance. With truly exceptional performance, it was possible to exceed a score of 2.00. He arbitrarily chose the maximum score to be 2.375.

He decided that an "average" performance across all four categories, that is, a score of 1.00 for each of the four measures, should be a rating of 66.7. Under this new rating system, it would be possible for a rating to exceed 100.0, but, he theorized that those instances would be rare.

(Note: I took a lot of this history from an article written by Don Steinberg, published in Slate magazine in 2001 - see here for the full article)

In summary then, this is how we end up with the current NFL passer rating system: we take four components of passing, convert each component, using league averages from the 1960's, to a score with a minimum of 0.000 and a maximum of 2.375, combine these scores by adding them, and convert to a rating system that has 66.7 as the "average".

Here's the math that does this:

Q = [ ( J + K + L + M ) * 100 ] / 6

where,

Q = Passer Rating

J = max [ min ( C, 2.375 ), 0 ]

K = max [ min ( Y, 2.375 ), 0 ]

L = max [ min ( T, 2.375 ), 0 ]

M = max [ min ( I, 2.375 ), 0 ]

and where,

C = [ ( Completions / Attempts ) * 100 - k1 ] / 20

Y = [ ( Yards / Attempts ) - k2 ] / 4

T = [ ( Touchdowns / Attempts ) * k3 ] * 20

I = 2.375 - [ ( Interceptions / Attempts ) * k4 ] * 25

and where,

k1 = 30

k2 = 3

k3 = 1

k4 = 1

In the first formula, you can easily see where if J = K = L = M = 1.000, the passer rating formula would yield Q = 66.7. In order to get J = K = L = M = 1.000, certain "transformations" were needed to each of the passing statistics to convert the averages to a score of 1.000. The transformations for each are shown as C, Y, T, and I, and, more specifically, k1, k2, k3 and k4. The figures for k1-k4 were derived using league statistics from the 1960's, and to make it somewhat easier, rounded. If you looked at data from 1960-1969, you get the following actual values for k1-k4:

k1 = 31.70

k2 = 3.24

k3 = 0.96

k4 = 0.99

The table below shows, for each NFL season from 1940-2008, how the averages, or, more precisely, the "scores" for those averages have changed from year to year (for a graphical illustration see my previous post on the subject here). For example, you can see that the average score for "J", in 2008 was 1.550, reflecting the fact that the "C" component, or, completions per attempt, for the average NFL quarterback was 61.0%. Compare this to the average score for "J" in 1968 of 1.079, reflecting the "C" component, or completions per attempt of that time of 51.6%. This is what happens when the transformations do not change over time, even when the actual game itself has undergone many transformations. If one wanted to keep everything in balance from year to year in the passer rating formula, in other words, to keep J = K = L = M = 1.000 for every year, then one would have to change the values of k1 - k4 every year. I have done that in the table below.

You can use these different k values every year to "adjust" the passer rating formula, so that you can make meaningful comparisons from year to year. If you didn't do that, and simply compared a quarterback rating from one year to the next, you wouldn't get an appropriate comparison, for the simple reason that the four components would be "out of balance".

While it was never Don Smith's intention to have a relative measure of performance - he wanted performance as measured relative to a fixed standard, that "fixed" standard has changed, and will continue to change over time. One way to minimize the effect of these changes is to look at a given performance in a given year to that season's average, using the standard deviation as a measuring stick. The other way, is to simply "adjust" the standard to reflect the averages for that year. In a previous post, I discussed the former method. In this, I am obviously discussing the latter.

This discussion is only to put Rivers' 2008 season in perspective. It is not intended as passing judgment on the NFL passer rating system. I have, in previous posts (see here and here) discussed why I do not like the system. In a post in the future, I'll elaborate further on why I think the NFL's passer rating system should be revised.

The table below shows the 75 greatest seasons in the NFL, both in terms of the current NFL passer rating system, and using an "adjusted" passer rating system, going through the transformations as I have described above. You can see that, using the current system, that Rivers' 2008 ranks as 13th best all-time. Using an "adjusted" passer rating system, however, results in a ranking of 60th best all-time. If we used the previously mentioned standard deviations from mean measurement, it would rank as the 16th best all-time.

We still have some unfinished business:

How does Rivers' stats translate to a rating of 105.5?

He completed 312 of 478 passes, or, a completion percentage of 65.3% - using k1 = 30, and Comp/Att = 0.653 in the formulas above, yields a J value of 1.764

He threw for 4,009 yards, or, yards per attempt of 8.4 - using k2 = 2, and Yards/Att of 8.4, we get a value for K of 1.347

He threw 34 TDs, or, TD/Att of 7.1% - using k3 = 1, and TD/Att = 0.071, we get a value for L of 1.423

He threw 11 INTs, or, INT/Att of 2.3% - using k4 = 1, and INT/Att - 0.023, we get a value for M of 1.800

Combining all, we get

Q = ( J + K + L + M ) * 100 / 6, or

Q = ( 1.764 + 1.347 + 1.423 + 1.800 ) * 100 / 6 = 105.5

There is no doubt that, whether we use the current NFL passer rating system, an "adjusted" passer rating system with different transformations for different years, or, the current system, but adjusted by looking at standard deviations from the mean, that Rivers' 2008 season was quite remarkable.

Whether the passer rating system is an accurate reflection of a passer's ability is a completely different question altogether. We shall, over time, address this question.

Anybody Looking for a Good QB?

2009-02-16T12:11:00.000-08:00

Available: Experienced quarterback with an accurate arm with ability to make immediate contributions to team; Ideally, would like to start, but not a requirement; may be perfect as back-up; durability maybe an issue.

I am referring of course to Jeff Garcia, released today by the Tampa Bay Buccaneers. I am personally a big fan of Garcia, because I believe he is underrated, and his achievements on the field have gone relatively unnoticed.

Let's take a look.

He began his NFL career in San Francisco in 1999, replacing not one, but two back-to-back Hall of Famers, and arguably the two greatest passers of all-time, Joe Montana and Steve Young. There was no way Jeff could live up to those lofty expectations. However, he did perform very well, as the table and chart below shows. While he may not have performed at Joe Montana and Steve Young levels, his years in San Francisco were very productive. After a sub-par year in 2003, he bounced around for three years, first as a starter in Cleveland (2004), then as a back-up in Detroit (2005) and Philly (2006). He finished the 2006 season as the starter in Philadelphia for the last 6 games after Donovan McNabb got injured. He showed that he could still play, and Tampa Bay, in need of a quarterback following an injury to Chris Simms, picked him up. He did well above average each of 2007 and 2008, as you can see from the table and charts (he doesn't show up in the charts in the years 2004-2006 since he didn't attempt enough passes to qualify).

Now, Tampa Bay, deciding to turn to the youth movement, have invested in Luke McCown, and, have deemed Garcia expendable.

Garcia has been a very accurate as a passer throughout his career. In a previous post, I discussed interceptions at length. His 2007 season ranks 18th all-time (see Exhibit 3 in this post) in interceptions per attempt, in terms of standard deviations from the mean (his 2008 season ranks 118th, out of the 1,451 seasons in the database since 1940). As a career passer, he ranks 12th all-time (see Exhibit 6 in this post). By not throwing interceptions, he will help your team win.

I believe that Garcia has a couple of good years still left in him, if not as a starter, as a back-up. His durability has been an issue the past two seasons, and hence, might be a liability as the starter.

So who might be worthwhile candidates?

The following charts show, for each of 2007 and 2008, how a team's passing game stacked up in terms of standard deviations from the mean. It's no coincidence that teams that were above average won more games than they lost, and teams that were below average lost more games than they won. The charts show how the teams did both in terms of NFL passer rating and CMI. Note that Garcia averaged 0.53 standard deviations above average in terms of NFL passer rating for the past two seasons, and averaged 1.23 standard deviations above average in terms of CMI.

To see which teams should be looking to get Garcia, let's start at the bottom, and move our way up.

Cleveland Browns - They have two young quarterbacks, Derek Anderson and Brady Quinn, both coming off season-ending injuries, that will compete for the starting job. Not a candidate. Look for another miserable year in Cleveland.

St Louis Rams - Marc Bulger is the starter, and for now at least, another soon-to-be 39 year-old - Trent Green, is the back-up. Not a candidate. By the way, what's happened to Bulger? Will 2009 be more like his first few years, or his last two? I suspect the latter. Another long year in St Louis.

Oakland Raiders - Paying a lot of money for JaMarcus Russell. They've also got Andrew Walter, Marques Tuiasosopo, and they just picked up Bruce Gradkowski off waivers. I haven't a clue what the Raiders are doing, and I don't think the Raiders do either. Not a candidate. Perhaps we'll start paying attention when Al Davis passes away.

San Francisco Forty Niners - They're not sure who their starting quarterback is. Shaun Hill? J.T. O'Sullivan? Alex Smith? The niners maybe a candidate. Garcia played here, and is originally from close-by Gilroy.

Detroit Lions - Daunte Culpepper is the starter. Or is it Dan Orlovsky? Didn't seem to matter in 2008, as they went 0-16. If I was GM, I'd replace Culpepper with Garcia, and draft a quarterback. But that's just me. Garcia may not want to go here, but that's a different matter altogeher.

Chicago Bears - Kyle Orton is the starter, and Rex Grossman is the back-up. Neither of whom are very good. I would think that Garcia would be an excellent choice to be a back-up here.

Kansas City Chiefs - Tyler Thigpen is the starter. Previous starter Damon Huard is the back-up. Garcia could be a potential back-up here.

Cincinnati Bengals - Ryan Fitzpatrick finished the year as the starter, having replaced the injured Carson Palmer. Not a candidate.

Of the remaining teams, perhaps the Minnesota Vikings might be interested. Gus Frerotte is simply not a good option. Maybe they'll lure Brett Favre out of retirement! The NY Jets might be an option for Garcia as well. Most likely, Kellen Clemens will be the starter, but Garcia could be a viable back-up here.

So these are teams that should be interested in Garcia. We'll wait and see where he actually ends up.

Brett Favre - Best Ever?

2009-02-14T22:06:00.000-08:00

Brett Favre retired earlier this week.

He played in the NFL for 18 years. He set numerous NFL records during the course of his career. The most impressive has to be the fact that, as a quarterback, he started a record 269 consecutive regular season games. Peyton Manning is second on the list, his streak currently standing at 176 games. Third on the list is Ron Jaworski, his streak having ended at 116 games. Tom Brady is next on the list, but his streak ended in the first game of last season at 110 games. Peyton Manning needs another 94 games - he would have to start every single game through the 14th game of the 2014 season to break Favre's record. In addition to the four mentioned above, only two other quarterbacks in history - Joe Ferguson (107 games), and Dan Marino (95 games) have started that many games consecutively. I suspect Favre's record will never be broken.

People are fascinated with Favre. Many consider him to be among the all-time greatest to ever play the game. Some consider him the greatest ever. I do not belong to either group.

The media loved Favre. And vice versa. He never met a camera he didn't like, always seeking an opportunity to get into the limelight (of course, the media loved that). He also seemed intensely competitive, and the combination is what I believe led him to his unfortunate decision to come back for one more season in 2008.

Let's start from the beginning.

In 1991, the Atlanta Falcons took Favre in the second round of the draft. With Chris Miller as their starting QB, in 1992, the Falcons traded Favre to Green Bay. Favre started the season as the back-up to the 'Magic Man', Don Majkowski. In the third game of the season, against the Cincinnati Bengals, he replaced the injured Majkowski in the first quarter. With 13 seconds left, he threw a 35-yard touchdown that helped the Packers come-from-behind to win 24-23. He started the next game against the Steelers, a game the Packers won. The rest, as they say, is history, as he started every game for the Packers until his trade to the Jets following the 2007 season (for a detailed history of, or, more aptly, a tribute to Favre, see here).

The table below shows how Favre did each year.

Before we go on, some explanations.

In order to be able to truly compare Brett Favre to other quarterbacks so that we can properly gauge his place in history, we can't simply look at the NFL's passer rating system. First, we know that the average NFL passer rating in the NFL has been increasing over time. So, by simply using the NFL's passer rating system, the more recent quarterbacks would appear to have done better than quarterbacks from earlier years (see here and here for evidence). Second, we have to agree that the NFL's passer rating system is an accurate, if not adequate measure of a quarterback's performance. I have previously introduced a metric, CMI, which I believe is a better, if not more understandable measure of performance. CMI, however, also has the same bias as the passer rating system. One way to adjust for, and remove the bias favoring more recent quarterbacks is to look at each year's performance for each quarterback, and relate that to the average performance that year. The measure that we use is the standard deviation. I have discussed previously how using this measure tends to "normalize" performances across years. In other words, when a given year's performance by a particular quarterback is "normalized" to the average for that year, and we do that for every year, we can very easily see how that particular performance compares to other performances across years. Since CMI is very similar to the passer rating system, when using standard deviations, it shouldn't really matter whether we use CMI or the NFL passer rating system (there will be differences, and, I often include both so that you can see the differences). The two graphs below show how using either the "normalized" NFL's passer rating system or "normalized" CMI as measures compare to a standard normal curve with 1,451 observations (that's how many qualified quarterback seasons there have been over the past 69 years).

Ok, so back to the table.

Favre had a "pretty good", or, "above average" first year in 1992. Using the NFL passer rating, he finished 0.57 standard deviations above the average that year. Using a standard normal curve, that would mean between the 71st and 72nd percentiles. If we used CMI as the standard, then his performance that year, 1.24 standard deviations above the mean, would place him between the 89th and 90th percentiles.

He followed that with a sub-par year in 1993, finishing between the 30th and 31st percentiles according to the NFL passer rating system, or between the 35th and 36th percentiles using CMI. Then followed several "above average" to "very good" years (1994-1998), depending on your perspective, followed by a couple of mediocre years (1999-2000), followed by four above average years (2001-2004), followed by his two worst years (2005-2006). He finished his Green Bay tenure with an "above average" year in 2007.

He should have taken the opportunity to retire then.

His last year, his first full season not with Green Bay, was "below average". So he finished his career with 3 "below average" years out of 4. Perhaps the Packers brain trust was on to something when they decided that Aaron Rodgers was their future at the end of the 2007 season.

The two graphical illustrations below show each of Favre's seasons in terms of standard deviations from the mean. In case you haven't noticed, as much as I love statistics, I find that graphs are better way to illustrate a point than a table of statistical measures. I show the CMI graph first and the graph based on NFL passer rating next, for reasons that will become plainly apparent when you read on.

Ok, so until now, we've been looking at Brett Favre's career in a bit of a vacuum. Yes, we have compared his performances to an average, and yes, we have tried to put it into historical perspective, but I haven't given you any 'perspective'.

First, let's compare Brett Favre's career to some individual careers. The biggest problem we face in comparing Favre's career to any other individual career, is obviously the length of his career. There have been only 5 individuals who have thrown enough passes to qualify in at least 15 different seasons. Those are Fran Tarkenton (18 years), Brett Favre (17 years), Dan Marino (16 years), John Elway (16 years), and Johnny Unitas (15 years). So how does Favre's "picture" compare to these quarterbacks, all of whom are in the Hall of Fame?

Fran Tarkenton - a picture of "excellence". With a couple of exceptions, Tarkenton exceeded his peers consistently for a very long time. Quite remarkable. Without question, Favre's picture does not resemble that of Tarkenton.

Dan Marino - Marino was "above average" for many of his years, tailing off towards the end of his career. He was "very good" very early in his career. He really didn't have a "bad" year until his last, and wisely decided to hang it up after that. Although closer in resemblance, Marino's picture still looks better Favre's overall.

John Elway - Elway underperformed his peers much of his early career, exceeding the average in only two of his first ten years. He then was "above average", or even "very good" his last six years. He went out on top, having retired following the Broncos having won back-to-back Super Bowls (Super Bowl XXXII and Super Bowl XXXIII). I think it's safe to suggest that Favre's picture looks better than Elway's.

Johnny Unitas - In my opinion, Favre's picture most closely resembles that of Johnny Unitas. They both started out well, had a couple of mediocre years, then had several years where they outperformed their peers, and ended their careers with a few sub-par years.

What if we compared Favre's average performance (in other words, average each of his year's performance in terms of standard deviations), to other players? Well, over his career, in terms of NFL passer rating, Favre averaged 0.42 standard deviations above the mean (0.26 when looking at CMI). Tarkenton averaged 0.65, Marino averaged 0.63, Elway averaged 0.06, and Unitas averaged 0.51 standard deviations above the mean.

Let's expand the comparison list to players who qualified in at least 10 different seasons. 41 different quarterbacks make that list. The two graphs below illustrate where Favre ranks on those lists.

The same graph as above, except looking at it Using CMI as the measure of performance.

Looking at his career statistically, it is hard to make the argument that he belongs in the truly elite group of all-time quarterbacks - those that would be ranked in the top 3 or top 5 all time. I think it is fair to suggest that Favre has had a very good career. It was certainly a lengthy one, and, in that regard, it was exceptional. But as far as his actual performance on the field, there isn't a measure, or set of measures, that suggest that it was exceptional.

People will suggest that he holds the all-time record for yards, or touchdowns, for example. But that is merely a function of his longevity, just as much as he holds the record for interceptions thrown, all-time.

I also don't discuss measures frequently used as evidence of greatness. MVP awards, for example - these are popularity contests more than a reflection of outstanding statistics or an individual's value to a team (the NFL MVP award is given by the Associated Press). Pro-bowl selections for example are another popularity contest - just look at who got selected this past year in the AFC. Another measure frequently used is Super Bowls. Dan Marino didn't win any Super Bowls. Trent Dilfer won a Super Bowl. Winning percentage is also frequently used - last time I looked, a quarterback doesn't play defense, return kicks or punts, or kick field goals. These pieces of evidence would never enter any serious discussion in terms of statistical evaluation.

My eventual goal is to identify the greatest passer of all time. I will use several statistical measures, many of which are mentioned in this post, to identify the most worthy candidates for consideration. As far as my research is concerned so far, Favre would not make that list. There are several other quarterbacks, not mentioned in this post, nor even displayed in either of the last two graphs that would make more worthy candidates, but that is for a different article.

The Importance of Interceptions (or lack thereof)

2009-02-06T17:15:00.000-08:00

I have spent quite a bit of time lately talking about interceptions. In case you have any doubt that an interception can make a difference you got that answer on Super Bowl Sunday. In probably one of the greatest plays in Super Bowl history (until the catch by Santonio Holmes with 35 seconds left that gave the Steelers a come-from behind victory), with Arizona on the Pittsburgh 1 yard line, first and goal, and 18 seconds left in the first half, the Steelers' James Harrison picked off Arizona's Kurt Warner and returned it all the way for a touchdown as time expired. This was, in effect, a 13-point play, as Arizona's expected points at the Pittsburgh 1 was about 6 points. Brian Burke, who I've highlighted before, has an excellent post on the subject on his site at advancednflstats.com.

Take a look at the graph below. The blue line (on the left scale) shows the league average QB passer rating (for those QBs who thew enough passes during the season to qualify) by year since 1940. As you know, the NFL's QB passer rating formula has four components - completion percentage, yards per attempt, touchdown percentage, and interception percentage (see my previous posts on the subject here and here). When it was designed in 1973, the formula used the 1972 season as a "base", and hence created adjustments to each component, such that the average would be a score of 1.00 for each component, resulting in a passer rating for 66.7 for a quarterback who had average statistics in each of the four categories. The actual calculations for each of the four components in 1972 yielded the following four figures - 1.085, 0.954, 0.897, and 1.043, respectively, which in turn yielded the average quarterback passer rating of 66.3 (the figure for the qualified leaders turns out only slightly higher - 67.9). Back then, each of the four components were essentially balanced.

Today, it's a different story. If the system was balanced, then we would expect the interceptions component to make up about 25% of the quarterback passer rating score. Looking at the graph again, and this time looking at the red line (on the right scale) shows how much the interceptions component influences the league's quarterback rating system. It hasn't been below 30% since 1983, and the last time it was "around" 25% was actually 1971. The point is that the NFL passer rating system is not what it used to be, and interceptions are the leading weight in today's NFL passer rating system.

Let's take a look at another example. Tom Brady's record-breaking 2007 season. That year, Brady completed 398 of 578 passes, for 4,806 yards, and threw for 50 touchdowns while only throwing 8 interceptions. His passer rating that year - 117.2, is second only to Peyton Manning's all-time best 121.1 in 2004. Most people will remember the 50 touchdowns that Brady threw. Impressive as it was, it wasn't that impressive (I'll have a post later on this subject). Most people will not remember that he only threw 8 interceptions in 578 attempts. That ranks as 14th best all-time in terms of single-season interception percentage (see Exhibit 2), and, it accounted for 29% of his quarterback passer rating that year (his touchdowns accounted for 25%).

There are 7 exhibits on interceptions attached to this post. Note that for all exhibits, I only considered those passers in any given year that threw enough passes to qualify - in other words, if a quarterback threw 20 passes during a season, that would not be enough to qualify him. On the nfl.com site, the minimum qualification standard is 14 passes per game (so, for the 2008 season, the standard would be 14 * 16 = 224 passes). The NFL uses this standard EVERY year.

I think this universal standard is inappropriate. Using this method for example, only three passers qualified in 1940. The 14 passes comes from the fact that during the 50's, 60's and 70's, the average number of passes attempted by a team during a game was about 28. Of course, we all know that this has changed over time. The average number of passes attempted in a game by a team was 32.3 in 2008. It has been below 30.0 once since 1979.

The standards that I use varies by year, and reflects the average # of passes attempted during these years. Here are the standards:

1940-1946 - 6.5

1947-1960 - 11.0

1961-1969 - 14.0

1970-1977 - 12.0

1978-1994 - 15.0

1995-2008 - 16.0

Using these standards, I get 1,451 quarterbacks in my database, with a low of 9 quarterbacks qualifying in each of 1941 and 1943, and a high of 32 quarterbacks qualifying in each of 1999 and 2005. For 2008, I had 30 quarterbacks in my qualified database, whereas the NFL.com has 32. So, not a big difference in recent years. I just think that applying a universal standard across all these years is silly, especially when the game has changed so much.

In any case, now that we got the some of the technical stuff out of the way, here are the 7 exhibits:

Exhibit 1 - Chronological list of NFL leader (lowest) in interception percentage

Exhibit 2 - Best seasons in terms of interception percentage

Exhibit 3 - Best seasons in terms of standard deviations from the mean

Exhibit 4 - Worst seasons in terms of standard deviations from the mean

Exhibit 5 - Best (lowest) career interception rate (minimum of 1,000 passes attempted)

Exhibit 6 - Best career interception rate relative to league average

Exhibit 7 - Worst (highest) career interception rate relative to league average

Exhibit 1 - Chronological list:

Nothing spectacular here, it's simply each year's best. I observe a few things:

Clearly, the average interception rate has been decreasing every decade.

The best quarterbacks in a given year seem to be around 1.8 standard deviations better than the average.

Slinging Sammy Baugh led the league 4 out of 6 years during the period from 1942 to 1947 - the only quarterback to have led the league on 4 separate occasions.

6 quarterbacks - Bart Starr, Bobby Thomason, Charlie Conerly, Ken Anderson, Ken O'Brien, and Roger Staubach have led the league on 3 different occasions.

5 quarterbacks led the league in consecutive seasons - Sammy Baugh, Bobby Thomason, Milt Plum, Ken Anderson and Ken O'Brien.

Only 2 quarterbacks in history, Steve DeBerg in 1990 (0.90%), and David Garrard in 2007 (0.92%) completed a season where less than 1 percentage of their attempted passes were intercepted (this is not entirely evident by looking at Exhibit 1, but can be confirmed by Exhibit 2). What's most unusual about DeBerg's performance that year was the fact that of the four interceptions he threw during the season, three of them were in one game! In other words, he threw 1 INT the rest of the season. Let's take a look at Exhibit 2.

Exhibit 2 - Best seasons in terms of interception percentage:

So indeed, only 2 quarterbacks have had seasons with fewer than 1% interceptions. The list below is the kind of list that would show up in a record book.

You'll see, for example, that Jason Campbell's 2008 season ranks in the top 5 all-time, and 2 other quarterbacks in 2008, Chad Pennington and Jeff Garcia, also had noteworthy seasons, with both finishing in the Top 30 all-time.

If you look carefully, you'll notice that the list is dominated by quarterbacks in the past 2 decades. Of the Top 50, 45 have occurred since 1990. 21 last decade and 24 this decade. The 5 seasons in the top 50 not to have occurred in the last 19 years are Steve Bartkowski's 1983 season (#3), Bart Starr's 1966 and 1964 seasons, respectively (#7 and #18), and Ken O'Brien's 1985 and 1988 season's respectively (#36 and #38).

So, what are we saying? Are we saying that quarterbacks prior to 1990 were not very good? No, not at all. The league has changed. From Exhibit 1, you can see that the average interception rate has been decreasing every decade. So, we simply can't just compare a quarterback from one decade to another. That's where Exhibit 3 comes in. Let's take a look at that.

Exhibit 3 - Best seasons - Interceptions percentage, ranked by how different the particular season was compared to the mean, using the standard deviation as the measuring stick:

Wow, what a difference. You can quickly see that this is a much better representation of the past 7 decades.

1940's - 3

1950's - 2

1960's - 10

1970's - 9

1980's - 7

1990's - 6

2000's - 13

Look at Bart Starr! Ranked twice in the Top 5, and 3 times in the Top 25.

And, this measure doesn't discount Steve DeBerg's and David Garrard's great seasons - they're both still in the Top 5. But it does give one a bit more perspective. In other words, this suggests that Bart Starr's 1962 season (which ranks #430 in absolute terms), when compared to everyone else's performance during that season, was better than David Garrard's performance, when compared to how all the other quarterbacks did in 2007.

Now for a little math. Why are we using standard deviation as a measure of separation? And why does using it make comparing quarterbacks across years more meaningful?

A non-technical definition of the standard deviation is that it is a measure of the dispersion of a set of data around the average. By dispersion we mean spread. Knowing the average of the data, and knowing how spread the data is, we can try to determine how likely a given observed value is. We can use this data to compare different data sets, and relate them to one another. So for example, in a data set where the average is 5, and the standard deviation is 2, an observation of 8, would mean 1.5 standard deviations ((8-5)/2 = 1.5) above the the mean. In a data set where the average was 7, and the standard deviation was 4, a value of 13 would also be 1.5 standard deviations from the mean. You can now see why the use of a standard deviation could be useful in comparing different sets of data.

As you can see from Exhibit 1, the league average interception rate has been changing over time. In addition, while I have not shown it explicitly, the standard deviation around the mean has also been changing. As a matter of fact, in the early 1940's, the standard deviations were quite high because there were fewer players involved, the number of attempted passes were fewer, and arguably the talent pool was not as great (i.e. passing was a fairly new concept).

By relating a given observation of an interception rate in 1943 to the mean interception rate that year, and the standard deviation of interception rate that year, one can then compare that particular observation to an observation of an interception rate in 2003, by its relationship to the mean and standard deviation of interception rates in 2003. If one assumes, in particular, that in any given year, that all observations about the mean are distributed normally (i.e. a "bell-shaped" curve), then the comparisons become that much more meaningful. For example we know that in a standard bell-curve, that approximately 68% of the observations will fall into a band +/- 1 standard deviation from the mean, and approximately 95% of observations will fall into a band +/- 2 standard deviations from the mean. Also, if, after relating the observations in each year to each year's mean and standard deviation, one aggregates the data across all years (since the data has been "normalized" to the same scale - a number in relation to a mean and standard deviation; in the example earlier, the observation of 8 in the first data set has the same value on a normalized basis as the observation of 13 in the second data set - 1.5), then the aggregated data should look like a standard normal curve, with a mean of 0, and a standard deviation of 1.

Well, I went through the trouble of doing that, and guess what - that is exactly what the looks like. All 1,451 qualified passers over the 69 years from 1940 to 2008 were analyzed in this manner, and the result is in the graphical illustration below. The average for the entire data set is -0.06 (close to 0), and the standard deviation is 1.00! Also, it turns out that 69% of the observations are within +/- 1 standard deviation, and 96% are within +/- 2 standard deviations. Quite a remarkable achievement! I've graphed a standard normal distribution as well, so that you can see for yourself how close the actual data is to the theoretical curve.

Exhibit 4 - Same as Exhibit 3, except ranks the worst seasons:

Terry Bradshaw's 1970 rookie season ranks as the all-time worst. And Vinny Testaverde's 1998 campaign is not that far behind. Although Testaverde's rookie season was 1987, he didn't "get exposed" until 1988. Most recently, Gus Frerotte's 2008 campaign was an absolute disaster. There's some names on the list that I would not have expected to see (let's face it, these are the 50 worst performances over the past 69 years - 1,451 quarterback seasons) - names like Favre and Aikman, along with the aforementioned Bradshaw. What was even more surprising to me was that they each showed up not once, but twice!

So now you've seen the best and the worst seasons. How about careers? Let's take a look at Exhibit 5.

Exhibit 5 - Lowest career interception rate - absolute figures - with a minimum of 1,000 passes attempted:

For reasons I discussed above, I don't like this list that much, as it is biased towards the more recent years. But, since people like looking at these types of lists, I have included it. Exhibits 6 (best) and 7 (worst) reorder the data after a player's career has been compared to the average during their career.

Exhibits 6 (best/lowest) and 7 (worst/highest) - career interception rate relative to league average during the same time:

First question is: why, after the long dissertation about standard deviations, am I not using that as the measuring stick? Simple answer - it's difficult! It was a relatively simply exercise to calculate standard deviations for each year. And it's not that difficult to do it for a given set of consecutive years. Where it becomes increasingly difficult is to do it for every combination of multiple years, over a 69-year period. At some point before the 2009 NFL season, I'll have it done. That will be a better reflection of who had good or bad careers, but, in the meantime, this will have to suffice. It's the next best thing. It is most certainly better than the absolute comparison shown in Exhibit 5.

Ok, on to a few observations.

Exhibit 6 is a who's who of quarterbacks in football history. Taking out the 13 players who are either currently playing, or who have retired in the past five years (i.e. not Hall of Fame eligible), 14 of the remaining 37 are in the Hall of Fame. Let's look at it another way. There are 27 quarterbacks who are in the Hall of Fame, who played football post 1940. And two of them, Arnie Herber, and Clarence (Ace) Parker, played much of their careers before 1940. Of the remaining 25, 14 show up in Exhibit 6.

Who are the 14?

Joe Montana (#3)

Roger Staubach (#9)

Steve Young (#14)

Sammy Baugh (#19)

Fran Tarkenton (#20)

Bart Starr (#22)

Sonny Jurgensen (#23)

Otto Graham (#32)

Dan Marino (#33)

Sid Luckman (#37)

John Elway (#39)

Len Dawson (#42)

Johnny Unitas (#46)

Norm Van Brocklin (#50)

And 2 more, Y.A. Tittle (#53), and Troy Aikman (#55) just missed being in the Top 50. Not a bad list.

What about Exhibit 7? 3 Hall of Famers, Joe Namath, George Blanda, and Terry Bradshaw are in the worst 50 all-time, in terms of their career interception percentage as it relates to the league average during the time that they played.

Super Bowl Quarterbacks

2009-01-31T18:39:00.000-08:00

Super Bowl XLIII is tomorrow. So, I thought it might be a good exercise to review quarterback performances in each of the past 42 Super Bowls. The first table below shows how each of the starting quarterbacks did. Here are a few observations.

Quarterbacks in the Super Bowl make lots of mistakes. Look at the quarterbacks on the losing team. Until Super Bowl XXV between the Giants and Bills, every starting quarterback on a losing team had thrown at least one interception. As a matter of fact, there have been only 4 times - Super Bowl XXV between the NY Giants and Buffalo Bills, Super Bowl XXXIV between the St Louis Rams and Tennessee Titans, Super Bowl XXXVIII between the New England Patriots and Carolina Panthers and last year's Super Bowl XLII between the NY Giants and New England Patriots, where the losing team's quarterback did not throw an interception. When you think about each of these four games, one could make the argument that they all rank as the best played, most exciting and closest Super Bowls of all time.

Starting QBs on the losing teams have thrown a total of 83 interceptions in the 42 games. This compares to only 22 for starting QBs on the winning teams. Don't throw an interception, and you give your team an excellent chance of winning the game. Conversely, throw a pick, and you significantly reduce your team's chances of winning.

Nine times the losing team's QB threw 3 picks; 4 times they threw 4 picks, and in Super Bowl XXXVII between the Tampa Bay Buccaneers and Oakland Raiders, Rich Gannon threw 5 picks. That's 14 times in 42 games where a QB threw 3 or more INTs. There's a 1/3 chance that tomorrow one of the two QBs will throw at least 3 picks!

Three times the winning team's QB has thrown for more than 2 INTs. In Super Bowl XIV, Pittsburgh's Terry Bradshaw threw 3, in Super Bowl XVII, Washington's Joe Theismann threw 2, and in Super Bowl XL, Ben Roethlisberger threw 2 INTs. Note that Johnny Unitas threw 2 INTs in Super Bowl V as a back-up in a win over Dallas.

There have been 9 occasions where the winning team's QB threw for 3 or more TDs. 4 times they have thrown 3, 3 times they have thrown 4, and in Super Bowl XXIV, San Francisco's Joe Montana threw 5 TDs and in Super Bowl XXIX, San Francisco's Steve Young threw for 6 TDs against the San Diego Chargers. It appears that not throwing INTs is probably more important than throwing TDs.

The classic example of this is Super Bowl III, where NY Jets' Joe Namath guaranteed a victory against the Baltimore Colts. He delivered. He was named the game's MVP. Most people don't realize that he didn't threw a single TD in that game. Most importantly, he didn't throw an INT either.

The table below shows (in my opinion) the 10 best and 10 worst performances by a starting QB over the past 42 Super Bowls. It's difficult to compare QBs over time. This is because the most commonly used measure to evaluate a QBs passing performance, the NFL Passer Rating system, has shown that the league average has been increasing over time (see here for the details).

I have come up with a measure, CMI (Completions Minus Interceptions, calculated as Completion % - 3 * Interception Percentage), which I believe is a better measure of a QBs passing performance. However, this measure also has the same problem that the Passer Rating formula has, in that it also shows that the league average has been increasing over time (see here). One way to adjust for this trend is to relate each QBs performance in a Super Bowl to the overall league average for that year. Then, one can compare that particular measure across Super Bowls. If I had game-by-game data, I would not only relate the measure to the mean, but would also take into account the standard deviation. However, in the absence of game-by-game data going back 42 years, the measure relative to the mean shall suffice. Also, I eliminated from consideration any QB who attempted fewer than 15 passes during the game.

With the technical stuff all out of the way, the single best performance by a QB in a Super Bowl is Phil Simms for the NY Giants in Super Bowl XXI, when he went a near-perfect 22 of 25 with no INTs. Interestingly, his QB Rating for that game was 150.9, close to the perfect rating of 158.3. Steve Young's 6 TD effort doesn't show up in the Top 10, although Joe Montana's 5 TD effort does.

At the other end of the Spectrum, Craig Morton's 4 for 15 effort with 4 INTs in Super Bowl XII is the absolute worst performance by a QB in a Super Bowl by any measure. He also has the dubious distinction of showing up twice on the 10 Worst Performances list. Ben Roethlisberger, Pittsburgh's QB in tomorrow's Super Bowl XLIII, is the only QB who shows up on the 10 Worst list, whose team actually won the game - Super Bowl XL against Seattle.

Super Bowl Squares - Part 2

2009-01-28T22:08:00.000-08:00

This post is in response to a comment made regarding my previous post on the same subject. The suggestion was made that I only consider games since 1994 since that is when the two-point conversion was introduced. The new distributions are shown in the charts below. There is also a chart at the bottom that compares the previous chart with the changes to the probabilities.

The distributions did change. Some of it I'm sure due to the introduction of the two-point conversion. Some of it is by random chance. And, the uncertainty has increased. The # of games over which this data is analyzed is 3,884, a reduction from the 9,509 in the previous sample. All else being equal, and they are rarely if ever equal, the reduction in the sample size alone creates a larger uncertainty in the estimated probabilities (technically speaking, if the sample size increases by a factor of n, then the error in the estimate increases by the square root of n).

Therefore I'm not necessarily sure that these are better estimates of the true probabilities, but it is now shown. What we may have gained in relevance, we may have given up in the uncertainty around the estimate.

Super Bowl Squares

2009-01-24T14:11:00.001-08:00

It's that time of the year, when seemingly uninterested people run around feigning interest in what is now a true American celebration - the Super Bowl. This year's Super Bowl features the Pittsburgh Steelers and the Arizona Cardinals. Not that anyone outside of Phoenix and Pittsburgh actually cares who is in it. That however won't prevent Super Bowl parties from cropping up across the nation on Sunday, February 1st. And, at most of those parties, someone will usually run around trying to get the last few stragglers to participate in the second greatest tradition of the Super Bowl - the "Super Bowl Squares" (the first has to be watching the Super Bowl commercials).

Basically, for a small donation, you get to place your name in a 10x10 square grid. For multiple donations, you may be able to place your name on multiple squares. Before the actual game starts, #s from 0-9 are drawn randomly and placed across each of the 10 columns. The same process is repeated to the left of each row. Then the name of team 1 is drawn and placed at the top, and the second team is placed on the left. Now you have a grid that has each of the possible last-digits of the scores of each team. When the game ends, you look at the score, then look at the last digits of each team's score, and see whose name corresponds to that permutation and that individual wins a pre-determined amount of the accumulated donations. This process doesn't have to be limited to the game-ending score. Many variations exist. For example, frequently some smaller amounts could be won based on the digit permutations at the end of each quarter.

This year, as I almost always do, I participated in one of these Super Bowl Squares. I donated for 2 squares. After all the squares were filled out, the #s were randomly drawn, the team's assigned, the coordinator of the game handed me my sheet. I had drawn the following 2 permutations:

Arizona 2 - Pittsburgh 5

Arizona 8 - Pittsburgh 2

I promptly tossed the sheet in the recycling bin.

Later on, I decided to see for myself the likelihood of my winning.

The analysis below shows the aggregated game-ending digit permutations and combinations for every NFL game played in the Super Bowl era, including playoff games. That's 9,509 games! That also means that there's a reasonable likelihood that the probabilities shown are close to the true probabilities. As a matter of fact, every single permutation has been "hit" at least once. A game ending in the 2 2 permutation has only happened once - on Sunday, December 5th, 2004 the Buffalo Bills beat the Miami Dolphins in Miami 42-32.

First, a little math. I refer to both permutations and combinations. There is a difference between the two. A combination refers to a sequence or collection without regard to order. A permutation is a combination with a specific order. Here's an example. Take what we commonly (and mistakenly) refer to as a "combination" lock. We say to unlock the lock, "use combination 472". Well, that's actually only mildly helpful. Knowing those three #s alone we wouldn't be able to open the lock. We need to know the specific order of that combination of #s. In other words, is it 274, 247, 427, 472, 724, or 742. So in this example, there is one combination. There are six permutations.

Ok, now on to the tables and charts below.

In TABLE 1, I show all 100 permutations of game-ending scores. So for example, one can see that the likelihood of the game ending with the winning team's score ending in a 4, and the losing team's score ending in a 3 is 2.94% (to see this, in TABLE 1, go down to the row with the digit 4, then across to the column with the digit 3). So, this specific permutation has a 2.94% likelihood.

If you didn't care about whether it was the winning or losing team that had the 3 or the 4 in the last digit, as long as there was a 3 and a 4, then look to TABLE 3. TABLE 3 shows the probabilities of each combination. As such, the 3 4 or 4 3 combination has about a 3.67% likelihood of occurring. CHART 2 graphically illustrates what's in TABLE 3.

TABLE 2 is not meaningful in and of itself, but simply shows the probability of any given digit occurring (note that in this table, the percentages add up to greater than 100.00% since "any 7" will include for example, the "1 7" combination, that will also show up under "any 1"). CHART 1 simply illustrates what's in TABLE 2.

Let's take a look at what my chances are. It's a little complicated so bear with me. Remember, I have 2 specific permutations. Ari 2/Pit 5 and Ari 8/Pit 2. However, since I don't know ahead of time who will win the game, I need to average the 2 permutations that yield the 2 5 combination for the first scenario and the two that yield the 8 2 combination for the second. I can do this by either going to TABLE 1, and adding the respective likelihoods of each of those permutations, or I can simply go to TABLE 3. From TABLE 3 I can easily see that the 2 5 combination shows a likelihood of 0.36% (this is made up by the 5 2 permutation likelihood of 0.23% and adding it to the 2 5 permutation likelihood of 0.13%). Therefore, the average expectation for the specific 2 5 permutation is 0.18%*. For the second scenario, from TABLE 3, I can see that the likelihood of this combination is 0.28%, and hence the average expectation for the specific permutation is 0.14%.

* (Technically, I shouldn't be averaging the permutations expectations. What I should be doing is weighing each permutation by the likelihood of Arizona (or Pittsburgh) winning or losing the game. So for example, if the likelihood of Pittsburgh winning the game is estimated to be 70%, then a truer expectation for my specific 2 5 permutation might be 0.7*0.23% +0.3*0.13% = 0.20%. However, if you assume that each team's likelihood of winning the game is close to 50%, then averaging is fine).

So there you have it, the combined likelihood that I would win ANYTHING is about 0.32% (0.18%+0.14%).

Hence why I tossed the sheet. Hope you have better permutations! Good Luck!

AFC and NFC Championships - Predictions for Sunday

2009-01-17T21:13:00.000-08:00

Like I said last week, I can't stay away from building predictive models, so here it is - last week's model applied to this week's championship games - Philadelphia Eagles at Arizona Cardinals, and Baltimore Ravens at Pittsburgh Steelers. I have included other predictions - these are from 3 sites/blogs I frequently follow. I have a lot of respect for the work these guys do. Their models of course, are much more sophisticated than mine - but give me a year, and I'll catch up! Also, I show you what Vegas thinks, just for sake of comparison.

Now, don't get fooled. Even though we all independently seem to agree on predicting who will win (lone exception being advancednflstats prediction re: Eagles), this does not mean that these teams will win. There is a big difference between a predicted outcome and actual outcome. Outcomes in football games are very difficult to predict. There are few scoring opportunities for each team during a game, and each score has huge variation (generally 3 pts or 7 pts). There are numerous factors that are simply unpredictable, or, if they are predictable, have a very high degree of uncertainty around them. In any case, the predictions are below.

A Logical Look at What Happened in the Carolina vs Arizona Game

2009-01-12T22:56:00.000-08:00

The Carolina Panthers, played in the toughest division in the NFC, the NFC South. The Arizona Cardinals played in the easiest, the NFC West. The Panthers completed a 12-4 season. The Cardinals were 9-7. The Panthers were 8-0 at home. The Cardinals were 3-5 on the road. The oddsmakers in Vegas decided that this was good enough to favor the Panthers by 9.5 to 10.0 points.

Result - Arizona beats Carolina 33-13

What Happened?

Jake Delhomme threw a career-high 5 INTs. That was unexpected. A shocker. But, how unexpected was it, really?

Let's take a look at some #s.

First, what are the chances of winning a playoff game, when a QB throws X interceptions? Here are the stats:

0 INTs - 184-51, a winning percentage of 0.783

1 INT - 140-112, a winning percentage of 0.556

2 INTs - 54-117, a winning percentage of 0.314

3 INTs - 17-76, a winning percentage of 0.183

4 INTs - 1-27, a winning percentage of 0.036

5 INTs - 0-11, a winning percentage of 0.000

6 INTs - 0-3, a winning percentage of 0.000

These are based on all playoff games in the Super Bowl era. The data is courtesy of Cold Hard Football Facts.

As I have shown before, a QBs likelihood of throwing interceptions has been decreasing over time (based on the increasing value of that component in the QB Rating). So, if one were to look at this season's regular season stats, I'm sure this table would look even more skewed. (I don't have 2008 game by game data handy). The point is still the same - throw INTs, and you dramatically reduce the likelihood that your team wins. The fact that someone threw 5 picks is actually quite irrelevant. Once you throw 2 picks, you're likelihood of winning is dramatically reduced, and it's less than 1 in 5 if you throw 3.

Ok fine, so let's look at both QBs in this game Jake Delhomme and Kurt Warner. According to the table below, in Jake's career (including prior post season games), he had never thrown 5 INTs in a game. However, he had thrown at least 2 INTs in a game 22% of the time, and at least 3 INTs in a game 6% of the time. What's perhaps most interesting is that Kurt Warner had thrown at least 2 INTs in a game 28% of the time, and 3 INTs in a game 12% of the time.

So, what does this all mean? We need to look at the second table (see below the first table for the analysis).

According to this table, based on the likelihoods shown in the previous table there was an 18% chance that at least one of the QBs in this game was going to throw at least 3 INTs (see the last column in the table below). That's a about 1 in 5. That's much higher than I would have thought. That's actually a higher likelihood than both QBs finishing the game having thrown 0 INTs. Now admittedly, the likelihood that it was Delhomme and not Warner that was going to throw the 3 or more INTs was 2-to-1 against (given at least 1 QB would throw at least 3 INTs). But the fact is, there was a pretty good chance that this was going to be a game where at least 1 QB threw multiple INTs. Heck, there was about a 1 in 14 chance that at least 1 of them was going to throw 4 or more INTs - again, much higher likelihood than I would have thought (and you too, I'm guessing).

So there you have it. An unexpected result. Certainly. But not that unexpected. What should also be clear is that if you throw a pick, you're chances of winning reduce quite significantly. And for each subsequent one, the odds of winning decrease exponentially. After 3 picks, it's lights out.

NFL DIVISIONAL PLAYOFFS - Predicting the Winners

2009-01-10T11:28:00.003-08:00

My intention with this blog was not to get into the prediction business, but I can't help myself. It is what I am trained to do. I like building models, and of course, "predicting" is a natural outcome. Although my focus with this blog is quarterbacks specifically, I thought I'd go out on a limb and expose a new game predicting model that I'm working on. Admittedly, my model doesn't have the sophistication of the models built by guys like Brian Burke over at AdvancedNFLStats.com, or Aaron Schatz and his team at the Football Outsiders (see their AFC predictions here, and their NFC predictions here). That all being said, I think I will continue to work on it over time, and, why not, let's put it to the test to see how mine stacks up.

The table below shows my predicted outcomes, assuming each team behaves 'as expected'. Of course, the football isn't round, and funny things happen when the ball bounces. The weather is another element that is not incorporated explicitly into the model.

Arizona at Carolina - All three of us agree that this is the 'easiest' game to predict - with Carolina winning handily. Vegas oddsmakers say a spread of 9.5/10 with the over/under at 48.5/49.0. My model concurs.

Baltimore at Tennessee - Should be a close game. FO calls it for Baltimore, and ANS calls it for Tennessee. My model suggests Tennessee in a close game. Vegas oddsmakers say a spread of 3.0, with the over/under at 34.0/35.0. I agree with the spread, but my model suggests more points (perhaps I need more work on my model)!

Philadelphia at NY Giants - Awfully close game - closest of all the games this weekend - FO on the one hand says Philly, but then he hedges and says its going to be really, really close (a pick 'em). ANS is a little more bullish on the Giants (he also hedges his bet and says, it's a 50-50 outcome if analyzed based on the whole season). My model agrees with both of these models/predictors, and says Philly in a squeaker. Vegas says G-men by 4, with the over/under at 40.0. My model says closer game than that, and a few more points (remember, Vegas needs to take 'perception' into account, as all they care about is getting half the money on one side and half on the other).

San Diego at Pittsburgh - FO says Pittsburgh, and so does ANS. I agree, with my model suggesting Pittsburgh buy more than a field goal. The oddsmakers in Vegas suggest a 5.5/6.0 point game, with the over/under at 37.5/38.0. I think it's a bit closer, but also a few more points.

This is the first public exposure of my model. I'll see how it does this weekend, and continue to build/improve it. Hopefully, at some point in the 2009 season, I'll be comfortable enough to share more details as to the inputs. Suffice it to say that I feel comfortable enough to expose it today.

Let's play the games!

We can evaluate how we did on Monday.

Stat of the Week - #1

2009-01-06T20:22:00.001-08:00

I'm introducing a new feature - Stat of the Week - little or no commentary - just a stat - mostly because I'm not sure I'll be able to post a blog every week because a) I love my day job, and that takes a lot of my time, and b) posting, especially an interesting, well-researched post, takes time, and I'm just not sure I have the time to do a detailed post every week. So enjoy....