Showing posts with label New Passer Rating. Show all posts
Showing posts with label New Passer Rating. Show all posts

Thursday, April 30, 2009

Better Than Jay Cutler


A lot of noise this past month about Jay Cutler. First, the Denver Broncos' new head coach, former New England Patriots offensive coordinator Josh McDaniels, apparently tried to work a trade to get Matt Cassel from the New England Patriots. This apparently didn't it well with the Pro-Bowl quarterback. After a lot of posturing from both sides, Cutler was eventually traded to the Chicago Bears for Kyle Orton and draft picks. This post isn't about who was right or wrong in the fracas, or which team got the better end of the deal in the trade. This post is about answering the following two questions:
1) How good is Jay Cutler?
2) Are there quarterbacks with similar experience that are better than Jay Cutler, and, if so, who are they?

The table below ranks each of the 30 quarterbacks that threw enough passes last year to qualify to be ranked. The ranking is based on CMI, and not the NFL's quarterback rating system. In numerous posts prior to this, I have discussed why CMI is a better measure of a quarterback's passing performance than the NFL passer rating system.



Just looking at last year, the year that Jay Cutler got named to his first Pro-Bowl, he was, by this measure, a below-average quarterback. His CMI of 0.536 was (0.14) standard deviations below the mean. It should be noted that even though he was below average relative to all quarterbacks in 2008, it wasn't unexpected. Previously, I discussed the notion of performance improving with experience. So, if you were to look only at quarterbacks and their second year performance, their average is (0.21) standard deviations relative to the mean. Relative to that measure, he actually did better than the typical second year quarterback.

Let's take a closer look at the table above. The last 2 columns indicate the # of years the quarterback has played since he first qualified, and the total # of years he has qualified. So, Cutler has played 2 seasons where he threw enough passes to qualify, and this is second consecutive year of being qualified. There are six other quarterbacks in this table who have similar experience.

They are (rank in parenthesis):

Jason Campbell - WAS (5)
Matt Schaub - HOU (7)
Trent Edwards - BUF (8)
David Garrard - JAX (12)
Jay Cutler - DEN (16)
Derek Anderson - CLE (30)

Of the six quarterbacks who were in their second year (Garrard has the most experience of all the quarterbacks, even though he has only qualified in two years), Cutler only outperformed Derek Anderson of Cleveland in 2008. Ok, so 1 year does not make a quarterback. Let's take a look at these six quarterbacks and see how they've done in each of their first two (qualifying) years.


According to this table, both Garrard and Schaub have outperformed Cutler in each of 2007 and 2008. And, arguably, over the two-year period, so did Campbell. The only quarterback in this class that has clearly underperformed the rest of the group is Derek Anderson. He will not last long in the NFL.

Admittedly, it's difficult to judge a quarterback after his first two years. But, so far at least, there appear to be several quarterbacks that have outperformed Cutler. It will be interesting to watch this group over the next several years, and especially Cutler, as he will be with a new team.

Saturday, February 14, 2009

Brett Favre - Best Ever?

Brett Favre retired earlier this week.

He played in the NFL for 18 years. He set numerous NFL records during the course of his career. The most impressive has to be the fact that, as a quarterback, he started a record 269 consecutive regular season games. Peyton Manning is second on the list, his streak currently standing at 176 games. Third on the list is Ron Jaworski, his streak having ended at 116 games. Tom Brady is next on the list, but his streak ended in the first game of last season at 110 games. Peyton Manning needs another 94 games - he would have to start every single game through the 14th game of the 2014 season to break Favre's record. In addition to the four mentioned above, only two other quarterbacks in history - Joe Ferguson (107 games), and Dan Marino (95 games) have started that many games consecutively. I suspect Favre's record will never be broken.



People are fascinated with Favre. Many consider him to be among the all-time greatest to ever play the game. Some consider him the greatest ever. I do not belong to either group.


The media loved Favre. And vice versa. He never met a camera he didn't like, always seeking an opportunity to get into the limelight (of course, the media loved that). He also seemed intensely competitive, and the combination is what I believe led him to his unfortunate decision to come back for one more season in 2008.


Let's start from the beginning.


In 1991, the Atlanta Falcons took Favre in the second round of the draft. With Chris Miller as their starting QB, in 1992, the Falcons traded Favre to Green Bay. Favre started the season as the back-up to the 'Magic Man', Don Majkowski. In the third game of the season, against the Cincinnati Bengals, he replaced the injured Majkowski in the first quarter. With 13 seconds left, he threw a 35-yard touchdown that helped the Packers come-from-behind to win 24-23. He started the next game against the Steelers, a game the Packers won. The rest, as they say, is history, as he started every game for the Packers until his trade to the Jets following the 2007 season (for a detailed history of, or, more aptly, a tribute to Favre, see here).


The table below shows how Favre did each year.



Before we go on, some explanations.


In order to be able to truly compare Brett Favre to other quarterbacks so that we can properly gauge his place in history, we can't simply look at the NFL's passer rating system. First, we know that the average NFL passer rating in the NFL has been increasing over time. So, by simply using the NFL's passer rating system, the more recent quarterbacks would appear to have done better than quarterbacks from earlier years (see here and here for evidence). Second, we have to agree that the NFL's passer rating system is an accurate, if not adequate measure of a quarterback's performance. I have previously introduced a metric, CMI, which I believe is a better, if not more understandable measure of performance. CMI, however, also has the same bias as the passer rating system. One way to adjust for, and remove the bias favoring more recent quarterbacks is to look at each year's performance for each quarterback, and relate that to the average performance that year. The measure that we use is the standard deviation. I have discussed previously how using this measure tends to "normalize" performances across years. In other words, when a given year's performance by a particular quarterback is "normalized" to the average for that year, and we do that for every year, we can very easily see how that particular performance compares to other performances across years. Since CMI is very similar to the passer rating system, when using standard deviations, it shouldn't really matter whether we use CMI or the NFL passer rating system (there will be differences, and, I often include both so that you can see the differences). The two graphs below show how using either the "normalized" NFL's passer rating system or "normalized" CMI as measures compare to a standard normal curve with 1,451 observations (that's how many qualified quarterback seasons there have been over the past 69 years).



Ok, so back to the table.


Favre had a "pretty good", or, "above average" first year in 1992. Using the NFL passer rating, he finished 0.57 standard deviations above the average that year. Using a standard normal curve, that would mean between the 71st and 72nd percentiles. If we used CMI as the standard, then his performance that year, 1.24 standard deviations above the mean, would place him between the 89th and 90th percentiles.


He followed that with a sub-par year in 1993, finishing between the 30th and 31st percentiles according to the NFL passer rating system, or between the 35th and 36th percentiles using CMI. Then followed several "above average" to "very good" years (1994-1998), depending on your perspective, followed by a couple of mediocre years (1999-2000), followed by four above average years (2001-2004), followed by his two worst years (2005-2006). He finished his Green Bay tenure with an "above average" year in 2007.


He should have taken the opportunity to retire then.


His last year, his first full season not with Green Bay, was "below average". So he finished his career with 3 "below average" years out of 4. Perhaps the Packers brain trust was on to something when they decided that Aaron Rodgers was their future at the end of the 2007 season.


The two graphical illustrations below show each of Favre's seasons in terms of standard deviations from the mean. In case you haven't noticed, as much as I love statistics, I find that graphs are better way to illustrate a point than a table of statistical measures. I show the CMI graph first and the graph based on NFL passer rating next, for reasons that will become plainly apparent when you read on.





Ok, so until now, we've been looking at Brett Favre's career in a bit of a vacuum. Yes, we have compared his performances to an average, and yes, we have tried to put it into historical perspective, but I haven't given you any 'perspective'.


First, let's compare Brett Favre's career to some individual careers. The biggest problem we face in comparing Favre's career to any other individual career, is obviously the length of his career. There have been only 5 individuals who have thrown enough passes to qualify in at least 15 different seasons. Those are Fran Tarkenton (18 years), Brett Favre (17 years), Dan Marino (16 years), John Elway (16 years), and Johnny Unitas (15 years). So how does Favre's "picture" compare to these quarterbacks, all of whom are in the Hall of Fame?


Fran Tarkenton - a picture of "excellence". With a couple of exceptions, Tarkenton exceeded his peers consistently for a very long time. Quite remarkable. Without question, Favre's picture does not resemble that of Tarkenton.


Dan Marino - Marino was "above average" for many of his years, tailing off towards the end of his career. He was "very good" very early in his career. He really didn't have a "bad" year until his last, and wisely decided to hang it up after that. Although closer in resemblance, Marino's picture still looks better Favre's overall.


John Elway - Elway underperformed his peers much of his early career, exceeding the average in only two of his first ten years. He then was "above average", or even "very good" his last six years. He went out on top, having retired following the Broncos having won back-to-back Super Bowls (Super Bowl XXXII and Super Bowl XXXIII). I think it's safe to suggest that Favre's picture looks better than Elway's.


Johnny Unitas - In my opinion, Favre's picture most closely resembles that of Johnny Unitas. They both started out well, had a couple of mediocre years, then had several years where they outperformed their peers, and ended their careers with a few sub-par years.



What if we compared Favre's average performance (in other words, average each of his year's performance in terms of standard deviations), to other players? Well, over his career, in terms of NFL passer rating, Favre averaged 0.42 standard deviations above the mean (0.26 when looking at CMI). Tarkenton averaged 0.65, Marino averaged 0.63, Elway averaged 0.06, and Unitas averaged 0.51 standard deviations above the mean.


Let's expand the comparison list to players who qualified in at least 10 different seasons. 41 different quarterbacks make that list. The two graphs below illustrate where Favre ranks on those lists.



The same graph as above, except looking at it Using CMI as the measure of performance.




Looking at his career statistically, it is hard to make the argument that he belongs in the truly elite group of all-time quarterbacks - those that would be ranked in the top 3 or top 5 all time. I think it is fair to suggest that Favre has had a very good career. It was certainly a lengthy one, and, in that regard, it was exceptional. But as far as his actual performance on the field, there isn't a measure, or set of measures, that suggest that it was exceptional.


People will suggest that he holds the all-time record for yards, or touchdowns, for example. But that is merely a function of his longevity, just as much as he holds the record for interceptions thrown, all-time.


I also don't discuss measures frequently used as evidence of greatness. MVP awards, for example - these are popularity contests more than a reflection of outstanding statistics or an individual's value to a team (the NFL MVP award is given by the Associated Press). Pro-bowl selections for example are another popularity contest - just look at who got selected this past year in the AFC. Another measure frequently used is Super Bowls. Dan Marino didn't win any Super Bowls. Trent Dilfer won a Super Bowl. Winning percentage is also frequently used - last time I looked, a quarterback doesn't play defense, return kicks or punts, or kick field goals. These pieces of evidence would never enter any serious discussion in terms of statistical evaluation.


My eventual goal is to identify the greatest passer of all time. I will use several statistical measures, many of which are mentioned in this post, to identify the most worthy candidates for consideration. As far as my research is concerned so far, Favre would not make that list. There are several other quarterbacks, not mentioned in this post, nor even displayed in either of the last two graphs that would make more worthy candidates, but that is for a different article.

Sunday, December 7, 2008

A Slight (but necessary) Digression

Brian Burke, who does some excellent analytical work at advancednflstats.com has made this observation regarding three first-year quarterbacks - Joe Flacco of Baltimore, Matt Ryan of Atlanta, and Matt Cassel of New England.  He suggests that their performance has been improving as the 2008 season has moved on. and shows their performance, graphically illustrating it in the following manner.  He uses a measure, "Adjusted YPA" or, Adjusted Yards per Attempt, defined as [Yards -40*Interceptions + 10*Touchdowns]/Attempt, and then looks at their 4-week moving average to compare the three quarterbacks.

Well, I decided to look myself.  I have been "casually observing" that Flacco and Ryan have been having some "pretty decent" games recently (Ryan is on my Fantasy team).  So, I decided to put my newly developed CMI to the test.

Here's what it shows:
It appears to be very consistent with what Brian found.  Matt Ryan has improved the most.  CMI shows that Matt Cassel has in fact had the least improvement, if any.  However, to his credit, he has been performing at a very high level from the beginning.  Looking at the table below, you see that both Ryan and Flacco have remarkably similar attempts and completions through the first 12 weeks, with their only difference being the # of interceptions thrown.  Also, you can see that Cassel attempts more than 5 passes per game more than the other two (partially reflecting the fact that New England has no running game).

Brian has, in the past, ventured into the "creating a new passer rating" space, and, in fact, created one.  He uses a concept called "Air Yards", defined as, [Yards - Yards After Catch] to relate "passer rating" to a team's wins and losses.  See his post here.  I agree with his approach generally.  In other words, if you look at his formula, it's a lot like CMI.

QB Wins Added = [(Air Yards - Sack Yards) * 1.56 - INTs * 50.5]/Pass Attempts - 3

where, (Air Yards - Sack Yards) is substituted for Completions.  In other words, he also takes out Touchdowns thrown and Yards per Attempt.  Brian is the first person I know of that does not include touchdowns as a part of a passer rating calculation, and he is to be commended for that.  My only real issue with Brian's calculation is that YAC, or, Yards After Catch, is not a readily available statistic, and hence, "Air Yards" is not easily calculated.

I am still developing CMI, and the next (enhanced) version will incorporate sacks.  Given that one of my goals is also to look back in time and be able to compare QBs over time, I will only be able to that back to 1969.

As I mentioned at the top, Brian is a very smart guy, and has done some very good work on a whole host of topics relating to the NFL, with particular attention to statistical rigor.  I will be referring to his site quite a bit in future postings.

Friday, December 5, 2008

Simplifying the QB Rating System

This graph illustrates a comparison between the QB Rating and only the components for Completions and Interceptions.  The components for Yards per Attempt and Touchdowns are ignored.  The data shows a correlation of 0.970.  Interestingly, the Yards per Attempt and Touchdowns components, when added together, have a slight negative (-0.17) correlation with the QB Rating calculation.  In a previous post, I had shown that correlations of just the Completions component with the QB Rating was 0.956, and the Interceptions component was 0.965, respectively.  What this shows, at least to me anyway, is that we could simplify the QB Rating to only include 2 components - Completion Rate and Interception Rate.

The QB Rating formula is as follows:

Q = (J+K+L+M)*100/6

where,
Q = QB Rating
J = max(min(C, 2.375), 0)
K = max(min(Y, 2.375), 0)
L = max(min(T, 2.375), 0)
M = max(min(I, 2.375), 0)

and where,
C = ([Completions/Attempts]*100 - 30)/20
Y = ([Yards/Attempts] - 3)/4
T = [Touchdowns/Attempts]*20
I = (2.375 - [Interceptions/Attempts]*25)

Here's how we will go about simplifying the QB Rating calculation:

As discussed above, we can eliminate the K and L terms, since the J and M terms have an almost perfect positive correlation with the QB Rating.

We get, 

Q ≈ k*(J + M)*100/6

where,
k = non-zero scalar

Let's further simplify the equation by doing the following:

Let's remove the restrictions on a max and a min for each term (in my opinion, these restrictions are arbitrary, and when looking at annual league average data, the restrictions haven't come into play since 1943).

And, by removing the approximation and by substituting back for the J and M terms, we get

Q*100/6 = ([Completions/Attempts]*100 - 30)/20 + 2.375 - [Interceptions/Attempts]*25

Re-arranging terms, we get

(Q*100/6 - 2.375 + 1.5)/5 = [Completions/Attempts] - [Interceptions/Attempts]*5

or,

Q(new) = C/A - b*I/A

where,

Q(new) = New QB Rating
C/A = Completions/Attempts
I/A = Interceptions/Attempts
b = factor to be determined (in my preliminary research, it appears that b would be a number close to 3)

So, there we have it, a (much) simplified new QB Rating System.  It only looks at completion rate and interception rate, and no fancy mathematical gyrations.  It is easy to understand, easy to explain, not arbitrary, simple to calculate and it also has the amazing beauty of never exceeding 100.0%!  It still does have the issue of not being comparable over time (i.e. players in the 1940's versus players from the 1990's).  I will address this issue in an upcoming post.

For simplicity, I'd like to call this new rating system CMI, or, short for Completions Minus Interceptions (sort of like the OPS statistic in baseball - on-base plus slugging).