Does Experience Improve Performance?

In my earlier post, "First Year Performance - An Evaluation Method", I discussed how, over the history of the NFL, from 1940-2008, a quarterback's passing performance, as measured using CMI, and more specifically, the standard deviation from the mean of CMI, changes over time (in that post, see in particular the third graphical illustration labeled "CMI - SDs from Mean - By Qualifying Year").  That illustration pointed out that it takes about 5 years for a quarterback to become "average" (or, better than average).  That's obviously only true for quarterbacks who qualify in at least that many years.  Meaning that since the graphical illustration includes all quarterbacks (who have thrown enough passes to qualify in at least 1 year), that the earlier years could be biased by quarterbacks who only qualify in  those earlier years, and not the later ones.  

This is the issue pointed out by a commenter regarding that post.  He (she) was suggesting that I have a biased sample, in that the later years are only populated by quarterbacks who throw for that many years, and hence, performance should be expected to improve, as the NFL weeds out the under-performers of the earlier years.

Here's the comment:

"This effect could be due to sampling bias.  It may be that there are a large pool of first year QBs.  Some are good and some are bad.  The good ones make it to year four, and the bad ones are cut, released, or benched before they can play four years...."

He (she) then went on to ask the following question:

"...if you restrict the analysis to only QBs that have played >= 4 years, do you get the same result?"

Before I answer the question, let's address the comment.  

First, the commenter does make a very valid (and what would seem to be an obvious) point, in that those QBs that make it to year 5 and beyond, must necessarily be the better ones, as the under-performers drop-off.  This point has several implications though.  The first is that the NFL does not look at quarterbacks (or, more accurately, their passing performance) the way I do.  So we can't simply assume that the point is in fact that obvious.  Second, perhaps the way I look at quarterbacks passing performance, CMI, is in fact, a decent (good?) measure of a their true passing ability, and, the graph validates that fact (this would be rather presumptuous of course, but that is what I am trying to prove after all - that I have, in fact, developed a method that appropriately evaluates a QBs passing performance, and, perhaps even more importantly, that I have developed a method by which one can identify a good passer versus a poor passer early on).

Second, the commenter suggests that my sample is biased because of this fact (i.e. it is biased towards the better players in the later years).  There are two ways we can look at that.  The first is that the sample isn't biased because it uses the entire population.  In other words it is what it is (and perhaps this again validates my measurement technique).  The second point is that the sample is biased (remember, however, the context in which the graphical illustration was shown - in that it was trying to look at all quarterbacks, and how their performances changed over time), and that we can, if we choose to do so, look at it differently.  This is, in fact, what the commenter was suggesting with their question.

So let's take a look.  As a matter of fact, let's take the question a bit further and look at it in four different groupings:  

QBs qualifying in at least 1 year (the original graph) - 345 Quarterbacks

QBs qualifying in at least 3 or more years - 180 Quarterbacks

QBs qualifying in at least 5 or more years - 121 Quarterbacks

QBs qualifying in at least 7 or more years - 84 Quarterbacks

The graphical illustration is followed by a tabular display of the same data.

There's no question about it.  Performance improves as a quarterback gains experience.  All groupings follow the same basic pattern:  Sub-par performance in year 1, followed by several years of gradually improving performance.  When the improvement takes place, and by how much, does vary by group.  For example, after three years, teams still haven't been able to markedly separate the good quarterbacks from the bad ones.  By the fifth year, it's now quite apparent who has "cut it", and who hasn't.  For all quarterbacks qualifying in at least 7 or more years, they appear to show their skill, or differentiate themselves, as early as year two.  So, to prove the obvious point that the commenter in the previous post making, those that don't turn in positive performances early do tend to get weeded out.  The process, however, happens to take a few years.

The question this prompts however, is, why does it take so long to weed out the poor performers?  

Well, there are at least two different answers.  

The first, something I alluded to earlier, is that the NFL doesn't evaluate passers the way I do.  I am a statistical analyst, and what I have created is a an objective methodology to evaluate a player's performance.  Teams in the NFL, aren't inclined to do the type of statistical analysis that I do.  In addition, it's just simply not clear, even using my statistical methods, that you can be sure that you can identify the good and bad future quarterbacks after year 1.  By year 2, however, year 3 the latest, my method does show who the keepers should be.

The second is, that teams pay a lot of supposedly smart people (scouts, GMs, etc) a lot of money to evaluate, and draft good talent.  Subsequently, these teams draft these "talented" players early in the draft, and pay them lofty sums of money, a lot of which is guaranteed.  Now what do you do?  You've got a poor-performing team (generally, a team's draft order is determined by their won-loss record, with the teams with poorer records drafting earlier, and vice-versa) that just drafted the next "Peyton Manning", and have paid them a lot of money, and they have a lousy first year.  You're not likely to blame the young quarterback for that year's performance.  The team still has a lot of weaknesses elsewhere, so you pick early again the next year (perhaps shoring up the defense).  Second year passing performance is better than the first, but still not better than average.  You even ignore the statistical evidence (he can't be that bad).  You're still paying him lots of money, you've seen improvement, team has won perhaps a couple of more games, and this gives you hope that, the third year is THE YEAR.  Then, much to your dismay, the quarterback that you paid millions of dollars for, doesn't pan out as you have another miserable year (the teams is "close" to 0.500).  You get impatient,  fire the head coach, try an aging veteran as your quarterback, he has a decent year, a quarterback controversy brews, ya da ya da ya da....and, you continue to ignore the statistical evidence....

Just off the top of my head, I can think of several recent obvious examples - Alex Smith of the Forty Niners (and, proving the corollary, the approach taken by the Green Bay Packers with Aaron Rodgers), and JaMarcus Russell of the Oakland Raiders.

Here are three quarterbacks that the NFL, teams, and fans, have fallen in love with, but, statistically have yet to prove themselves.

Eli Manning (I can hear the argument - but they won the Superbowl)

Tony Romo

Michael Vick

So there you have it.  My method is the Moneyball approach.  What actually happens in the NFL is quite different.  NFL teams could save themselves a lot of money (and perhaps even win a few more games) if they were to adopt a statistical approach to their business (I think a lot of NFL teams actually believe they take a statistical approach, but I don't think that most actually go through rigorous analyses the way I do).  

My next post will be my preseason rankings for 2009.