One of my favorite Bill James essays is one that he did on Boston Red Sox second baseman Marty Barrett in his 1985 Baseball Abstract. He spoke about a time when he was scouting Barrett and a situation arose in which the pitcher was almost certain to throw an inside pitch. James said that after Barrett smashed the expected inside pitch out of the park, he was really impressed because it showed that Barrett had the ability to handle that type of pitch. At the same time, he also knew this sentiment was a bit inaccurate because he had predetermined that this was the play he was going to judge Barrett on. James ended the essay by noting, "this is a good example of the kind of gut-level decision making that sabermetrics intends to render obsolete."
The same type of gut-level decision making can happen in tape grading NFL players if one isn't careful. Nothing illustrates this better than a recent six-game breakdown I did on Alabama linebacker Rolando McClain (at Kentucky, at Mississippi, vs. South Carolina, vs. Tennessee, vs. LSU, at Mississippi State).