Just how important is "clutch", really?

I didn't say weather is the explanation, just that it may be one that should be taken into account. Your research confirms that -- nice find on those late season stats. But I wouldn't say it is a negligible effect compared to the purported gap between regular season and playoffs. First, as I pointed out, that HOF QB data was wrong in some respects and omitted other guys. Some of the ones on that list distort those numbers based on small sample size and weather conditions. YA Tittle, for example, had 5 career playoff games, two of which were played in horrific weather conditions (sub-10 degrees wind chill, 60%+ relative humidity, etc,.) and he had horrific numbers in those two games. George Blanda had 4 career playoff starts. When you take out the outliers (20+ changes in 5 or fewer games dominated by 1-2 bad performances) and add in the guys I pointed out were missing from he list, the average drop in rating for that group is 3.9 pts. Your late season weather data suggests that 2+ of those pts could well be due to weather. Since weather is generally worse in January than December, the effect in the playoffs could very well be greater. And yes, strength of competition is likely a factor too. Teams and defenses are generally better in the playoffs and could very easily explain a 5-10 pt (or even more) drop in rating, which really isn't all that much.

The examples of other stats you say can't be explained aren't very compelling IMO. Tom Brady's 4th quarter rating is the lowest of any quarter for him. And his rating when trailing with less than 4 mins to go is 83.7. The 4th quarter is also Joe Montana's lowest rating quarter. I can't find meaningful trailing with less than 4 mins stats for him -- it only shows 3 pass attempts. Joe Flacco's career 4th quarter rating is slightly higher than his overall rating, but if you include OT then it is lower. And his trailing with less than 4 minutes numbers are worse than his overall numbers. So what exactly is the data you are referring to when you imply those stats prove the existence of clutch?

 

It's for all. It's one of the issues that occurs when you collapse data across different conditions.

You can go back to the debate I had with Fineas earlier in this thread about how to calculate drop-off in HoF QB vs. non-HoF QB performance in close vs. non-close games (concluded on page 5 with post #92) to see the same issue pop up. He was arguing overall win % in either condition must be 50% because for every win you have a loss, while I was arguing that's true if you collapse across condition, but not true if you calculate win % on a QB-by-QB basis because the smaller denominators will skew the stats, meaning average win % for non-HoF QB's in close games will be far lower than 50%.

Which approach is correct depends on the precise question asked. Ask about average performance in either condition and you should collapse. Ask about average drop-off in performance for any randomly chosen QB from one condition to the other and you shouldn't collapse, and that's really the question we were asking back then and also here.

Fast forward to this debate and there's another (obvious) problem you and jdang307 bring up in that collapsing across condition often has the property of including QB's that are not represented in one or the other condition. Certainly true with "regular season" vs. "playoff", but also true with many non-HoF QB's playing "non-close" games but no "close" games or vice versa (because there are so many non-HoF QB's that played only a few games).

So the only way Fineas can be consistent across both debates (because he stated in post #90 that he doesn't find any real value to asking the "drop-off per QB" question for which collapsing isn't the right math) is IF his baseline in this debate is to collapse across condition, irrespective of whether a QB was represented in both conditions. And yes, if you do that what he said is correct, but I think it's pretty clear that shouldn't be the calculation. Either way, that calculation was useful because showing both conditions having similar passer rating means playoff QB's performed on average worse in the playoffs using the drop-off per QB method (without having to actually do the calculation).

 

Go back to comparing an individual QB's drop-off vs. average drop-off from one condition to another, or better yet where that lies in the distribution of drop-off's across the league. If the drop-off is much less than average (even better if you have the distribution, then you can calculate the probability that QB was "average" in his drop-off performance), then you have evidence that QB is better at something than the others given changing conditions.

Brady's 4th quarter trailing stats with <4 minutes is 83.7 yes, but that compares to what? 85.1 with less than <4 minutes while either tied or ahead (collapsing that data). NFL average in 2015 for <4 minutes trailing? 78.5. NFL average in 2015 for <4 minutes either tied or ahead? 91.6.

So NFL average drop-off is 13.1 passer rating points, while Brady's is 1.4!! I don't have the full distribution on that so I can't calculate a probability that was due to chance, but clearly Brady is "clutch" based on that one stat (again, one needs to see a pattern across all stats, but this is one), and I don't think calculating it across all NFL years instead of just using 2015 numbers will change the conclusion much.

Now.. try explaining that away with weather, strength of competition, time of year, etc..

 

It's not clutch that Brady is better in the 4th than the NFL average. It's that he's better overall. I expect Brady to be better in than the average QB, because he is better. His drop should be less, because he is a better QB.

 

The "drop" is the effect of external influences (otherwise why would there be a drop). If that external influence let's say is "increased pressure", then yes Brady has less of a drop because he responds better to pressure.

That's all we're saying. He responds better to pressure because he's a better QB. He's more "clutch" than other QB's.

 

So Brady's rating drops from 96.4 overall for his career, to 83.7 trailing in the 4th with less than 4 minutes. That is a 12.7 pt drop. According to your numbers, the average NFL QB (whose cumulative overall rating in 2015 was 88.4) dropped to 78.5 when trailing with less than 4 minutes left. That's a drop of 9.9 pts. So Brady had a bigger dropoff when trailing with less then 4 minutes to go than the average QB. But again, it is very close and not a significant difference.

Here, you are comparing the wrong numbers. Whether winning or losing at the end of games, there is presumably still "pressure." As it turns out from those numbers, Brady is not especially good in those late game situations whether winning or losing. But he's not as good in those late game situations as he is in non-late game situations. Not sure how/why you feel that shows he is clutch. He plays better in the first 56 minutes than in the last 4 minutes and his dropoff in that regard is bigger than the average NFL QB.

 

But he doesn't have less of a drop he has more of a drop. His drop from first 56 minutes to last 4 minutes is bigger than the average QB. It doesn't suggest he responds better to pressure. Indeed, if "pressure" is the only variable associated with the last 4 minutes, he performs worse and is less clutch. It's a very, very small difference, however and in terms of % decrease it is about the same as the average NFL QB. And, of course, "pressure" is not the only variable associated with last 4 minutes. Size of lead, strength of running game, conservativeness of play calling, use of no huddle, prevalence of prevent defenses, etc. are all potential other variables.

 

Yeah drop-off percentage-wise for league in 2015 from whole-game passer rating to last 4 minutes trailing conditions is 11% and for Brady across his career is 13%. So you're right there that Brady is around average. There are other stats like that. Just look at his drop-off in 4th quarter performance relative to previous quarters and it's also about the same as league average.

You'll find that for EVERY QB. That is, there is no QB for which every single "clutch" stat shows they are well above average. What's different with "clutch" QB's is that you find multiple "clutch" stats where they are way above average while you'll find none or almost none where they are way below average.

In Brady's case, his <4 minute trailing vs. <4 minute non-trailing stats is definitely one such example. You say that's not the right comparison, but if anything that's a better comparison than to overall rating because it conditions on whatever the effect <4 minutes left in the game has on a QB. Another "clutch" stat with Brady is that he is ahead 50% of the time at the end of the 3rd quarter in "close" games (games that END with 0-7 points difference) yet he wins 69% of the time. Sure, the rest of the team and coach have something to do with that, but obviously the QB is an important component too.

Here's the challenge. Find as many "clutch" stats for Brady where he is way below average as he is way above. You'll find Brady is generally average in drop-off or well above average, which fits with the eye test that he's the type of QB you can count on much more than the average QB to win the game when (extra) pressure is on the QB to deliver.

 

Again, because if those other QBs were better, they wouldn't be average. "Clutch" doesn't magically manifest at the end of games. Brady is a well above average QB at all times, so he's going to be above average in supposed "clutch" situations to.

 

Two things: 1) not all above average QB's are well above average in clutch situations (e.g. Peyton Manning), and 2) just because a QB is above average at all times doesn't mean he's not clutch. He is based on how we're defining this.

 

Clutch isn't real. If you're above average, you're going to probably be above average in high pressure situations. Manning got a bad rap for playoff performances, but he certainly had regular season "clutch" success.

 

Forget playoffs.. let's do the same comparisons for Peyton Manning in the regular season:

Career <4 minute trailing rating vs. <4 minute tied/ahead rating: 75.6 vs. 103.2!!

That's a drop-off of 27.6 points vs. 13.1 points for league in 2015. Percent comparisons are more relevant but it's still 26.7% for Peyton vs. 14.3% for the league, and yes this is regular season.

Even Peyton's 4th quarter rating drop-off vs. other quarters is twice league average. For the league you generally get around 3-4 passer rating points drop-off, while Peyton is 8 points drop off. That's also regular season.

Of course, not every clutch stat with Peyton is bad. He has great 4th quarter comeback numbers. But to think that any above average QB must in general be above average in "clutch" situations is wrong. Yes, clutch is real (probably )

 

Obviously they're not ALL going to be great numbers. These guys are still human.

What I'm getting at, is great QBs are generally going to be better than average or below average QBs, especially in pressure situations.

Because they're better.

 

Yeah, but the causality is usually: BECAUSE they are better than average in pressure situations they are better than average QB's. That is, it's because clutch exists (by definition of being better than average in response to pressure) that a QB is more likely to be good overall.

You're somehow trying to argue the measure (of clutch) doesn't exist because it's correlated with other measures. The measure exists independent of the other measures resnor.

 

If that were true, people would take Flacco over Manning.

 

"more likely" is the key phrase. There are many factors you'd consider in choosing a QB (and yes I'd choose Peyton in his prime.. not now obviously), and how he responds to pressure is one of the factors.

 

"More likely" seems like an awful imprecise phrase to wrap up all the "precise" work done up until this point...no?

 

Haha. Yeah, but all I'm doing is making a derivation from what resnor assumed, which in stats translates to saying different measures of what makes a QB better correlate with each other to a good degree (obviously not 100%). To quantify it, I'd need the data that shows the actual correlations.

Either way, IF we assume what resnor said is correct, then "more likely" is imprecise but accurate, which will suffice for now.

 

No...generally, as in, not in EVERY situation, but in most. These guys are human, so, you may have a HoF caliber QB play poorly in a pressure situation, but an average QB on the other team plays well. Statistically, that would be unlikely to happen frequently. However, generally, we would expect the better QB to play better, no matter the situation. Doesn't matter if it's first play of first quarter, or last play of the fourth quarter.

Good or great QBs exhibit those qualities, regardless of situation. Poor QBs exhibit poor qualities, regardless of situation. Again, generally speaking, as these guys can have some anomalies in either direction.

 

I wasn't disagreeing. Now, knowing I wasn't disagreeing, please read the post you quoted again, especially the 2nd paragraph.

 

I still don't understand why you are comparing numbers when they are leading with less than 4 minutes to when trailing with less than 4 minutes. They are both late game, arguably pressure situations when the game is close. When the game isn't close, neither is a pressure situation. A QB with a 1 pt lead and 4 minutes left has much more pressure on him than one who is down by 35 with less than 4 minutes left. In the same vein, a QB with a 35 pt lead in the last 4 minutes has virtually no pressure on him.

You brought up the late game situations, and for the reason mentioned above, I don't agree that is a strong measure of clutch. But to the extent it is, the more relevant comparison would be overall rating (average, normal pressure) with late game pressure (last 4 minutes). And better than that would be to look at numbers when tied with 4 minutes left. At least there we know it is a close game and mistakes likely cost you the game and big plays likely win you the game. So let's see how a few examples of supposedly clutch QBs' tied with 4 minutes or less ratings compare to their overall rating.

Tom Brady -- 96.4 overall; 68.3 when tied with less than 4 mins; -28.1

Aaron Rodgers -- 104.1 overall; 91.3 when tied with less than 4 mins; -12.8

Peyton Manning -- 96.5 overall; 93.7 when tied with less than 4 mins; -2.8


Ryan Tannehill -- 85.2 overall; 97.5 when tied with less than 4 mins; +12.3

Jay Fiedler -- 77.1 overall; 134.8 when tied with less than 4 mins; +57.7

Clearly, when faced with obvious pressure situations like a tied game with less than 4 minutes left, Tannehill and Fiedler are more clutch than Brady, Rodgers or Manning. While Fiedler and Tannehill elevate their games in such situations, Brady, Rodgers and Manning's games diminish.

Of course, I am being facetious. These tied game stats are pretty small sample sizes and it is silly to jump to a conclusion that Fielder's facially impressive numbers in such situations are any kind of proof of clutch. But that's part of the danger of (a) using small sample sizes, and (b) using apparent correlations as basis for a conclusion of clutchness.

 

For any given score differential, I think it's pretty solid to argue that a QB with a negative such differential has more pressure on him than one with a positive one, especially when the game is on the line. I mean if you're trailing you NEED to score. If you're leading you don't, and you'll probably rely on the run game more. So yeah it makes sense to compare tied+leading vs. trailing in <4 minutes.

Right, which is why one should always account for sample size. Once you do that, the objection doesn't hold. It doesn't hold for Flacco's postseason numbers being only 0.21% likely to be due to random variation (GIVEN the sample size = 15), and for passer rating in general, I showed in another thread how the variance changes as a function of attempts (can't remember where that was.. might be in the Luck vs. Tannehill thread). My argument using the <4 minute tied + ahead vs. trailing stats for Tannehill did account for sample size, and that was a smaller sample size than with Brady, so the effect is solid.

In any case, though I'll admit you found some examples of where HoF (or future HoF) QB's don't have good numbers in arguably more pressure situations, by and large you'll see they are either around average in drop-off or well above it. It's statistical evidence of clutch if one defines it as simply better than average response to pressure.

 

You are conflating the issues on that Flacco sample size. If you are counting all 15 of his post-season games, then the difference between his playoff numbers and regular season numbers is minimal -- about 4 rating points. The 0.21% number came from isolating his 10 game stretch of good playoff games and looking at them in isolation. That is inappropriate because it pretends that those games stand alone. It would be like selecting the 10 best home games of a player's career and purporting to show that being at home is what caused his great play. I asked you to run your numbers for his 9-10 game stretch in 2010 and his numbers against the NFC South to see what your numbers say is the likely he would have had those stretches due to random variation. I don't recall seeing the results of that so I'll ask again. Assuming you are consistent in your approach, the numbers for that 2010 stretch will look an awful lot like the numbers for that selection of 10 playoff games. And while you are at it, please run your numbers for his full 15 game playoff sample as compared to his regular season numbers. I can guarantee that the likelihood that his playoff rating is 4 pts higher is due to random chance will be much, much greater than 0.21%.

 

Dude.. my numbers are for the entire 15 game stretch! Go back and read post #407 on pg 21. 0.21% probability for all 15 games.

Oh, and post #420 answered your 9 game question long time ago (you never asked me to run the NFC South thing btw). I guess you need to read my posts better


EDIT: OK.. just for completeness, to answer your NFC South question, I did what's called a 2-sample t-test with unequal sample sizes for all of Flacco's regular season games vs. all 8 of his NFC South games. It tests whether the two sets came from the same distribution. Note this automatically takes sample size into account but is a different test than I did for the correlations in post #407 and #420. Anyway, probability that NFC South record is due to chance is 0.23%. So with that one you found something almost perfectly comparable to how statistically unlikely his playoff run was. Good find btw.. and it's only the NFC South, all others are normal.

 

My bad. We are talking about 2 different things. Your 0.21% number purported to show whether his supposed playoff improvement from the first 5 games to the last 10 was due to chance. It was not to show whether his 15 game set of playoff games shows that he is clutch when compared to his regular season games. I was interpreting your latest post as suggesting that.

Yes, I missed post 420, but you did it with the wrong games. I wasn't talking about the first 9 games, but rather games 6-14. Compare those to the rest of Flacco's career and tell me what the probability is that those 9 games were due to chance. It's not problematic at all that they are just like every other stretch of regular season games with no distinguishing characteristics. Indeed, that's the point. He randomly had a stretch of 9 great games. It appears to be completely random, yet I'm willing to bet that your calculation is going to show that the likelihood that it is random is negligible.

Yes, that NFC South data set is an outlier, but those happen. Indeed, I found one for the exact same player with almost the exact same probability as that playoff set. Probably not as dramatic as that one, but I bet if you compared his rating in stadiums with a retractable roof with those in non-retractable roofs (or no roofs) it will also appear to suggest that his poor numbers in retractable roof stadiums is not likely due to random chance. Yet I can't for the life of me think of any reason why one would play worse in retractable roof stadiums than outdoors and in indoor stadiums.

 

I've said this before to someone else, it's just this easy, what you are saying, is that anxiety, nervousness and the like do not exist, or you are saying that all people react equally to these things.

Either way, that is obviously flawed logic.

 

Yeah, the 0.21% was only referring to the probability the improvement was due to random chance. I never calculated the probability the set (NOT sequence btw.. ordering goes out the window when you do this) of ratings themselves was likely/unlikely to have come from the distribution of Flacco's regular season ratings (didn't do that because I know the answer).

Anyway, let's just do the calculations, once again with the 2-sample t-test with unequal sample sizes. Probability those 15 ratings came from the same distribution as Flacco's regular season games is 84.6%. Probability the last 10 games came from it is 3.15%, and probability games 6-14 in 2010 came from it is 1.06%.

Of course, except for that first number, 84.6%, I'd say the others are just cherry picking data because you really can't independently identify a separate condition there (playoffs are a separate condition so 84.6% I'd accept). Anyway, that was obvious, and just to be clear what I did in post #420 was also answering the question of probability the improvement was due to chance.

Regarding the NFC South. I wouldn't just dismiss that. There's probably something real there. Those ratings occurred in two years, 2010 and 2014, and in one case NFC South was strong, other case weak, and Flacco still had great ratings. There's probably something about Flacco and the Ravens' style that just matched up well with those teams but what that is someone who is more familiar with the NFC South during those years needs to try and answer.

 

I don't think that's because of truncating the data tbh. I think it's because playoff teams are in general stronger and have better records, so the range of win/loss records is reduced (all have a lot of wins), which on its own would reduce any correlations. So I bet if you had the same sample size for playoff teams vs. regular season teams, the correlation also goes down by a similar amount.

 

No, I'm arguing that "clutch" doesn't just appear at the end of games. "Clutch" players play well all game long. Their norm is to play at a higher level than average, so they tend to play at higher level than average in supposed "clutch" situations, toO.

Yes, anxiety, nervousness, etc are real, and affect people differently. I just think you guys are severely overestimating it's effect on professional football players.

 

Clutch is a term that applies to pressure situations, where anxiety and nervousness are most prevalent, and football players are people, and as people, they will all react differently, these are undeniable truths.

 

I actually think we're closer to a consensus than when this whole thing started.

I doubt anyone can really disagree with the following: IF you define "clutch" as performing better than average in response to pressure, then there are many stats that measure "clutch".

Truthfully, that's all the "clutch" crowd (as far as I can tell) is asking people to accept. For us, it really doesn't matter if QB's that are above average in "clutch" measures are also generally above average in other measures (they are, but that's really beside the point). The measure itself is what we're arguing for, and as far as I can tell the only technical uncertainty there is that you can't directly measure the differential amounts of pressure in two conditions, including whether pressure exists at all, but that part doesn't seem too controversial anyway.

Maybe we can boil this debate down to those who specifically disagree with that assertion.

 

This can also mean that HoF QBs play on better teams.

 

I think this is THE problem. I'd strenuously argue against it if someone pushed "clutch" = "elevating performance above norm" too far. Define it as better than average performance in response to pressure and I think all other issues go away. For example, define it that way and I bet they find evidence of "clutch" in other sports (the ones where you point out they say it doesn't exist).

Also.. in general you start with operational ways of measuring things first, then afterwards try to figure out what the underlying physical/mental mechanisms are that give rise to the measure. Start the other way and you might not make any progress. Otherwise I agree with what you wrote.

 

An above average QB, showing to be above average in pressure situations, makes him "clutch?" Even if they have a bigger overall drop than the average QB?

Think about it this way: if I was to play someone in golf, who hadn't played much, they might hit some good shots, and I might hit some bad shots. Overall, though, I'm going to beat them, but you wouldn't say that was a function of me being "clutch." Even if the match came down, miraculously, to the last hole, it would be expected that I would win, because I'm better overall.

 

No.. obviously they on average have to have a smaller drop in performance from their normal level of performance (that's what I mean by "in response to pressure", not "in pressure situations" which would just be looking at absolute performance levels). Not sure where you've been this whole time if you think I was arguing anything else. Maybe to prevent confusion for those who didn't debate, just explicitly define "clutch" = "smaller than average drop in performance in pressure situations".

Regarding comparing to a novice, the definition probably still works unless the novice is learning (hopefully) at a large enough rate that it might be difficult to figure out what normal level of performance is. But IF performance is relatively stable, yeah it works for the novice too.