The car seat world freaked out a couple of months ago when Consumer Reports announced that 10 out of 12 major US car seat models didn't perform well on more stringent crash tests required for new cars. Then just two weeks later, CR very publically withdrew the announcement after the NHTSA discovered flaws in the testing methodology. CR also commissioned an independent investigation into how those errors occurred and where CR and its specialized testing contractor screwed up.
Those experts, former presidents of the NHTSA and the Insurance Institute for Highway Safety, sifted through the wreckage and have delivered their report: in short, Consumer Reports made significant methodological errors in their testing; miscommunication between the testing lab and CR was compounded by CR's lack of internal expertise with side impact testing; and CR's practice of pursuing independence by limiting contacts with government and industry officials helped the errors go undetected.
CR screwed up, and they say so. And then the outside experts say so. And then they let their critics say so, too. All in all, the findings highlight the errors in Consumer Reports' testing, but they also point out the lack of consensus surrounding side impact testing. And the announcement shows a remarkable candor on CR's part, which is vital, of course, to repairing a serious blow to their credibility.
I can't think of any other example of an organization being so critical of itself and so upfront about its mistakes. So hats off to them for that. The only thing that seems to be missing is a discussion of what changes CR will make, either in designing its tests or consulting with manufacturers and governments as part of their process. Not that I doubt for a second CR's desire to keep a fiasco like this from happening again, I'm just saying.
[update from the comments: DT announces an investigation into how it missed the The Road Ahead link about CR's next steps.]
How our car seat tests went wrong [consumerreports.org]
Previously: AWKWARD! Consumer Reports re car seat study: Never Mind!
Uhh, consider my ass bitten: Consumer Reports: Daddy Types was right about the car seat!
You mean a discussion like this letter from the president of CR that was linked from the section of the report titled "The Road Ahead?"
Not that I think you had any desire to only read the overview page of the report and not click any of the links. I'm just saying. :)
[clearly, mistakes were made in how this information was communicated. I'm going to personally oversee an investigation by my paid consultants at the JPMA and several employees who depend entirely on me for their children's health care coverage, and they're going to get to the bottom of this. -ed.]
Gotta give credit to CR. When they screw up, it's a doozy. But when they own up to it, they really OWN UP to it. Try to think of any other organization that would admit to an error this large, apologize, and set out a system to try to prevent it from happening again. Most other pubs would bury a retraction inside the next issue and get defensive about it. Kudos to CR.
Doesn't this mean that the two car seats that CR said did perform well enough are still the safest? My understanding is not that the test itself was flawed, just that it was done at 70mph instead of 35mph.
[yeah, it was flawed in the sense that CR and the lab interpreted the 35mph thing differently, which resulted in the equivalent impact speed of 70mph. So on the one hand, two car seats passed the test at those speeds. The catch, though, is that 70mph side impacts are extremely rare, and when they do happen, the real risk is from intrusion, aka highspeed entry of the passenger safety zone by the other car or parts of your own car. And they didn't test for that. And no car seat is required to protect against it. -ed.]
wow. i think you actually used the word "methodological" correctly, as in, the study of studying (carseats in this case) nice job.