Initial Comments by Steve Bryan, President & Founder of Vigillo, A SambaSafety Company
June 28, 2017
Yesterday, June 27, 2017, 18 months after the FAST Act removed CSA Scores from public view, the National Academies of Science, Engineering and Medicine (NAS) published their preliminary findings outlining their recommendations to reform the Compliance, Safety and Accountability program we all know and love as CSA.
Their 132 page report starts with a nod to the FMCSA for the work they have done on CSA and for the funding for this study (amount undisclosed). NAS starts by thanking Joseph DeLorenzo, director, FMCSA Office of Enforcement and Compliance, for his presentations, which were not just useful, but “enormously useful”, way to go Joe!
That was followed by acknowledgements to the other twenty-five presenters the first of which was yours truly (it was alphabetical). We are all thanked but our contributions are not characterized on a usefulness scale. Item Response Theory (IRT) applied to the relative informational value of each of us would have been nice, but hey, more on that later.
NAS categorizes their report into the following four sections:
In one paragraph, I’ll summarize. My take on this entire 132 page report is stated as follows.
“Good job FMCSA on what you have done with CSA so far, we (NAS) give you a solid 5 out of 10 for your efforts so far. But there needs to be a sound statistical model applied to replace the ad hoc nature of how CSA is formulated today. More, and more reliable data needs to be gathered, and the data utilized needs to be more transparent and software developed to make such data more useful by all stakeholders. Only then should CSA Scores be released to the public.”
My take on the Report
So I’m going to very briefly summarize my thoughts on each of the 4 categories of the NAS Report. I have distilled 132 pages to 5 ½. There are some specific topics down in the trees such as how to blend relative and absolute scoring, I’ll leave those for a follow up, I’ve had 24 hours with the report and so has the Vigillo team, I wanted to get out some thoughts at a high level, so here we go…
NAS Recommendation: SMS is structured in a reasonable way, and its method of identifying motor carriers for alert status is defendable. However, much of what is now done is ad hoc and based on subject-matter expertise that has not been sufficiently empirically validated. This argues for FMCSA adopting a more statistically principled approach that can include the expert opinion that is implicit in SMS in a natural way.
My Commentary: NAS states that Roadside Inspection data (MCMIS) cannot be utilized to build a predictive model that identifies the crash risk of individual motor carriers. Identifying the crash risk of individual carriers is a specific requirement of the FAST Act. NAS has, in its first conclusion supported the structure of SMS but called out the lack of sound statistical underpinnings. The industry has always supported the concept of CSA but identified early on that FMCSA’s ad hoc approach to the methodology, severity weights and BASICs largely lacked any statistical support.
My Summary: I agree that the CSA concept is sound. I also agree that there has never been an identifiable statistical model that FMCSA has ever relied upon. NAS gives more credit than I would to the subject matter expertise that originally architected CSA. Something like CSA makes sense, it needs science behind it.
NAS Recommendation: FMCSA should develop the suggested IRT model over the next 2 years. If it is then demonstrated to perform well in identifying motor carriers for alerts, FMCSA should use it to replace SMS in a manner akin to the way SMS replaced SafeStat.
My Commentary: NAS proposes an IRT Model. So what is an IRT model? Item Response Theory has been largely used for years in transportation, aviation, health care, and education and it can be explained easily how it might be applied to CSA. Below the surface, IRT lies in the domain of PhD statisticians and is monstrously complex. But the concept is easily explained.
Think of a test you’d take in school. It has ten questions. Sally takes the test and gets 2 incorrect. Sally gets an 80%. Steve takes the test and gets 2 incorrect, a different 2. Steve gets an 80%. Little can be learned about Steve vs Sally because we got the same score but got different questions (items) wrong. What is hidden and what IRT reveals is that Sally’s questions were far more difficult than Steve’s. Only by applying sound statistics to the items, the individual questions, can we begin to get insight into the differences between Sally and Steve. IRT does this analysis
There is a very good analogy to CSA. Carrier A has a BASIC % of 80% in the Maintenance BASIC and Carrier B has an 80% in the same BASIC. They are equivalent, right? Maybe not. Carrier A operates a proportionately high number of miles in Texas and receives many violations because of the Texas focus on Maintenance. Carrier B does not operate in Texas, they just have really crappy trucks. IRT will help explain this, and can begin to explain and balance some of the defects in CSA like disparate State Enforcement. NAS also believes that IRT can assist in the areas of:
My Summary: IRT can improve CSA, even with data that is not as rich as we’d like it to be, and even in a complex environment like trucking, with 50 enforcement regimes, millions of trucks, differing cargo types (tasks) and a huge variety of operating environments, weather, mountains and congested cities.
Warning: IRT, while well understood by statisticians and widely used for years and in many different applications, is far more complex than the current CSA methodology. Safety professionals in trucking are going to have to learn to explain to drivers, customers, bosses and shareholders how all of this works. Let’s look at the existing methodology for Vehicle Maintenance vs the IRT Modeling Strategy.
Existing Vehicle Maintenance Methodology:
Proposed IRT Model (excerpt from NAS Report)
The Science of the IRT model is far superior to the ad hoc pseudo-science of the current methodology. Great statistically. Now its Monday morning, your phone just rang, a shipper wants to know why you have an Alert in Vehicle Maintenance, what do you say?
NAS Recommendation: FMCSA should continue to collaborate with states and other agencies to improve the quality of MCMIS data in support of SMS. Two specific data elements require immediate attention: carrier exposure and crash data. The current exposure data are missing with high frequency, and data that are collected are likely of unsatisfactory quality. Further, to improve the exposure data collected involves not only collecting higher-quality VMT data, but also collecting this information by state and by month. This will enable SMS to (partially) accommodate existing heterogeneity in the environments where carriers travel. Crash data are also missing too often. Also, there is information available from police reports currently not represented on MCMIS that could be helpful in understanding the contributing factors in a crash. Such information could help to validate the assumptions linking violations to crash frequency. To address these issues, FMCSA should support the states in collecting more complete crash data, and in universal adoption of the Model Minimum Uniform Crash Criteria, as well as developing and supplying the code needed to automatically extract the data needed for the MCMIS crash file.
My Commentary: NAS breaks down the goal of improving data into quality of data that is currently collected, and quantity of new data that might be collected.
Quality of existing data focuses on VMT (miles) and APU (truck count). These are both reported today on the MCS-150, but as NAS correctly points out the “impact of flawed or out-of-date VMT and APU data on SMS percentile ranks may not be fully appreciated by the carriers”. Better reporting of these two data elements would improve the data quality of CSA and make an IRT model more trustworthy.
When we move to Quantity, the NAS suggestion that more data be provided, while a worthy goal statistically, becomes problematic (huge understatement) in practice. NAS suggests that VMT by State, cargo type, driver pay and pay model and driver retention rates all be reported to FMCSA. There is also a recognition of the data that could be available as a result of the ELD mandate, and NAS suggests that that data could be reported to FMCSA. I’m trying to think of the right word here…
My Summary: “Ain’t not never gonna happen”. Where I drive, what cargo I carry, what and how I pay my drivers and how successfully I retain them, combined with HOS data that would show location, speed, direction, time/date, perhaps shipper detail, is never, on this planet, in this universe, going to be provided by motor carriers to the FMCSA. This is the soul of a very competitive industry and this data is solid gold and will never be made available to FMCSA. The competitive nature, combined with years of distrust of the FMCSA makes this, in my opinion, the weakest component of this entire 132 page report. If this data is to be aggregated, FMCSA is not the one to do it, they have no authority and their reputation for aggressive interventions based on flawed data precludes them as a trusted candidate. Sorry Joe, you’ll never see this data.
NAS Recommendation: FMCSA should structure a user-friendly version of the MCMIS data file used as input to SMS without any personally identifiable information to facilitate its use by external parties, such as researchers, and by carriers. In addition, FMCSA should make user-friendly computer code used to compute SMS elements available to individuals in accordance with reproducibility and transparency guidelines.
Steve Commentary: MCMIS data is a mess, its always been a mess. Vigillo spends 20-30% of our time cleaning the data before we even begin to analyze it. NAS has seen this, and that combined with the far more complex IRT model, will require a new focus on MCMIS. Better and more consistent collection from the states, a much better understanding of some of the flaws in how its reported and now combined with the need to give non-PhD’s the ability to run the analysis, is absolutely critical if CSA is to move forward.
My Summary: I hope the FMCSA accepts the recommendations of NAS and seriously begins to work on the next phase of CSA, or whatever we call it. FMCSA must strike a more collaborative relationship with the industry, and demonstrate absolute transparency in how they are collecting, analyzing, and taking action on this new data model. NAS suggests that outside experts be hired, perhaps from academia, maybe there are some of us in the industry that know a little about this too. It’s a moment in time where perhaps, a less combative, more inclusive partnership can be developed and we can re-boot from the very controversial launch of CSA in 2010.
I’m hopeful, I’m appreciative of the work NAS has done, now we need to see what FMCSA’s response is in 120 days… oops 119. Then we can discuss when scores are public again.
Happy Friday everybody,
It seems to be pretty rock solid now that the CSA Reform Report, 18 months in the making, will be released to the press on Monday, and the Public on Tuesday, June 27.
I’ll be reading the report as soon as I can get a hold of it, I have a team at Vigillo ready to dive into it, and we’ll be issuing comments by Wednesday, perhaps sooner. Watch this blog for those comments. Where possible, we’ll be running the numbers in addition to issuing opinions.
I posted a status report last December HERE and outlined the 8 items that the FAST Act cited as areas that need review. Those 8 areas of reform are:
1. The accuracy with which the BASIC’s identify high risk carriers and predict or are correlated with future crash risk, crash severity, or other safety indicators for motor carriers. (Safety Event Groups)
2. The accuracy of safety data, including the use of crash data from crashes in which a motor carrier was free from fault. (Crash Accountability)
3. Whether BASIC percentiles for carriers of passengers should be calculated separately from motor carriers of freight.(Freight vs Passenger systems)
4. The differences in the rates at which violations are reported to the FMCSA by various enforcement authorities, including States, territories, and Federal inspectors (State Difference)
5. How members of the public use the SMS and what effect making the SMS information public has had on reducing crashes
6. Whether the SMS provides comparable precision and confidence, through SMS alerts and percentiles, for the relative crash risk of individual large and small motor carriers; (Small Carriers largely unscored)
7. Whether alternatives to the SMS would identify high risk carriers more accurately
8. FMCSA has fully implemented or satisfactorily addressed the issues raised in the report titled ‘‘Modifying the Compliance, Safety, Accountability Program Would Improve the Ability to Identify High Risk Carriers’’ of the GAO dated February 2014.
As I conduct my review, these are the areas I’ll be looking for action.
Have a good weekend, I’ll talk to you all early next week