ISD Certification Testing

isd

The Problem

When 10 gallons of gasoline are pumped into your gas tank, 10 gallons of gasoline vapor are displaced. To prevent the vapor from escaping into the atmosphere, gasoline pumps are equipped with Enhanced Vapor Recovery Systems (EVR systems), which recover the displaced vapor. The gasoline pump is also equipped with an In-Station Diagnostic System (ISD system), which monitors the EVR to make sure that it is working properly. If the EVR fails, the ISD system responds by shutting down the pump.

When a company develops an ISD system, it must be tested and certified before it can be put on the market. The California Air Resources Board (CARB) has set regulations that require an ISD system to catch at least 95% of all failures in order to become certified. The purpose of this project is to design statistical certification tests that CARB can run on new ISD systems to efficiently test their accuracy. CARB tests ISD systems by instituting artificial failures at the pump. The failure in the EVR should cause the ISD to shut down the pump. CARB runs a certain number of trials, and the accuracy of the system is determined from the number of failures that the system catches versus the number it misses. The certification test procedure must include how many trials to perform, how many failures must be caught in order to pass a system, and how stringent the test can be before it begins to cause error in the certification procedure.

Background Information

Veeder-Root, Inc. of Simsbury, Connecticut is a leading manufacturer of In-Station Diagnostic Systems. In order to sell its diagnostic systems in the state of California, Veeder-Root must demonstrate that their products comply with the regulations of the California Air Resources Board (CARB). Veeder-Root and CARB must agree on certification tests that demonstrate compliance. Working together, engineers, statisticians, and technicians at Veeder-Root and at the California Air Resources Board are involved in developing and performing these certification tests.

The California Air Resources Board (CARB) defines the performance levels for diagnostic systems in gasoline stations. If the vapor recovery system is not working properly, then the alarm system should shut down the pump. CARB requires that the alarm system catch at least 95% of all true failures with no more than 1% false alarms.

Many companies face the challenge of demonstrating that their products satisfy some type of quality or reliability or safety criteria. As this example illustrates, in these situations there is not one right answer but a trade-off between two competing objectives of maximizing safety while controlling costs. The primary tools needed to make decisions regarding quality and safety on the one hand, and the costs involved on the other, include basic probability statistics, including the Binomial theorem. In addition, because this kind of problem is very subtle but arises in virtually every arena of public policy where it often has to be argued with words rather than numbers communication skills are especially important. This task requires the ability to conduct research, give and follow directions, summarize technical text, and  in its presentation to the regulatory board make effective arguments and develop ideas systematically.

There are two types of error to consider in this problem:

  • Fail a system that is actually good
  • Pass a system that is actually bad

The same dichotomy applies to every medical test (e.g. mammograms) that can fail in two ways: miss real problems or falsely flag problems that do not exist. The first is of real concern to the patient, the second to the health care system as a whole.

Veeder-Root is most concerned about limiting the chances of the first type of error. CARB is most concerned about limiting the chances of the second type. The Veeder-Root engineers argue that a sequence of 3 tests is sufficient. The statisticians at CARB argue that this test is not sufficient to support the company�s claim of 95% accuracy. What would the statisticians have to show to prove their point? What is the probability that a bad system, one working at only 80% accuracy, would pass this test?

Materials Included

  • Project and testing description
  • Explanation of binomial distribution
  • Possible approach