The Bug’s Underside

Earlier this year, engineering consultant David Hall began a series of year 2000 tests at a large refinery in the Southwest that had been temporarily shut down for maintenance.

One by one, the date settings on systems to monitor waste, storage and other parts of the refining process were rolled forward to Jan. 1 and allowed to enter a simulated version of the new millennium.

While most continued on without missing a beat, a relatively minor part of the refinery controlling the supply of steam heat came to a halt within minutes of the new year. The culprit was a group of microchips buried deep in the plant’s electronic innards that controlled temperature and pressure of the steam--none of which had an obvious reason to be affected by a date change.

At worst, Hall said, a real failure of the system would have made the refinery function less efficiently and might have eventually led to a plant shutdown if allowed to continue for a long period.

But the lesson he took away from the experience was one that has now begun to creep into the already saturated minds of engineers grappling with the year 2000: “If it uses electricity, it is guilty until proven innocent,” said Hall, a senior engineer for the information consulting firm CARA Corp.

The tale of the refinery marks a subtle but important distinction in defining the ever-broadening scope of the year 2000 bug, known as Y2K for short.

Since the discovery of the so-called millennium bug a few years ago, the impending collision of technology and the year 2000 has been largely framed as a problem of software--operating systems, databases and other programs--that are open to potential errors because of the ambiguity of the two-digit year “00.”

What engineers and consultants like Hall have begun to realize is that the software problem is only part of a broader issue that also encompasses the vast array of digital devices that have proliferated into every nook and cranny of modern life.

These devices are used to control the operation of any system that requires a touch of machine intelligence, ranging from the tiny chips used in automobile fuel injection systems to the networks of chips that handle the operation of mammoth power or chemical plants.

These smart devices are found in factories to control machinery and assembly lines. Water treatment plants use other types of chips to monitor and guide the flow of waste. Even seemingly low-tech elevators now rely on chips to control their movements.

The heart of the problem is the tens of billions of generic microprocessors--the brains of modern smart devices--that have been strewn across the globe since the beginning of the digital revolution. Billions of these embedded chips are produced each year; they are the dust of the electronic age.

“The software stuff is just the tip of the iceberg,” said Ken Owen, vice president of business development for the system integration consulting firm TAVA Technologies. “The rest of the iceberg--the big blue mass underneath--that’s the real problem.”

The chips, which include such common types as ASICs (application-specific integrated circuits), PLCs (programmable-logic controllers) and general-purpose microprocessors, suffer from essentially the same vulnerability as software--the inability to handle the two-digit abbreviation for the year 2000. The problem was created by programmers and chip designers who never imagined that their products would have to confront the turn of the millennium.

*

The awareness of the embedded systems problem came slower than its software counterpart because of the invisibility of the chips and their obscure functions.

James Brainerd, the chief information officer for the Los Angeles Department of Water and Power, said the software problem with the year 2000 was fairly easy to discern, since it would reveal itself through the normal process of forecasting with financial information. Chips, on the other hand, were just little pieces of silicon tucked away in facilities among tens of thousands of other components.

“Not many thought there was much time logic until a little later,” he said. “You don’t see date-sensitive components in facilities.”

Once the department realized the scope of the problem, it began to review all its critical plants--a process covering 100 facilities from Utah to the Pacific Ocean that the department is still working on a year later.

The method companies typically use to evaluate their operations begins with a complete inventory of every electronic device in their operations, from security badge readers to computer-controlled machinery.

Xavier Roy, a general manager for Litton Enterprise Systems, which has launched an extensive year 2000 consulting business, said his group’s scan of a single medium-size hospital turned up about 30,000 devices--with about 500 of them having some date function.

He said it took about 10 people over a month just to conduct the initial inventory of devices and compare that with their ever-growing list of suspect machines.

Eric Trapp, head of the year 2000 program for Southern California Edison, said from 25 to 40 engineers worked for four months to pare down a list of 190,000 devices at the San Onofre Nuclear Generating Station to 32,000 electronic components and finally to 450 items that had some potential date connection. It will take the company another year to analyze those devices and fix the ones that will fail in the year 2000.

“The problem is the geographic dispersion of these devices,” Trapp said. “Instead of having some automated tool, you have to use a person with drawings that physically has to [locate] all these systems. The actual fix is relatively easy.”

After completing an inventory, companies then check with information from the manufacturer or other companies to see if their equipment has been identified as having a year 2000 problem.

*

The actual number of failing devices, according to many year 2000 consultants, is small--perhaps no more than 3% to 5%. The reason is that the vast majority of chips exist in a sort of timeless state where the particular date has no meaning to their tasks.

Foxboro Co., one of the largest manufacturers of automated control systems, reports that out of 17,000 devices made over the last 40 years, fewer than a handful were susceptible to a year 2000 failure.

But even in chips that seem to have no outward connection with time, there can be hidden problems, since generic chips often include a time function whether it is used or not.

“Computers are like anything else,” said Jerry Kilpatrick, project manager of the Central Illinois Manufacturing Extension Center at Bradley University. “They have accessories that not everyone needs. If one of the accessories is a real-time clock, the programmer may never use it, but it’s still there ticking away.”

Even a tiny rate of failure can have an impact, given that there are an estimated 35 billion to 40 billion logic chips floating around. The high level of integration between devices in modern plants and factories opens the possibility of cascading failures, where the one faulty device starts a chain reaction by feeding confusing information to the next device.

Donald LeMaster, president of Seattle-based TAVA/R.W. Beck, a joint venture specifically created to deal with embedded year 2000 problems in utilities, related the story about a power-generating plant that conducted a test while shut down for maintenance. The problem took hours to manifest itself but gradually spread to various parts of the operation like a virus.

“The system just got overloaded,” he said, adding that if the failure had occurred for real, the plant would have shut down, since even the backup system would have been affected by the date change.

How the percentages will ultimately play out on Jan. 1, 2000, is uncertain, and those who have spent the last few years working on the problem are torn about the potential impacts.

A recent survey by a Senate committee investigating year 2000 found that out of 10 of the largest power utilities in the country, only two had completed an assessment of their operations.

*

But Trapp, of Southern California Edison, said he believes that most large utilities will fare well in the process, since they have the resources to devote to the issue. He added that in the critical area of nuclear power, the situation is actually more encouraging, since most of the facilities were built years ago using analog controls, which are essentially immune to year 2000 problems.

“Frankly, I don’t believe we have anything that gives us much concern,” he said. “I am a little worried, though, about smaller companies that don’t have the resources.”

For some, the year 2000, which will begin on a Saturday, will simply be a matter of sitting back and watching how all the billions of chips react to the new millennium.

“I’m going to stay home, drink a glass of wine and celebrate,” said Roy of Litton Enterprise Solutions. “I’m going to buy a box of candles. I hope I find a girlfriend by then to celebrate with.”