By Brian Fung, CNN
A massively disruptive computer shutdown at the Federal Aviation Administration this week that caused thousands of flights to be canceled or delayed has put Americans uncomfortably face to face with the technology behind US air travel — at least for the second time in a month.
As the country once again picks up the pieces, frustrated air travelers may wonder why flying suddenly seems vulnerable to devastating IT problems.
The answer involves not only aging hardware and software, but also institutional failures that make updating technology more challenging, according to current and former industry officials, government reports and outside analysts.
Over the years – and in the face of exploding demand for air travel – bureaucratic snafus and deferred maintenance have contributed to an increasingly fragile system, even as it grows ever more sophisticated and has far more points of failure than many users may realize.
The recent collapse of Southwest Airlines — in the middle of a winter storm and during the most critical travel period of the year, no less — and Wednesday’s widespread flight disruptions may have put many of these problems front and center for U.S. passengers, but they are just the latest manifestation. of longstanding problems and enormously complicated.
Airline travelers get a month from Hell
The glitch at the center of this week’s confusion is a damaged database file in the pilots’ advisory system that issues warnings, known as NOTAMs, of various dangers that can affect the flight, ranging from notices of closed runways to the presence of construction equipment nearby. . The damaged files are also in the FAA’s backup system, a source familiar with the matter told CNN, which first reported the details Wednesday.
Officials moved to reboot the main NOTAM system early Wednesday morning, but it failed to be completely restored by early rush hour on the East Coast, leading to the FAA ground stop. A senior US official told CNN on Wednesday that there was no evidence of foul play in the incident, details that were later confirmed by the FAA.
“The FAA is continuing a thorough review to determine the cause of the Notice to Air Missions (NOTAM) system malfunction,” the agency said in a statement Wednesday evening. “Our initial work has traced the outage to a corrupted database file. At this time, there is no evidence of a cyber attack. The FAA is working diligently to further pinpoint the cause of this problem and take all necessary steps to prevent such disruptions from occurring again.
The FAA said Thursday evening that the data files were “damaged by personnel who failed to follow procedures.”
The NOTAM issue comes just days after the FAA said an “air traffic computer problem” was responsible for an hours-long flight delay to a Florida airport on Jan. 2. The system, known as ERAM, is responsible for tracking hundreds of flights at a time and is considered a critical component of the FAA’s efforts to modernize US airspace.
In the case of Southwest, an outdated scheduling system that cannot automatically adjust to disruptions caused by severe winter weather requires painstaking manual intervention, which makes the airline’s weather-related problems particularly pronounced.
‘Old, old system’
Despite the move to modernize their equipment, in some cases airlines and the US government may still rely on technology that may be years or even decades old.
The FAA software that failed this week is 30 years old and at least six years away from being updated, US government officials told CNN on Thursday, although Transportation Secretary Pete Buttigieg has been pushing to speed up that timeline since the meltdown, officials said.
The FAA’s NOTAM system is “Jurassic,” said Kathleen Bangs, a spokeswoman for flight tracking company FlightAware. “It’s a clumsy system that often burdens pilots with pages and pages of less important notices, written in antiquated code that sometimes bury very important safety information that pilots need.”
The FAA has acknowledged the age of the NOTAM system. In its latest budget request to Congress, the agency called for money to help “eliminate the failed vintage hardware” behind it.
In early 2012, the FAA decided it wanted to replace the aging legacy voice switches used in air traffic control communications with a new internet-based communications technology. But thanks to a contract dispute, the FAA now intends to keep using the old switch until at least 2030, according to a report by the Department of Transportation Inspector General last year.
The ERAM air traffic system at the center of the disturbance on January 2 is much younger, and only became fully operational in 2015. But according to the 2020 report of the Inspector General, this system was supposed to have been fully implemented five years earlier, as a replacement for another system that has been running for more than 40 years. The FAA is currently working to update the ERAM software and hardware, after at least seven ERAM failures since 2014, a track record that has prompted congressional scrutiny. But it may not be until 2026 that the ERAM upgrade is completed, according to the 2020 report.
Meanwhile, many of the IT systems airlines rely on are custom-built, with some running on legacy mainframe computers, and not designed to handle large surges of information, aviation experts say.
“This is not your standard Windows server or modern VMware architecture,” said Seth Miller, IT consultant, aviation journalist and editor of the PaxExAero travel publication. “It’s an old, old system.”
As a result, the acute crisis can easily overwhelm these fragile setups, according to an aviation industry official, speaking on condition of anonymity to discuss the issue more freely.
“These systems were built at a time when airlines could be smaller, and they weren’t necessarily built to handle a lot of incoming data at once,” the official said. “When you have something like a big winter storm over the holidays, it can’t handle the volume of changes at one time, because it’s on a system that’s not built to handle large data sets.”
It is not always that the age of our technology is inherently a problem, said industry experts. That’s what age means: An inability to scale to meet new demands, and a lack of proper support as the rest of the world moves forward. The use of custom-built technology, as opposed to off-the-shelf solutions, exacerbates the problem, said Miller, as maintenance requires more specialized parts and know-how.
Trying to integrate old systems with newer ones – always in real time, because the global aviation industry never sleeps – can also create its own opportunities for serious mistakes.
Many ways to fail
While all flight delays and cancellations tend to result in similar experiences for air travelers, the source of the illness may vary. More things can go wrong than you might expect – highlighting the complexity of the airline industry, and underscoring how there are no quick fixes for IT-related travel disruptions.
Getting a flight off the ground involves complex information, industry experts say, and a disruption in any part of that information supply chain can cause delays.
Vulnerability is magnified due to the large number of companies involved in the ecosystem – not just airlines, but their vendors, and their vendors’ vendors.
“There are so many systems talking to each other,” said Ross Feinstein, a former spokesman for American Airlines and the Transportation Security Administration.
For example, Feinstein said, TSA vets airline manifests. “If TSA has a problem, it stops the vetting process for reservations, which means that passengers can’t check in, and they can’t take boarding. Maybe the weather company has a problem, and pilots can’t get the latest weather data for their departure, en route, or come
In 2019, computer problems at a third-party company whose flight planning tools help airlines calculate weight and balance for their planes caused delays for several national airlines.
In 2021, disruptions at Sabre, one of the world’s largest airline reservation companies, caused disruption globally.
The interconnected nature of the aviation sector, involving dozens of countries, companies, agencies and databases creates several points of failure. Backups and redundancies can help, but it’s a very complex system.
bureaucratic challenges
Beneath the surface-level symptoms of the aviation sector’s IT problems are deeper, messier and more human challenges.
Take the FAA’s effort to replace its air traffic sound switch. According to the report of the Inspector General, the main source of the breakdown came when the FAA and its potential seller got into a dispute about the terms of the contract. The dispute centers on possible software defects in the new switches, and whether the vendor can still deliver a good product on time.
The root of the problem is not, in itself, a technological problem. It’s a procurement issue. But it has had a lasting effect on FAA technology. The eventual termination of the contract means the FAA will have to spend more than $270 million through 2030 to keep using its aging legacy voice switches, the report said.
“Continued dependence on these switches creates the risk that communications will be disrupted,” the report concluded.
A similar dynamic has been at play in the debate over 5G wireless technology near airports, which last year threatened to cause major disruptions. Bureaucratic divisions and years of delayed avionics upgrades have led to a crisis in which US aircraft are not equipped with technology that can handle potential 5G disruptions.
Meanwhile, the FAA continues to be led by an acting administrator, and lacks a Senate-confirmed chief. That has real-world ramifications for IT upgrades and other projects, according to people familiar with the agency, who spoke on condition of anonymity to discuss the matter more freely.
“It’s very difficult to set a direction and a vision when you don’t know if you’re going to be there for a week or you’re going to be there for 18 months,” the person said.
Much of the unpaid technical debt in the airline industry, meanwhile, can be traced to multiple mergers and bankruptcies after 9/11, when many airlines focused more on finances than technological upgrades, industry officials said.
That bureaucratic myopia is itself the cause of the current technological malaise in the aviation industry. In some cases, institutional inertia and commercial priorities have outranked investment in expensive and boring infrastructure.
But the increasingly connected and digitized nature of today’s systems means that when things go wrong, they can do so in much more serious ways.
Aviation experts say only more investment, and better planning, can meet the challenge.
“[The FAA] do more resources, and they need more funding to modernize,” Feinstein said. “In Washington, we’ll talk about it for the next 24 to 48 hours, forget about it, and it’ll be a fight again when the FAA reauthorization bill comes up.”
–— CNN’s Pete Muntean and Gregory Wallace contributed to this report
The-CNN-Wire
™ & © 2023 Cable News Network, Inc., a Warner Bros. Discovery Company. All rights reserved.
