Networking

It Wasn't a Hack: What the FAA Router Incident Teaches About Network Change Control

← Back to all articles

On Friday 13 November, the US Federal Aviation Administration (FAA) halted flights across the country for nearly five hours. According to Reuters, more than 20,000 flights were disrupted over the following days. Officials said on 14 November that they had found no evidence of a cyberattack.

The trigger appears to have been a network change. Investigators have linked the outage to a newly installed router at a remote FAA radio site, and the questions they are now asking apply to almost any business network.

What we know so far

The investigation is still under way, so some details may change. Based on reporting by Reuters and Men's Journal, which cites The Wall Street Journal:

  • It started with routine work. US Transportation Secretary Sean Duffy said the outage began during work to move lines from copper to fibre at a site in El Dorado, Arkansas.
  • A new router flooded the network. When the new router was switched on, a router in Amarillo, Texas, which had been replaced about three weeks earlier, also began flooding the network. The FAA's acting chief technology officer described more than 1,000 connection attempts every four to five seconds.
  • A feature is in the spotlight. FAA officials believe a feature that broadcast data more widely than necessary caused the overload. The FAA has switched that feature off on all 69 routers of that type.
  • The cause is not settled. The router was a Juniper device installed by contractor L3Harris, which runs the network. HPE, which owns Juniper, says the devices "did not fail". Officials are still working out whether the problem was a faulty device, a faulty installation or the configuration.

We are not drawing conclusions about who was at fault. What matters for everyone else is how a single change at one remote site was able to affect a whole national system.

Why this matters for your network

Most network outages are not caused by attackers. They are caused by changes: a new device, a firmware update, a feature switched on by default, a configuration copied from another site. Older networks are especially sensitive, because equipment designed years ago may not cope with traffic that newer devices generate.

The FAA case brings together several risks that UAE and GCC businesses face too.

  • New devices on old networks. Modern routers and switches can introduce protocols and features that legacy equipment was never designed to handle.
  • Defaults nobody reviewed. A feature that is harmless in one network can overload another.
  • Work done by a third party. When an integrator installs and configures equipment, the business still carries the impact if something goes wrong.
  • Problems that appear later. The Amarillo router had been in place for weeks before it caused trouble, which makes the link between change and outage harder to see.

Change control steps that reduce the risk

  1. Test new features before rollout. Try new devices and features in a lab or an isolated segment first, with traffic that resembles your real network.
  2. Review default settings. Before a device goes live, check which features are enabled out of the box and switch off what you do not need.
  3. Stage your deployments. Roll changes out to one site, watch it, then expand. Do not replace every device of a type on the same day.
  4. Review integrator configurations. Ask your installer for the configuration they applied and have someone on your side, or an independent partner, check it before go-live.
  5. Monitor for unusual traffic. Alert on sudden spikes in connection attempts or broadcast traffic, so a misbehaving device is spotted in minutes rather than hours.
  6. Have a rollback plan. For every change, agree in advance how to undo it quickly, including keeping the old configuration and, where practical, the old device to hand.
  7. Keep a change log. Record what changed, where and when. When something breaks weeks later, that log is often the fastest route to the cause.

Our view

Firewalls and security tools get most of the attention, and rightly so. But the FAA ground stop is a reminder that a disciplined change process is part of keeping a network running. The time to plan a rollback is before the change, not during the outage.

If you are planning a network upgrade or migration and would like help with staging, configuration review or monitoring, contact our team.

Drawing on reporting from Reuters, Men's Journal.

Chat with us WhatsApp