Major BGP leak disrupts thousands of networks globally

A large BGP routing leak that occurred last night disrupted the connectivity for thousands of major networks and websites around the world.

Although the BGP routing leak occurred in Vodafone's autonomous network (AS55410) based in India, it has impacted U.S. companies, including Google, according to sources.

BGP leak causes 13x spike in misdirected traffic

Yesterday, Cisco's BGPMon spotted a discrepancy in an internet routing system, potentially indicating some BGP hijacking activity taking place:

"Prefix 24.152.117.0/24, is normally announced by AS270497 RUTE MARIA DA CUNHA, BR."

"But beginning at 2021-04-16 15:07:01, the same prefix (24.152.117.0/24) was also announced by ASN 55410," stated BGPMon's announcement.

Doug Madory, director of Internet analysis at Kentik further confirmed these findings stating that the autonomous system ASN 55410 was seeing a 13 times spike in inbound traffic directed to it.

This occurred from the network mistakenly advertising that it supported over 30,000 BGP prefixes or routes, when it didn't, causing the internet to flood this network with traffic that was not meant to go through it.

The said autonomous system (AS55410) belongs to Vodafone India Limited.

According to Kentik's analysis, some U.S. companies, including Google, were also affected as a result of this incident which appears to have lasted from a little before 13:50 to around 14:00 UTC on April 16, 2021.

Kentik shared some more insights with BleepingComputer:

"This incident only affected traffic for about 10 minutes, but during that time there were likely countless internet connection problems for users around the world."

"Anyone trying to reach web resources configured with the IP addresses in the routes that were leaked would have had their traffic misdirected to AS55410 in India and then dropped," Doug Madory from Kentik told BleepingComputer in an email interview.

Although the leak primarily impacted the misdirection of internet traffic for the 30,000+ advertised routes and Indian internet users, the leak did spread throughout the world, says Madory.

Madory has compared the unexpected surge in inbound traffic experienced by Vodafone India (AS55410) to essentially a self-inflicted DDoS attack which also likely caused problems for Vodafone's users to reach the internet during this time.

Further analysis by a BGP expert Anurag Bhatia, has identified over 20,000 prefixes from global autonomous networks that were impacted as a result of this incident.

What are BGP, BGP hijacking, and BGP leaks?

BGP or Border Gateway Protocol is what makes the modern-day internet work.

It is akin to having a "postal system" for the internet that facilitates the redirection of traffic from one (autonomous) system of networks to another. 

The internet is a network of networks, and for example, a user based in one country wanted to access a website based in another, there has got to be a system in place that knows what paths to take when redirecting the user across multiple networked systems.

This is similar to a letter being transited through multiple postal branches between its source and destination.

And, that is the purpose of BGP: to direct internet traffic correctly over various paths and systems between the source and destination to make the internet function.

But, BGP is fragile, and any disruptions or anomalies in even a few intermediary systems can have a lasting impact on many.

For the Internet to work, different devices (autonomous systems) advertise the IP prefixes they manage and the traffic they are able to route. However, this is largely a trust-based system with the assumption that every device is telling the truth.

Given the massive interconnected nature of the Internet, it is hard to enforce honesty on every single device present on the network.

BGP route hijacking occurs when a malicious entity manages to "falsely advertise" to other routers that they own a specific set of IP addresses when they don't. When this happens, chaos occurs.

This route confusion would create a lot of trouble on the Internet and lead to delays, traffic congestion, or total outages.

But, BGP route leaks are similar to BGP route hijacking, except the latter more specifically refers to instances of malicious activity taking place. 

Whereas, route leaks can be, more likely than not, accidental.

In either case of a BGP route leak or BGP hijacking, an Autonomous System (AS) announces that it knows "how" or "where" to direct the traffic meant for certain destinations (ASes) that in actuality it does not know.

This can lead to the user being taken over an internet route that will offer suboptimal performance or outright cause disruptions and potentially serve as a front for eavesdropping or traffic analysis activities, in cases of malicious hijacking.

For example, last year, as reported by BleepingComputer, IBM's global outage was caused because of an erroneous BGP routing configuration. 

Prior to this, we had seen a significant case of BGP hijacking in 2008, when YouTube had gone offline for its global audience due to some of its traffic getting redirected through Pakistani servers.

Over the next few years, we have reported similar incidents. 

How could this BGP leak have been prevented?

Although the incident is now over, it could happen again at any time.

Kentik shared some common safeguards with BleepingComputer that companies can use to protect against BGP leaks like this one, which says Kentik were not in place yesterday when the leak had occurred.

"Obviously, AS55410 should not have leaked 30,000 routes, but adjacent networks should have deployed filtering mechanisms to contain the damage," Madory further told BleepingComputer, explaining that these mechanisms include:

1. Setting MAXPREF: AS1273 (also registered to Vodafone) and AS9498 (registered to Bharti Airtel Ltd.) could have set a MAXPREF on their connection to AS55410. This is a simple mechanism that automatically disables a BGP connection when a downstream network suddenly starts sending an unexpectedly large number of BGP routes.
    
2. Configuring filters: Using internet routing registry data, AS1273 and AS9498 could have been filtering the routes accepted from AS55410 knowing that they shouldn't receive US BGP routes, for example.
    
3. Deploying RPKI: Leaked routes with Route Origin Authorizations (ROAs) configured in the RPKI should have been dropped.

"In every BGP routing leak, there is more than one party involved. There is a leaker, of course, but there is also always another network that distributes leaked routes out onto the internet."

"Mistakes inevitably occur, we have to do better to look out for each other. The internet, after all, is a team effort!" concluded Madory in his interview with BleepingComputer.

Update 11:50 AM ET: Added comments from Kentik's Doug Madory received after publishing.