Enlarge image
A technician checks network cables.
The Telekom malfunction had nothing to do with a wrongly connected cable
Photo: Felix Kästle / picture alliance / dpa
For many employees, the working day in the home office began on Thursday with error messages. They were unable to access Office 365 and Outlook.com. Video conferences via Skype, Microsoft Teams or Cisco's WebEx had to be canceled. Confusion was caused by the fact that those affected could easily access other Internet services and that colleagues with connections to other providers had no problems using Microsoft's offers. So if neither the Telekom networks nor the Microsoft servers were down - what was the problem?
At the request of SPIEGEL, Microsoft and Deutsche Telekom did not comment on the reasons for the failure or its exact scope.
The provider attaches importance to the fact that the problem was not his fault.
"There was no fault in the Telekom network," explains a spokesman.
Rather, a misconfiguration at another provider caused the outages that lasted several hours.
But what does that mean in concrete terms?
Data packets ended in Bulgaria
The trail leads to Bulgaria, but first of all to Dortmund.
This is where the small provider and network specialist "rrbone" sits, who looks after 1000 corporate customers across Germany specifically for such cases.
When the disturbance spread through the Telekom network on Thursday morning, the systems that constantly monitor the availability of its customers suddenly reported an alarm.
The reason was quickly found: "At 8:33 am, a Bulgarian provider claimed five large Deutsche Telekom networks for itself," says network architect Dominik Bay, managing director of the Dortmund company.
The result: Microsoft services that tried to communicate with the affected IP addresses suddenly no longer sent their data packets to Germany, where they were expected, but to the Bulgarian network.
The connections were then broken.
After a good three hours, the disturbance had dissipated with pleasure.
The Bulgarian provider Telehouse reversed the change and the data traffic was gradually able to resume its normal course.
Telekom support on Twitter advised customers who still had problems to restart their routers.
The incident shows the decentralized nature of the internet.
Because how a data packet can travel across entire continents in milliseconds is the result of complex technology.
The Domain Name System (DNS) regulates the IP address under which a website can be reached.
But how a connection is established is determined by the so-called Border Gateway Protocol (BGP).
This enables providers to find out which route is the fastest way to transport data packets from their network to another network.
These so-called routes are a core technology of the Internet that enables data packets to be sent from one provider to another without having to be directly interconnected.
The data packets find their destination by looking for a route from Internet node to Internet node.
But BGP works to a certain extent on the basis of trust. Each provider announces himself which routes he wants to take over. The neighboring providers carry the information from Internet node to Internet node until the new connection is established. However, if the advertised link is wrong, the consequences can be enormous. In 2008, for example, a Pakistani provider disrupted YouTube traffic worldwide because it wanted to block its own customers' access to the video portal. Experts are talking about »BGP hijacking« - the hijacking of IP addresses. Since then, there have been a handful of such incidents, each time causing major disruptions.
That is why the providers have introduced security mechanisms.
Although each provider can still advertise its own routes, the other providers subject such messages to a plausibility test.
"There are actually filters that are supposed to prevent someone from gaining unauthorized access to such large IP ranges," says network expert Bay.
The German Internet node operator DE-CIX, which is connected to almost 2,300 network operators worldwide, also received the wrong message from Bulgaria.
But the automated filters recognized the obvious error, so that the data traffic continued as usual.
Why the system failed at Microsoft and some other vendors is unclear.
The Bulgarian provider Telehouse, which was the cause of the malfunction, has not yet commented on SPIEGEL's request.
