As the coronavirus places increasing pressure on the internet’s infrastructure, companies that have become essential to people’s daily lives are racing to ensure that the lights stay on.
This involves adapting to changing usage patterns and increased traffic demands. Behind the scenes, it also means keeping services online even as the teams charged with doing so are facing the same mandates to work from home.
In a webinar this week hosted by Kentik, a network analytics company, top managers from Dropbox, Equinix, Netflix and Zoom discussed the challenges they are facing amid a rapidly evolving crisis.
“This is all super fresh for everyone,” said Dave Temkin, vice president of networks at Netflix. “Everyone’s just trying to figure out what the new normal is.”
With the coronavirus still on the rise in most nations, more people are facing mandatory quarantine orders that have closed schools and forced those who can to work from home. People are adapting by implementing distance learning, turning to professional and personal video conference services and apps, and relying on video streaming services for entertainment.
“I had a call with a regulator yesterday from a European country who literally said to me, ‘In an ideal world, every single one of our citizens would be sitting at home on a couch watching Netflix right now,’” said Temkin. “And, you know, obviously, I don’t think that’s something that any government official would have said before the current situation.”
The result is a big shift in usage patterns that is quite different than what many large companies had forecast. While people tend to think of the internet as a single system, it is really a patchwork of datacenters and delivery networks stitched together by a wide range of companies. Far from being virtual, making these physical networks run smoothly means betting on where the traffic demands will originate and to where they will travel in order to strategically place network capacity over time.
Bill Long, senior vice president of product management at datacenter giant Equinix, said the companies have seen spikes in traffic ranging from 10% to 40% depending on the region.
Overall, the news is good. In the short term, many of these companies said they had already built excess capacity while also increasingly automating management of their networks. That has given them headroom to manage their service during a period of wild unpredictability.
“We’ve been architecting it for many years,” said Alex Guerrero, senior manager of SaaS operations at Zoom. “And we want it to be extremely scalable, especially in situations like we’re dealing with right now. So we have automation in place so that we can create quickly scale our infrastructure with very little human intervention.”
Still, this surge has made governments nervous, particularly in Europe, where politicians asked streaming companies like Netflix to help lighten the congestion. Netflix, YouTube, Facebook, Amazon, and others responded by reducing their streaming resolution for the coming weeks.
“We’d certainly been seeing issues in some of the early wave European countries like Italy and Spain, where they had had a shelter in place for lockdown orders earlier than than other places,” said Temkin. “At the end of the day, I think they just had the best interest of their citizens and their networks in mind. No regulator wants to be the one to tell their citizens that they made the decision to make Netflix stop working during a pandemic when everyone is isolated at home. And so all they’re trying to do is just make sure that their country’s networks are stable. And we said, ‘Hey, we feel like we’ve got a few levers we can pull. Let’s pull those levers and let’s go with it.’”
These companies are carefully monitoring a big shift in internet traffic patterns caused by the move toward working at home. For instance, in many cases they had prepared their networks for large amounts of centralized traffic coming from corporate campuses, universities, or office parks. Now that traffic has been dispersed and scattered into residential neighborhoods.
“There will be a different pattern, where we used to have a lot of traffic coming from one university and from thousands of accounts, and now we’ll see a lot of traffic coming from a thousand different places, from many different networks, from the same people,” said Dzmitry Markovich, director of engineering at Dropbox.
Adjusting their networks to adapt to these new patterns presents several challenges. The first is just a question of capacity and flexibility. Netflix has been able to call on its partner Amazon Web Services to allocate greater capacity when needed to help keep its video streaming stable, Temkin said.
At the same time, Netflix has tried to accelerate its own plans to build out parts of its network. Infrastructure roadmaps through the winter holidays are being implemented now. That has caused some crunches because suppliers of network equipment have sent their employees home.
“We have had multiple fires at this point with our supply chain trying to figure out how and where we’re going to get servers to deploy into networks,” Temkin said. “We’ve resolved all of them at this point. We’re fairly comfortable with our supply chain. But as an example, our primary server manufacturer is located in Santa Clara County in California. And so they got a shelter-in-place order and [we] had 24 hours to figure out how to get as many of the boxes out of there as we possibly could.”
The concern is what happens if every company starts rushing to put in place more capacity. Temkin said Netflix is trying to weigh its own needs in terms of equipment and capacity against other more urgent uses to make sure suppliers aren’t put in the uncomfortable position of having to select one client over another.
“We recognize there’s a lot of other things going on that are potentially more important than Netflix, frankly,” Temkin said. “And we want to be sympathetic to those causes and allow people who have those immediate needs, whether it’s e-learning or health care or anything else like that, to have the ability to get the capacity they need. So we don’t rush and try to get it all at the same time and put people in an awkward situation.”
The other riddle that is difficult to solve is whether these usage patterns will be temporary, or whether this crisis will provoke a more permanent shift in working, learning, and entertainment habits. Forecasting when and where to build additional infrastructure is going to be a much more complicated exercise.
“How much of these shifts that we’re seeing now are going to stick and become the new normal?” Long said. “And then how many are going to actually revert to old ways of doing things? I’ve sat online with my kids and they’re doing classes and it works pretty well. So I think there’s gonna be a lot of new patterns that we’re all gonna adjust to and it’s just gonna be interesting to see what stays and how we creatively adapted.”
Work at home
These challenges are arising as many of these companies are also adapting to their own teams working virtually.
“Equinix operates a lot of physical infrastructure, and people have to come into and out of our datacenters all the time,” Long said. “We made the hard decision in Germany, Italy, Spain, and France to restrict access into those datacenters for those exact reasons of health and safety of our employees, our customers, but also the long-term viability of these assets.”
Not surprisingly, adjustment hasn’t been too bad for Zoom’s employees, though even they have been sharing tips on how to work at home.
“Like waking up and getting dressed as you would if you went to the office, stuff like that,” said Guerrero. “It’s just little tips and tricks like that.”
For some, that also means learning to manage work and family demands when spouses and kids are in the house. But for others who live alone, there is also concern about long-term isolation.
“Some people are missing out on social interactions,” said Dropbox’s Markovich. “And for many people to go to working from home and almost zero social interactions, it’s very, very painful. And they require a lot of support.”
At Netflix, the work from home and network management is taking on another dimension as the company tries to figure out how to resume work on some productions. In general, the entertainment industry has ground to a halt. But some projects that were in the post-production phase might still be able to be completed.
But whether employees’ home networks or their networks in their neighborhoods can handle the demands of such large files transfers could pose a problem.
“One of the big challenges we are trying to figure out is what parts of it can we restart,” Temkin said of production efforts. “These might be things like post processing for visual effects, animation, things that traditionally people have not done from home because they require significant amount of compute power and a significant amount of bandwidth to move. We’re undergoing efforts to figure out how can we get that operating out of people’s houses.”