Computerworld

How Cisco's TacOps unit restores communication after natural disasters

How small business units and volunteers re-establish communication infrastructure in areas hit by hurricanes and other natural disasters

Before Ron Snyder joined Cisco's 'Tactical Operations' unit as a solutions architect he was in the US Navy, giving him plenty of experience working in adverse, austere and high-pressure conditions.

Now he guides the 11-person team's technical direction to stand up communications infrastructure in areas hit by natural disasters where it's desperately needed.

The 'TacOps' unit provides secure communications to deliver mission-critical services to first responders and populations in areas afflicted by disaster. It works in the USA and internationally - recently travelling in 'rotations' to the towns, cities and municipalities tragically affected by Hurricane Maria in Puerto Rico in 2017.

NetHope - a coalition of charities and technology providers - spearheaded the response efforts from the technology community, with 70 sites set up by the organisation.

"Tactical Operations was directly involved in the implementation of more than 30 of those sites," says Snyder, speaking with Computerworld UK.

"We teamed up with the core network team, and also industry partners to set up these links - we used satellite communications to bring those up, we worked with local service providers to tap off their cell towers, and provide the last-mile solutions to municipalities and city governments."

This helped the administrations and aid responders to carry out their work in extremely harsh conditions.

Getting to Puerto Rico was a challenge in of itself though.

There were no commercial flights available to reach the island, but one of the advantages of working for an enormous firm like Cisco is the availability of resources.

Cisco chartered private jets from Fort Lauderdale in Florida, whose mission was to evacuate the families of Cisco employees in the capital.

"With the difficulty of our team finding commercial flights, we were able to hitch a ride on the planes as they were rotating in and out of San Juan," says Snyder. "We were able to hitch a ride with our gear and personnel to go to San Juan and start coordinating all of the operations, and get on the ground up and running."

When Snyder landed there was no electricity in the majority of the city. He adds that most of the Cisco employees were staying at the company's office there due to a lack of running water and power - in other words, a "lot of the things we take for granted".

"Just something as simple as bottled water is an important item over there as a means to survive," he says. "It really gave a feel for the type of severity of the disaster that affected Puerto Rico at that time."

The team deployed its gear while it was waiting for a large shipment that NetHope had ordered, which would eventually be transported to 70 of these sites.

Some of it had been donated by service providers from Mexico, such as satellite communications equipment. A unified command post was established at the city's convention centre.

"We started driving out to these sites and just kind of going down the list," he says, "because there's a priority list of sites that needed to be deployed and have connectivity. We ran non-stop for the first two or three weeks going down the list and getting to as many sites as we can."

Difficulties on the ground included unreliable lat-long coordinates in locating buildings like clinics. While these minor setbacks were easily righted by talking to the locals, a more serious issue was the limited connectivity.

"Whenever you go out to these remote sites you lose connectivity," says Snyder.

"There were still some service providers up and running in San Juan... when you go out to the remote sites you don't have anything, so you had to rely on things like satellite phones. It just makes it more difficult to coordinate with the back office that served as our operation centre - back at the Cisco office."

Food, water, web

"You know, we try our best to reassure - that we are restoring a bit of normalcy to a really bad situation," Snyder says. "Whenever our team is called in to help it really means something bad has happened."

By providing much-needed access to the wider world, the team and their partners try to provide "a little bit of reprieve" for the hardships experienced by people in the affected areas.

"It is sometimes a little bit difficult to witness the suffering that is happening," he adds. "But we are there for a job - we are trying to do what we do best, which is set up the networks, get that up and running, and provide some kind of relief.

"Food and water is at the top of the list for the type of relief goods that are needed. Connectivity is close, right behind all of that, because they want to let people know - they want their families to know how everything is going, whether they're OK or not - just giving them the status of how they're doing."

Once the team has stood up its networks it also has to take local technical knowledge into account for when they leave - for instance, during a deployment to the Philippines following Typhoon Haiyan, most local personnel were not familiar with the Cisco IOS syntax.

"When the personnel pulled out of the Philippines it was difficult to walk them through how to check the status of the routers," says Snyder.

"What we did is we started implementing cloud-based routing security. We started using [Cisco's cloud management solution] Meraki, which eliminated the need of knowing really complex command structures with Cisco IOS.

"It streamlined operation with them - a really simple, easy-to-use web-based graphical user interface. The majority of our overseas deployments now are using Meraki cloud-based technology, just because it's easier to support and scale.

"The lessons learned from all of our disaster responses... we talk about some of the things that can be improved on. From these lessons learned that we get from each disaster is really what drives the technical direction of what the team is going to be, for the next iteration of kits that we deploy."

Familiar faces

Not only do the vendors that have their own response teams share a sense of collaboration, there also exists a "brotherhood and sisterhood" within the responders. Because it is a small community, the responders tend to run into each other frequently.

A spokesperson for Cisco confirmed that the company employs a core full-time TacOps team, however much of the work is carried out by volunteer groups from the company around the world.

The time they take to volunteer is in addition to the usual holiday allocated for their day jobs. The spokesperson added that there are "no direct business benefits to Cisco".

British Telecom (BT) is one such other vendor that had a presence in two of the major deployments Snyder had been part of - a "really good response team," he says - in the Philippines and also in Vanuatu following Cyclone Pam.

"It's good to see familiar faces, people that have the same passion, that have the same drive to help people," he says. "You can share information: like where are some of the cities or municipalities that need help in a particular area?

"Having that good relationship established whether that be within government or industry response groups... it's only good.

"It's good for the population that we're serving, because the faster we can get that information out to where help is needed, the more attention would be put on those places, so help can get there faster."

He adds that although the technology industry already does contribute both financially and with technical knowledge, there is still room for improvement, as well as further participation from companies that might not be suited to put boots on the ground.

"From my observation there is definitely a place for industry, for companies, to set up and have a programme such as this," he says.

"There's always an opportunity to be creative and help in times of disasters or crises by these different companies, using their own core competencies.

"There's always more - there's always more that we can do, there's always more that the technology community can do to alleviate some of these burdens that are experienced not only by the people that are affected by crises but also the people that are responding to help them get there faster, get the supplies there faster, and rebuild the community faster using the tools that this industry can provide."