2 Savvy Employees "MacGyver" Emergency Flood Protection

By Lauren Garner, AT&T Insider

Eric Hendricks, network process and quality area manager at AT&T, is no stranger to flooding in his neighborhood southwest of St. Louis. There are 3 major rivers in the area. In 2008, the water rose to 38 feet. Then in 2015 during the 100-year flood, the water rose to 44 feet and they lost electricity, the Ethernet switch and all of the equipment providing our network services to the neighborhood.

"I was on vacation and out of town that week," Eric said. "I learned my neighborhood had turned into an island from a Facebook site my neighbors set up for the flood. They were complaining about AT&T service and our response. We lost some customers that year."

Another flood predicted

When rain started accumulating again this year, the National Weather Service predicted the same flood levels as 2015. Eric reached out to his colleagues for help.

First, he called a colleague and neighbor 3 doors down, Mike Brennecke, senior-equipment design engineer at AT&T, to get construction and engineering contractor contacts. His calls brought the contractors out with sandbags and plastic sheeting to surround the switch for protection and a pump for the rising water. His colleagues also brought another pump and some wood to fortify the barrier.

After a day, the water levels were rising and the contractors had to get out with their vehicles to avoid being stranded.

Eric and Mike realized the sand bags were not maintaining a barrier around the equipment. They asked permission to go through neighbors' backyards since the site was no longer accessible by road. They made their way through the woods and down a steep hill to the box.

"I couldn't live with another outage …"

"We knew we needed to save the equipment," Eric said. "Our neighbors were starting to chatter on Facebook again. I couldn't live with another outage and lose them as customers."

"We were working remotely, so we wanted to keep the internet running," Mike said. "It was like a neighborhood snow day with everyone home. They were counting on us to keep their TV, internet and wireless services running."

Mike added his pool pump to get the water out of the area. But the circuit couldn't handle all 3 pumps at once. The water was coming in faster than they could pump it out. If they only used 2 pumps, they'd lose the battle.

At this point, the neighborhood was surrounded by water like an island. Eric and Mike found another outlet to a different breaker in the cell tower hut. They borrowed power cords, a wet vacuum from their neighbors and another pump was brought in by boat from a contractor.

Practically swimming on the job

"We kept rearranging the sand bags to stop the water leaks," Mike said. "The rain started up again and we wondered if we would ever get the water levels back down."

"I can't tell you how many hundreds of times we filled up the wet vac and dumped it," Eric said. "It would fill up in 30 seconds, I'd dump it and start over again. This went on for a few hours. We were out there most of the night with a lantern we had hauled in for lighting. We were exhausted."

The water rose to 43 feet. It kept penetrating the sand bags and finding new ways in. Eric and Mike kept plugging the holes.

"We didn't know where the water was coming from," Eric said. "Then we saw it was coming through the cracks in the floor on the concrete pad. We were standing in water up to our knees. We wore our waterproof work boots. Mike made his own hip waders.

"We wanted to be safe and not get electrocuted. We made a rule: no one comes alone to the site."

At 12:30 a.m., Eric and Mike were able to slow down and maintain the water levels. They took a break and agreed to meet in a few hours.

"We saved it!"

"It was a relief to come back and see the equipment was still OK," Eric said. "It was the best feeling ever. We saved it!"

"I couldn't believe it," Mike said. "We weren't sure it would make it through the night. Our neighbors speculated that AT&T might have another service outage like the prior flood."

The water was still rising and crested around 2:00 p.m. Mike and Eric continued their efforts until the water began to recede and all was safe.

"It's about the customer base," said Eric. "Losing assets is one thing. If you lose customers, they don't come back."

Plus, he says, "It beats sitting at a desk on a conference call all day. It's great to get back in the field and see my accomplishments right in front of me."