There is a distinct difference between thinking you are ready should a disaster strike and having the confidence of knowing you are prepared because you’ve practiced.
That is exactly why what would normally be an empty parking lot at Chicago’s Soldier Field in mid-May was instead transformed into a bustling network technology compound for one action-packed week of fun, hard work and hands-on learning.
Although our AT&T Network Disaster Recovery (NDR) team holds multiple “dress rehearsals” every year to prepare for disasters– something we’ve done for 23 years in various locations across the U.S., Canada, and Europe, this one took on an added level of awareness for me after being selected to serve as Incident Commander (or IC).
While I tracked many observations during the week, the one that stuck the most aligns with the saying, it takes a village. I know that to be especially true during an NDR exercise or a real-life deployment. In Chicago, we had over 50 people working at this exercise—NDR volunteers and full-time staff each fulfilling different roles.
NDR manages disaster events using an Incident Command System (ICS) that breaks up the responsibilities into different roles or sectors. Depending on the severity of a situation, it may take a handful of people, dozens, or more. When an AT&T Emergency Operations Center sets up for a large hurricane response, for example, it takes on average 70 people working around the clock.
Following many weeks of planning, when the big day arrived and our equipment started rolling into the Soldier Field parking lot, the IC responsibility required my full attention on the team’s immediate needs. My awareness focused on any possible risks to the team’s welfare, such as weather, and risks to the success of the exercise. Next was setting and overseeing each day’s action plan.
The rest of the team had even more work to do. For example:
- Our logistics officer took care of the physical needs of the team and of the site — food, water, sanitation, lodging, fuel for the generators and insurance.
- Safety officers monitored the work site every day. They’re critical during the construction and tear down days. And they handle training at the site.
- Our medical officer monitored the site, prepared to respond if needed, and led the team in first aid and CPR training.
- Our Operations Sector (or Ops team) deals with the heavy lifting – either at an exercise or real-life response. They’re jacks-of-all-trades. First, they’re a construction team—positioning and grounding the trailers, pulling power and communications cables, powering up and turning up the technology recovery trailers. Then they morph into technology teams working inside the recovery trailers on more technical tasks.
In the span of one week, we teamed up to create, test, and break down a working technology compound in an empty parking lot at Soldier Field – it was fast-paced at times, awe inspiring at others and overall just a pretty cool process to lead.
There were great moments throughout the week--from seeing the equipment staged with the Chicago skyline in the background to photographing the finished site from a crane cage.
The best part was time spent working with the team and the chance to keep our recovery processes sharp and watching the last of the trailers leave the site after a safe and successful event.
Kelly Morrison, Network Disaster Recovery (NDR)