Incident Summary
At approximately 1:30 p.m. on Wednesday, April 26, 2017, a cut fiber optic cable interrupted internet connectivity for six locations in the Georgia Department of Education K-12 Network (Clarke County, Greene County, Lincoln County, Morgan County, Taliaferro County, and Wilkes County) as well as the Augusta University School of Nursing at Athens (SONAT). Diagnosis and remediation efforts began immediately, with AT&T repair crews restoring internet connectivity to all six DOE locations by 8:50 p.m. that same evening. A separate, unrelated hardware issue at the AT&T Central Office prevented SONAT’s internet connectivity from being restored at that time.
As AT&T technicians worked to replace the affected hardware, ITS technicians restored SONAT’s connectivity at approximately 11:55 a.m. on Thursday, April 27 by rerouting their internet traffic to an alternate PeachNet circuit. By approximately 12:35 p.m. on Friday, April 27, 2017, AT&T technicians completed their repairs and SONAT was restored to its original PeachNet circuit.
Because we recognize that interruptions of service impact institutions across the state, we are communicating this post-outage analysis of what occurred and the measures being taken to address the factors resulting in this incident.
Incident Cause
The root cause of the outage has been identified as a cut aerial fiber cable (in Barrow County, at Bert Day Road and Atlanta Highway).
The SONAT circuit did not recover when AT&T completed initial repairs to the fiber cable. Through additional testing, it was determined there was a faulty optical circuit pack in the AT&T Central Office which may have been due to the fiber cut. Replacement of this circuit pack was delayed as AT&T had it shipped to their Central Office.
Incident Response
AT&T is investigating the root cause of the equipment failure. Once these activities are complete, AT&T will take appropriate steps necessary to increase the reliability of their services. ITS is taking steps to improve the timeliness and quality of information communicated to DOE and PeachNet customers during events that impact their network connectivity.