SYSTEMS STATUS
Network Outage Affecting Multiple DOE Locations
Incident Report for USG Services
Postmortem

Incident Summary

At approximately 1:30 p.m. on Wednesday, April 26, 2017, a cut fiber optic cable interrupted internet connectivity for six locations in the Georgia Department of Education K-12 Network (Clarke County, Greene County, Lincoln County, Morgan County, Taliaferro County, and Wilkes County) as well as the Augusta University School of Nursing at Athens (SONAT). Diagnosis and remediation efforts began immediately, with AT&T repair crews restoring internet connectivity to all six DOE locations by 8:50 p.m. that same evening. A separate, unrelated hardware issue at the AT&T Central Office prevented SONAT’s internet connectivity from being restored at that time.

As AT&T technicians worked to replace the affected hardware, ITS technicians restored SONAT’s connectivity at approximately 11:55 a.m. on Thursday, April 27 by rerouting their internet traffic to an alternate PeachNet circuit. By approximately 12:35 p.m. on Friday, April 27, 2017, AT&T technicians completed their repairs and SONAT was restored to its original PeachNet circuit.

Because we recognize that interruptions of service impact institutions across the state, we are communicating this post-outage analysis of what occurred and the measures being taken to address the factors resulting in this incident.

Incident Cause

The root cause of the outage has been identified as a cut aerial fiber cable (in Barrow County, at Bert Day Road and Atlanta Highway).

The SONAT circuit did not recover when AT&T completed initial repairs to the fiber cable. Through additional testing, it was determined there was a faulty optical circuit pack in the AT&T Central Office which may have been due to the fiber cut. Replacement of this circuit pack was delayed as AT&T had it shipped to their Central Office.

Incident Response

AT&T is investigating the root cause of the equipment failure. Once these activities are complete, AT&T will take appropriate steps necessary to increase the reliability of their services. ITS is taking steps to improve the timeliness and quality of information communicated to DOE and PeachNet customers during events that impact their network connectivity.

Posted Apr 28, 2017 - 16:31 EDT

Resolved
ITS has confirmed that The Augusta University School of Nursing at Athens (SONAT) has been restored to its preferred PeachNet circuit. SONAT internet connectivity is functioning normally at this time. No further interruptions of service are anticipated.

In the event that you experience any further issues with internet connectivity through the Ga DOE K-12 or PeachNet Networks, please contact the ITS Helpdesk at helpdesk@usg.edu or 706-583-2001.
Posted Apr 28, 2017 - 12:44 EDT
Update
AT&T has installed replacement hardware in their Central Office, but are unable to restore SONAT's regular connection. AT&T technicians continue to troubleshoot the situation and work towards a permanent solution.

In the meantime, ITS will continue to keep its workaround in place to ensure SONAT retains internet connectivity.

Additional information will be provided here as it is made available.
Posted Apr 27, 2017 - 18:05 EDT
Monitoring
ITS technicians have successfully restored internet connectivity to SONAT by redirecting the location to an alternate PeachNet circuit. This is a temporary workaround. ITS will work with SONAT and AT&T to move back to the primary circuit when repairs are complete.

Additional information will be provided here as it becomes available.
Posted Apr 27, 2017 - 11:54 EDT
Update
AT&T has indicated that replacement hardware needed to restore internet service to SONAT will arrive at their Central Office by noon today. They will begin installation shortly thereafter.

There is currently no estimated time to restore service.

Additional information will be provided here as it becomes available.
Posted Apr 27, 2017 - 10:54 EDT
Identified
AT&T has identified a hardware issue in their Central Office that is affecting internet connectivity at SONAT. AT&T technicians are currently working to address the issue.

There is currently no estimated time to restore service.

Additional information will be provided here as it becomes available.
Posted Apr 27, 2017 - 10:13 EDT
Update
As of 6:30 a.m., AT&T technicians are still working to restore internet connectivity to SONAT.

There is currently no estimated time to restore service.

Additional information will be provided here as it becomes available.
Posted Apr 27, 2017 - 06:44 EDT
Update
ITS has confirmed that all affected DOE sites have stable internet connections and are functioning normally.

ITS continues to work with AT&T technicians to resolve ongoing internet connectivity issues with SONAT.

This incident will remain open and continue to be updated until SONAT's connectivity is fully restored.
Posted Apr 26, 2017 - 22:44 EDT
Monitoring
Internet connectivity has been restored to all DOE sites, but fiber repair work is ongoing. It is possible that affected sites could experience internet connectivity issues until the work is complete.

AT&T estimates that repairs will be complete by approximately 9:00p.m.

The Augusta University School of Nursing at Athens (SONAT) continues to experience issues.

Additional information will be provided here as it becomes available.
Posted Apr 26, 2017 - 20:52 EDT
Update
ITS has confirmed that as of 6:11 p.m. an AT&T construction crew is onsite and repairing the damage.

There is currently no estimated time to restore service.

Additional information will be provided here as it becomes available.
Posted Apr 26, 2017 - 19:14 EDT
Update
ITS has confirmed that this cable cut is also affecting The Augusta University School of Nursing at Athens (SONAT).

Additional information will be provided here as it becomes available.
Posted Apr 26, 2017 - 17:10 EDT
Identified
AT&T has confirmed that an aerial cable cut (at Bert Day Road and Atlanta Highway) is responsible for the service interruption. AT&T crews are on-site and assessing the damage.

There is currently no estimated time to restore service.

Additional information will be provided here as it becomes available.
Posted Apr 26, 2017 - 17:04 EDT
Update
ITS has confirmed that AT&T’s investigation is ongoing. An AT&T Critical Issue Resolution Manager has been assigned. The issue has been escalated and technicians have been engaged.

No estimated uptime has been provided at this time.

Additional information will be provided here as it becomes available.
Posted Apr 26, 2017 - 15:42 EDT
Investigating
ITS has been notified that a number of K-12 schools in Northeast Georgia have lost access to the Internet. AT&T has been notified and is working on the issue. The following counties are affected:

Clarke County
Greene County
Lincoln County
Morgan County
Taliaferro County
Wilkes County

Additional information will be provided here as it becomes available.
Posted Apr 26, 2017 - 13:43 EDT
This incident affected: GA DOE K-12 Network and PeachNet Network (Augusta University).