Network outage impacting multiple DOE locations
Incident Report for USG Services

Incident Summary

At approximately 12:30 a.m. on Friday, April 21, 2017, 79 locations in the Georgia Department of Education K-12 Network lost internet connectivity due to an outage at the Digital Realty Data Center in Atlanta. Diagnosis and repairs began immediately, with internet connectivity being restored to the majority of affected locations by 11:45 a.m. that same morning. By approximately 1:30 p.m., only five DOE sites were still affected. By approximately 11:00 a.m. on Saturday, April 22, 2017, the last affected DOE location was brought back online.

Because we recognize that interruptions of service impact institutions across the state, we are communicating this post-outage analysis of what occurred and the measures being taken to address the factors resulting in this incident.


Incident Cause

An AT&T equipment failure caused a power outage at the Digital Realty’s Data Center (56 Marietta Street) in downtown Atlanta, where multiple network providers such as AT&T, Google, Sprint, etc. co-locate to connect with each other. Because ITS contracts with AT&T to provide network circuits to individual DOE school districts, the PeachNet DOE circuits housed at the facility were affected by the power outage.

Incident Response

AT&T is investigating the root cause of the equipment failure and continues to repair and replace equipment affected by the outage. Once these activities are complete, AT&T will take appropriate steps necessary to increase the reliability of their services. ITS is taking steps to improve the timeliness and quality of information communicated to DOE K-12 customers during events that impact their network connectivity.

Posted 3 days ago. Apr 24, 2017 - 16:33 EDT

Resolved
ITS has determined that the following schools continue to have Internet problems; Cartersville City, Fayette County, Rockdale County and Putnam County. If your school is not included in the list of schools and are still experiencing issues, please contact the ITS Helpdesk at helpdesk@usg.edu or 706.583.2001.
Posted 6 days ago. Apr 21, 2017 - 12:08 EDT
Monitoring
ITS has determined that the AT&T power issue has been resolved. There are a few sites that are still experiencing problems, ITS will work with them individually. Please contact us if you have questions, we will continue monitoring throughout the business day.
Posted 6 days ago. Apr 21, 2017 - 11:48 EDT
Update
AT&T has identified some rectifiers that need to be replaced. We do not have an estimated time of repair (ETR) at this time. ITS will provide additional updates as they become available.
Posted 6 days ago. Apr 21, 2017 - 09:39 EDT
Update
Repairs are underway. Many sites are back up. AT&T continues to work to restore service to the remaining sites. ITS will provide additional updates as they become available.
Posted 6 days ago. Apr 21, 2017 - 09:23 EDT
Update
AT&T has identified a power outage at the co-location where the Bellsouth circuits terminate and continues to work on the issue. ITS will provide additional updates as they become available.
Posted 6 days ago. Apr 21, 2017 - 08:27 EDT
Identified
ITS has been notified that a large number of schools in the North Georgia area have lost access to the Internet. This issue is related to an AT&T circuit outage in the area. AT&T has been notified and is working on the issue. The following schools are affected;

Atlanta Area School for the deaf
Baker County Board of Education
Bartow County Board of Education
Bremen City Board of Education
Brookhaven Innovation Academy
Buford City Board of Education
Butts County Board of Education
Calhoun City Board of Education
Candler County Board of Education
Carroll County Board of Education
Carrollton City Board of Education
Cartersville City Board of Education
Charter Conserv. For Liberal Arts & Tech
Chattahoochee County Board of Education
Cherokee Charter Academy
Cirrus Charter Academy
Clarke County Board of Education
Clayton County Board of Education
Cobb County Board of Education
Colquitt County Board of Education
Coweta Charter Academy
Coweta County Board of Education
Decatur City Board of Education
DeKalb County School System
DOE HQ
Douglas County Board of Education
DuBois Integrity Academy
Elbert County Board of Education
Fayette County Schools
Floyd County Board of Education
Foothills Charter High School (Madison)
Forsyth County Board of Education
Fulton County Board of Education
Fulton Leadership Academy
Gainesville City Board of Education
Georgia School for Innovation and the Classics
Georgia School for the Deaf
Gilmer County Board of Education
Gordon County Board of Education
Greene County Board of Education
Griffin-Spalding County Board of Education
Gwinnett County Board of Education
Habersham Board of Education
Hall County Board of Education
Haralson County Board of Education
Harris County Board of Education
Hart County Board of Education
Heard County Board of Education
Henry County Board of Education
International Charter School of Atlanta
Ivy Prep Academy at Gwinnett
Ivy Prep Academy at Kirkwood
Jasper County Board of Education
Jones County Board of Education
Lamar County Board of Education
Liberty Tech Charter School
Lincoln County Board of Education
Marietta City Schools
Meriwether County Board of Education
Mitchell County Board of Education
Morgan County Board of Education
Muscogee County Board of Education
Newton County Board of Education
Oconee County Board of Education
Odyssey Charter School
Paulding County Board of Education
Pike County Board of Education
Polk County Board of Education
Putnam County Board of Education
QTS (Data Center)
Rockdale County Public Schools
Rome City Board of Education
Scintilla Charter School
Social Circle Board of Education
Talbot County Board of Education
Taliaferro County Board of Education
Troup County Board of Education
Utopian Academy
Wilkes County Board of Education
Posted 7 days ago. Apr 21, 2017 - 07:58 EDT