SYSTEMS STATUS
UGA Power Outage Affecting Multiple ITS Systems
Incident Report for USG Services
Postmortem

Incident Summary:
At approximately 5:35 a.m. on Friday, February 10, 2017, the University of Georgia experienced a power event that interrupted Georgia Power’s electrical feed to a portion of campus that included the UGA Boyd Data Center. The data center’s backup generator was engaged, but it failed several minutes later. The data center's uninterruptible power supplies (UPS) continued to provide power for approximately 30 minutes until their batteries were exhausted. The resulting loss of backup power impacted multiple systems, including ITS’ hosted instances of the GeorgiaVIEW Online Learning QPROD environment and SharePoint.

Georgia Power restored utility power to the data center by approximately 7:25 a.m. At this time, technical staff began bringing the affected systems back online. Due to a failure in the UPS environment, power wasn't immediately available to some of the systems including networking equipment required by the GeorgiaVIEW QPROD service and SharePoint. Service for USO SharePoint was restored at approximately 9:50 a.m. Service for QPROD was restored at approximately 11:00 a.m.

Because we recognize that interruptions of service impact institutions across the state, we are communicating this post-outage analysis of what occurred and the measures being taken to address the factors resulting in this incident.


Incident Cause:
The initial loss of power within the Boyd Data Center was caused by the failure of the data center's backup generator after the loss of Georgia Power utility power.


Incident Response Measures:
UGA and their generator support vendor are investigating the root cause of the generator's failure. Based on this research, UGA plans to take any appropriate steps necessary to increase the reliability of the generator. ITS is taking steps to improve the timeliness and quality of information communicated to our customers during events that impact the availability of our systems.

Posted Feb 16, 2017 - 09:35 EST

Resolved
Information Technology Services (ITS) has been monitoring the power issue at UGA’s Boyd Data Center which impacted GeorgiaVIEW QPROD and USO SharePoint. Power has remained stable throughout the afternoon without further interruption of any ITS services.
If you are still experiencing issues, please contact the ITS Helpdesk at helpdesk@usg.edu or 706-583-2001.
Posted Feb 10, 2017 - 16:31 EST
Monitoring
ITS has restored full functionality of GeorgiaVIEW QPROD We will continue to monitor this system for stability for the remainder of the day. If you are still experiencing issues with GeorgiaVIEW QPROD, please contact the ITS Helpdesk at helpdesk@usg.edu or 706-583-2001.
Posted Feb 10, 2017 - 10:56 EST
Update
Power at UGA's Boyd Data Center has been restored. As of 9:50AM (ET) access to USO SharePoint has been restored. GeorgiaVIEW QPROD is still unavailable. We will provide further updates regarding QPROD as they become available.
Posted Feb 10, 2017 - 10:56 EST
Update
Information Technology Services (ITS) has confirmed that a power outage at UGA's Boyd Data Center is affecting the availability of several ITS systems, including USO SharePoint and the GeorgiaVIEW "QPROD" Online Learning Environment. Users at the following Institutions and organizations may be impacted:

Albany State University
Atlanta Metropolitan State College
Darton State College
Georgia College and State University
Georgia Gwinnett College
Georgia Highlands College
Georgia Perimeter College
Gordon State College
Kennesaw State University
Savannah State University
South Georgia State College
University of Georgia
USG Training Initiatives
GeorgiaVIEW Community
GeorgiaVIEW Functional Evaluation

Additional information will be posted here when it becomes available.
Posted Feb 10, 2017 - 09:13 EST
Identified
Information Technology Services (ITS) has confirmed that a power outage at UGA's Boyd Data Center is affecting the availability of several ITS systems, including USO SharePoint and the GeorgiaVIEW "QPROD" Online Learning Environment. Users at the following Institutions and organizations may be impacted:
Albany State University
Atlanta Metropolitan State College
Augusta University - School of Nursing at Athens
Darton State College
Georgia College and State University
Georgia Gwinnett College
Georgia Highlands College
Georgia Perimeter College
Gordon State College
Kennesaw State University
Savannah State University
South Georgia State College
University of Georgia
USG Training Initiatives
GeorgiaVIEW Community
GeorgiaVIEW Functional Evaluation

Additional information will be posted here when it becomes available.
Posted Feb 10, 2017 - 08:02 EST
This incident affected: GeorgiaVIEW Online Learning (GeorgiaVIEW Online Learning Environment (QPROD)) and Internal USO Services (USO SharePoint / OneDrive).