Incident Summary:
At approximately 12:50 p.m. on Thursday, December 7th, 2017, the GeorgiaVIEW Online Learning XPROD environment experienced a disruption of service that impacted customers’ ability to login. This persisted until approximately 2:55 p.m., when full functionality of the service was restored.
Because we recognize that interruptions of GeorgiaVIEW service impact institutions across the state, we are communicating this post-outage analysis of what occurred and the measures being taken to address the factors resulting in this incident.
Incident Cause:
Upon investigation, ITS technical staff determined that between 12:50 p.m. and 1:10 p.m. an excessive amount of deadlocks in XPROD negatively impacted overall system performance and caused several application nodes to lose connectivity with SQL servers. Due to a known bug in the D2L Learning Environment, some of these nodes were unable to reestablish that connectivity again.
Incident Response Measures:
To correct the issue, ITS technical staff initiated a full restart on all of XPROD’s application nodes. These restarts were completed at approximately 2:55 p.m.
ITS continues to improve monitoring in order to take more timely actions to minimize impact.