GeorgiaVIEW QPROD Issues Reported

Incident Report for USG Services

Postmortem

Incident Summary

At approximately 11:15 a.m. on Thursday, March 10, 2016, Information Technology Services’ (ITS) QPROD server experienced a significant decrease in performance. This intermittently prevented USG Institutions from accessing their instances of D2L hosted on the server. These issues persisted up until approximately 11:30 p.m., when full functionality was restored.

Because we recognize that interruptions of service impact institutions across the state, we are communicating this post-outage analysis of what occurred and the measures being taken to address the factors resulting in this incident.

Incident Cause

The CPU on QPROD was running at elevated levels for the duration of this incident and negatively impacting service performance. The root cause was identified as a caching service that was not working properly. To restore functionality of the service, ITS performed a primary database failover. D2L and ITS are continuing to analyze the cause of the caching service's failure.

Incident Response Measures

In addition to restoring the service, ITS is working with D2L to take further steps to improve troubleshooting capabilities, improve overall situational awareness, and reduce incident response times.

Posted Mar 16, 2016 - 08:52 EDT

Resolved

The emergency QPROD maintenance was successfully completed.

ITS can confirm that QPROD is functioning normally at this time.

In the event that you experience any further issues with QPROD, please contact the ITS Helpdesk at helpdesk@usg.edu or 706-583-2001.

Posted Mar 10, 2016 - 23:29 EST

Update

[CORRECTION] ITS will be taking QPROD offline for emergency maintenance from approximately 10:35p.m. Thursday night to 12:30a.m. Friday morning. The system will be unavailable during this time.

Additional information will be posted here once the emergency maintenance is completed. Thank you for your continued patience.

Posted Mar 10, 2016 - 23:01 EST

Identified

ITS will be taking QPROD offline for emergency maintenance from approximately 10:35p.m. Friday night to 12:30a.m. Saturday morning. The system will be unavailable during this time.

Additional information will be posted here once the emergency maintenance is completed. Thank you for your continued patience.

Posted Mar 10, 2016 - 22:38 EST

Update

ITS is cooperating with D2L support staff in their continued investigation into ongoing issues with QPROD. Affected users may be experiencing performance and accessibility issues. Additional information will be posted here when it becomes available.

Posted Mar 10, 2016 - 16:05 EST

Investigating

ITS is currently investigating reports of issues with the GeorgiaVIEW "QPROD" Online Learning Environment. Users at the following Institutions and organizations may be affected:

Albany State University
Atlanta Metropolitan State College
Darton State College
Georgia College and State University
Georgia Gwinnett College
Georgia Highlands College
Georgia Perimeter College
Gordon State College
Kennesaw State University
Savannah State University
South Georgia State College
University of Georgia
USG Training Initiatives
GeorgiaVIEW Community
GeorgiaVIEW Functional Evaluation

Affected users may be experiencing performance issues. Additional information will be posted here when the issues are confirmed.

Posted Mar 10, 2016 - 11:40 EST

This incident affected: GeorgiaVIEW Online Learning (GeorgiaVIEW Online Learning Environment (QPROD)).