SYSTEMS STATUS
GeorgiaVIEW QPROD Slowness Issues Reported
Incident Report for USG Services

Incident Summary:
On October 11, 2017, the GeorgiaVIEW Online Learning QPROD environment began experiencing periodic disruptions of service due to high CPU usage and application errors. This negatively impacted performance for USG Institutions accessing their instances of D2L hosted on our server. These disruptions include the following dates and times:

-October 11, between 10:30 a.m. and 12:47 p.m.

-October 18, between 10:32 a.m. and 12:41 p.m.

-October 24, between 10:18 a.m. and 3:52 p.m.

-October 24, between 7:54 p.m. and 11:01 p.m.

-October 25, between 3:02 p.m. and 5:01 p.m.

-October 26, between 10:51 a.m. and 12:46 p.m.

In response to each disruption, ITS and the vendor (D2L) successfully restored the system to normal operation.

Because we recognize that interruptions of GeorgiaVIEW service impact institutions across the state, we are communicating this post-outage analysis of what occurred and the measures being taken to address the factors resulting in this incident.


Incident Cause:
Ongoing, joint investigations between ITS and D2L Support have discovered an application query that occasionally results in a bad query plan being executed. This bad query resulted in significant spikes in CPU utilization which significantly decreased system performance and application errors which impacted system availability.


Incident Response Measures:
On October 26, 2017, D2L instituted a technical fix that ensures a valid query plan remains in place on QPROD. This fix will remain in place as D2L develops a code-based solution to address the underlying issue. This solution will be deployed into production at a future date. Additional information regarding the deployment schedule will be communicated to GeorgiaVIEW customers as they become available.

In addition to working with D2L, we are also reviewing internal ITS processes to improve internal/external communications and reduce incident response times.

Posted about 1 month ago. Oct 31, 2017 - 10:42 EDT

Resolved
After monitoring the situation throughout the afternoon, Information Technology Services (ITS) has confirmed the stability of GeorgiaVIEW QPROD. The system should continue operating normally.

In the event that you experience any further issues with GeorgiaVIEW QPROD, please contact the ITS Helpdesk at helpdesk@usg.edu or 706-583-2001.
Posted 2 months ago. Oct 11, 2017 - 16:47 EDT
Monitoring
Information Technology Services (ITS) has restored full functionality of GeorgiaVIEW QPROD. We will continue to monitor this system for stability until the end of the business day today, October 11, 2017. If you are still experiencing issues with GeorgiaVIEW QPROD, please contact the ITS Helpdesk at helpdesk@usg.edu or 706-583-2001.
Posted 2 months ago. Oct 11, 2017 - 12:47 EDT
Investigating
Information Technology Services (ITS) is currently investigating reports of slowness issues with the GeorgiaVIEW "QPROD" Online Learning Environment. Users at the following Institutions and organizations may be affected:

* Albany State University
* Atlanta Metropolitan State College
* Georgia College and State University
* Georgia Gwinnett College
* Georgia Highlands College
* Gordon State College
* Kennesaw State University
* Savannah State University
* South Georgia State College
* University of Georgia
* USG Training Initiatives
* GeorgiaVIEW Community

Affected users may be experiencing slowness. Additional information will be posted here when the issues are confirmed.
Posted 2 months ago. Oct 11, 2017 - 10:30 EDT
This incident affected: GeorgiaVIEW Online Learning (GeorgiaVIEW Online Learning Environment (QPROD)).