SYSTEMS STATUS
GeorgiaVIEW QPROD Issues Reported
Incident Report for USG Services

Incident Summary:
At approximately 11:47 a.m. on Thursday, November 30th, 2017, the GeorgiaVIEW Online Learning QPROD environment experienced a disruption of service due to blocking. While no outage occurred, this potentially impacted system performance for a small subset of users actively working on quizzes. These issues persisted until approximately 12:45 p.m., when full functionality of the service was restored.

Because we recognize that interruptions of GeorgiaVIEW service impact institutions across the state, we are communicating this post-outage analysis of what occurred and the measures being taken to address the factors resulting in this incident.

Incident Cause:
Upon investigation, ITS technical staff determined that the application code related to quiz copy was causing excessive database blocking. Blocking, a term that describes database processes systematically waiting their turns so as to ensure database consistency as multiple changes are requested at the same time, is completely normal. Excessive blocking, however, which can occur for many reasons including poor application/query design, stale indexes, and unexpected changes in the use of the application, can result in significant performance problems.

Incident Response Measures:
To correct the issue, ITS technical staff identified and terminated the so-called "lead blocker" process. To help prevent this situation from recurring, ITS staff forced a refresh of the statistics relevant to the blocking event and also added these tables to the job that will automatically refresh the same statistics on a more frequent and ongoing basis. (Microsoft SQL Server automatically keeps these internal statistics and uses them to help optimize database activity.)

Posted 3 days ago. Dec 12, 2017 - 16:26 EST

Resolved
After monitoring the situation through the evening, Information Technology Services (ITS) has confirmed the stability of GeorgiaVIEW QPROD. The system should continue operating normally. ITS will provide a postmortem just as soon as all of the details and timelines are gathered.

In the event that you experience any further issues with GeorgiaVIEW QPROD, please contact the ITS Helpdesk at helpdesk@usg.edu or 706-583-2001.
Posted 14 days ago. Dec 01, 2017 - 09:20 EST
Monitoring
Information Technology Services (ITS) has restored full functionality of GeorgiaVIEW QPROD. We will continue to monitor this system for stability through the night. If you are still experiencing issues with GeorgiaVIEW QPROD, please contact the ITS Helpdesk at: helpdesk@usg.edu or 706-583-2001.
Posted 15 days ago. Nov 30, 2017 - 15:15 EST
Investigating
Information Technology Services (ITS) is currently investigating reports of issues with the GeorgiaVIEW QPROD Online Learning Environment. Users at the following Institutions and organizations may be affected:

* Albany State University
* Atlanta Metropolitan State College
* Georgia College and State University
* Georgia Gwinnett College
* Georgia Highlands College
* Gordon State College
* Kennesaw State University
* Savannah State University
* South Georgia State College
* University of Georgia
* USG Training Initiatives
* GeorgiaVIEW Community

Affected users may be experiencing internal and runtime errors. Additional information will be posted here when the issues are confirmed.
Posted 15 days ago. Nov 30, 2017 - 12:20 EST
This incident affected: GeorgiaVIEW Online Learning (GeorgiaVIEW Online Learning Environment (QPROD)).