Network Outage - Multiple Campus Outage
Incident
Report for USG Services
Resolved
After monitoring the situation, Information Technology Services (ITS) has confirmed the stability of network access at the affected sites. The system should continue operating normally.
In the event that you experience any further issues with network access, please contact the ITS Helpdesk at helpdesk@usg.edu or 706-583-2001.
In the event that you experience any further issues with network access, please contact the ITS Helpdesk at helpdesk@usg.edu or 706-583-2001.
Monitoring
All locations are currently passing traffic, though some are running in protect. Efforts are ongoing to restore service to normal functionality. We will continue to monitor the situation until all sites are restored.
Update
The cut has been repaired and Abraham Baldwin Agricultural College (ABAC) is back online.
Update
Abraham Baldwin Agricultural College (ABAC) is down due to a new cut along the CRISP-FITZ path. Accelecom is investigating. This is ABAC’s primary path. Their backup path is affected by the WRENS-QAUG cut.
Update
Valdosta State University (VAL) is currently up, although the fiber is still down.
The cut from WREN to QAUG is still being worked.
The cut from WREN to QAUG is still being worked.
Update
An AT&T outage has brought down the backup path for Valdosta State University (VAL), bringing the site offline given the ongoing fiber issues with the primary path. We’re working with the provider to resolve the issue.
Update
Georgia Southern University - Armstrong Campus (ARM2) - Site is currently online. Power levels look at or better than prior to the break.
Update
We are continuing to work on a fix for this issue.
Update
Georgia Southern University - Armstrong Campus (ARM) - The QSAV to SAV span has gone from degraded to fiber cut and is taking the protect path for Georgia Southern University - Armstrong Campus down and therefore campus offline. Accelecom has been informed and they are investigating.
Valdosta State University - VAL-ADEL - As long as the power company allows, Accelecom hopes to have someone starting on repairs in the area tomorrow morning. There are 2-3 miles of severely damaged fiber in Valdosta, and the damage is in multiple areas. Fiber repairs will likely take until sometime next week.
Valdosta State University - VAL-ADEL - As long as the power company allows, Accelecom hopes to have someone starting on repairs in the area tomorrow morning. There are 2-3 miles of severely damaged fiber in Valdosta, and the damage is in multiple areas. Fiber repairs will likely take until sometime next week.
Update
The bypass at AUG/Summerville campus has been removed. Augusta Cybersecurity Initiative (AUG.RTR/GRUSEC) is back online.
At this point, all campuses' PeachNet gear is currently online. Several are running in protect due to the remaining fiber cuts, a few still have their local gear powered down, and some are on generator power.
At this point, all campuses' PeachNet gear is currently online. Several are running in protect due to the remaining fiber cuts, a few still have their local gear powered down, and some are on generator power.
Update
Correction: South Georgia State Collage (SGA) is running on generator power.
Update
Fiber cuts:
WREN-QAUG – Fiber is still broken and may be so for days.
VAL-ADEL – Fiber is still broken and may be so for days.
All other cuts have been repaired. Some repairs are temporary and at risk for further damage.
Sites:
East Georgia State College (EGA) – Power has been restored to parts of campus which brings the southern path back online. Campus itself still has their gear powered down, but the PeachNet rack is operational.
Georgia Southern University Statesboro (GASO1), GASO2, South Georgia State College (SGA), WAY – Now that power has been restored to SGA, EGA, and WAY, as well as the WAY-SGA fiber cut repaired, all of these campuses can now use the southern route through north Florida to get traffic to our core. WAY is on generator power, but they believe it will hold.
Valdosta State University (VAL) – Still on the metro-e.
Augusta University (AUG)/GRUSEC – Power has been restored, but GRUSEC will not come online until we undo the optical fix that we put in place to bring up AU. That will take place tonight.
College of Coastal Georgia (BRU2) – This path came up when WAY had power restored.
WREN-QAUG – Fiber is still broken and may be so for days.
VAL-ADEL – Fiber is still broken and may be so for days.
All other cuts have been repaired. Some repairs are temporary and at risk for further damage.
Sites:
East Georgia State College (EGA) – Power has been restored to parts of campus which brings the southern path back online. Campus itself still has their gear powered down, but the PeachNet rack is operational.
Georgia Southern University Statesboro (GASO1), GASO2, South Georgia State College (SGA), WAY – Now that power has been restored to SGA, EGA, and WAY, as well as the WAY-SGA fiber cut repaired, all of these campuses can now use the southern route through north Florida to get traffic to our core. WAY is on generator power, but they believe it will hold.
Valdosta State University (VAL) – Still on the metro-e.
Augusta University (AUG)/GRUSEC – Power has been restored, but GRUSEC will not come online until we undo the optical fix that we put in place to bring up AU. That will take place tonight.
College of Coastal Georgia (BRU2) – This path came up when WAY had power restored.
Update
As of 4:30 p.m. Monday, AGA is still working on clearing hurricane debris and repairing pole lines in the area related to the Wrens to Augusta fiber cut. They are making progress and field crews are on site re-splicing fibers. No ETR at this time.
Update
SGA - South Georgia State College (Douglas) has come back online. The WAY-SGA fiber appears repaired, but we have not received a hands-off from the vendor at this point.
Update
Alma Telephone has found the fiber cut between Waycross (WAY) and Douglas (SGA) and has crews on the way with an ETA of 11:30 a.m. The current ETR is 3:00 p.m.
Update
Fiber Cuts:
VAL-ADEL -- Still waiting on power company.
WREN-QAUG -- Still waiting on power company.
WAY-SGA -- Cut as of 5:41 p.m. on 9/29. Blocks south path for GASO1, GASO2, EGA, and SGA.
Sites:
South Georgia State College (SGA) -- Has power, but there is a new cut between Waycross and Douglas. ATC is en route to investigate.
Augusta Cybersecurity Initiative (AUG/GRUSEC) -- Power Outage
Power has been restored to AUG but two optical shelves remain unequipped. Communications Consulting Group {CCG} is working with Augusta University (AU) to schedule reseating/replacing the affected components and normalizing traffic through AUG in order to bring GRUSEC back online.
Georgia Southern University Statesboro (GASO1, GASO2) -- On alternate ISP. PeachNet access depends on repairs to either WREN-QAUG path or power at EGA and repairs to WAY-SGA path.
East Georgia State College (EGA) -- No power, but would be down due to cuts at WAY and WREN.
Valdosta State University (VAL) -- Still on backup metro-e. Primary path depends on VAL-ADEL fiber cut.
Augusta University (MCGN, MCGA) -- Previously reported optical fix still in place.
Apogee services to Augusta are down. -- PeachNet access depends on repairs to either WREN-QAUG path or power at AUG.
VAL-ADEL -- Still waiting on power company.
WREN-QAUG -- Still waiting on power company.
WAY-SGA -- Cut as of 5:41 p.m. on 9/29. Blocks south path for GASO1, GASO2, EGA, and SGA.
Sites:
South Georgia State College (SGA) -- Has power, but there is a new cut between Waycross and Douglas. ATC is en route to investigate.
Augusta Cybersecurity Initiative (AUG/GRUSEC) -- Power Outage
Power has been restored to AUG but two optical shelves remain unequipped. Communications Consulting Group {CCG} is working with Augusta University (AU) to schedule reseating/replacing the affected components and normalizing traffic through AUG in order to bring GRUSEC back online.
Georgia Southern University Statesboro (GASO1, GASO2) -- On alternate ISP. PeachNet access depends on repairs to either WREN-QAUG path or power at EGA and repairs to WAY-SGA path.
East Georgia State College (EGA) -- No power, but would be down due to cuts at WAY and WREN.
Valdosta State University (VAL) -- Still on backup metro-e. Primary path depends on VAL-ADEL fiber cut.
Augusta University (MCGN, MCGA) -- Previously reported optical fix still in place.
Apogee services to Augusta are down. -- PeachNet access depends on repairs to either WREN-QAUG path or power at AUG.
Update
SGA - South Georgia State College (Douglas) is down. The site appears to be down due to loss of power.
Update
Power has been restored at South Georgia State College - Waycross Campus (WAY). Once the power was restored, Waycross Campus and South Georgia State College (SGA) came back online.
Update
Power has been restored to the Fitzgerald hut. Abraham Baldwin Agricultural College (ABAC) is back online.
Update
Communications Consulting Group (CCG) is reporting an issue with the generator and UPS has been fixed. Power is restored to our equipment. Georgia Southern University - Armstrong Campus (ARM) and Savannah State University (SAV) are back online.
Update
It appears that the Fitzgerald hut has lost power. This has caused Abraham Baldwin Agricultural College (ABAC) to go down.
Communications Consulting Group (CCG) has contacted AGA and they are investigating.
Communications Consulting Group (CCG) has contacted AGA and they are investigating.
Update
There is a fiber cut between QSAV and ARM. This (and the fiber cut in Augusta) is causing Georgia Southern University - Armstrong Campus (ARM) and Savannah State University (SAV) to be down. Communications Consulting Group (CCG) is reporting this fiber break to AGA.
We suspect that this is a power outage at Savannah State University because both spans are down. There is a fold, so a double cut is possible, but a power outage is more probable. This also takes down the Apogee service to Georgia Southern’s Armstrong campus. Campus and Apogee services to the Armstrong campus will return either when Savannah State’s power returns or the issues between Sandersville-Wrens-Augusta are resolved.
We suspect that this is a power outage at Savannah State University because both spans are down. There is a fold, so a double cut is possible, but a power outage is more probable. This also takes down the Apogee service to Georgia Southern’s Armstrong campus. Campus and Apogee services to the Armstrong campus will return either when Savannah State’s power returns or the issues between Sandersville-Wrens-Augusta are resolved.
Update
The two fibers have been repaired and Abraham Baldwin Agricultural College (ABAC) is back online.
Update
Fiber cuts:
* CRISP-FITZ fiber cut - Mediacom and Accelecom are working on restoration. In the area, we have multiple cuts across fiber belonging to multiple vendors. No ETR.
* ADEL-VAL fiber cut - The crew in the FITZ area will head to the Valdosta area after repairs are completed in Fitzgerald. No ETR.
* SSC - WREN - QAUG fiber cuts - WoW is working to restore the WREN-QAUG fiber and Accelecom has 3 technicians working on the Sandersville to Wren span. No ETR.
Campus Power Outages:
* SGA - South Georgia State College (Douglas)
* WAY - South Georgia State College (Waycross) - Last report from campus (late Friday afternoon) was that they expected power at both sites to be down for days.
* EGA - East Georgia College - Reports from Friday morning indicated a power outage at campus. We saw them come up briefly Friday, but our gear indicates power failed again last night -7:15pm.
* AUG/GRUSEC - Campus confirmed a power outage, although we were able to get to one of our shelves. Possibly a partial power outage or breaker.
Other affected Sites:
* VAL - Power has been restored and they are online via our backup metro-e. Their primary fiber path is still down due to the ADEL-VAL cut mentioned above.
* GASO1, GASO2 - They are up on their backup ISP although with limited functionality (no VPN access). They should come online when either the Waycross power outage or the SSC-WREN-QAUG cuts are fixed.
* MCGN, MCGA - The Augusta University primary paths go through the SSC-WREN-QAUG path and are down due to the cut. The backup path for MCGN and MCGA both go through AUG, which has a power outage. We were able to make some modifications to the optical gear to route a wave away from the shelves without power at AUG and bring MCGN and AUG back online. This will need to be undone once things normalize, but they are online for the time being.
* ABAC - Abraham Baldwin Agricultural College is down due to the combination of the CRISP-FITZ cut and SSC-WREN-QAUG cut. Once either one of them is repaired, ABAC should come back online.
* BRU2 - The College of Coastal Georgia is currently online via our metro-e backup circuit. Their primary fiber path is down due to the power outage at Waycross.
* Apogee services - Apogee services to the College of Coastal Georgia and to Abraham Baldwin Agricultural College are down and will come back online when a fiber path to those campuses are restored.
If you have reason to believe that your campus is listed incorrectly (i.e. you do have power), please call our HelpDesk at 706-583-2001.
* CRISP-FITZ fiber cut - Mediacom and Accelecom are working on restoration. In the area, we have multiple cuts across fiber belonging to multiple vendors. No ETR.
* ADEL-VAL fiber cut - The crew in the FITZ area will head to the Valdosta area after repairs are completed in Fitzgerald. No ETR.
* SSC - WREN - QAUG fiber cuts - WoW is working to restore the WREN-QAUG fiber and Accelecom has 3 technicians working on the Sandersville to Wren span. No ETR.
Campus Power Outages:
* SGA - South Georgia State College (Douglas)
* WAY - South Georgia State College (Waycross) - Last report from campus (late Friday afternoon) was that they expected power at both sites to be down for days.
* EGA - East Georgia College - Reports from Friday morning indicated a power outage at campus. We saw them come up briefly Friday, but our gear indicates power failed again last night -7:15pm.
* AUG/GRUSEC - Campus confirmed a power outage, although we were able to get to one of our shelves. Possibly a partial power outage or breaker.
Other affected Sites:
* VAL - Power has been restored and they are online via our backup metro-e. Their primary fiber path is still down due to the ADEL-VAL cut mentioned above.
* GASO1, GASO2 - They are up on their backup ISP although with limited functionality (no VPN access). They should come online when either the Waycross power outage or the SSC-WREN-QAUG cuts are fixed.
* MCGN, MCGA - The Augusta University primary paths go through the SSC-WREN-QAUG path and are down due to the cut. The backup path for MCGN and MCGA both go through AUG, which has a power outage. We were able to make some modifications to the optical gear to route a wave away from the shelves without power at AUG and bring MCGN and AUG back online. This will need to be undone once things normalize, but they are online for the time being.
* ABAC - Abraham Baldwin Agricultural College is down due to the combination of the CRISP-FITZ cut and SSC-WREN-QAUG cut. Once either one of them is repaired, ABAC should come back online.
* BRU2 - The College of Coastal Georgia is currently online via our metro-e backup circuit. Their primary fiber path is down due to the power outage at Waycross.
* Apogee services - Apogee services to the College of Coastal Georgia and to Abraham Baldwin Agricultural College are down and will come back online when a fiber path to those campuses are restored.
If you have reason to believe that your campus is listed incorrectly (i.e. you do have power), please call our HelpDesk at 706-583-2001.
Update
Emergency services are still in the process of clearing roads to get to the WRENS and SSC fiber cuts. There is no ETR at this time.
Identified
The fiber between Wrens and Augusta is down. This in connection with the Waycross outage is causing the following campus outages:
- Georgia Southern University (GASO1)
- Georgia Southern University - NOC (GASO2)
- East Georgia State College (EGA)
- South Georgia State College (SGA)
- South Georgia State College - Waycross Campus (WAY)
- Georgia Southern University (GASO1)
- Georgia Southern University - NOC (GASO2)
- East Georgia State College (EGA)
- South Georgia State College (SGA)
- South Georgia State College - Waycross Campus (WAY)
This incident affected: PeachNet Network (Abraham Baldwin Agricultural College, Apogee, Augusta University, College of Coastal Georgia, East Georgia State College, Georgia Southern University, Savannah State University, South Georgia State College, Valdosta State University) and GeorgiaVIEW Online Learning (Augusta University, Georgia Southern University, Savannah State University (QPROD), South Georgia State College (QPROD), Abraham Baldwin Agricultural College (XPROD), College of Coastal Georgia (XPROD), East Georgia State College (XPROD), Valdosta State University (XPROD)).