UPDATE: 12/23/22 (7:00pm): The temperature in the CCB data center (CCB 247/246) is holding and TSO has powered on all servers. Most services have been restored.
The following services are still offline:

  • Ta-app.cc.gatech.edu 

  • Various research and instructional VMs

TSO has notified stakeholders of problem servers.
 

Feel free to contact the TSO Help Desk (CCB, 404.894.7065, helpdesk@cc.gatech.edu) with questions/concerns. Please note that you will not receive an automated response with a ticket number.

UPDATE: 12/23/22 (2:45pm): GT Facilities has been able to restore chilled water to CCB. TSO staff is onsite monitoring the temperatures to determine when it will be safe to begin turning on servers. TSO anticipates services will begin to come online in the next few hours.

UPDATE (12/22/22 at 12:25pm): TSO and GT Facilities are actively troubleshooting the situation. Some servers and services may come online temporarily while we troubleshoot. Please do not anticipate services to be available until we send out an update with an all clear indicating the data center temperatures are holding.

UPDATE (12/21/22 at 11:00pm): GT Facilities has confirmed there is no chilled water pressure coming into CCB. They have shut off the chilled water pumps to prevent them from burning out. TSO is continuing to shut down everything in the CCB data center (CCB 246 & 247). The issue has been escalated to the Holland Plant management staff.

 

WHAT’S HAPPENING? 

We have been notified of networking equipment, server hardware, shutting down from high ambient temperature. Switches began to shut down on their own and TSO members are working to shut down servers & clusters.

Affected servers & equipment: 

Everything inside of CCB 247/Data Center 

A follow-up of all servers and equipment will be provided. 

WHEN IS IT HAPPENING? 

At roughly 06:00:00PM EST on December 21st, 2022, reports of high temperatures and VMs crashing were reported. 

WHY IS IT HAPPENING? 

It is unknown at this time; facilities, GT Police, and TSO members have been notified. 

WHO IS AFFECTED? 

Everyone who has equipment, virtual machines, inside of the College of Computing Data Center (CCB 247). 

Affected services: 

All servers and equipment inside the College of Computing Data Center, CCB 247. 

WHAT DO YOU NEED TO DO? 

If there is anything you can turn off inside the datacenter to reduce load, please let us know by emailing helpdesk@cc.gatech.edu with the server/vm hostname in the subject line.  

WHO SHOULD YOU CONTACT FOR QUESTIONS? 

Feel free to contact the TSO Help Desk (CCB, 404.894.7065, helpdesk@cc.gatech.edu). 

 

Owner of Alert
TSO