Update #15

12/22/2022 at 04:42pm

We completed all electrical work associated with the B-Side UPS repairs and the A-Side electrical modernization project at 2pm this afternoon, and the building was returned to normal operations. Both A and B side are on UPS utility power and the generators are back in standby mode. The team monitored all systems the past few hours for stability, and we are confident that everything is functioning as designed. Please contact DC Operations at x59194 with any further questions on this.

Update #14

12/14/2022 at 09:10am

Our vender partner Vertiv along with DC Facilities completed commissioning of the A-side UPS Friday evening. The UPS continues to operate as designed. Final acceptance testing will take place Monday. A-side continues to run on generator, and B-side continues to operate as designed.

Update #13

12/14/2022 at 05:22pm

Our vender partner Vertiv along with data center facilities continues to work on commissioning the A-side UPS. No estimated time for completion at this time. A-side continues to run on generator, and B-side continues to operate as designed.

Update #12

12/14/2022 at 10:08am

Our Vendor partner Vertiv along with Data Center facilities will be energizing the A-side UPS systems momentarily. This is the first step that must take place in order to move the A-side from generator to UPS/utility. Further updates will be provided this afternoon.

Update #11

12/13/2022 at 3:51pm

The primary data center A-side power continues to run on generator. Steady progress is being made with A-side commissioning. B-side system continue to perform as designed. All power to the compute floor continues to be stable and fully redundant.

Update #10

12/12/2022 at 10:07pm

The B-side has been successfully transferred back to UPS and RG&E utility power. All B-side power systems are now operating as designed. The A-side will continue to run on generator with plans to power on the new A-side UPS tomorrow. The next update will be provided by end of day tomorrow.

Update #9

12/12/2022 at 6:46pm

Repairs to the B-side UPS have been completed. Commissioning is taking place at this time. We have not transitioned the B-side back to UPS. Further updates will be provided as we have them.

Update #8

12/12/2022 at 12:03am

We successfully moved the A-side power back to the A-side electrical distribution. The Primary Data Center compute floor is currently running on both the A and B-side generators, restoring redundancy to the compute floor. All necessary resources will be back on-site tomorrow to repair the B-side UPS. Further updates will be provided late tomorrow afternoon.

Update #7

12/11/2022 at 9:47pm

DC Facilities and Operations along with our vendor partners have successfully brought a key portion of the new A-Side electrical distribution online. This enables us to move the A-Side power back over and run on A-Side generator. Once reviewed we plan to complete this work at 11:30 PM tonight.

Update #6

12/11/2022 at 9:30am

Repair parts for the B-Side UPS are not scheduled to arrive until 11am tomorrow. Due to the ETA for parts and the long ETR, Facilities, vendor partners and Data Center Operations are focusing all efforts today on bringing the new A-Side electrical distribution online. We will provide an update later today.

Update #5

12/11/2022 at 8:20am

Primary Data Center continues to run on B-side generators at this time. All available resources continue to work on the issue. Further updates will be provided as we have them.

Update #4

12/10/2022 at 11:25pm

Our Vendor partner Vertiv continues troubleshooting the issue with the B-side UPS and the UPS continues to be offline. We were unable to complete the procedure to move A-Side compute floor power back to the A-Side electrical distribution. All compute systems remain powered on the B-Side generator. IT resources will reconvene with facilities at 9am.

Update #3

12/10/2022 at 9:42pm

Our Vendor partner Vertiv continues troubleshooting the issue with the B-side UPS and the UPS continues to be offline. To regain redundancy in support of compute systems, we have been approved to transfer A-Side compute floor power back to the A-Side electrical distribution. While this procedure is performed we will have a brief interruption of A-Side power to the compute floor. All compute systems should remain unaffected and powered on the B-Side generator while we transition. The building load including the compute floor will then continue to run on both A and B generators until the UPS has been repaired. A technical conference bridge will be opened shortly.

Update #2

12/10/2022 at 8:14pm

Upgrading to Priority 2.

Our Vendor partner Vertiv is on-site troubleshooting the issue with the B-side UPS. The UPS continues to be offline and the building load including the compute floor continues to run on generator. Data Center Facilities has engaged additional resources and is in the process of re-energizing portions of the A-side electrical distribution in the event that it is needed. Further updates will be provided as we have them.

Update #1

12/10/2022 at 4:53pm

The primary data center has experienced an issue with the B-Side UPS. DC Facilities has transferred the building load to generator at this time. All resources are troubleshooting the issue. Further updates will be provided as we have them.