UPS Investigation

On this day we discovered a strange smell in the data center. The smell was like something burning and it was quite strong. Of course, having anything with a burning smell is never good with networking equipment. Troubleshooting this would seem to be something easy, you just follow your nose. However, due to the moving air in the room and the strong ventilation, tracking down the smell was quite difficult. To aid our search we used a heat gun to locate areas of abnormal heat.

We inspected all of the UPS’s in the rack we believed the smell was coming from. In this photo you cannot see, but we have several Dell Power Edge devices at the top of the rack along with the core router. In order to maintain constant up time we needed to trace each power cable and their redundant power connection. There were two different two-pack UPS units in this rack and we carefully removed the power cables from each two-pack UPS that we wanted to inspect one at a time.

We pulled out the UPS units and removed their panels, inspected for anything out of the ordinary (burn marks, melting, battery pack swelling, acid leaks). Additionally, we tested for abnormal heat signatures. Unfortunately, we did not find anything wrong with these units and put them back in place and reconnected the main and redundant power connections to the UPS units for the devices operating in that network rack.

There are some older UPS’s in this datacenter for some lesser important non client devices that we believe were part of the issue which was a failed capacitor. These UPS’s were investigated by our WAN team. The smell has never returned, and all of our equipment is working without issue.