« Reading List: La Force de la Raison | Main | Fourmilab Framed! »
Friday, December 3, 2004
Hello, Dali
This is a picture (click the thumbnail for an even more frightening enlargement) of a pair of batteries removed on October 10th, 2004 from the American Power Conversion Smart-UPS 1000RMI which had been mounted in the Fourmilab communications rack since 1996, providing power protection for the leased line modem, router, firewalls, hubs, and switches, as well as the Alcatel phone central. The original set of batteries which came with the UPS lasted four years and were replaced in August 2000 after the self-test began to fail. Those batteries were, in turn, replaced by those you see here, a genuine APC RBC-6 set bought new from an authorised APC dealer, with a manufacture date within a couple of months of when I installed them, August 13th, 2002. (I don't know why the first set of batteries lasted four years and the subsequent sets only two; I usually get four years or more from UPS batteries, and this UPS is only trivially loaded and we rarely have more than momentary power outages. Perhaps the short life is a clue.)
Anyway, I'd never had any trouble with previous battery replacements in this UPS or any other of the dozen or so installed here, apart from two minor acid leaks which were easily cleaned up, but this time when I tried to extract the batteries, they wouldn't budge more than a centimetre or so. The battery compartment in the rack-mount Smart-UPS is a really tight fit, and it was hard to see what was going on in there, but close examination with a bright light made it clear that the top of the batteries bulged upward, apparently enough to jam them in the compartment. However, other than the UPS displaying the "Replace batteries" light and squawking daily to get my attention, everything continued to function perfectly.
I tried lots of stratagems to dislodge the batteries without pulling the UPS. Over the years, its nice, cool, flat top surface had been a convenient place to stack lots of gear that wasn't rack mountable (or I didn't feel like mounting), and below the UPS was a pile of ISDN terminal adaptors sitting on the floor of the rack which would be instantly crushed if I unbolted it from the rack. Finally I concluded there was no alternative and swapped over everything to a new Smart-UPS 1500 tower model behind the rack (after this experience, I will never again rack mount a UPS), and re-cabled everything so I could install a shelf above the UPS and transfer the gear sitting on top of it to there. Then I gingerly teased the ISDN adaptors out onto the floor in front of the rack, placed a pile of books beneath the UPS to support it when I unscrewed it from the rails, dropped it, and removed it from the
rack. When I tore it apart, this is what I found in the battery compartment. (The fuse in the series connection between the two batteries was disconnected during my attempts to get the batteries out through the front access door.) There is clearly no way whatsoever I could have gotten those batteries out of the UPS without removing the top and dismantling most of the internal structure of the battery compartment.
Have you ever seen anything like those batteries? They're something right out of a Salvador Dali painting! I certainly haven't, and to me it's a total mystery why one or more of the caps didn't pop and spew acid all over--there was no leakage whatsoever. There were no indications of heat, and the batteries in this model are well-separated from the transformer and circuit board. None of the other components in the rack emit any significant heat. Since I'd already bought a new set of batteries for this UPS, just to see what would happen, I re-assembled the UPS, installed the new batteries, and it's been working just fine ever since, powering a retired laptop I'm planning to use to test bleeding-edge Linux kernels and the like. I stuck elastomeric feet to the bottom, and it's sitting on the concrete floor where I can occasionally feel the top for, as they say, "excess heat", of which to date there has been no evidence. The UPS passes self- and plugs-out tests with flying colours.
It's a rule of engineering that whatever you don't understand is sure to bite you again, so this is one of those inscrutabilities I'd very much like to unscrew. But for the moment I haven't the remotest clue what happened to those batteries. (They have now been recycled because I was uncomfortable with something like that sitting around the shop, so further forensic examination of the batteries is not possible.)
Posted at December 3, 2004 16:41