Unplanned downtime Nov 17th

There was a scheduled power outage of UWM’s physics building for November 17 but since we did not expect any disruption to the E@H servers we did not announce any downtime. The plan was to run E@H off of the data center’s UPS but unfortunately the batteries did not last nearly as long as expected. We apologize for any inconvenience caused by this outage.

Comments

Jim Overheul
Jim Overheul
Joined: 30 May 10
Posts: 1
Credit: 1026038
RAC: 0

Unplanned downtime Nov 17th

Not a problem sir. Thank you for the update!

Mario S.Garcia
Mario S.Garcia
Joined: 29 Nov 09
Posts: 1
Credit: 149501
RAC: 0

Ok, no problem. We'll

Ok, no problem. We'll working! Good job guys!

CLYDE
CLYDE
Joined: 19 Sep 09
Posts: 1
Credit: 2719790
RAC: 0

Murphy's Law #57: Batteries

Murphy's Law #57: Batteries never last as long as expected. :)

Bright Libra
Bright Libra
Joined: 20 Mar 08
Posts: 1
Credit: 450683
RAC: 0

Dunderheads! How could you

Dunderheads! How could you let this happen?!!?!

Seriously, no harm no foul and no real loss of any data... so how are we doing?

;)

Snoopp2
Snoopp2
Joined: 22 Oct 12
Posts: 1
Credit: 565065
RAC: 0

It's always good to test the

It's always good to test the UPS batteries occasionally...time to swap them out, perhaps??

Mario Marcel Salas
Mario Marcel Salas
Joined: 17 Aug 10
Posts: 3
Credit: 1019732
RAC: 0

Hope is good and cooperation

Hope is good and cooperation essential--hope we get back up soom

Bruce Allen
Bruce Allen
Moderator
Joined: 15 Oct 04
Posts: 1119
Credit: 172127663
RAC: 0

RE: Hope is good and

Quote:
Hope is good and cooperation essential--hope we get back up soom

As far as I can see, the project is up and running correctly again. Are there still issues?

Director, Einstein@Home

edjcox
edjcox
Joined: 10 Dec 05
Posts: 11
Credit: 141337850
RAC: 0

Tell your techs to get new

Tell your techs to get new batteries but to parallel them with the old. This way you'll get the amperage from both and last a lot longer.. Most UPS system can trickle charge multiple batteries..

Bob Moore
Bob Moore
Joined: 5 Dec 05
Posts: 7
Credit: 24983877
RAC: 31360

You have the other type of

You have the other type of UPS - unreliable power supply..... ;-)

Mac.teh.Knife
Mac.teh.Knife
Joined: 4 Feb 13
Posts: 17
Credit: 865586
RAC: 0

RE: Tell your techs to get

Quote:
Tell your techs to get new batteries but to parallel them with the old. This way you'll get the amperage from both and last a lot longer.. Most UPS system can trickle charge multiple batteries..

Actually, depending on what condition the old batteries in, this might be very bad advice. I am assuming we're talking about lead-acid storage batteries here.

A lead-acid battery contains sulphuric acid. If not properly maintained the sulphur precipitates (goes out of solution) and plates onto (attaches itself to) the lead plates inside the battery. The process is generally known as "sulphation". As sulphation advances the battery's ability to hold charge decreases. That means both the voltage and the amperage the battery is capable of delivering decrease. If allowed to proceed long enough the battery becomes useless. Except in the very, very early stages sulphation cannot be reversed.

It is generally believed that one need only apply a trickle charge to a battery to maintain it. That is partially true. The full truth is that one must occasionally apply a voltage slightly over 16 volts for short duration to the battery to prevent sulphation. Normally a trickle charge is about 14.5 volts and that will slow the sulpahtion process considerably but to do the job properly the voltage must be boosted to 16 for a few minutes once a month.

If the batteries in the UPS in question here did not last nearly as long as expected then it certainly sounds like they might be in bad condition. The other explanation is that they are in good condition and someone simply over-estimated how long they could keep the server powered up. If they are actually in bad condition then that can be determined through simple procedures. The standard procedure is to charge the battery for a sufficiently long time, the time depending on the battery's capacity and the charger's capacity. After the charge period you then measure the density of the acid with a hydrometer, more commonly known as a battery tester. Most testers have an analog scale and depending on where the indicator falls in the scale you can determine the condition of the battery. Another method is to charge the battery for a sufficient time then remove the charger and wait for 1/2 hour for localized charge to dissipate away from the terminals. The localized charge is due to presence of the charging voltage and gives a false, higher than normal, reading. Within half an hour the charge will have dispersed into general solution and then you can measure the true voltage. A 12 volt battery in excellent condition should read 14.5 volts. If 14.5 volts the battery is capable of delivering its rated amperage. If 12 volts then the battery is in poor condition an incapable of taking the charge it is designed to take and hold. It's like a bucket that has a hole in it's side somewhere between the top and bottom. You can pour as much water as you want into that bucket but after a few minutes, when everything "normalizes", the bucket will not contain as much water as it could.

If the reading is somewhere between 12 and 14.5 volts then the condition of the battery is somewhere between poor condition and excellent condition. Whether it should be kept in service depends upon the application. If it's being used to start an engine in very cold weather then a battery at 13 volts is questionable though in warm weather (where engines start a lot easier) it might work just fine. If it's in a UPS attached to a computer then it will be capable of powering a computer but not nearly as long as a battery that will charge up to 14.5 volts. Many inverters (the thing that converts the battery DC voltage to 120/240 AC volts) are capable of operating and delivering enough voltage to keep a computer running with a battery voltage as low as 11.25. Others are not that capable. Some computer power supplies and probably mother boards too are more tolerant to low voltage than others.

Getting back to the notion of connecting the old batteries in parall to the new batteries.... the old batteries will simply drag the new batteries down to their level. If the new batteries are at say 14.5 volts and the old batteries are say 13 volts then there is a 1.5 volt potential which will cause current to flow from the new batteries through the old batteries and, since the old batteries will not be capable of holding that charge it will simply boil off water and the power will be dissipated as heat. There will be a small increase in the charge available for the inverter to convert to 120/240 volts but the proper way to do this is to determine the condition of the old batteries first. Standards vary from one person to another but in my opinion if the difference in voltage (after the battery has been charged and allowed to dissipate localized charge) is more than .5 volts then the old and the new should not be married. Others are more stringent and never marry new to ld regardless of their condition. They insist that only brand new batteries should be married and they have very valid point.

This not to say the old are junk and should be discarded, they should be assigned to a new purpose if they can hold reasonable charge but they are not suitable for marrying to a battery that is in excellent condition. Indeed you will find many techs who scoff at the idea because they believe a battery is a battery is a battery and they have no fundamental understanding of how batteries work and what they do does appear to work if you don't examine too closely but it does not work nearly as well as they believe.

Now a word about UPS in general... any UPS you buy off of a store shelf is junk. The reason is because they sit on the store shelf or in a warehouse for months and even years (check the date of manufacture if you don't believe me, I have and that's why I'm writing this) without a maintenance voltage applied to the terminals. As stated above, without a proper maintenance voltage (aka trickle charge) the battery will sulphate. Sulphation is irreversible damage. You cannot repair it simply by taking the UPS home and allowing it to charge the battery. No, it does not work that way. The only way is to replace the battery with a new one. If you use the old one then it might be capable of powering you computer long enough to allow it to shutdown or it might not.

Lead-acid batteries should not be filled with acid until just before they are delivered to the user. They should be stored in the warehouse and in the store empty. If it is to be filled with acid before delivery then it must placed on a maintenance charge until delivered. I will not buy a lead-acid battery unless I see with my own eyes that it is empty when removed from its box and that the vendor then fills it with acid. They won't do that at most retailers so I go to a battery specialty shop where they know batteries.

For that same reason, never buy a "maintenance free" battery. You need a battery that allows you to remove the caps and check the acid level and add distilled water if necessary. Yes, distilled water not de-mineralized water and absolutely NOT tap water.

The best UPS you can buy is one you can build yourself. Go to a battery shop and buy a big, deep-cycle AGM lead-acid battery. AGM is a technology not a brand name. Buy a full sine wave inverter. Buy a battery charger capable of delivering 1.5 times the power your computer needs to operate. The idea is your computer will run continuously off the inverter which will be continuously supplied by the battery which is continuously charged by the charger. The battery will clamp any voltage spike better than any surge protector on the market as well as filter line noise. You can also buy devices that will monitor the battery voltage and send a "shutdown command" to your computer if the voltage drops too low. This would be in the event the mains power goes off and does not come back on for a while which would of course not allow the charger to run which in turn would cause the battery voltage to slowly decrease. The setup is not cheap but you get a huge battery that is in top condition as opposed to a piece of junk like you'll get in any UPS you buy off the shelf, you get much longer run time when the mains power goes off and you get the best possible surge protector/line filter going.

You might get the idea that you can just buy a UPS from the store and replace the junk battery in it with a new battery. The problem is that the charging system in most UPS are braindead. They assume the battery has a certain capacity and so they charge the battery for just long enough to charge such a battery. If you install a larger battery the charger won't charge it to full capacity and voltage and then it will sulphate. I have also heard that some UPS do not apply a trickle charge and that they simply wait for the battery voltage to drop below a certain level then they cut in and charge for long enough to top up the size of a battery they were designed to work with. That's bad from a sulphation aspect as well as negating the possibility of replacing the original with a bigger, better battery. You could buy a UPS, toss the battery and replace it with the same spec battery but you're still left with what is IMHO an inadequate charger. I'd rather build my own and know what I have than risk buying garbage that doesn't do what the manufacturer and the media claim it can do. Actually they don't make claims, they just give subtle hints that lead you to believe. When your belief turns out to be false they reply, "We never claimed it would do that, you got the wrong impression." Yeah right!