Unplanned downtime Nov 17th


Advanced search

Message boards : News : Unplanned downtime Nov 17th

AuthorMessage
Profile David Hammer
Forum moderator
Project administrator
Project developer
Send message
Joined: Oct 15 04
Posts: 361
Credit: 1,672,886
RAC: 0
Message 120137 - Posted 18 Nov 2012 3:42:32 UTC

    There was a scheduled power outage of UWM’s physics building for November 17 but since we did not expect any disruption to the E@H servers we did not announce any downtime. The plan was to run E@H off of the data center’s UPS but unfortunately the batteries did not last nearly as long as expected. We apologize for any inconvenience caused by this outage.

    Jim Overheul
    Send message
    Joined: May 30 10
    Posts: 1
    Credit: 1,026,038
    RAC: 0
    Message 120138 - Posted 18 Nov 2012 4:25:14 UTC - in response to Message 120137.

      Not a problem sir. Thank you for the update!
      ____________

      Mario S.Garcia
      Send message
      Joined: Nov 29 09
      Posts: 1
      Credit: 48,634
      RAC: 28
      Message 120142 - Posted 18 Nov 2012 9:31:10 UTC - in response to Message 120138.

        Ok, no problem. We'll working! Good job guys!

        Profile CLYDE
        Avatar
        Send message
        Joined: Sep 19 09
        Posts: 1
        Credit: 1,900,096
        RAC: 7,090
        Message 120143 - Posted 18 Nov 2012 12:23:07 UTC

          Murphy's Law #57: Batteries never last as long as expected. :)
          ____________

          Bright Libra
          Send message
          Joined: Mar 20 08
          Posts: 1
          Credit: 450,683
          RAC: 0
          Message 120153 - Posted 18 Nov 2012 16:11:11 UTC

            Dunderheads! How could you let this happen?!!?!

            Seriously, no harm no foul and no real loss of any data... so how are we doing?

            ;)

            Snoopp2
            Send message
            Joined: Oct 22 12
            Posts: 1
            Credit: 565,065
            RAC: 0
            Message 120164 - Posted 18 Nov 2012 19:44:44 UTC

              It's always good to test the UPS batteries occasionally...time to swap them out, perhaps??

              Mario Marcel Salas
              Send message
              Joined: Aug 17 10
              Posts: 3
              Credit: 796,034
              RAC: 1,310
              Message 120223 - Posted 19 Nov 2012 14:42:04 UTC

                Hope is good and cooperation essential--hope we get back up soom

                Profile Bruce Allen
                Forum moderator
                Project administrator
                Project developer
                Project scientist
                Avatar
                Send message
                Joined: Oct 15 04
                Posts: 1102
                Credit: 171,768,380
                RAC: 175
                Message 120226 - Posted 19 Nov 2012 16:36:52 UTC - in response to Message 120223.

                  Hope is good and cooperation essential--hope we get back up soom


                  As far as I can see, the project is up and running correctly again. Are there still issues?
                  ____________

                  Profile edjcox
                  Send message
                  Joined: Dec 10 05
                  Posts: 5
                  Credit: 17,581,985
                  RAC: 35,823
                  Message 120311 - Posted 22 Nov 2012 8:58:54 UTC - in response to Message 120226.

                    Last modified: 22 Nov 2012 8:59:59 UTC

                    Tell your techs to get new batteries but to parallel them with the old. This way you'll get the amperage from both and last a lot longer.. Most UPS system can trickle charge multiple batteries..
                    ____________

                    Bob Moore
                    Send message
                    Joined: Dec 5 05
                    Posts: 5
                    Credit: 536,488
                    RAC: 7,494
                    Message 120364 - Posted 23 Nov 2012 13:36:51 UTC

                      You have the other type of UPS - unreliable power supply..... ;-)
                      ____________

                      Mac.teh.Knife
                      Send message
                      Joined: Feb 4 13
                      Posts: 17
                      Credit: 865,586
                      RAC: 0
                      Message 122840 - Posted 8 Feb 2013 23:25:39 UTC - in response to Message 120311.

                        Tell your techs to get new batteries but to parallel them with the old. This way you'll get the amperage from both and last a lot longer.. Most UPS system can trickle charge multiple batteries..


                        Actually, depending on what condition the old batteries in, this might be very bad advice. I am assuming we're talking about lead-acid storage batteries here.

                        A lead-acid battery contains sulphuric acid. If not properly maintained the sulphur precipitates (goes out of solution) and plates onto (attaches itself to) the lead plates inside the battery. The process is generally known as "sulphation". As sulphation advances the battery's ability to hold charge decreases. That means both the voltage and the amperage the battery is capable of delivering decrease. If allowed to proceed long enough the battery becomes useless. Except in the very, very early stages sulphation cannot be reversed.

                        It is generally believed that one need only apply a trickle charge to a battery to maintain it. That is partially true. The full truth is that one must occasionally apply a voltage slightly over 16 volts for short duration to the battery to prevent sulphation. Normally a trickle charge is about 14.5 volts and that will slow the sulpahtion process considerably but to do the job properly the voltage must be boosted to 16 for a few minutes once a month.

                        If the batteries in the UPS in question here did not last nearly as long as expected then it certainly sounds like they might be in bad condition. The other explanation is that they are in good condition and someone simply over-estimated how long they could keep the server powered up. If they are actually in bad condition then that can be determined through simple procedures. The standard procedure is to charge the battery for a sufficiently long time, the time depending on the battery's capacity and the charger's capacity. After the charge period you then measure the density of the acid with a hydrometer, more commonly known as a battery tester. Most testers have an analog scale and depending on where the indicator falls in the scale you can determine the condition of the battery. Another method is to charge the battery for a sufficient time then remove the charger and wait for 1/2 hour for localized charge to dissipate away from the terminals. The localized charge is due to presence of the charging voltage and gives a false, higher than normal, reading. Within half an hour the charge will have dispersed into general solution and then you can measure the true voltage. A 12 volt battery in excellent condition should read 14.5 volts. If 14.5 volts the battery is capable of delivering its rated amperage. If 12 volts then the battery is in poor condition an incapable of taking the charge it is designed to take and hold. It's like a bucket that has a hole in it's side somewhere between the top and bottom. You can pour as much water as you want into that bucket but after a few minutes, when everything "normalizes", the bucket will not contain as much water as it could.

                        If the reading is somewhere between 12 and 14.5 volts then the condition of the battery is somewhere between poor condition and excellent condition. Whether it should be kept in service depends upon the application. If it's being used to start an engine in very cold weather then a battery at 13 volts is questionable though in warm weather (where engines start a lot easier) it might work just fine. If it's in a UPS attached to a computer then it will be capable of powering a computer but not nearly as long as a battery that will charge up to 14.5 volts. Many inverters (the thing that converts the battery DC voltage to 120/240 AC volts) are capable of operating and delivering enough voltage to keep a computer running with a battery voltage as low as 11.25. Others are not that capable. Some computer power supplies and probably mother boards too are more tolerant to low voltage than others.

                        Getting back to the notion of connecting the old batteries in parall to the new batteries.... the old batteries will simply drag the new batteries down to their level. If the new batteries are at say 14.5 volts and the old batteries are say 13 volts then there is a 1.5 volt potential which will cause current to flow from the new batteries through the old batteries and, since the old batteries will not be capable of holding that charge it will simply boil off water and the power will be dissipated as heat. There will be a small increase in the charge available for the inverter to convert to 120/240 volts but the proper way to do this is to determine the condition of the old batteries first. Standards vary from one person to another but in my opinion if the difference in voltage (after the battery has been charged and allowed to dissipate localized charge) is more than .5 volts then the old and the new should not be married. Others are more stringent and never marry new to ld regardless of their condition. They insist that only brand new batteries should be married and they have very valid point.

                        This not to say the old are junk and should be discarded, they should be assigned to a new purpose if they can hold reasonable charge but they are not suitable for marrying to a battery that is in excellent condition. Indeed you will find many techs who scoff at the idea because they believe a battery is a battery is a battery and they have no fundamental understanding of how batteries work and what they do does appear to work if you don't examine too closely but it does not work nearly as well as they believe.

                        Now a word about UPS in general... any UPS you buy off of a store shelf is junk. The reason is because they sit on the store shelf or in a warehouse for months and even years (check the date of manufacture if you don't believe me, I have and that's why I'm writing this) without a maintenance voltage applied to the terminals. As stated above, without a proper maintenance voltage (aka trickle charge) the battery will sulphate. Sulphation is irreversible damage. You cannot repair it simply by taking the UPS home and allowing it to charge the battery. No, it does not work that way. The only way is to replace the battery with a new one. If you use the old one then it might be capable of powering you computer long enough to allow it to shutdown or it might not.

                        Lead-acid batteries should not be filled with acid until just before they are delivered to the user. They should be stored in the warehouse and in the store empty. If it is to be filled with acid before delivery then it must placed on a maintenance charge until delivered. I will not buy a lead-acid battery unless I see with my own eyes that it is empty when removed from its box and that the vendor then fills it with acid. They won't do that at most retailers so I go to a battery specialty shop where they know batteries.

                        For that same reason, never buy a "maintenance free" battery. You need a battery that allows you to remove the caps and check the acid level and add distilled water if necessary. Yes, distilled water not de-mineralized water and absolutely NOT tap water.

                        The best UPS you can buy is one you can build yourself. Go to a battery shop and buy a big, deep-cycle AGM lead-acid battery. AGM is a technology not a brand name. Buy a full sine wave inverter. Buy a battery charger capable of delivering 1.5 times the power your computer needs to operate. The idea is your computer will run continuously off the inverter which will be continuously supplied by the battery which is continuously charged by the charger. The battery will clamp any voltage spike better than any surge protector on the market as well as filter line noise. You can also buy devices that will monitor the battery voltage and send a "shutdown command" to your computer if the voltage drops too low. This would be in the event the mains power goes off and does not come back on for a while which would of course not allow the charger to run which in turn would cause the battery voltage to slowly decrease. The setup is not cheap but you get a huge battery that is in top condition as opposed to a piece of junk like you'll get in any UPS you buy off the shelf, you get much longer run time when the mains power goes off and you get the best possible surge protector/line filter going.

                        You might get the idea that you can just buy a UPS from the store and replace the junk battery in it with a new battery. The problem is that the charging system in most UPS are braindead. They assume the battery has a certain capacity and so they charge the battery for just long enough to charge such a battery. If you install a larger battery the charger won't charge it to full capacity and voltage and then it will sulphate. I have also heard that some UPS do not apply a trickle charge and that they simply wait for the battery voltage to drop below a certain level then they cut in and charge for long enough to top up the size of a battery they were designed to work with. That's bad from a sulphation aspect as well as negating the possibility of replacing the original with a bigger, better battery. You could buy a UPS, toss the battery and replace it with the same spec battery but you're still left with what is IMHO an inadequate charger. I'd rather build my own and know what I have than risk buying garbage that doesn't do what the manufacturer and the media claim it can do. Actually they don't make claims, they just give subtle hints that lead you to believe. When your belief turns out to be false they reply, "We never claimed it would do that, you got the wrong impression." Yeah right!

                        Post to thread

                        Message boards : News : Unplanned downtime Nov 17th


                        Home · Your account · Message boards

                        This material is based upon work supported by the National Science Foundation (NSF) under Grants PHY-1104902, PHY-1104617 and PHY-1105572 and by the Max Planck Gesellschaft (MPG). Any opinions, findings, and conclusions or recommendations expressed in this material are those of the investigators and do not necessarily reflect the views of the NSF or the MPG.

                        Copyright © 2014 Bruce Allen