Results 1 to 10 of 10
  1. #1
    Join Date
    Oct 2010
    Location
    Orlando, FL
    Posts
    311

    Question How To Handle Power Outages

    Can anyone tell me what the best practice is for handling a data center that has excessive power outages? I've been called in now 3 times this month due to the power failing in the building my database is housed in. Obviously the APC 3000 units can only hold 'x' amount of systems for 'x' amount of time but after that time is reached, the servers just cold die. Is there a way I can have my Linux servers mainly the ones managing my database systems cleanly power off / shutdown once a threshold of battery level on the UPS is reached? Is this something the UPS must support or can it be done some other way?

  2. #2
    Join Date
    Jan 2003
    Location
    Massachusetts
    Posts
    5,799
    I remember NT 4.0 had a method of powering off when the power went out. It required an install of some monitor from the UPS vendor. I can not imagine that Linux does not have something similar. Check with the vendor of your UPS system to see what they offer.

  3. #3
    Join Date
    Sep 2009
    Location
    Ontario
    Posts
    1,057
    Assuming APC3000 refers to APC the corp.

    Software / Firmware

    How long are the power outages lasting? You should be able to install the software on one server only, and have that server initiate shutdown on all the other machines.
    Last edited by kitaman; 10-19-12 at 16:01.

  4. #4
    Join Date
    Feb 2004
    Location
    In front of the computer
    Posts
    15,579
    Full disclosure, I work for a company that builds and runs data centers so I have both expertise and also bias on the subject.

    A Hosting DataCenter (which hosts computers for other companies) nomrally has battery backup for a very short period (roughly an hour or so). This is only intended to support the systems while the operations crew determines what went wrong, is there a known, short-term fix, etc. In other words, the battery backups are important and ought to provide absolutely reliable but short term "bridging" when primary power fails.

    There should be two or more utility company provided power systems feeding the data center. Each of these systems ought to be completely isolated from each other with NOTHING shared between them outside of the data center itself.

    As a fail-safe, there ought to be onsite power generators with at least a week's supply of everything they need to operate. We typically use our own privately owned diesel powered APUs, exactly the same as the utility companies use for spot production.

    I would be motified if I ever had to tell a customer that their systems lost power in our Data Center for any reason other than human intervention. Losing power due to an acciident would horrify me.

    The short answer is that the customer should never need to ask this question. A data center ought to take this problem completely off the table except in the case of a natural disaster.

    -PatP
    In theory, theory and practice are identical. In practice, theory and practice are unrelated.

  5. #5
    Join Date
    Jun 2003
    Location
    Ohio
    Posts
    12,592
    Quote Originally Posted by CarlosinFL View Post
    Can anyone tell me what the best practice is for handling a data center that has excessive power outages?
    Who's basement is this being hosted in?
    Obviously you need to contract with a new data center.
    If it's not practically useful, then it's practically useless.

    blindman
    www.chess.com: "sqlblindman"
    www.LobsterShot.blogspot.com

  6. #6
    Join Date
    Sep 2009
    Location
    Ontario
    Posts
    1,057
    Now you want to rob his mommy of her source of income

  7. #7
    Join Date
    Oct 2009
    Location
    221B Baker St.
    Posts
    486
    Yup, probably time to look for a new place to park the box(es). . .

    How much has to be kept running? Running your own generator might be of use. When the building power is interrupted, the generator(s) automatically kick in. The battery backups can carry the load during the "switch" and be recharged while the power is being locally generated also.

    Even with a new "room", there is still the issue of a damaging crash due to loss of power.

  8. #8
    Join Date
    Jun 2003
    Location
    Ohio
    Posts
    12,592
    We have ours hosted on a cluster, which is then mirrored to a local cluster that we manage.
    Haven't quite got automatic fail-over of all applications working yet, but we're close and we run a scenario about once each quarter.
    If it's not practically useful, then it's practically useless.

    blindman
    www.chess.com: "sqlblindman"
    www.LobsterShot.blogspot.com

  9. #9
    Join Date
    Sep 2009
    Location
    Ontario
    Posts
    1,057
    Maybe something like this:
    PrimePact 50LP 5000 Watt Propane RV Generator

    With the remote start option, you might even be able to have the UPS start it.

  10. #10
    Join Date
    Aug 2004
    Location
    Dallas, Texas
    Posts
    831
    Quote Originally Posted by blindman View Post
    We have ours hosted on a cluster, which is then mirrored to a local cluster that we manage.
    Haven't quite got automatic fail-over of all applications working yet, but we're close and we run a scenario about once each quarter.
    We have ours clustered and then replicated which is mirrored and then replicated to the primary node1 in our cluster which is backup hourly and replicated every ten seconds. Can't get the cpu bottlenecks aligned with our service until the differentials are done, but this approach has not succeeded yet so we're ok for now.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •