Results 1 to 10 of 10
  1. #1
    Join Date
    Aug 2011
    Posts
    4

    Unanswered: Patches in a Cluster environment ?

    Hello,

    I'm not sure if this is the correct place to post this question, so if it's not then i appologize in advance.

    I just recently applied to a job posting with the role of installing patches on servers , standalone and cluster environment. And i got an interview.
    To my surprise it's a Level 3 support job for patching up servers when level 1 and level 2 fails to do using Bladelogic or WSUS. My job would be to install the failing patches manually...

    My question is : how hard can it be to install Microsoft patches (only) on a server ? being either standalone or Clustered ? What kind of extremely hard situations can i encounter for this job to be a level 3 ????

    Thanks in advance for your answers.

  2. #2
    Join Date
    Jan 2003
    Location
    Massachusetts
    Posts
    5,800
    Provided Answers: 11
    Depends on what goes wrong with the patch.

    Loss/crash of the cluster service is a possibility, albeit quite rare in my experience. Just remember to patch and confirm any patches on a single node before moving on to the others.

  3. #3
    Join Date
    Aug 2011
    Posts
    4
    Thanks for your reply. Correct me if i'm wrong but to manually install a patch on a Failover cluster you simply need to connect to each node (the passive ones first) and install the patch, then install it on the active node and reboot?

    To manually install a patch on a Network Load Balancing cluster, simply connect to each server and install the patch in any order and reboot it?

    I still don't see how hard this can be
    Last edited by Alexoman; 08-12-11 at 16:49.

  4. #4
    Join Date
    Feb 2004
    Location
    In front of the computer
    Posts
    15,579
    Provided Answers: 54
    If you know enough that you're leary about the concept of a "level three" patching tech/manager, then you're about ten percent of the way to understanding the problem. This puts you light years ahead of the pack, but probably not ready for the job.

    The exact mechanics of this kind of position seem to vary greatly from one organization to another, so without knowing a LOT more about the position I can't be very specific. This type of position is typically a low level people manager, and a powerful administrator cum architect within the IT organization.

    -PatP
    In theory, theory and practice are identical. In practice, theory and practice are unrelated.

  5. #5
    Join Date
    Aug 2011
    Posts
    4
    Thanks for the reply. I will be getting a lot more technical details next tuesday, then i can make a decision.

  6. #6
    Join Date
    Mar 2007
    Location
    Holmestrand, Norway
    Posts
    332
    Alexoman: To be extremely honest with you: Have you ever installed patches in a cluster, or just read the theory? OS patches is installed on the passive node, rebooted, and validated before failover, yes. But SQL Server patches must typically be installed on the ACTIVE node. If you have ever installed patches in a clustered environment, you should know this. Ah, and by the way Alexoman: In a 4-node cluster with 3 SQL Server 2008 instances, how many times do you have to install a patch for it to apply to all instances on all servers?
    Ole Kristian Velstadbråten Bangås - Virinco - MSSQL.no - Facebook - Twitter

  7. #7
    Join Date
    Feb 2004
    Location
    In front of the computer
    Posts
    15,579
    Provided Answers: 54
    roac raises good points, but these are what we call "the small end of the stick" in that they are relatively trivial in terms of what I think that a patch manager needs to do.

    Knowing the hardware inside and out to know what hardware level patches conflict with, are compatible with, or are required by OS patches requires a truck load of understanding of every piece of hardware in your Data Center. Knowing which Windows patches are compatible with the HBA interfaces for multiple SANs would make the average admin go even more insane.

    Knowing which applications can tolerate which patches is another nightmare, and many of the vertical market applications that drive a major business are VERY picky about what patches you can apply to the hardware that runs them... ERP packages are notorious culprits in this area. Regulatory compliance packages can be even worse!

    This only scratches the surface of the kinds of problems that I've seen. If you have an enormous budget that allows all hardware to be relatively similar (same manufacturer, similar sizing, all machines still under manufacturer's warranty) then patch management is trivial. That doesn't seem to happen in the real world, so at least in my opinion patch management is an arcane art!

    -PatP
    In theory, theory and practice are identical. In practice, theory and practice are unrelated.

  8. #8
    Join Date
    Mar 2007
    Location
    Holmestrand, Norway
    Posts
    332
    Pat is absolutely right. I've never been working full time with patching (Personally I don't want to, and Norway barely has any companies large enough for it). Still, I've come across patches which does not work with specific driver or firmware versions. If you have a dedicated QA environment with identical hardware, these are easy to pick up. If the hardware is different, it is hard. I guess I don't have to mention that virtual QA and physical production makes this even harder.

    You mention specifically windows updates, put let us nevertheless take this a bit further. Specific updates for VMware tools makes SQL Server stop (hey, that is cool or?). If you're working with these patches as well, you should definitely discover this, and figure out how to re-add the DLL that WMware deleted.

    And, the boring side: SQL Server hotels (which of course tend to be clusters): How can you ensure that ALL software using an instance is supporting the new update before you add it?. Although an update is "just" a bugfix, that bug may have been used by software developers to achieve something specific. I don't remember any specific here, but in SQL Server 2000 the FOR XML syntax allowed creating an element with two attributes with the same name (invalid xml), this was fixed in SQL Server 2005. You can clearly imagine, that a patch fixing this could have broken software that were dependent on this bug.

    A more interesting point is, how is updates affecting your PERFORMANCE? It may be tempting to believe that a small patch cannot do much with performance. If so, you can easily be proven wrong. I know a lot of software vendors who publish patches to their application, which drops ALL indexes on their tables, do the update, and recreate all but the custom built indexes (Why on heavens earth don't they just disable the indexes). It can have a tremendous impact on the performance. Again, if you work with this kind of patches, you should find out before the patch goes to production.

    Safe patching is not simple. I guess there are even more to take into consideration, patching as never been the main focus of my job.

    Now, by all means, I do not want to scare someone off such a position. It would indeed be an interesting and challenging job, but it is important to know that patching is not simply a walk in the park.
    Ole Kristian Velstadbråten Bangås - Virinco - MSSQL.no - Facebook - Twitter

  9. #9
    Join Date
    Aug 2011
    Posts
    4
    Roac: I only have installed patches on file server clusters, and only on very small clusters. The rest i only know in theory. And to answer your question, i would say you have to install the patch twice for all instances. Once for the SQL servers and once for the other. It won't be my job to Verify and approve which patches will be installed. That will be done at the 1st or 2nd level. My job will mainly be to install the patches manually if using Bladelogic or patchlink fails. So i think that very much simplifies my job at this point but i could be wrong.

    Pat: My current job is mainly hardware support for servers and SAN's etc.. i am very much aware some patches might not work with certain firmware and driver levels..well mostly for IBM servers. did not have the chance to touch other kind of servers.

    Well guys tomorrow i will have much more technical details on what the job is and i'll have a better idea if i should accept it or it... the only downside i can see for now is pager duty two weeks a month !!

  10. #10
    Join Date
    Jan 2003
    Location
    Massachusetts
    Posts
    5,800
    Provided Answers: 11
    Pfft. I am the only SQL DBA here. I am on call all year long. As a result, I work hard at being lazy as hell. ;-)

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •