Here's something which might cause some conversation. A lot of company systems out there have a Dev, Test, and Production (Live) system. Everything you develop you do on Dev, and everything you test data-wise you do on the Test system and then you move it to Live, NEVER touching Live. Seems like a perfect world.

Problem is....and I've asked this question (with these type of systems), and always get the same answer: Have you ever tested a Restore of the LIVE system? Have you ever tried this, or that on the Live system which might not be quite the same as your Test system?

The answer I always get is No, No, No....you never touch the LIVE system!! LIVE is the untouchable where you only touch it if absolutetly need be. But other than that, something you never mess with.

Now say all you're restores worked great for Dev and Test (you've done your job and tested them 2-3 times a year)....and now your Live system has somehow crashed (after all - you're working with 16 gigs of data verses 2 gigs). You go to your Live backups and they DONT work! You're not familiar with the data in Live so even if they do work, you're not sure if you got everything restored correctly! What do you do?

Seems like a trivial question but I've asked many, many people if they have actually tested out the restores on the Live system and I get the same answer: "I don't need to - I test out the restores on my Test system all the time and they always work perfectly."

Now also say you add a field to Dev and Test and it works great. You add that same field to Live and it brings down your system for 1/2 the day (odd - it worked quickly with only 2 gig of records but now I do this on Live with 16 gig of data and I've brought down the system for 1/2 day. Is this something I must only do on weekends (when hopefully no one else is working on it) or is this something I would do with a modular system where I work on that specific "piece" of the Live system? My "central" Live system was something which was worked on cautiously but it was still worked on. Almost all the important (i.e. financial, evaluation, etc.) data on the modular system would get written to the central system but the modular system would stay "modular", storing data not pertainenant to the the central system.

My point is that you have to get your hands dirty and you NEED to work on Live (Object-Oriented type programming does help). Sure it might be nice to have a Dev and Test system or even another "testing" type system, etc...etc...time permitting if you're not in a fast-paced environment where things don't need to get done quickly. Or you could simply have a modular system based around your Live system where you work on "pieces" of the live system bringing down the pieces you need to work on - modular - without bringing down the entire system.

But if you're like everyone I talk to who says "You never touch the Live system," in my opinion, you're in for a few surprises (i.e. you train for war and when the "real" war comes into play, it's one heck of a shock to find things don't work the same as training?).

I didn't have the luxury, space, or time to have a Dev, Test, and Live system at my last job. But I did have time to create a very good auditing system around the Live system where I knew every piece of data that got changed or deleted. I could easily immediately know and restore 1000 records which got deleted because a developer accidently ran a delete query instead of a select query (I got in touch and KNEW my data). Our code was worked on a "Dev" type system (i.e. MSAccess mdb files which were eventually converted to mde files to the Live system) but the data worked on was based around "Live" data. And it worked very well with 100% up-time (I know - some of you out there are thinking I'm crazy.) But you need to get to know your data on Live, not Test (i.e. how many records, time it takes to restore (if it works), queries which might take a long time, etc., etc...!!) A concept my new company doesn't seem to grasp as I always get the question: "Why do you need to see the tables and data on Live (and even Test!) - what good does that do you?"

So I guess the question is what kind of system do you have and what are your arguements for why you chose that system (and never testing out the restore of "Live" if you haven't?) I'm really very curious of other thoughts on this issue.

Hi Paul

My Test and Dev machines are restores of Live. Every so often we overwrite the Test and Dev dbs with the latest production data backup - we plan in advance and ensure that we have no loose development ends for the Big Day.

You can't develop & test on systems with only the fraction of the data on the production database IMHO. Your query plans & optimisations are dependent on the data. The same queries will be executed differently if applied to the same database with different data.

So that's my answer - what is the point of testing on a system that is not comparable to the system that really matters (i.e. prod)?

BTW - any decent disaster recovery plan requires occassional "dry runs" to check that it is robust, documentation is accessible & unambiguous, all gotchas are documented etc.

HTH

pootle flump ur codings are working excelent.

Hi Paul,

I've worked on extremely large databases for credit card companies. We create a copy of the the entire live system for test purposes. If we are offshoring development the test data is scrubbed so that sensitive columns contain impersonal but coherent information. ie. names are replaced with random, but real words.

So, I guess what I'm saying is the client should if they have half a brain be willing to fork out for a test system on the same scale as the live system whether they arrange to lease or rent the additional hardware.

Thanks guys/gals. I was beginning to think that the concept of "never touch the live system" was something everyone else followed (I just got out of a unix class where my classmate (as well as my new company) has told me I am crazy for ever messing around with the live system. To me it seems like you have to get your hands dirty at some point or another and I'd rather get them dirty when I have the time and I don't have the accounting department (and every other department) breathing down my back telling me "I need to get back up and running so HURRY!". I'd like to be prepared in advance and know my backups work (on live). Cloning the live system to a test system is nice and is a good test but I guess I'm just used to "crashing" and bringing back up the actual live system. Good point on the queries of 16 gig verses 2 gig pootle and thanks for the reply Certus!

I was beginning to doubt my concept with other views I was hearing but I still firmly believe you've got to crash and test out LIVE to know if LIVE will and can be restored to it's original state.

Anyone out there still think that "you never touch the LIVE system" and if so, I'd still like to know why? Maybe I'm just too used to in-house programming verses using another software package to do our thing (where I'm working now). It does seem to make a difference in that it gives you more accountability and a sense of responsibility and security when you've designed all the programming and databases yourself.

I agree with the others. I never test stuff out on Production, but my DEV and QA regions are restores from Prod. I am of the firm belief that if the data in those systems is fairly recent, theres rarely a good reason to do much in DEV or QA.