Page 1 of 2 12 LastLast
Results 1 to 15 of 19
  1. #1
    Join Date
    Jan 2011
    Posts
    8

    Question When to destroy data

    I'm having a debate with some of my colleagues about when to destroy/delete data.

    I'm in favour of not destroying data unless absolutely necessary. My colleagues are in favour of destroying data without considering alternatives.

    Can anyone offer, or point me in the direction of, some recommendations on when it makes sense to destroy data?

    Any help is appreciated.

  2. #2
    Join Date
    Oct 2009
    Location
    221B Baker St.
    Posts
    486
    In well-run organizations, purge criteria is usually part of the initial design. Very few things are supposed to be kept/used "forever".

    When considering when data could/should be purged, the "owner(s)" of the data should explain what the legal and company requirements are for this particular data.

    The question is rather nebulous and would possibly be better dealt with if some specifics were provided. Not all data is the same.

  3. #3
    Join Date
    Jan 2011
    Posts
    8
    This is well before the point where records would be purged.

    The data in this particular case are customer records. Each time that a customer makes an online order our application searches for them and if a customer record exists it will link the order record to the existing customer record. If the customer enters different details then they have multiple records in the database as a new customer record is inserted.

    When multiple records are discovered they are 'merged'. During this process one of the customer records is deleted. The only trace of the old record is a reference that is entered in a field in the surviving record. This reference is unique but not the primary key. Many other applications, eg a customer satisfaction survey, refer to the primary key of our customer records. This can result in survey records in other applications that contain a foreign key which refers to a referrent that no longer exists.

    My argument is that the old customer records should be archived rather than deleted. My colleagues argue that instead of one foreign key our applications should be rewritten to have two. That is to say store the primary key and the unique reference of each customer. That would mean that applications, such as the survey, could reconcile merged records. The same outcome could be achieved by archiving records with no change to the application. One of the reasons that I prefer the archiving approach is that I think that there may be circumstances, at present unknown, where the archived record would be of use. Both approaches cover the need to reconcile records after merger but only one covers the possibility that the data in the old record may be of some other use.

    As I say I appreciate any recommendations.

  4. #4
    Join Date
    Oct 2009
    Location
    221B Baker St.
    Posts
    486
    Suggest this needs to be based on what will best serve the business and the owner of the data rather than having technical people with different opinions choose the direction.

    Is there no business rule or legal consideration that would require keeping "deleted" info for some amount of time? Several of my clients have kept customer service info for several years and their decision was not to keep or discard, but how to best support the current online need as well as have the ability to bring back the picture of the customer as of some particular prior date with minimal impact to the online transactions.

    With dasd storage getting cheaper and cheaper, cost should not be an issue.

    Something to keep in mind is that there is a cost getting the info in the first place and once completely deleted, it is gone.

  5. #5
    Join Date
    Jun 2003
    Location
    Ohio
    Posts
    12,592
    Data is cheap to get, but expensive to throw away.
    If it's not practically useful, then it's practically useless.

    blindman
    www.chess.com: "sqlblindman"
    www.LobsterShot.blogspot.com

  6. #6
    Join Date
    Sep 2009
    Location
    Ontario
    Posts
    1,057
    If the merge process that your business has such a great deal of faith in is so good, why don't you put at the front of the customer add process and prevent the duplicates.

  7. #7
    Join Date
    Jan 2011
    Posts
    8
    Quote Originally Posted by papadi View Post
    Something to keep in mind is that there is a cost getting the info in the first place and once completely deleted, it is gone.
    Quote Originally Posted by blindman View Post
    Data is cheap to get, but expensive to throw away.
    I agree, I wish I could point to an authoritative source to help persuade my colleagues though.

  8. #8
    Join Date
    Jan 2011
    Posts
    8
    Quote Originally Posted by kitaman View Post
    If the merge process that your business has such a great deal of faith in is so good, why don't you put at the front of the customer add process and prevent the duplicates.
    Because it is triggered manually when it is discovered that a customer has more than one record.

  9. #9
    Join Date
    Jun 2003
    Location
    Ohio
    Posts
    12,592
    Quote Originally Posted by The User View Post
    Quote Originally Posted by blindman View Post
    Data is cheap to get, but expensive to throw away.
    I agree, I wish I could point to an authoritative source to help persuade my colleagues though.
    I AM an authoritative source.
    If it's not practically useful, then it's practically useless.

    blindman
    www.chess.com: "sqlblindman"
    www.LobsterShot.blogspot.com

  10. #10
    Join Date
    Sep 2009
    Location
    Ontario
    Posts
    1,057
    Need an authoritative source? Have one of your colleagues call up the local sales tax auditor, and tell him that you are deleting customer records.

  11. #11
    Join Date
    Jan 2011
    Posts
    8
    Quote Originally Posted by kitaman View Post
    Need an authoritative source? Have one of your colleagues call up the local sales tax auditor, and tell him that you are deleting customer records.
    They are not deleting any orders.

  12. #12
    Join Date
    Sep 2009
    Location
    Ontario
    Posts
    1,057
    So now I have a question.
    I have two credit cards, a visa and a mastercard, the name on each is subtly different (one has just my middle initial, the other has my full middle name) as is the address ( I live where both the rural route and the street address is required).
    If I purchased something from you using the visa card, then I would have to use the address that the visa card is billed to.
    If I tried to use that address with the mastercard, the transaction would be (and has been) rejected.
    By merging the two records are you going to create a bigger problem than the one you have by leaving them alone?

  13. #13
    Join Date
    Jan 2011
    Posts
    8
    Quote Originally Posted by kitaman View Post
    So now I have a question.
    I have two credit cards, a visa and a mastercard, the name on each is subtly different (one has just my middle initial, the other has my full middle name) as is the address ( I live where both the rural route and the street address is required).
    If I purchased something from you using the visa card, then I would have to use the address that the visa card is billed to.
    If I tried to use that address with the mastercard, the transaction would be (and has been) rejected.
    By merging the two records are you going to create a bigger problem than the one you have by leaving them alone?
    The type of information that you mention is kept in other records. These records link to the surviving record after the merge process.

    Why do you make so many assumptions?

  14. #14
    Join Date
    Jun 2003
    Location
    Ohio
    Posts
    12,592
    Quote Originally Posted by The User View Post
    Why do you make so many assumptions?
    Because we are not privy to all the details. What aren't you more clearly framing your question?
    As Papadi already pointed out: "The question is rather nebulous and would possibly be better dealt with if some specifics were provided. Not all data is the same. "
    If it's not practically useful, then it's practically useless.

    blindman
    www.chess.com: "sqlblindman"
    www.LobsterShot.blogspot.com

  15. #15
    Join Date
    Jan 2011
    Posts
    8
    Quote Originally Posted by blindman View Post
    Because we are not privy to all the details. What aren't you more clearly framing your question?
    As Papadi already pointed out: "The question is rather nebulous and would possibly be better dealt with if some specifics were provided. Not all data is the same. "
    That's a fair point to an extent. I want to know more about general approaches to destroying data rather than specifics. I am usually capable of figuring out what to do in particular cases on my own. Some of the responses that I have received have helped to stimulate further thought on the matter which is helpful enough.

    Even if I thought it would be suitable to offer more details I would be understandably reluctant to do so at the behest of kitaman. He started off with a sarcastic tone and has moved onto something bordering on accusatory.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •