Results 1 to 5 of 5
  1. #1
    Join Date
    Apr 2007
    Posts
    130

    Unanswered: efficiency comparision between memory access and database access(hard disk access)

    Dear All,

    I am working on a project towards a text retrieval system. In the system, a document is represented by features (i.e., some words and phrases conveying the essence of the document). And the similarity between a query and a certain document is dertermined by counting the matching of features. Currently, at retrieval time, features of all documents are held in memory. However, holding all feature data in memory will become impossible as the number of documents dramatically increases.


    I have thought of this problem for a long time. The only way I can figure out at this time is to use database technology. However, the problem that follows is that the drop-off in efficiency (retrieval response time will increase as accessing hard disk, where database is located, is slower than accessing memory ) by using database technology. I do not have any experience and idea about this. Or there may be other better technologies available?

    Sometimes, I wonder how Google to solve this problem.

    Please help. Thanks!

  2. #2
    Join Date
    Aug 2003
    Location
    Where the Surf Meets the Turf @Del Mar, CA
    Posts
    7,776
    Provided Answers: 1
    >(retrieval response time will increase as accessing hard disk, where database is located, is slower than accessing memory )
    True, but from where would the data in memory originate?
    You can lead some folks to knowledge, but you can not make them think.
    The average person thinks he's above average!
    For most folks, they don't know, what they don't know.
    Good judgement comes from experience. Experience comes from bad judgement.

  3. #3
    Join Date
    Aug 2007
    Location
    Malta
    Posts
    10
    True but applying the 80/20 rule, there may be some gains.

    If a part of the system is used a lot, it 'might' make sense to store the most used data in memory.

  4. #4
    Join Date
    Apr 2007
    Posts
    130
    Quote Originally Posted by anacedent
    >(retrieval response time will increase as accessing hard disk, where database is located, is slower than accessing memory )
    True, but from where would the data in memory originate?


    Thanks for your reply.

    When start up the retrieval system, in initial stage all data (features of documents, features are extracted from documents in a dataset by another program in advance) are read into memory. This will take about 20-30 seconds.

  5. #5
    Join Date
    Aug 2003
    Location
    Where the Surf Meets the Turf @Del Mar, CA
    Posts
    7,776
    Provided Answers: 1
    Rarely does the system hosting any production RDBMS application have enough RAM memory to hold the complete database in memory.

    If the whole DB can be read in under 30 seconds, then the backend data store is miniscule compared to any production application about which I have 1st hand knowledge.
    You can lead some folks to knowledge, but you can not make them think.
    The average person thinks he's above average!
    For most folks, they don't know, what they don't know.
    Good judgement comes from experience. Experience comes from bad judgement.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •