View Poll Results: Database vs Directory

Voters
15. You may not vote on this poll
  • Database

    3 20.00%
  • Directory

    12 80.00%
Page 1 of 2 12 LastLast
Results 1 to 15 of 21
  1. #1
    Join Date
    Jun 2007
    Location
    London
    Posts
    117

    Question Database vs Directory

    Hello all,

    (not sure if this has been asked)

    I would like to store some videos and images. A lot of them and i am guessing they will be accessed frequently and i want to minimise the strain on my host/server. So which one shall i use to store them:

    1) Put them all in a directory and use the database to store the path

    2) Store them in a database [if so which database and how - please provide link if you can]

    Thanks for any replies.

    (i hope the poll isn't pointless! )

  2. #2
    Join Date
    Feb 2004
    Location
    One Flump in One Place
    Posts
    14,912
    Quote Originally Posted by compsci
    (i hope the poll isn't pointless! )
    Good lord man - NO poll is pointless. I even polled people on that very topic but the poll was pulled (not mentioning any names Pat PHELAN).

    Anyhoo - I bet you get 80-20 split. Maybe more like 90-10.
    Testimonial:
    pootle flump
    ur codings are working excelent.

  3. #3
    Join Date
    Jun 2007
    Location
    London
    Posts
    117
    Quote Originally Posted by pootle flump
    Good lord man - NO poll is pointless. I even polled people on that very topic but the poll was pulled (not mentioning any names Pat PHELAN).

    Anyhoo - I bet you get 80-20 split. Maybe more like 90-10.
    lol - so i am guessing the (80-20) or (90-10), is in this form (directory-database)???

  4. #4
    Join Date
    Feb 2004
    Location
    One Flump in One Place
    Posts
    14,912
    Quote Originally Posted by compsci
    lol - so i am guessing the (80-20) or (90-10), is in this form (directory-database)???
    I'll leave you to find out.

    ...and you can't ask a question and then be the second person to vote in your own poll
    Testimonial:
    pootle flump
    ur codings are working excelent.

  5. #5
    Join Date
    Jun 2007
    Location
    London
    Posts
    117
    Quote Originally Posted by pootle flump
    I'll leave you to find out.

    ...and you can't ask a question and then be the second person to vote in your own poll
    I think vBulletin is to blame since the poll allows the poll setter to vote!

  6. #6
    Join Date
    Feb 2004
    Location
    One Flump in One Place
    Posts
    14,912
    I'm allowed to buy an electric drill but it does not mean it is the hardware store's fault when I drill clean through the wood and into my thigh
    Testimonial:
    pootle flump
    ur codings are working excelent.

  7. #7
    Join Date
    Jun 2007
    Location
    London
    Posts
    117
    Quote Originally Posted by pootle flump
    I'm allowed to buy an electric drill but it does not mean it is the hardware store's fault when I drill clean through the wood and into my thigh
    In the same way, if your employer provides un-safe work area for you to work in and you use the wrong ladder to do something, you can sue your employer.

    [I cant help feeling we are straying from the topic ]

  8. #8
    Join Date
    May 2004
    Location
    Seattle
    Posts
    1,313
    Quote Originally Posted by pootle flump
    I'm allowed to buy an electric drill but it does not mean it is the hardware store's fault when I drill clean through the wood and into my thigh
    right, it's the drill-makers fault. or perhaps the utility company for supplying the power to the drill.

    the rule is: whoever has the most money, it's their fault. ask any lawyer, they'll tell you i'm right on this.

  9. #9
    Join Date
    Oct 2002
    Location
    Baghdad, Iraq
    Posts
    697
    Look, this isn't "who wants to be a millionaire." Either the DBMS works properly and you can store the files as BLOBs or it doesn't and you have to store them as files.

    There are only downsides to using the filesystem but if you have to do it, you have to do it.

    The downsides are, if you haven't heard the litany, app must track orphans, backup is more complicated, accessing files is now different from accessing database, filesystem concurrency works different from dbms, etc.

    The reasons you have to do it are usually: your connector doesn't handle BLOBs properly (e.g. perl DBI requires that you pull the whole BLOB in all at once) or your DBMS doesn't handle BLOBs (e.g. bad performance, MS Access's wierd OLE shennanigans).

    One non-reason: you're running a website and the CGI that fetches the blob is too slow. Set up proper caching so this isn't an issue, e.g. Squid.

  10. #10
    Join Date
    Jun 2004
    Location
    Arizona, USA
    Posts
    1,848
    To summarize:
    1. Store videos and Images
    2. a lot of videos and images...
    3. frequent access
    4. desire to minimize strain (load) on host/server
    Now, I'm assuming that host/server indicates a web server app.

    By storing the images on a folder, there are more advantages than sco08y indicates. For instance, if images must be edited, an authorized user may more easily edit images in a folder than in a database record. Loading an image from file is more efficient than loading an image from a table (editing apps normally require that the image be in a standard graphic file, which necessitates moving the image from the database to an intermediate file)

    Depending upon the database server, blobs are subject to varying data size limits by the server itself. SQL Server, for instance, breaks up images to fit into database pages. The image must be reassembled before it can be displayed. A couple of other bottlenecks also exist, which together cause serving images from a database to a web page less efficient than loading directly from a file.

    Heck, Microsoft itself recommends saving images in file form...

    Permissions are easily set by the server administrator to control unauthorized access.

    Ref Images in Database or as PathReference
    Lou
    使大吃一惊
    "Lisa, in this house, we obey the laws of thermodynamics!" - Homer Simpson
    "I have my standards. They may be low, but I have them!" - Bette Middler
    "It's a book about a Spanish guy named Manual. You should read it." - Dilbert


  11. #11
    Join Date
    Jun 2007
    Location
    London
    Posts
    2,527
    I can't see any real advantage to storing video files in a database. Even if you store the videos in a directory on your web server then you're still going to swamp your web server pretty quick (storage space and bandwidth). If you have a lot of files and frequent access then I don't think either solution will work well.

    Couldn't you just store the files on your users servers and then link your users to the correct files. It should be easy to manage the load so no user gets overused. Popular files should be duplicated many times and perhaps split into sections across different machines. This would push the load from your server out to your users.

    Could you look into the methods that BitTorrent and YouTube use? Out of curiosity can I ask what type of videos they might be?

    Mike

  12. #12
    Join Date
    Oct 2002
    Location
    Baghdad, Iraq
    Posts
    697
    Quote Originally Posted by loquin
    For instance, if images must be edited, an authorized user may more easily edit images in a folder than in a database record.
    You just reminded me: you also have to synchronize permissions and user accounts between the RDBMS and the file system. Have fun with that!

    Loading an image from file is more efficient than loading an image from a table (editing apps normally require that the image be in a standard graphic file, which necessitates moving the image from the database to an intermediate file)
    Two arguments against this: 1. You're ignoring the fact that a good solution implements a caching system, which can minimize those advantages for frequently accessed items. 2. With a DBMS, the middle tier can convert file formats on the fly allowing you maximum flexibility for whatever graphics software you prefer. 3. There's no requirement to write to an intermediate file; we've had FIFOs since the 60s.

    Depending upon the database server, blobs are subject to varying data size limits by the server itself. SQL Server, for instance, breaks up images to fit into database pages. The image must be reassembled before it can be displayed. A couple of other bottlenecks also exist, which together cause serving images from a database to a web page less efficient than loading directly from a file.
    Filesystems have to do the exact same thing, these are known as file extents. Furthermore, RDBMS physical implementation is generally far more advanced than filesystems because they don't have the same constraints of providing binary compatibility.

    Heck, Microsoft itself recommends saving images in file form...

    Permissions are easily set by the server administrator to control unauthorized access.

    Ref Images in Database or as PathReference
    Fine, SQL Server might be bad at it. Neither source provides any proof of these claims in terms of reproducible tests. The second link suggests a straw man: using the DBMS to serve web pages, which is stupid. Obviously a web server will be better optimized to serve web pages. And then they ignore an obvious alternative setup, that of shifting your big blobs over to another machine so that you have a pipe for small traffic and one for BLOBs.

  13. #13
    Join Date
    Oct 2002
    Location
    Baghdad, Iraq
    Posts
    697
    Quote Originally Posted by mike_bike_kite
    I can't see any real advantage to storing video files in a database.
    The real advantage is that it's in a database. It's that simple.

    Could you look into the methods that BitTorrent and YouTube use? Out of curiosity can I ask what type of videos they might be?
    Wikipedia has an entry on BitTorrent which doesn't prefer any file format, video or otherwise. The entry on YouTube talks about their file format; to learn how they get videos to you, look up "Content Delivery Network."

  14. #14
    Join Date
    Jun 2007
    Location
    London
    Posts
    2,527
    Originally Posted by sco08y
    Wikipedia has an entry on BitTorrent which doesn't prefer any file format.
    But it does use files! You can see them if you go to your torrent directory. To be fair I have no idea how YouTube stores things but it does use much more compressed video images so it may be possible to store the data on their own servers - I still doubt very much they store the data in a database though.

    Originally Posted by compsci
    I would like to store some videos and images. A lot of them and i am guessing they will be accessed frequently
    He didn't mention how big the videos were but the average size of avi files on my sons system is around 600mb. I'm not sure how many "a lot" is but let's assume it's more than 1000. That means you'd need a huge 600GB database to handle this + large scale backup facilities + a similarly sized hot standby. Also bear in mind the amount of traffic you're hoping to get - each access for a video file in this case is going to be a 600mb download. This all implies huge costs. Whoever is paying for this would want to know why it should be done this way.

    Originally Posted by sco08y
    The real advantage is that it's in a database. It's that simple.
    For a nasty moment I thought you were going to convince me with a good point

    A few pro's for a distributed file based system
    Keeping the files on end user machines (like BitTorrent) and using a database to keep track of all the files is probably best if you're planning of keeping lots of video files because

    • It will be quicker to get to the files as you're not relying on one server.
    • Less bandwidth is required because rather than sending a 600mb video out each time from your server you can just point to 100 users with the file and let them spread the load.
    • It's easy to expand. If your system took off and you needed to store 1 million videos then there would be no issues storing 1m pointers to external files but storing 1m avi files on your database would require 600,000GB - you'll have to wait a long time for Moore's Law to catch up.
    • It will be more reliable as the same files are spread across multiple machines.
    • It will be cheaper as no need to store huge videos so you can use a pint sized server. No need for a huge backups, huge hot standby systems etc.


    Mike

  15. #15
    Join Date
    Jun 2007
    Location
    London
    Posts
    117
    Quote Originally Posted by mike_bike_kite
    Couldn't you just store the files on your users servers and then link your users to the correct files. It should be easy to manage the load so no user gets overused. Popular files should be duplicated many times and perhaps split into sections across different machines. This would push the load from your server out to your users.
    Could you explain your idea more, or do you have somewhere i can read your idea it sounds very interesting!

    Quote Originally Posted by mike_bike_kite
    Could you look into the methods that BitTorrent and YouTube use? Out of curiosity can I ask what type of videos they might be?
    Mike
    I would like to know what YouTube uses actually!!! Since they have a storage that has already exceeded 60TB?

    The type of videos will mostly be flash...but it can also be mpeg, avi etc

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •