Hello all,

(not sure if this has been asked)

I would like to store some videos and images. A lot of them and i am guessing they will be accessed frequently and i want to minimise the strain on my host/server. So which one shall i use to store them:

1) Put them all in a directory and use the database to store the path

2) Store them in a database [if so which database and how - please provide link if you can]

Thanks for any replies.

(i hope the poll isn't pointless! )

Good lord man - NO poll is pointless. I even polled people on that very topic but the poll was pulled (not mentioning any names Pat PHELAN).

Anyhoo - I bet you get 80-20 split. Maybe more like 90-10.

pootle flump ur codings are working excelent.

lol - so i am guessing the (80-20) or (90-10), is in this form (directory-database)???

I'll leave you to find out.

...and you can't ask a question and then be the second person to vote in your own poll

pootle flump ur codings are working excelent.

I think vBulletin is to blame since the poll allows the poll setter to vote!

I'm allowed to buy an electric drill but it does not mean it is the hardware store's fault when I drill clean through the wood and into my thigh

pootle flump ur codings are working excelent.

In the same way, if your employer provides un-safe work area for you to work in and you use the wrong ladder to do something, you can sue your employer.

[I cant help feeling we are straying from the topic ]

right, it's the drill-makers fault. or perhaps the utility company for supplying the power to the drill.

the rule is: whoever has the most money, it's their fault. ask any lawyer, they'll tell you i'm right on this.

Look, this isn't "who wants to be a millionaire." Either the DBMS works properly and you can store the files as BLOBs or it doesn't and you have to store them as files.

There are only downsides to using the filesystem but if you have to do it, you have to do it.

The downsides are, if you haven't heard the litany, app must track orphans, backup is more complicated, accessing files is now different from accessing database, filesystem concurrency works different from dbms, etc.

The reasons you have to do it are usually: your connector doesn't handle BLOBs properly (e.g. perl DBI requires that you pull the whole BLOB in all at once) or your DBMS doesn't handle BLOBs (e.g. bad performance, MS Access's wierd OLE shennanigans).

One non-reason: you're running a website and the CGI that fetches the blob is too slow. Set up proper caching so this isn't an issue, e.g. Squid.

To summarize:

1. Store videos and Images
2. a lot of videos and images...
3. frequent access
4. desire to minimize strain (load) on host/server

Now, I'm assuming that host/server indicates a web server app.

By storing the images on a folder, there are more advantages than sco08y indicates. For instance, if images must be edited, an authorized user may more easily edit images in a folder than in a database record. Loading an image from file is more efficient than loading an image from a table (editing apps normally require that the image be in a standard graphic file, which necessitates moving the image from the database to an intermediate file)

Depending upon the database server, blobs are subject to varying data size limits by the server itself. SQL Server, for instance, breaks up images to fit into database pages. The image must be reassembled before it can be displayed. A couple of other bottlenecks also exist, which together cause serving images from a database to a web page less efficient than loading directly from a file.

Heck, Microsoft itself recommends saving images in file form...

Permissions are easily set by the server administrator to control unauthorized access.

Ref Images in Database or as PathReference

I can't see any real advantage to storing video files in a database. Even if you store the videos in a directory on your web server then you're still going to swamp your web server pretty quick (storage space and bandwidth). If you have a lot of files and frequent access then I don't think either solution will work well.

Couldn't you just store the files on your users servers and then link your users to the correct files. It should be easy to manage the load so no user gets overused. Popular files should be duplicated many times and perhaps split into sections across different machines. This would push the load from your server out to your users.

Could you look into the methods that BitTorrent and YouTube use? Out of curiosity can I ask what type of videos they might be?

Mike

You just reminded me: you also have to synchronize permissions and user accounts between the RDBMS and the file system. Have fun with that!

Two arguments against this: 1. You're ignoring the fact that a good solution implements a caching system, which can minimize those advantages for frequently accessed items. 2. With a DBMS, the middle tier can convert file formats on the fly allowing you maximum flexibility for whatever graphics software you prefer. 3. There's no requirement to write to an intermediate file; we've had FIFOs since the 60s.

Filesystems have to do the exact same thing, these are known as file extents. Furthermore, RDBMS physical implementation is generally far more advanced than filesystems because they don't have the same constraints of providing binary compatibility.

Fine, SQL Server might be bad at it. Neither source provides any proof of these claims in terms of reproducible tests. The second link suggests a straw man: using the DBMS to serve web pages, which is stupid. Obviously a web server will be better optimized to serve web pages. And then they ignore an obvious alternative setup, that of shifting your big blobs over to another machine so that you have a pipe for small traffic and one for BLOBs.

The real advantage is that it's in a database. It's that simple.

Wikipedia has an entry on BitTorrent which doesn't prefer any file format, video or otherwise. The entry on YouTube talks about their file format; to learn how they get videos to you, look up "Content Delivery Network."

But it does use files! You can see them if you go to your torrent directory. To be fair I have no idea how YouTube stores things but it does use much more compressed video images so it may be possible to store the data on their own servers - I still doubt very much they store the data in a database though.

He didn't mention how big the videos were but the average size of avi files on my sons system is around 600mb. I'm not sure how many "a lot" is but let's assume it's more than 1000. That means you'd need a huge 600GB database to handle this + large scale backup facilities + a similarly sized hot standby. Also bear in mind the amount of traffic you're hoping to get - each access for a video file in this case is going to be a 600mb download. This all implies huge costs. Whoever is paying for this would want to know why it should be done this way.

For a nasty moment I thought you were going to convince me with a good point

A few pro's for a distributed file based system
Keeping the files on end user machines (like BitTorrent) and using a database to keep track of all the files is probably best if you're planning of keeping lots of video files because

• It will be quicker to get to the files as you're not relying on one server.
• Less bandwidth is required because rather than sending a 600mb video out each time from your server you can just point to 100 users with the file and let them spread the load.
• It's easy to expand. If your system took off and you needed to store 1 million videos then there would be no issues storing 1m pointers to external files but storing 1m avi files on your database would require 600,000GB - you'll have to wait a long time for Moore's Law to catch up.
• It will be more reliable as the same files are spread across multiple machines.
• It will be cheaper as no need to store huge videos so you can use a pint sized server. No need for a huge backups, huge hot standby systems etc.

Mike

Could you explain your idea more, or do you have somewhere i can read your idea it sounds very interesting!

I would like to know what YouTube uses actually!!! Since they have a storage that has already exceeded 60TB?

The type of videos will mostly be flash...but it can also be mpeg, avi etc