If this is your first visit, be sure to check out the FAQ by clicking the link above. You may have to register before you can post: click the register link above to proceed. To start viewing messages, select the forum that you want to visit from the selection below.

 
Go Back  dBforums > General > Database Concepts & Design > A very big number of columns

Reply
 
LinkBack Thread Tools Search this Thread Display Modes
  #1 (permalink)  
Old 02-16-10, 05:37
poplop poplop is offline
Registered User
 
Join Date: Feb 2010
Posts: 2
A very big number of columns

How to deal with a very large number of columns?

This my problem: I need to track an object which emits 1 million different variables several times a day (in average 10 times a day).

Technically, it is possible to split the variable into groups, and then put each group in a table. but I can't imagine myself create the schema with one million columns...
Are relational databases the best choice to deal with that?

I thought I can create a table like a dictionary with two columns <key, value> and though have 1 million rows instead of 1 million columns... But it will be difficult to write queries...

Last edited by poplop; 02-16-10 at 05:51. Reason: message was trunked
Reply With Quote
  #2 (permalink)  
Old 02-16-10, 05:44
pootle flump pootle flump is offline
King of Understatement
 
Join Date: Feb 2004
Location: One Flump in One Place
Posts: 14,905
What do you mean by "deal"? What do you think is a large number? Is this in one table and is it normalised?
__________________
Testimonial:
Quote:
pootle flump
ur codings are working excelent.
Reply With Quote
  #3 (permalink)  
Old 02-16-10, 07:54
MarkATrombley MarkATrombley is offline
Registered User
 
Join Date: Jul 2009
Location: Michigan
Posts: 125
I can't imagine an object that has 1 million different attributes. Can you provide more detail about what it is you are trying to track?
Reply With Quote
  #4 (permalink)  
Old 02-16-10, 09:54
poplop poplop is offline
Registered User
 
Join Date: Feb 2010
Posts: 2
It is a kind of robot explorer and the variables describe its state and the environment.
No matter what are the variables, and for sure, it won't be 1 million variables but maybe 990000 or something like that...

I wonder if the present DB software will be useful for that
Reply With Quote
  #5 (permalink)  
Old 02-16-10, 10:24
pootle flump pootle flump is offline
King of Understatement
 
Join Date: Feb 2004
Location: One Flump in One Place
Posts: 14,905
You aren't really looking at a typical relational solution.

The sort of solution for this sort of problem are traditionally either EAV or an XML column. You will need to research these terms. EAV is just an extension of your two column <key value>. Note that both of these solutions come with health warnings, but with a problem like this you have to compromise somewhere.
__________________
Testimonial:
Quote:
pootle flump
ur codings are working excelent.
Reply With Quote
  #6 (permalink)  
Old 02-16-10, 13:48
mike_bike_kite mike_bike_kite is offline
vaguely human
 
Join Date: Jun 2007
Location: London
Posts: 2,519
I still don't believe there's a million variables. Can you just list a few of the variables (not all of them) so we can get an idea of what data you're holding. Is it the environment around the robot? - in which case it should just be the time, x pos and y pos, the type of sensor and the value read. If it's internal sensors on the robot then it's just time, sensor id, reading. If it's camera or ultrasonic data then shouldn't you be analysing the data before storing any info.

What environment does the robot explore? what type of sensors are you using? What will you do with the data? Is it a real robot (photo please) or just a pretend one?

Mike
Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are On