Results 1 to 6 of 6
  1. #1
    Join Date
    Feb 2010
    Posts
    2

    A very big number of columns

    How to deal with a very large number of columns?

    This my problem: I need to track an object which emits 1 million different variables several times a day (in average 10 times a day).

    Technically, it is possible to split the variable into groups, and then put each group in a table. but I can't imagine myself create the schema with one million columns...
    Are relational databases the best choice to deal with that?

    I thought I can create a table like a dictionary with two columns <key, value> and though have 1 million rows instead of 1 million columns... But it will be difficult to write queries...
    Last edited by poplop; 02-16-10 at 06:51. Reason: message was trunked

  2. #2
    Join Date
    Feb 2004
    Location
    One Flump in One Place
    Posts
    14,912
    What do you mean by "deal"? What do you think is a large number? Is this in one table and is it normalised?
    Testimonial:
    pootle flump
    ur codings are working excelent.

  3. #3
    Join Date
    Jul 2009
    Location
    Michigan
    Posts
    125
    I can't imagine an object that has 1 million different attributes. Can you provide more detail about what it is you are trying to track?

  4. #4
    Join Date
    Feb 2010
    Posts
    2
    It is a kind of robot explorer and the variables describe its state and the environment.
    No matter what are the variables, and for sure, it won't be 1 million variables but maybe 990000 or something like that...

    I wonder if the present DB software will be useful for that

  5. #5
    Join Date
    Feb 2004
    Location
    One Flump in One Place
    Posts
    14,912
    You aren't really looking at a typical relational solution.

    The sort of solution for this sort of problem are traditionally either EAV or an XML column. You will need to research these terms. EAV is just an extension of your two column <key value>. Note that both of these solutions come with health warnings, but with a problem like this you have to compromise somewhere.
    Testimonial:
    pootle flump
    ur codings are working excelent.

  6. #6
    Join Date
    Jun 2007
    Location
    London
    Posts
    2,527
    I still don't believe there's a million variables. Can you just list a few of the variables (not all of them) so we can get an idea of what data you're holding. Is it the environment around the robot? - in which case it should just be the time, x pos and y pos, the type of sensor and the value read. If it's internal sensors on the robot then it's just time, sensor id, reading. If it's camera or ultrasonic data then shouldn't you be analysing the data before storing any info.

    What environment does the robot explore? what type of sensors are you using? What will you do with the data? Is it a real robot (photo please) or just a pretend one?

    Mike

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •