Since I need it to be as close as live data and due to some limitation on protocol for getting data, the fastest I can update is in 5 seconds.
Every 5 seconds I will grep around 200 of market data for one market_name and one currency
So in one day I'll get
( 86400 / 5 ) * 200 = 3,456,000 row
In one month I'll get around
3456000 * 30 = 103,680,000
That's a lot .... of row
And for now I got 5 market_name with one same currency but it will be more later.
After that I need to grep the last data imported and like month of data from different market for analysis.
I never work with this much data before max is only 5 mill and MySQL still can handle that. I'm not sure about this one. Do any of you have experience that can be share