Initially I would say look at the amount of data your application is supposed to query each time it calls the data. And then look at how often it does that.
Then look at how much work it takes to request and deliver the data (incl. network time and server and client I/O etc. in your environment).
Generally, I would suggest always query only the exact data you need, never the whole bunch at a time if it's really a lot, like thousands of rows or more.
Unless you have very many concurrent transactions and your database and server can't handle all the database concurrent connections (like hundreds of users at once doing all kinds of things with the table you need). in that case you may want to store the needed data temporarily somewhere else (another object or temporary table or whatnot).
Also of course it always depends on what DB and System you're using...
Maybe someone else here can go into more specifics...