Databases operate by caching data so that instead of reading the contents from disk it reads from memory. The more memory you have the higher the cache hit rate. In most databases the cache hit rate should be > 95%. Meaning that 95% of hits read from memory rather than reading from disk. If you reduce the amount of cache space available your hit rate will go down which means responses from your queries will take longer.
So to answer your question, yes 128MB should be fine but have a small memory footprint is at the expense of performance.