Can someone tell me why the following sql statement cannot retrieve the
maximum count of the record:

SELECT col, max(count(*)) FROM table
GROUP BY col

And also, can I use one sql statement to retrieve the maximum count of the
record? If not, how?

Thank you more!!

You cannot give a function as argument for max().
Try this

SELECT col, max(colcount) from
(SELECT col, count(*) as colcount
FROM TABLE
GROUP BY col) AS aa
GROUP BY col

Could you explain in details why we cannot give a function as argument for max()

It makes no sense. You can give a scalar function as an argument for a column function, but not a column function as argument for a column function (look at SQL Reference).