But what is the reason behind it?

I was able to use a single command in my second example.
Here is one way to resolve the error:
db2 "alter tablespace dmsts reduce (all containers 976)"
DB20000I The SQL command completed successfully.
db2 "alter tablespace dmsts reduce (file 'dmsts1' 8)"
DB20000I The SQL command completed successfully.