Our customers updated from 8.1 to 8.6.
Now they have to reboot their servers or restart the server-engines after some days, because the applications can't connect to the database.
It seems ntdbsmgr.exe occupies more and more memory until the described problem appears. Anyone who has a solution or experienced the same?

Haven't seen that behavior. What error do they get when trying to connect? Also, is there anything in the PVSW.LOG on the server? If it continues, you should open an incident with Pervasive.