I had a similar issue years ago with Visual FoxPro (chills down the spine) and SQL Server. I think the issue you are experiencing is because SQL Server has a max memory setting and when something is running in process it can not exceed this setting. If you allow something to run out of process, it can gobble up as much memory on the server that it wants to.
As for what we did to fix it, I do not remember. Sorry. You might want to check if your sql server dynamically configures memory. The max memory a sql server can handle depends on the edition the OS and the hardware.
If one brings so much courage to this world the world has to kill them or break them, so of course it kills them. The world breaks every one and afterward many are strong at the broken places. But those that will not break it kills. It kills the very good and the very gentle and the very brave impartially. If you are none of these you can be sure it will kill you too but there will be no special hurry. Earnest Hemingway, A Farewell To Arms.