Quick fix, not using FM Server:
If you're NOT using FileMaker Server, then create an .fp5 file that exits the application at start up (write a script that exits application, then set that as the startup script). Create another file that opens all the files. Create windows scheduled tasks that will run launch these new Closing an Opener files.
Quick fix, using FM Server:
Send FM Server the "FMServer STOP" and "FMServer START" commands (in the command shell) using windows scheduled tasks.
Assuming you're using FM Server, you can setup scheduled tasks in FileMaker Server that will pause the service, backup the files, and then restart the service. (You don't have to do anything in the command shell, just point and click to what files to backup, when days to backup, what times to backup, and where to backup to). Then, your tape backup would just grab the data from the FM Server backup directory, not the live data.
I don't think this method will work, but I'm dying to know. Win 2003 server has an implementation of shadow copying (VSS, I think), whereby the OS silently (and invisibly to applications) grabs open/in-use files out from under services and applications and creates backups of them - so you don't have to interrupt services or applications.
Could you try using this to create a backup of FileMaker files while they're open, then try opening the back up files created by the shadow copy? I think the files will open with errors ("This file was not closed correctly, FileMaker is performing a consistency check, etc..."), since the backup was made while some flag in the FMP app was still set to "dirty" (the new file is created with the flag telling it that it is still open). Let us know what you find Thanks!