...so, as you can see if we get one error (typically for us it a SMALLINT or DECIMAL field and the number trying to be loaded is too big) the script errors out, loads messages to the message file and induces a page out to support people. I am trying to think of some way to not have the script fail yet still populate a messages file. In the morning the support people could then look at this messages file and pull the bad rows from the text file, fix the bad data (as messages file would tell us rows and error) and then have it be loaded on the next run of the job. These jobs only run at 2-4am and thus the need to at least look for other ways than job failure and immediate fixing of the bad data.
I can increase the warningcount so the script would keep loading but when i do that nothing is written to the messages file (which i believe is working as designed).
One approach is to pre-process the data file(s) before load, with a perl script or awk script, and just find the broken lines and either exclude them from the output file of the script, or do a data-repair before writing the corrected line to the output file of the script. Then load the modified file.
Other approaches exist...