I haven't used MySQL full-text search, but I'm used to Oracle Text, and I think I understand your problem. I think you are facing two distinct issues :
- It is normal not to retrieve words with special characters (such as 'ä') when you try to replace these characters with standard ones (such as 'a' for 'ä'), because the indexing was only done on the word with 'ä', not the word with 'a'. MySQL may allow you to define rules such as 'ä'='a' for indexing, but I doubt so. I believe that you will have to replace programmatically special characters with standard ones BEFORE indexing and BEFORE querying, so that you never insert or query with special characters.
- If you get no result or strange results when querying with special characters, then I suppose you don't use the same character set when querying as the one you used when inserting (I speak of the client character set), hence a misinterpretation of special characters which are not converted the same way before being sent to the server.