As someone who works on installation routine for a living I would say that a true uninstall should remove all files, folders, shortcuts, and registry keys. But if there is the possibility of an re-install or an upgrade you have three options.
1) Do nothing remove the config file along with the rest of the application.
2) Prompt for back up of the config files so the user has access to them if they choose to re-install or upgrade.
3) Automatically back up the config files for the user and let them how where there are in a message box.
Personally I would go for option 2 and give the user the ability to specify where the backups should be created.