ok i have made a homemade H shifter and in lfs i got everything setup and i have it on manual clutch but i have to hold it in gear to keep it in gear if i let go of it it goes back into nutral without me touching the clutch.
if the homemade shifter is a FFB-joystick like mine, you may need to turn down the centering spring settings. I hope i helped, you didn't give much info on the shifter...
its not a ff shifter it was made from a game pad with switches that are engaged when put into gear. theres no return sprind but i want it to stay in gear so that i can consintrade more on stering instead of holding the shifter in place.
Unless your shifter keeps the button pressed while in gear it will not work. I get the feeling your system presses the button as you slot into gear but then it does not lock the stick in place to keep the button pressed.
simple test. Go into the control panel for the controller. Put in gear and see if it stays in gear. The button in question should still be lit after you let go of the stick