Well, here is my 0.2 hryvnia (2 cents US) -
I drove with a controller for years (big caluses on my thumbs - I looked deformed) and when I switched to a wheel it took me 3 months to get back close to my controller pace ......
Like you, I already knew the racing lines, so obviously the slower times with the wheel were because I couldn't "control the controller" (wheel) - not because I was a noob that didn't "know". You just need to learn the "feel" with the different kind of controller .....
To get that "feel" of the wheel and pedals, what worked for me was to drive quite a few different cars and NOT worry too much about lap times ..... just have fun, "toss" the car around, play with the lines, do things like it was "Bump and Jump". Maybe even battle with the AI in races.
For example, when learning with the wheel, you brake and just don't "feel" the car's deceleration. You brake, you look and then have to think: "Am I slowing? How much am I slowing? Am I slow enough now to start turn?" Your mind is relying on more than the visual (I see I am slowing down) - you need to learn that feel with the wheel almost as if you had never driven with some other controller. Same applies to the steering and gas.
IOW, you already know how to drive - you need to learn the new feel so that you sense the changes in the car's behavior more accurately and without having to "think".
Stick to it - you will get there. And the experience of immersion is so much better once you are comfortable with the wheel.