This is not like Intel poaching Jim Keller from AMD. The U.S. had only a nascent rocket program at the end of the war. They did more than recruiting scientists, they also came back with loads of hardware. Within a year they were launching seized V-2s at altitudes that matched the Germans, and within 7-8 years it was manufacturing rockets that matched Germany. The only rival to the U.S. rocket program after the war was the Soviet Union which had their own program to recruit Nazi scientists[1].
Every field is "hobbyist level" before certain breakthroughs are made that allow it to take off. Look at computers before and after the invention of integrated circuits, transistors, or even vacuum tubes.
In this case those initial breakthroughs were made by the Nazis. Nobody is disputing that. But there is quite a leap to be made between lobbing explosives at London and putting live humans on the moon and then retrieving them, and many things besides scientists with dubious pasts were needed to make that leap. I do not understand what drives somebody to downplay those accomplishments every time the subject comes up. Your statement that those scientists were "what actually made the moon mission possible" is worded in a way that implies that they were the only thing that made it possible, rather than one factor among many, and that is objectively false. It's like saying that a spark plug is "what actually makes a car run".
We essentially imported the Nazi rocket program and continued investing in it. Obviously that continued investment was essential, but I don't think it's overstating it to say it's what made it possible. Many of the countries that have nukes have them as a direct result of espionage, and I don't think many people would object to saying that it was the espionage that made it possible, even though that is far less than importing large groups of scientists and equipment.