It’s impossible to know for sure, but I have a theory: The most bitter opponents of Ryan are motivated almost solely by the issue of immigration reform — but the Freedom Caucus has a lot of members who are (or have been) open to some type of reform (which makes sense, considering the libertarian framing of the name “Freedom Caucus”).
In a joint 2013 letter written to Sen. Rand Paul , signed by Reps. Mick Mulvaney , Justin Amash , Thomas Massie , Jeff Duncan , and Mark Meadows , the group advocated “expanding legal immigration” and finding a way to “reasonably address the reportedly 11 million people who came here knowingly and illegally…”
Rep. Daniel Webster , the original Freedom Caucus candidate for Speaker, has advocated immigration reform (so long as it comes after the border is secured). Consider his quote in this 2013 article from the AP: “‘I think as a country we need to do something,’ Webster said in an interview, echoing the rhetoric of Florida Sen. Marco Rubio and other prominent Republicans. ‘Doing nothing is amnesty.’”