Faced with the possibility of the congressional equivalent of a no confidence vote, Ryan would probably decide to pull a Boehner and immediately resign as speaker, announce that he won’t run again as speaker in the next Congress or announce that he is resigning from Congress at the end of this session.
Any of these options would throw the House into almost immediate chaos and severely limit what will get done in the lame duck.
Ryan is not likely to do what Boehner did by cleaning out the barn and making deals with House and Senate Democrats and the Obama administration on his way out the door. He has presidential ambitions and any last-minute deals on spending and other issues would greatly complicate a run for the GOP nomination.
Ryan might also want to stay in Congress and resume being chairman of the House Ways and Means Committee. Last-minute closed-door arrangements with Democrats would likely make that impossible.