Rangers have failed to reach the last 32 of the Europa League after Dejan Ljubicic's late winner gave Rapid Vienna victory in Austria.
Steven Gerrard's side needed a win to qualify, but instead it is their hosts who reach the knock-out stage.
As both sides found it difficult on a rutted pitch, Rangers' Connor Goldson struck the bar with a header.
Home goalkeeper Richard Strebinger saved from James Tavernier before substitute Ljubicic slid in the winner.
That secured a first win in four home games for Rapid, who finish as runners-up to Villarreal after the La Liga side beat Spartak Moscow 2-0 in Spain.
It was Rangers' second defeat in 14 Europa League games and ends a journey that began in the first round of qualifying against Shkupi in July.
This was a massive opportunity missed for Rangers against a distinctly average Rapid team.
Their European campaign petered out after the early promise of a point in Villarreal and the win at home against the Austrians.
Allan McGregor was a virtual spectator until he was beaten by Ljubicic's late finish. It came at a time when Rangers were throwing everything forward in search of the goal they needed.
But it was the goalkeeper's poor kick-out that gave Rapid possession and, with a rare piece of incisive passing, they cut through the visitors' back-pedalling defence, giving Ljubicic the time and space to slide the ball into the net.
Ultimately, Rangers barely troubled Strebinger in the Austrian goal other than that late Tavernier drive that forced the goalkeeper into a good diving save.
That aside, Rangers struggled to create meaningful opportunities on a dreadful playing surface.
In truth, the state of the pitch meant neither team could put together a fluent move in their search for goals.
The closest Rangers came to finding the goal they so badly needed came from Tavernier's pinpoint free kick that Goldson headed against the crossbar.
The thousands of Rangers supporters who had managed to get tickets for what might have been a momentous night stood in appreciation of the players at the final whistle. The dejected men in blue had put plenty into the match but came up short.
They will reflect on a campaign that brought results they may not have anticipated at the start of the season, but also one that could have ended better.
It was a dearth of creativity that haunted them on Thursday. In Moscow, it was poor defending and an assistant's flag that denied them a fourth, potentially winning, goal. And at Ibrox, there was the late Eros Grezda chance to beat Spartak but instead was scooped over the bar.
Rangers are a work in progress - Gerrard has been quite clear about that. A European run has been enormous fun for the supporters, but the priority has to be trying to stay within touching distance of Celtic in the league and challenging for domestic silverware.
To do that, they must quickly regroup and find a better, more consistent level of form, starting against Hamilton Academical on Sunday.