Australia steal it at the death
They outscored the Scots by five tries to three, but required a last-minute penalty by flyhalf Bernard Foley to seal the deal.
It was a questionable call, one of several contentious decisions by South African referee Craig Joubert and his fellow match officials.
Foley, who missed his first three goal-kicks, held his nerve amidst raucous boos from Scotland fans unhappy with the call by Joubert.
It had seemed Mark Bennett's intercept try seven minutes from time had set up a huge World Cup upset by Scotland, who lost all five of their matches in this year's Six Nations.
Bennett intercepted a pass from Wallaby replacement prop James Slipper and ran in under the posts for a try converted by Scotland captain Greig Laidlaw, who kicked 19 points in total.
But Australia, the 1991 and 1999 World Cup champions will now face Argentina who beat Ireland 43-20 earlier, in next weekend's semifinal.
Scotland's defeat means the last four will be an entirely Southern Hemisphere affair – a mini Rugby Championship.
"We were stuck in a dark spot for a while but we managed to get out of it which was very pleasing," said Drew Mitchell, scorer of two of Australia's five tries.
Australia had a measure of control at 25-19 heading into the final quarter.
But Finn Russell's charge-down of opposing flyhalf Foley's kick led to a try for wing Tommy Seymour.
Laidlaw missed the difficult conversion, but Australia's lead had been cut to a point at 25-24.
Australia appeared to have wrapped up the match in the 63rd minute.
They won a line-out off a Scotland throw and outside centre Tevita Kuridrani, bursting through a gap in the cover, surged over for a try converted by Foley.
That gave Australia a 32-24 lead. But Laidlaw's fifth penalty cut the lead heading into the last 10 minutes as rain started to pelt down upon Twickenham.
Australia, eight points down at one stage early on, were 15-16 behind at the break but rallied soon after when Scotland wing Sean Maitland was sin-binned for a deliberate knock-on.
The Wallabies were without two influential players in the injured duo of fullback Israel Folau and No.8 David Pocock.
But despite this, and losing two of their last three Tests against Scotland, Australia still started as odds-on favourites in the match in which skipper Stephen Moore and centre Matt Giteau became the seventh and eighth Wallabies respectively to win 100 Test caps.
Scotland made two late changes to their side, recalling hooker Ross Ford and lock Jonny Gray after the pair won their appeals against three-week bans imposed for a dangerous tackle against Samoa.
Australia had their first try in the ninth minute when Kuridrani burst clear and was then missed by Seymour.
Kuridrani's pass gave right wing Adam Ashley-Cooper a clear run into the corner.
Foley missed the difficult conversion but Australia led 5-0.
Laidlaw got Scotland on the board with a 12th-minute penalty and five minutes later they scored a remarkably simple try.
Powerful carries by WP Nell and David Denton took them to within five metres of the Australia line before centre Peter Horne simply picked up and burst through a static Wallaby defence for a try under the posts.
Laidlaw's conversion and penalty then made it 13-5 to put Scotland two scores in front.
But Scotland's defence, so porous in a 36-33 win over Samoa, was soon breached again.
Drives by lock Kane Douglas and prop Sekope Kefu took Australia to within five metres of Scotland's line.
The ball was worked back from Foley to Kurtley Beale, in for Folau and his pass found wing Drew Mitchell on an overlap out on the left
Foley, however, again failed to convert and Scotland led 13-10.
That became 16-10 when Laidlaw's penalty extended Scotland's lead.
But on the stroke of half-time Australia had their third try.
Giteau booted a kickable penalty into touch for a line-out and, from the ensuing driving maul, Michael Hooper crossed.
Foley, however missed his third successive goal-kick and Scotland were still 16-15 in front.
Maitland was yellow carded early in the second half for a deliberate knock-on.
That decision was made even worse for Scotland when it led to Australia's fourth try, with Mitchell – off another line-out move, going into the space vacated by the sin-binned Maitland for his second try of the match.
This time, Foley converted to put Australia 22-16 in front only for scrumhalf Laidlaw's fourth penalty to keep Scotland in touch.
Foley replied in kind and then came the most dramatic finale possible.
Man of the match: Stuart Hogg showed why he is regarded as one of the best fullbacks in the tournament, Greig Laidlaw was the general that kept his team in the game, Dave Denton was a massive influence all over the field and Willem Nel was a monster in the scrums. Kurtley Beale looked to inject himself into the game and at times looked very threatening, Adam Ashley-Cooper was always hard to stop. Michael Hooper and Scott Fardy and Ben McCalman were very much involved in all aspects of the game. However, our award goes to barn-storming Wallaby centre Tevita Kuridrani, who produced some big plays – including a crucial try.
Moment of the match: There were eight tries, but Bernard Foley's match-winning penalty at the death gets this one.
Villain of the match: Nothing nasty, not even Sean Maitland's yellow card – it was one of several contentious calls.
The scorers:
For Australia:
Tries: Ashley-Cooper, Mitchell 2, Hooper, Kuridrani
Cons: Foley 2
Pens: Foley 2
For Scotland:
Tries: Horne, Seymour, Bennett
Cons: Laidlaw 2
Pens: Laidlaw 5
Yellow card: Sean Maitland (Scotland, 42 – professional foul, slapping the ball down)
Teams:
Australia: 15 Kurtley Beale, 14 Adam Ashley-Cooper, 13 Tevita Kuridrani, 12 Matt Giteau, 11 Drew Mitchell, 10 Bernard Foley, 9 Will Genia, 8 Ben McCalman, 7 Michael Hooper, 6 Scott Fardy, 5 Rob Simmons, 4 Kane Douglas, 3 Sekope Kepu, 2 Stephen Moore (captain), 1 Scott Sio.
Replacements: 16 Tatafu Polota-Nau, 17 James Slipper, 18 Greg Holmes, 19 Dean Mumm, 20 Sean McMahon, 21 Nick Phipps, 22 Matt Toomua, 23 Quade Cooper.
Scotland: 15 Stuart Hogg, 14 Sean Maitland, 13 Mark Bennett, 12 Pete Horne, 11 Tommy Seymour, 10 Finn Russell, 9 Greig Laidlaw (captain), 8 Dave Denton, 7 John Hardie, 6 Blair Cowan, 5 Richie Gray, 4 Jonny Gray, 3 Willem Nel, 2 Ross Gord, 1 Alasdair Dickinson.
Replacements: 16 Fraser Brown, 17 Gordon Reid, 18 Jon Welsh, 19 Tim Swinson, 20 Josh Strauss, 21 Henry Pyrgos, 22 Richie Vernon, 23 Sean Lamont.Referee: Craig Joubert (South Africa)
Assistant referees: Glen Jackson (New Zealand), Pascal Gauzère (France)
TMO: Ben Skeen (New Zealand)
Agence France-Presse & @rugby365com
