The Astana rider went on his own in the long climb to Chamrousse to open up a lead of three minutes and 37 seconds over Spain's Alejandro Valverde (Movistar) in the overall standings.
The Italian, aiming to complete a set of grand tour titles, having won the Giro d'Italia last year and the Vuelta a Espana in 2010, said: "I wanted to control the race. It was a very long climb, I thought it would never end. But I wanted to gain time on Valverde and I knew Porte had been dropped. After joining [Rafal] Majka and [Leopold] Konig, there was not much collaboration in the group so I decided to go on my own."
Majka (Tinkoff-Saxo) claimed second place ahead of Konig (NetApp-Endura), with Valverde 50 seconds down in fourth place.
Nibali also assumed the polka dot jersey from Joaquim Rodriguez (Katusha) but he admitted the race is far from won. The 29-year-old said: "For the coming days, I only know that I have to remain quiet - there are still many mountains ahead.
"I've suffered a lot today because of the heat but at least as we were climbing I felt better and better. My intention was just to control in the final climb but when I saw Richie Porte in trouble . . . my goal became to gain important seconds on Valverde."
Romain Bardet, the leading French hope, remains third overall, ahead of Thibaut Pinot, despite losing 30 seconds to his fellow countryman during the 197.5km trek from Saint-Etienne, and France is one step closer to having a rider on the podium in Paris for the first time since 1997.
Bardet, who holds the white jersey for the best under-25 rider, is 4:24 off the pace, with Pinot 16 seconds back ahead of today's second stage in the Alps, a 177km ride from Grenoble to Risoul. Porte, who started the day in second place overall, cracked early in the last ascent, losing more than eight minutes to drop out of contention, 11 minutes 11 seconds behind Nibali overall.
Porte lost contact with the yellow- jersey group 12.5km from the finish as Nibali went for his third stage win of the Tour and second in as many hilltop finishes after his 10th-stage victory on Monday's 161.5km ride from Mulhouse to La Planche des Belles Filles.
Temperatures reached 35C yesterday and Porte said: "I don't think I dealt with the heat very well. It's one of those things. It's a massive shame but we'll just have to see what happens tomorrow. I feel more for my team-mates who have been brilliant for me every day but if it happens to me I think it can happen to other guys too, so I'll just have to keep on wishing."
Sir Dave Brailsford, the Team Sky principle, admitted their hopes of making it three in a row, after the triumphs of Sir Bradley Wiggins and Froome, are over. He said: "We've had plenty of good days in the Tour de France over the last couple of years and today wasn't one of them.
"We've had two difficult days, obviously with Chris crashing out, and everyone could see it was a tough day for Richie. We'll take stock tonight and see if we can get into a situation where we can animate the race - there's no point us just coming here and riding round. We have to take the blows - it's not easy, it's disappointing - and think about the coming days in the mountains and what opportunities we have to do other things in the race."
There will be no respite for Porte today with the 14th stage of the race taking the riders over the Col d'Izoard, at 2,360m the highest point on this year's Tour. Valverde admitted he would have to recuperate if he is to eat into Nibali's increasing lead.
He said: "I rode full gas to do the best I could but when Nibali attacked I couldn't follow him. Tomorrow will again be difficult and I will need to recover some strength.
The peloton held a minute's silence at the start of yesterday's stage in memory of those killed in when the Malaysian Airlines jet crashed in Ukraine. Many of them were Dutch citizens and riders from the Dutch teams Belkin and Giant-Shimano wore black armbands.