Ancelotti, who took charge of PSG in January, had seen his team rescue draws and scrape wins with very late goals in recent matches, but this time PSG's luck ran out and it has now conceded 13 goals in the last six games — including a 4-4 draw away to Lyon last month.
Nene put the French leader ahead in the 19th minute from the penalty spot, but Kim Kallstrom equalized with a sweetly struck freekick soon after and Lisandro Lopez gave Lyon the lead shortly before the interval.
"The first defeat has come but we need to keep looking forward," Ancelotti said. "We didn't play well in the first half, we showed no aggression, no intensity. It was different in the second half, where we were in total control.
"We're disappointed to go out, but we'll focus on the league now, there are 10 games left. I'm not worried, but we need to win our next game (against Bordeaux on Sunday)."
PSG pressured Lyon's defense throughout the second half, but France goalkeeper Hugo Lloris kept his cool to make several saves and Bafetimbi Gomis punished Ancelotti's side for those missed chances when he stroked home a penalty in injury time.
"No team had won here this year, so it's a great win for us, but we have to keep our feet on the ground," Lyon coach Remi Garde said. "We struggled a bit in the second half, PSG pushed us all the way. I take my hat off to the players."
Lyon looked comfortable until center half Samuel Umtiti was harshly judged to have handled Jeremy Menez's cross from the left and Nene sent Lloris the wrong way from the spot.
After Kallstrom curled a sublime freekick over the wall, PSG went close to regaining the lead in the 35th when Menez burst down the left and set up striker Kevin Gameiro, but his shot was smothered by Lloris.
Lyon rocked PSG when Lopez scored from close range after a mix up in PSG's defense, and Lloris then denied Nene before PSG left the field to jeers.
Ancelotti brought on record signing Javier Pastore for the second half, and PSG immediately pressured Lyon's defense, with center half Alex, Menez and then Pastore forcing Lloris into smart saves.
The action was almost entirely in Lyon's half and, in the 60th minute, Pastore's pass gave midfielder Blaise Matuidi the perfect chance to equalize, but he blazed over the bar.
Lopez missed an even easier chance as Lyon hit PSG on the break down the right flank. Gomis squared the ball to him, but he smacked it over with goalkeeper Nicolas Douchez stranded.
Gomis wrapped up the win with moments left after Alex tripped Michel Bastos as he cut inside the area.
Earlier, semiprofessional team Gazelec Ajaccio produced another upset to reach the semifinals for the first time, beating league title contender Montpellier 1-0 from Yohan Bocognano's goal.
The defender volleyed in from a corner with 15 minutes remaining to stun Montpellier, which is in second place in the French league behind PSG.
"It's huge. It's very satisfying for me as I scored," Bocognano said. "We had nothing to lose. We didn't ask ourselves any questions and in the end you couldn't tell the difference between us and a first division team. Our fabulous adventure continues."
The Corsican club's win followed Marseille's 3-2 loss to Quevilly in extra time on Tuesday. Both Gazelec and Quevilly play in the semiprofessional National league, three tiers below the first division.
"The pitch didn't help us, but that's no excuse," Montpellier midfielder Karim Ait-Fana said. "It's a match we have to forget."
Rennes won 3-1 at Valenciennes in an all-first division match.
Winger Jonathan Pitroipa put Rennes ahead in the 12th minute before Mamadou Samassa equalized for the home side. John Boye and forward Youssouf Hadji sealed the win for Rennes.
"Sometimes we were on top and at other times we were up against it a bit," Rennes coach Frederic Antonetti said. "But on the whole I think we deserved to win."
Gazelec plays Lyon, and Quevilly face Rennes in next month's semifinals.