Two-time defending champion Alcaraz needed nearly three hours to defeat Jan-Lennard Struff 6-3 6-7 (5-7) 7-6 (7-4) in a rematch of last year's final in the Spanish capital and reach the quarter-finals.
Nadal, a record five-time champion in Madrid, struggled from the start in a 7-5 6-4 loss to 31st-ranked Czech player Jiri Lehecka.
Jiri Lehecka powers a forehand during his straight-sets win over Rafael Nadal in Madrid. (AP PHOTO)
Both Spaniards are coming off injuries and are trying to get back in shape ahead of next month's French Open.
The 20-year-old Alcaraz had missed tournaments in Monte Carlo and Barcelona, and the 37-year-old Nadal had played two matches in Barcelona after not playing competitively in nearly three months.
The 22-time grand slam champion arrived in Madrid saying he wasn't fully fit and was worried about his condition, but in the end was celebrating having won three consecutive matches without showing significant physical limitations.
Nadal had only one break opportunity against the big-serving Lehecka, who has won his three matches in Madrid in straight sets.
Second-seeded Alcaraz squandered four match points while serving for victory at 5-3, but converted on his first opportunity to seal the victory in the deciding tiebreak at the Caja Magica centre court.
"I wasn't at my best physically toward the end of the match, but I'm happy that in the end I found my game," Alcaraz said.
Carlos Alcaraz was stretched to the limit in beating Jan-Lennard Struff at the Madrid Open. (AP PHOTO)
"I fought for every ball and didn't let down despite some difficult moments when things didn't go my way."
Alcaraz is trying to become the first player to win three straight Madrid Open titles. He also needed three sets to beat Struff in last year's final. The world No. 3 will next face seventh-seeded Andrey Rublev at the claycourt tournament.
Top-seeded Jannick Sinner defeated 16th-seeded Karen Khachanov 5-7 6-3 6-3 to reach the Madrid quarter-finals for the first time. He is the only player to make it to the last eight at all four ATP 1000 events this season.
"I think today was really tough, because at some points he served really well, so it was tough to return," the Italian said.
"I made a couple of mistakes in the first set when he broke me, but this can happen. In the second set I tried to stay focused immediately, I broke him, and the confidence level raised a bit."
Francisco Cerundolo, ranked 22nd, upset two-time champion Alexander Zverev 6-3 6-4 for his third career win over a top-five player. Cerundolo next plays Taylor Fritz after the American defeated Hubert Hurkacz 7-6 (7-2) 6-4.
Third-seeded Daniil Medvedev was given the run-around by Alexander Bublik before winning 7-6 (7-3) 6-4. The Russian next plays 30th-seeded Lehecka.
