"I feel like I was still playing good points," Pegula said.
"Elise was just being really tough, not making a lot of errors and making me play every single ball.
"With the windy conditions, I felt like it definitely played into her game.
"She returned well and she used the wind on both sides to her advantage."
Former doubles world No.1 Mertens used her complete arsenal, including sliced forehands, drop shots, lobs and deep groundstrokes to move her opponent around and pin her back.
Mertens, seeded 28th, was near flawless at the start, breaking the American twice in a row to race to a 5-0 lead and bag the first set in just 26 minutes.
Pegula, who had advanced into the third round after her ailing Italian opponent Camila Giorgi retired after the first set, broke Mertens early in the second but the Belgian responded in the next game.
The American earned another rare break point at 3-2 but sunk three consecutive backhands into the net.
The 27-year-old Belgian took advantage of Pegula's low first-serve percentage to break her again and move 5-3 clear, leaving her frustrated opponent to shake her head and look at her coaching box for help.
Her ordeal was over after 82 minutes, following her 28th unforced error.
Another US player, Peyton Stearns, also crashed out on Friday - beaten 6-0 6-1 in less than an hour by Russia's Daria Kasatkina.
- with AP