The Welsh held their nerve in a frantic Pool C clash that was tense throughout, especially when Fiji scored twice in the last seven minutes and centre Semi Radradra knocked on close to the line in the dying seconds.
At 32-14 ahead deep inside the final quarter, Wales looked home and dry, but Fiji had other ideas.
Asked if he had his heart in his mouth during the closing stages, Gatland said: "Absolutely. With seven minutes to go I wondered about just going down to the changing rooms and waiting until the final whistle.
"I am absolutely delighted with the result. It is pretty significant for us.
"We've always been a team that builds on confidence and get better in tournaments. So that is exciting.
"I was pretty frustrated and angry about the last period of the game, but it's about being honest and making sure we learn from it.
"I am delighted with the win, but we made it a lot harder for ourselves than we needed to, made some dumb decisions in the last 15 minutes.
"A few of their players looked a bit tired and we looked in control, but typical Fiji - they never say die, get a couple of sniffs of a try and come back at you."
Wales ultimately prevailed through tries from Josh Adams, George North, Louis Rees-Zammit and Elliot Dee, with fly-half Dan Biggar adding two penalties and three conversions in a bonus-point success.
Fiji claimed tries through captain Waisea Nayacalevu, flanker Lekima Tagitagivalu and replacements Josua Tuisova and Mesake Doge yet Wales gained the victory they craved ahead of remaining Pool C appointments with Portugal, Australia and Georgia.
Wales were close to a staggering 250 tackles in the game.
"It was absolutely exhausting. The last 10 minutes felt like the clock never moved," Biggar said.
"We put everything into it, Fiji came strong at the end, but I am really thrilled for this group of players that we managed to see it through because the work we've put in over the past few months has just been incredible."
Captain Jac Morgan said Wales won through thanks to a "mentality of never giving up, always working hard for each other".
Fiji took two losing bonus points from the game, which might prove critical in the race for a last-eight spot.
"We had our opportunities," head coach Simon Raiwalui said.
"We weren't clinical enough to finish off, and we move on to Australia next week.
"We made mistakes at critical moments, and we had to chase it at the end."
The Wallabies are next up for Fiji, with the teams due to meet in Saint-Etienne on September 18 (1.45am AEST), while Wales take on Portugal the previous day.
