The five-year-old has spent several weeks at his usual twice-yearly haul-out spot in southern Tasmania.
Videos of Neil crushing signs and poles and lounging on suburban streets near fences have gone viral online and attracted streams of visitors to the otherwise sleepy beach spot.
On Thursday, Tasmania's Department of Natural Resources and Environment said it appeared Neil had left the area.
"Neil the Seal returned safely to sea last night. This is his natural behaviour and was expected," it said.
He may still return or haul-out at a nearby location in the short-term, or be off to southern feeding grounds.
Wildlife authorities recently issued a firm warning, telling people to keep their distance from Neil and saying euthanasia was a last resort if public safety couldn't be guaranteed.
There were reports of people getting up close with young children for photos and leaving food for the apex predator.
The local council, which is in the process of tallying up Neil's damage bill, was forced to manage traffic and a 24-hour security presence was hired.
Neil was born in southern Tasmania without a colony and is pre-programmed to return to the area twice a year to rest and moult.
Experts have said his "jousting" with posts and other objects is natural behaviour and something seals of his age usually do with each other.
Neil was rescued by wildlife officers from a sandbar and is expected to reach three tonnes when fully grown.
There were once southern elephant seals in Tasmania but they were wiped out by hunting in the early 1800s - the species is currently listed as vulnerable in Australia.
"Neil is potentially one of the first southern elephant seal pups to be born back in Tasmania," department wildlife expert Kris Carlyon has said.
"Regardless of the resource burden and the challenges that Neil throws, we're pleased to see him."
People need to keep 20m away from Neil, keep dogs on a lead at least 50m away and avoid blocking his access to water.
"We'll keep an eye out for him but otherwise the traffic cones and street signs of Tasmania can breathe a small sigh of relief," Tasmanian Premier Jeremy Rockliff said.
"A big well done for everyone that has treated big Neil with respect and caution while he's been on land."
10°C