Seven-time grand slam winner and two-time Open finalist Williams was handed a wildcard, becoming the oldest ever women's player in the Melbourne Park main draw.
She held a 4-0 third-set lead over 24-year-old Serb Olga Danilovic but lost the next six games to go down 6-7 (5-7) 6-3 6-4 and exit in the first round.
"I felt good. Also it's the biggest lead I've had since I've been back," Williams said with a smile.
"In a lot of ways, I'm having to relearn how to do things again, if that makes any sense.
"I'm really proud of my effort today because I'm playing better with each match, getting to the places that I want to get to.
"It's kind of weird, but it's super exciting to have played that well and to get myself in that position and come very close."
An emotional Danilovic pounded her chest in celebration after sealing her epic comeback victory over a tennis legend.
"These things don't happen every day and playing against Venus Williams is something I cannot take for granted," Danilovic said.
"But there was a lot of nerves. At 4-0 I just said to myself, 'OK, just play. Just take everything out and just play point by point', and I'm very happy I managed to get this one.
"But it was such a pleasure playing against such a legend."
World No.69 Danilovic will next face either world No.3 Coco Gauff or Uzbek Kamilla Rakhimova.
Crowd favourite Williams went up 5-4 then teed up a set point on Danilovic's serve with a vicious forehand winner - but the Serb saved it and took the set to a tiebreak.
The American claimed victory in the tiebreak, to the crowd's delight.
Danilovic composed herself and raced away to claim the second set, before Williams ruthlessly clubbed her way to a 4-0 lead in the decider.
But she let Danilovic back into the match.
Williams' service game at 4-4 dragged on for more than 14 minutes before Danilovic broke, then successfully served for the match.
As Williams bowed out, she smiled graciously as the adoring crowd applauded her.
"I had to appreciate the crowd. They were right there with me. They deserved that moment. We deserved that moment together, so..." she said.
"It was an amazing journey on the court today."
Williams will now turn her focus to doubles with Ekaterina Alexandrova.
