The 43-year-old woman was part of an extended family group seen struggling in the ocean off Newhaven on Phillip Island about 3.30pm on Wednesday.
Two off-duty life guards who were surfing in the area pulled three people out of the water, while a rescue boat was required for the fourth.
All were unconscious and unresponsive, with a 23-year-old man, a 20-year-old woman and the 43-year-old woman dying at the scene.
Another 20-year-old woman was flown to the Alfred Hospital in a critical condition but later died.
It was the state's worst beach tragedy since 2005 when five people drowned at Stingray Bay near Warrnambool, Lifesaving Victoria said.
"I certainly say this is horrific," Eastern Region Assistant Commissioner Karen Nyholm told reporters on Thursday.
"For those people that witnessed it ... I'm sure it's something they will unfortunately have to recall for the rest of their lives."
A woman in her 20s was flown to the Alfred Hospital. (HANDOUT/7NEWS MELBOURNE)
Police initially identified the victims as three women in their 20s and a man aged in his 40s.
But on Thursday afternoon they clarified the victims were a 43-year-old woman from India, and a 23-year-old man and two women aged 20 from Clyde in Melbourne's southeast.
Ms Nyholm would not say what their family connection was, as next of kin were still being notified.
The Indian high commission in Australia confirmed it was providing assistance to the victims' family and friends.
Police are investigating what led to the drownings, although the Forrest Caves beach in Newhaven is a well known surf spot with numerous rips.
Heart breaking tragedy in Australia: 4 Indians lost their lives in a drowning incident at Phillip Island, Victoria. Deepest condolences to families of the victims. — India in Australia (@HCICanberra) @cgimelbourne team is in touch with friends of the deceased for all necessary assistance.@MEAIndia @DrSJaishankarJanuary 25, 2024
It is not patrolled although lifesavers are stationed in two neighbouring areas, Life Saving Victoria's Kane Treloar said.
"Forrest Caves is approximately four kilometres from both the patrolled site at Smiths Beach to the west and Woolamai Beach to the east," he told reporters.
"The beach at Forrest Caves is not for swimming - the rip currents ... are quite significant and only a very, very experienced swimmer should be entering the water to swim."
Lifesavers have urged anyone planning on going into the water off Phillip Island to stick to the area's patrolled beaches.
"It's certainly an absolutely tragic, tragic event," the agency's general manager of lifesaving operations Liam Krige said.
"It just goes to highlight that important safety messaging of making sure when you are heading out to the coastline, you do prioritise heading to a patrolled location."
Mr Krige said lifesavers would work with the local council to assess whether the beach needed to be patrolled in the future.
Victorian Premier Jacinta Allan said there will be an investigation into the drownings. (Joel Carrett/AAP PHOTOS)
"But we need to remember that water safety is more than just rescue," he said.
"It is about being really aware of the dangers and in the first instance, trying to prevent those rescues from even happening."
Premier Jacinta Allan extended her condolences to the families of the victims and thanked first responders who reacted quickly.
"There will be an investigation and reports on yesterday's incident in terms of what further actions need to be taken at that location," she told reporters on Thursday.
"The locals who know this area well don't swim there. It just reminds us of when we're visiting a place, is to be aware of the local conditions."
The island is a popular holiday destination southeast of Melbourne, famous for its penguins and the MotoGP.
The Visit Phillip Island tourism body said the caves were formed by cliff erosion and only accessible at low tide.
Police will prepare a report for the coroner and are not treating the deaths as suspicious.
14°C