An extended family group was seen struggling in the ocean off Newhaven about 3.30pm on Wednesday before emergency services rushed to the scene.
Two off-duty life guards pulled three people out of the water, while a rescue boat was required for the fourth.
All were unconscious and unresponsive, with a man and woman in their 20s and a 43-year-old woman dying at the scene.
Another woman aged in her 20s was flown to the Alfred Hospital in a critical condition but later died.
It was the state's worst beach tragedy in almost 20 years, Lifesaving Victoria said.
"I certainly say this is horrific," Eastern Region Assistant Commissioner Karen Nyholm told reporters on Thursday.
"For those people that witnessed it ... I'm sure it's something they will unfortunately have to recall for the rest of their lives."
A woman in her 20s was flown to the Alfred Hospital. (HANDOUT/7NEWS MELBOURNE)
Police initially identified the victims as three women in their 20s and a man aged in his 40s.
But on Thursday afternoon clarified the victims were a 43-year-old woman from overseas and a man and two women aged in their 20s from Clyde in Melbourne's southeast.
Ms Nyholm would not say what their family connection was, as next of kin were still being notified.
Police are investigating what led to the drownings, although the Newhaven beach is a well known surf spot with numerous rips.
Lifesavers have urged anyone planning on going into the water off Phillip Island to stick to the area's three patrolled beaches.
"It's certainly an absolutely tragic, tragic event," the agency's general manager of lifesaving operations Liam Krige said.
"It just goes to highlight that important safety messaging of making sure when you are heading out to the coastline, you do prioritise heading to a patrolled location and you do prioritise water safety."
Mr Krige said lifesavers would work with the local council to assess whether the beach needed to be patrolled in the future.
Victorian Premier Jacinta Allan said there will be an investigation into the drownings. (Joel Carrett/AAP PHOTOS)
"But we need to remember that water safety is more than just rescue," he said.
"It is about being really aware of the dangers and in the first instance, trying to prevent those rescues from even happening."
Premier Jacinta Allan extended her condolences to the families of the victims and thanked first responders who reacted quickly.
"There will be an investigation and reports on yesterday's incident in terms of what further actions need to be taken at that location," she told reporters on Thursday.
"The locals who know this area well don't swim there. It just reminds us of when we're visiting a place, is to be aware of the local conditions."
The island is a popular holiday destination southeast of Melbourne, famous for its penguins and the MotoGP.
The Visit Phillip Island tourism body said the caves were formed by cliff erosion and only accessible at low tide.
Police will prepare a report for the coroner and are not treating the deaths as suspicious.
6°C