Emergency service crews rushed to Vincent Street, Daylesford, northwest of Melbourne, at about 6pm on Sunday following reports an SUV had crashed into the beer garden.
Victoria Police said the deaths occurred after the white BMW mounted the kerb and struck patrons who included two men, one woman, a boy thought to be six years old and a teenage girl.
Emergency authorities said a number of other people were injured, including the driver of the car, a 66-year-old Mount Macedon man.
Superintendent John Fitzpatrick from Victoria Police said they were waiting to speak to the driver, who had non-life-threatening injuries, and were still trying to determine what happened.
A boy was flown to hospital with critical injuries while a man, a woman and a boy had non-life-threatening injuries.
Police have asked the public to avoid Daylesford so emergency services can access the scene.
Victoria Ambulance regional director Trever Weston told Nine's Today show emergency crews were confronted with "an extremely chaotic scene".
He said members of the public did a "fantastic job" to help the injured until ambulance paramedics arrived.Â
"We had about seven ambulances and four ambulance helicopters dispatched and conveyed those patients off to hospital," Mr Weston said.
Infrastructure Minister Catherine King, whose federal Ballarat electorate includes Daylesford, asked the community to follow advice from emergency services as crews carry out their work.
"My thoughts are with all those at the scene, their friends and families and the broader Daylesford community," she said in a statement on X.
Anyone who witnessed the incident or has dashcam or CCTV footage is urged to contact Crime Stoppers.