The masked men dressed in black marched down Sturt Street in Ballarat on Sunday chanting and holding a banner reading "Australia for the White Man".
Police cars were travelling behind and in front of the group.
The demonstration was unplanned and police say they are investigating reports of a Nazi salute being performed.
A 15-year-old boy who was not attached to the group was spoken to by police but charges are yet to be laid.
Premier Jacinta Allan condemned the "hateful" behaviour in the strongest possible terms.
"Ballarat is one of the birthplaces of our state's rich multiculturalism - people came to the gold fields from every corner of the world in search of a better life," she said in a statement on Monday.
"Every Victorian deserves to live free from fear of racism and bigotry.
"These disgraceful and cowardly acts have no place in Victoria."
Local MP and federal Infrastructure Minister Catherine King MP was similarly scathing.
"Bigotry, racism and Nazism are never welcome in Ballarat," she wrote on X, formerly known as Twitter.
Ballarat Community Alliance, an association of progressive community organisations, said the city was a proud multicultural one.
"We condemn this group of blow-ins and their message of hate," the alliance said in a statement on Facebook.
"They have come to Ballarat to co-opt the Eureka legacy on the inclusive and peaceful commemoration of the anniversary of Eureka stockade."
In October, Victoria made it illegal to intentionally display or perform a Nazi gesture or symbol in public.
The change was triggered a group performing the salute outside Victorian parliament earlier this year.
The ban applies to anything closely resembling a Nazi gesture or symbol but exceptions apply if either is displayed in good faith for a genuine academic, artistic, educational or scientific purpose.
Those who break the law face fines of more than $23,000 or 12 months in prison.
8°C