Wild scenes erupted as police wrangled crowds outside Parliament House, where a group of masked men dressed in all black performed the Nazi salute, sparking a bitter war of words.
Members of a Campaign Against Racism and Fascism group chanted "refugees welcome, Nazis are not" as they faced off with a group holding a banner for the far-right National Socialist Network.
A Victoria Police spokesperson said there was a heavy police presence across the city.
"Officers are equipped and well prepared to deploy resources to ensure the safety of the community and to keep the peace," they said.
"Hate and prejudice hold no place in our community and we will not tolerate any offensive and abhorrent anti-social behaviour."
Earlier on Saturday, Police declared the CBD a "designated area" from 7am to 7pm in response to the planned "stop immigration rally".
The designation means officers are empowered to search a person or their vehicle for weapons.
Greens deputy leader Mehreen Faruqi condemned the Nazi salute display.
"Disgraceful to see Neo-Nazis on the streets of Melbourne again, this time rallying against immigration," she tweeted.
"Fascism & racism have no place in our multicultural society & must be destroyed."
Most states and territories have enacted bans or are in the process of outlawing the display of Nazi symbols, with the salute covered in some jurisdictions.
Victoria has banned Nazi symbols in public and is moving to include the Nazi salute.