The national director of equality advocacy group Democracy in Colour said on Tuesday Nazi Hakenkreuz, or hooked cross, was enough to make people of colour and members of the LGBTQI+, Jewish and other communities fearful.
"It made me feel unsafe and that myself and others around me could be in danger of physical harm," Neha Madhok told a parliamentary inquiry looking into legislation for penalties over use of the symbol.
She said that while she supported fines for using the symbol, there could be unintended consequences for sending people to prison.
"By putting people into the prison system, they are far more likely to be exposed to other radical elements, people with far more extremist views," Ms Madhok said.
Australia-Israel and Jewish Affairs Council executive director Colin Rubenstein said jail and fines could have an impact, especially on recruitment to extremist groups.
"This legislation is one of the strategies to deal with the evil,'' Mr Rubenstein said.
He said political leadership and education were also needed.
However, Larry Stillman from the Australian Jewish Democratic Society said he was concerned someone would pay the fines on behalf of offenders.
Mr Stillman said the media needed to be educated on how to talk about Nazism and extremism to avoid amplifying the hatred and ideology.
"You're not just going to be able to stop them by throwing them in jail," he said.
The Australian Muslim Advocacy Network's Rita Jabri Markwell said one way to combat hate against Muslims was to avoid conflating Islam with terrorism through the term "religiously motivated" when referring to groups such as ISIS.
Ms Markwell said acts of violence, such as the Christchurch massacre in New Zealand, were conducted by ethno-nationalists who do not necessarily brand themselves as Nazis.
"It's fuelled racist nationalist movements," she said.
"No religion supports terrorism, no religion supports sexual slavery."
The Australian Federal Police says the new laws could close the current gap, with the public display of the Nazi symbol normalising the hateful ideology that can lead to radicalisation.
Counter-terrorism investigations commander Nicholas Read told the inquiry radicalisation online has increased, including through games that simulate scenarios from Nazi Germany.
"It's very similar in terms of how we've seen pedophiles groom youth online," he said.
Counter-terrorism assistant commissioner Krissy Barrett said problems could arise with policing the ban because far-right groups could alter the symbol.
But she added taking away a well-known symbol would have an impact.
"A symbol can be much more powerful than a lot of words can be," Ms Barrett said.
"They may look at using other symbols, but primarily they're using those really well-known symbols to communicate the ideology."
She said some groups could go underground or contravene the law to gain notoriety.
The attorney-general's department said there needs to be a clear definition of the symbol and the context it is used to ensure it can be accurately enforced.
Buddhist groups are calling for a carve-out when it connects to Buddhism, Hinduism or Jainism due to the swastika being derived from an ancient symbol for wellbeing or peace.
The inquiry was told the value of the legislation was also about sending a message to the community about what was right and wrong, as opposed to securing prosecutions.
"Criminal sanctions have an important role to play in setting standards," Race Discrimination Commissioner Chin Tan said.
20°C