At a symposium in Adelaide on Monday, both the Public Health Association of Australia and the Alliance for Gambling Reform said the unit should sit within the Department of Health and Aged Care.
They also called for a properly resourced national regulator and for a mandatory pre-commitment scheme for online gambling services.
Rather than tackling gambling harms, governments across Australia consistently failed to take action, the two groups said.
"The tens of billions of dollars that the gambling industry sucks out of households across the country each year is shameful, and the harms and stresses caused to individuals, families and the wider community is preventable," the public health association's chief executive Terry Slevin said.
"We urge governments who receive revenue through gambling, and political parties who receive donations from gambling companies, to set aside their pecuniary interest and act on behalf of the most vulnerable members of the community, who are those who most often pay the biggest price."
The symposium urged all state and territory governments to commit to introducing cashless gambling cards with mandatory and binding loss limits when used on poker machines in all casinos, pubs, and clubs.
It also called for pokie venues to close between midnight and 10am, the period when people were considered most vulnerable to gambling harm.
Alliance for Gambling Reform chief executive Carol Bennett said Australian governments had been asleep at the wheel when it came to regulating the gambling industry.
"It is beyond time for federal, state and territory governments to step up and address gambling harm as a public health problem that is exacting a huge health, social and economic toll on communities, especially our most vulnerable," she said.
9°C