Mr Dutton is in Alice Springs and has been highlighting crime and unrest in the central Australian community as a reason why local solutions are needed instead of a national voice.
But he came under fire for saying there were children in the town who were being sexually abused yet people were being told nothing could be done about it.
Acting Prime Minister Richard Marles said both sides of politics had worked to improve the situation in Alice Springs and problems in the Northern Territory town shouldn't be politicised.
"It's really important that we're not making Alice Springs a political football. A lot of work has gone on here to try and improve the situation in Alice Springs," he told Nine's Today program on Friday.
"We need to be working with each other to try to improve the situation."
The Northern Territory police minister on Thursday slammed Mr Dutton's comments as a "dog act", adding his visit was opportunistic after his party's decade in government.
Kate Worden challenged Mr Dutton to come forward to police and report any crimes he was aware of given the NT's mandatory regime for reporting child abuse.
"What we've seen ... from Peter Dutton in central Australia is absolutely opportunistic, political game-playing and using the most vulnerable people here in the heart of our nation as a pawn in that game," she said.
Liberal senator Kerrynne Liddle said there needed to be care about politicising any form of assault, but she fired back at people who had attacked Mr Dutton for failing to provide evidence to back up his sexual assault claims.
"I say prove it's not happening and then we can have a conversation about the kind of language we can actually use for this," Senator Liddle told ABC radio on Friday.
"You've got the statistics, which everyone accepts are under-reported and under-represented."
Indigenous senator Jacinta Price, the former deputy mayor of Alice Springs, also defended Mr Dutton, saying Ms Worden's comments were disgusting.
"I know of children in my own extended family, I know of children in other people's families, everybody knows what's going on," the Country Liberal Party senator said.
Senator Price said the opposition leader, a former police officer, wanted to see a reduction in child sexual abuse.
Mr Dutton has also accused the NT government of playing politics, saying the chief minister has refused to act. He has called for the Australian Federal Police to be sent in.
17°C