The embattled firm is accused of leaking information on tax avoidance to clients, for financial gain.
Former PwC partner Peter Collins has been referred to federal police to investigate the allegations while nine other partners have been stood down.
But Peter Dutton said a decision to cut other government contracts with the firm would be disruptive to the delivery of essential services.
"Obviously there's a very serious situation that continues to unfold and maybe we're not to the bottom of it yet," he told ABC Radio.
"Where people have breached the contract, they've breached trust, there's a penalty and a price that should be paid.
"I don't believe they should be cut from all government work ... I think the company has taken very significant action in sacking a number of staff who were involved in a very egregious act."
This week a Senate committee labelled PwC's conduct in the tax advice scandal as a calculated and egregious breach of trust.
The committee's interim report recommended that PwC publish the names of partners who were involved, as well as cooperate with investigations by the federal police and the Tax Practitioners Board.
A NSW government inquiry is also examining its own use of PwC consultancy and temporarily suspended taxation-related contracts with the firm.
14°C