Mr Wyatt, the first Aboriginal person to be Indigenous minister, rejected claims from Opposition Leader Peter Dutton that the voice would be a "Canberra voice" and the body would be elitist.
"It was a hard decision, I had to weigh up my love of the Liberal Party and its values and what it stands for against a stubborn position of not wanting to give Aboriginal people a seat at the table," he told the ABC's 7.30 program.
"I wanted to fight for the voice because this is something we've asked for over many decades."
Mr Wyatt resigned on Thursday from the party he had served as minister, after Mr Dutton said it would oppose a constitutionally enshrined voice to advise parliament and executive government.
The former minister said the decision by the opposition leader was a mixture of genuine opposition and political opportunity.
"It's a tough decision to make but you've got to weigh this up against a section of our community who live with gaps in their lives that are so substantial," he said.
"It is about local community designing their regional voice and then the regional voice selecting people they want to represent them in Canberra. It's not a Canberra voice. It is not elite. It is people from the grassroots."
The comments came on the same day the coalition's Indigenous Australians spokesman Julian Leeser quit the shadow ministry also due to the opposition to the voice.
Mr Leeser said he believed the time for a national voice had come.
"I believe the voice can help move the dial on Indigenous education, health, housing, safety and economic development," he said, announcing his resignation on Tuesday.
"I believe better policy is made when the people affected by it are consulted."
Mr Dutton said while he and Mr Leeser agreed on the need for regional Indigenous people to be heard, they differed when it came to constitutional change.
He lauded his colleague as a man of strength and character but maintained the Liberal Party would continue to chase local and regional voices.
"We want to listen to those people in the communities to get the best possible outcome for them," he said.
"Julian had a view about the voice being enshrined in the constitution. He is at odds with the overwhelming majority of the Liberal party room."
Mr Leeser will push the government to amend the proposed wording it has put forward in a bid to improve the chances of the referendum's success.
"The government must seriously engage with coalition voters and it hasn't done so to date," he said.
"This failure could ultimately put the referendum at risk. An all-or-nothing approach could deliver nothing. That's why we must find common ground."
But he said he would ultimately support the voice even if his amendments weren't taken on board.
"It's right that it has a place in the constitution," he said.
Indigenous Australians Minister Linda Burney welcomed the news, saying Mr Leeser had joined a growing choir of Liberals coming out to support the voice.
"Julian Leeser has shown strength in putting his principles ahead of politics," she said.
"Aboriginal and Torres Strait Islander people have extended a generous invitation to all Australians to walk together to a better future."
Senior minister Penny Wong said she expected more Liberals to break ranks and support the 'yes' campaign against "Mr Dutton's partisan narrow position".
10°C