The Liberals are facing criticism from within and outside the party over the decision to oppose enshrining a national Indigenous voice in the constitution.
Ken Wyatt, the first Aboriginal person to hold the Indigenous Australians portfolio, said he had resigned from the Liberals as he did not believe in what the party had become.
He said it had rejected the invitation of Aboriginal people who had reached out and asked to be heard.
The Liberal Party decided to support constitutional recognition for Indigenous people but rejected the government's proposal to enshrine a national voice.
The party has instead called for legislated regional and local voices to be established.
Deputy Liberal leader Sussan Ley said she was saddened by Mr Wyatt's decision, but she wouldn't criticise her former colleague for the course he took.
"I respect the decision that every Australian will come to in the referendum," she told Seven's Sunrise program on Friday.
"We are asking questions, it's not good enough for the prime minister to issue demands that people follow his lead."
The Liberals argue a constitutionally enshrined voice will divide the nation.
Cabinet minister Jason Clare hit back at the Liberals for accusing the government of not putting out enough detail about the voice.
He said while the prospect of the referendum passing had become harder, he still had faith Australians would vote 'yes'.
"The Australian people are bigger and better than the mean and negative Liberal Party," he said.
"I have faith they will meet this moment later this year."
Victorian Premier Daniel Andrews accused the federal opposition of adopting a "mean and nasty position".
"We are big enough in this country, we are generous enough, we're smart enough to take this step forward," he said.
"For far too long, we've told Aboriginal people how it's going to be and that has to stop.
"We have to do more listening... and the voice is a critically important part of that."
Tasmanian Liberal moderate Bridget Archer said she was not the only member who felt their party was at a crucial juncture and she pledged to campaign in support of the voice despite her party's position.
Mr Dutton has said he has the support of 99 per cent of the party room.
Liberal frontbencher Simon Birmingham, who is bound to the party position as a member of the shadow cabinet, refused to say whether he would actively campaign against the voice.
The senior moderate told ABC Radio his approach would be "one of respecting the Australian people as they go about making their decision".
"At the republic referendum John Howard campaigned for a 'no 'vote, Peter Costello campaigned for a 'yes' vote. Some were vocal, some were not, and that may well be replicated on this occasion," he said.
Referendum working group member Marcia Langton also attacked the opposition leader for suggesting it would be a "Canberra voice" that would be stacked with academics and not represent the views of Indigenous people around Australia.
"He couldn't be further from the truth and I deeply resent that deceitful opinion that he's expressed," she told the ABC's 7:30 program.
"His opposition to the voice relies absolutely on deceit and misrepresentations and, I have to say, a great deal of ignorance."
Professor Langton also attacked the opposition leader for suggesting the government's proposed wording went against the solicitor-general's advice.
"That is absolutely not correct. I'm really astonished a member of the Australian parliament would be so deceitful and propagate what is essentially a lie," she said.