Mr Albanese was speaking at the Woodford Folk Festival in Queensland on Thursday, where he was advocating for the voice before a member of the crowd interrupted him.
"What's the point of a voice if people don't f***ing listen," a man shouted from the crowd.
In response, the prime minister said it was a "very good point".
Mr Albanese dismissed criticism the voice lacked detail, saying the proposed body was described in a more than 200-page document which had undergone consultation over five years.
He said the voice had since been scrutinised by parliamentary committees and other inquiries.
Mr Albanese promised the referendum to enshrine the voice and constitutionally recognise Indigenous people would be held by the end of next year.
"It will be when we join with our continent's mosaic of ancient nations that our modern nation will find its greatest strength, achieve its fullest potential and realise its truest self," he said in a speech to the festival.
But the opposition has accused the government of failing to provide voters with adequate detail on the mechanics of the body.
Mr Albanese's speech on Wednesday was marked by climate change protests.
19°C