All 44 of the city's high-rise public housing estates will be demolished and redeveloped by 2051 under Victoria's housing statement unveiled on Wednesday.
Five towers in Flemington, North Melbourne and Carlton will replaced by 2031.
The urban renewal project is expected to lift the number of residents across the sites from roughly 10,000 to 30,000 and its social homes by 10 per cent.
Merri-bek City Council, a Greens-led local government in Melbourne's inner north, said the 10 per cent commitment fell far short of combating growing public housing demand.
"Fundamentally, this fails to address crippling shortages throughout the state," council mayor Angelica Panopoulos said.
Federal Greens leader Adam Bandt and state party leader Samantha Ratnam have suggested the plan could signal the "end of public housing in Victoria".
"Labor's demolishing public homes and selling off public land with no guarantee all - or any! - new homes will be public housing," Mr Bandt wrote on social media.
Selling his government's housing statement atop a Box Hill high-rise apartment building on Thursday, Mr Andrews accused the Greens of spreading misinformation and scaring people.
"This is a really big project and a critically important one," he told reporters
"And if your answer is just the status quo and to scare people, that's not leadership."
Asked why all the extra homes couldn't be earmarked for public tenants, Mr Andrews pointed to the "enormous" cost of redeveloping the towers.
"If you don't have a partnership with the private sector, where will that money come from?" he said.
"Lots of people want to live on those sites and we can get a mix of housing, a mix of different families living side by side."
Construction timelines and costs have not been outlined for the other 39 towers, nor how residents will be housed during the redevelopments.
Mr Andrews insisted all displaced residents will be found alternative accommodation and invited to return once construction is finished.
"Those who want to come back will be on the same terms they left, that'd be our aim," he said.
"Except it will be a brand new unit they move back into, not a place that's 50C in summer and Arctic in the winter."
The long-awaited housing statement was headlined by a commitment to build 800,000 homes across Victoria over the next decade and an Australian-first statewide levy on all short-term accommodation bookings with online platforms.
The consumer-facing 7.5 per cent levy on short-stay providers like Airbnb and Stayz is slated to come into effect from 2025 and raise $70 million a year to build and maintain social housing.
The premier dismissed a warning from Stayz that owners could avoid the levy by taking their properties off platforms and listing them elsewhere.
"There are many different ways in which people try and avoid paying tax and the State Revenue Office are expert in finding those people and making sure that people do pay their fair share," he said.
"If you're eligible to pay the tax then you will be paying that tax."
Other highlight reforms include closing a rental bidding loophole, slashing planning red tape and bolstering apartment design standards.
