Specialists operating heavy machinery have begun the difficult process of knocking down what is left of the seven-storey, 1910s-era brick building in inner-city Surry Hills on Tuesday.
NSW Police say they have spoken with a fourth boy over the incident, after three other young teens handed themselves in to police after the massive fire last week.
"Detectives have identified and spoken to four teenage boys as they continue to appeal for anyone else to come forward," NSW Police said.
"No charges have been laid at this time."
NSW Fire and Rescue Superintendent Adam Dewberry said the demolition plan could be reworked throughout the day if contractors encounter problems.
Two large excavators began picking over the rubble as Fire and Rescue's major structural collapse urban search and rescue unit truck arrived.
Specialist heavy machinery is shifting piles of bricks from the building's collapsed walls at the building's front facade and a burnt-out ute parked on the street will also be removed.
A 60-tonne excavator with a reach of 23 metres will be used to demolish unstable side brick walls from a safe distance.
The site needs to be made safe before it can be thoroughly inspected and residents in neighbouring buildings can return home.
"We want to make sure everyone remains safe but we also don't want to cause any damage to adjoining structures and utilities that are in and around and under the road,"Â Mr Dewberry earlier told 2GB.
"We've got gas mains underneath, we've got electricity and water so we just need to make sure we don't cause any more problems."
The site will be handed over to police to allow arson squad investigators and cadaver-detection dogs to examine the scene for any human remains or evidence as rubble is removed.
"While police have not received any reports of missing people, investigators are unable to definitively say there is no one inside," police said.
The brick and timber building and a neighbouring structure formerly home to karaoke bar Ding Dong Dang, was known for regularly housing 15 rough sleepers but two remain unaccounted for.
The blaze lit up the city sky on Thursday night, closing streets, diverting public transport and displacing more than 100 people who live in the vicinity.
Welfare services will continue to update displaced residents and keep them informed.
The brick hat factory was built in 1912 and operated until 1954 when proprietor RC Henderson went into liquidation.
The property was later leased to individual tenants in the 1960s and '70s, and plans were afoot to use the brick structure for a $40 million redevelopment of the site into a 123-room hotel.
