The city of Petropolis was slammed by a deluge on Tuesday and Castro says almost 400 people were left homeless. Searchers picked through the wreckage throughout the day and 21 people were recovered alive.
Civilians joined official recovery efforts. Among them were Priscila Neves and her siblings, who looked through the mud for any sign of their disappeared parents, but found only clothing.
Neves told the Associated Press she had given up hope of finding her parents alive.
And Rosilene Virgilio, 49, was in tears as she recalled the desperate pleas from someone she couldn't save.
"There was a woman screaming, 'Help! Get me out of here!' But we couldn't do anything; the water was gushing out, the mud was gushing out," Virgilio told AP. "Our city unfortunately is finished."
Petropolis is a German-influenced city named for a former Brazilian emperor. Nestled in the mountains above the coastal metropolis, for almost two centuries it has been a refuge for people escaping summer heat and tourists keen to explore the so-called "Imperial City".
But its population has grown haphazardly and many are in areas unfit for structures and rendered more vulnerable by deforestation and inadequate drainage.
The stricken mountain region has seen similar catastrophes in recent decades, including one that caused more than 900 deaths. In the years since, Petropolis presented a plan to reduce risks of landslides, but works have been advancing only slowly.
The governor told reporters earlier the situation "was almost like war" and he was mustering all the state government's heavy machinery to help dig out the buried area.
The state fire department said late on Tuesday the area received 25.8 centimetres of rain within three hours on Tuesday - almost as much as during the previous 30 days combined. Petropolis' civil defence authority said moderate rain was expected on Wednesday.
Video on social media showed cars and houses being dragged away by landslides, and water swirling through Petropolis and neighbouring districts.
On Wednesday, houses were buried beneath mud while appliances and cars were piled on streets where they had been swept the night before. Some people had attempted to flee the perilous hillsides.
Petropolis' city hall declared three days of mourning. Brazil's President Jair Bolsonaro expressed solidarity while on a trip to Russia, as did his counterpart Vladimir Putin.
Southeastern Brazil has been punished with heavy rains since the start of the year, with more than 40 deaths recorded between incidents in Minas Gerais state in early January and Sao Paulo state later the same month.