The 70-year-old, believed to be from Paynesville, was transferred by ambulance to Bairnsdale Regional Health Service in the state's East Gippsland region on Monday.
With no beds available, the man remained in paramedics' care on a stretcher inside the hospital for about three and a half hours before he went into cardiac arrest and died.
Bairnsdale Regional Health Service and Ambulance Victoria are investigating the circumstances of his death.
Victorian Ambulance Union General Secretary Danny Hill said four or five crews had patients ramped at the hospital on Monday.
"The emergency department was completely overwhelmed. A lot of patients were unable to be moved up to wards because they were full," he told AAP on Wednesday.
A three- to four-hour wait for hospital treatment is usually more common at Melbourne hospitals than those in the regions, Mr Hill said.
It is the latest fatal episode linked to Victoria's ailing health system, with at least 12 people dying since October after calls for an ambulance to the state's triple-zero service went unanswered or were not picked up quickly enough.
Mr Hill said the COVID-19 pandemic has exacerbated, not created, problems within the system, and the lack of alternative health services in the bush often makes residents feel like their only option is to call triple zero.
"The system seems to focus on the only two options which is ambulance and emergency department," he said.
"Opening up access and improving access to 24-hour GPs, aged care in-reach ... is really where the attention should be focused. Not just on putting on more ambulances and more ED beds."
While offering his condolences to the man's family, Mr Andrews said 2000 hospital staff across the state are off work because of COVID-19 on any given day.
"They're sick or they're having to care for somebody who is sick. That's put extraordinary pressure on our system," he said at Kyneton.
"Every part of Victoria has seen a boost in health funding, a boost in support for our nurses, our ambos, our doctors. You can't generate health professionals in a matter of weeks and months."
Opposition Leader Matthew Guy said 10 per cent of Bairnsdale hospital staff were unavailable to work due to COVID on Monday and the government could not evade responsibility for the state of the health system.
"It's on their watch. They've had years to prepare for this circumstance," he said.
Victorian health workers such as doctors, nurses and paramedics can be exempted from COVID-19 isolation rules for close contacts if they are asymptomatic and return daily negative rapid antigen tests.
