Coveney was taken off stage by officials during a speech at a peace-building event on Friday and quickly driven away from the venue in his government car.
The event organiser said a suspicious device had been discovered in a hijacked van in the car park of the venue.
The van driver was ordered at gunpoint to drive to the venue in north Belfast, one of the event's organisers told Reuters.
The device placed by the two gunmen was declared a hoax but was clearly designed to cause maximum disruption, Assistant Chief Constable Mark McEwan told a news conference.
"The victim believed he was driving a van with a live bomb and that his family were being threatened," Asst Chief Constable McEwan said.
"At this early stage of the investigation our assessment is that these crimes were carried out by loyalist paramilitary groups.
"We're keeping an open mind but one of the primary lines of investigation is the UVF (Ulster Volunteer Force)."
The UVF were among the loyalist groups that last year temporarily withdrew their support for a 1998 peace deal in protest at the trade border Britain's exit from the European Union created between Northern Ireland and the rest of the United Kingdom.
A small number of militant groups remain active and carry out occasional attacks but their capacity is tiny relative to the 30-year conflict between nationalists seeking unification between Northern Ireland and the Irish Republic, and the British Army and pro-British loyalists determined to keep Northern Ireland ruled from London.
Friday's incident came three days after the UK lowered its Northern Ireland-related terrorism threat level for the first time in more than a decade, with police saying operations against Irish nationalist militants were making attacks less likely.
Coveney, who was taken to a secure location after the event, said on Twitter he was "saddened and frustrated" someone had been attacked and victimised.
The driver was in tears inside the venue after alerting security officials to the incident and apologising to attendees for being forced to drive to the site, journalists said. He was later taken to hospital, police said.
"I spoke to the poor man whose van was hijacked. He's lost his memory. He's traumatised. It's just unreal," Father Aidan O'Kane, the manager of the Houben Centre where the event was being held, told Reuters.
A funeral in the adjacent church also had to be evacuated and continued in a car park, Father O'Kane said.
Britain's Northern Ireland Minister Brandon Lewis said the incident was a reminder there is a small minority willing to use violence to advance their goals.
"These actions are reprehensible. I utterly condemn those involved," Lewis said in a statement.
