Brimmer's marvellous, dipping half-volley from outside the box in the 74th minute at AAMI Park proved the difference in a clash that had plenty of aggression but few quality chances.
Crucially, it hands Tony Popovic's Victory a one-goal advantage ahead of Saturday's reverse fixture at the same venue, as they eye their first grand-final appearance in four years.
There was also controversy with United arguably unlucky not to receive a penalty in the fifth minute when Jason Davidson went into the back of Lachie Wales.
Victory almost struck early with Brimmer whipping a free kick just outside the near post in the second minute.
From there United, who were coming off a three-day turnaround compared to Victory's nine days between games, dominated.
Three minutes after Wales' penalty appeal, Ben Garuccio whipped in a corner and Victory goalkeeper Ivan Kelava did brilliantly to bat away Aleksandar Prijovic's near-post header.
From there Victory had several half-chances, with Nick D'Agostino flashing a volley wide in the 23rd minute.
In the 40th minute, Victory defender Roderick Miranda outmuscled Dylan Wenzel-Halls inside the area but his appeals were ignored.
United created the first opening after the break with Garuccio picking out Leo Lacroix from a corner only for the big centre-back to head the ball straight into Kelava's gloves.
Then, as the game looked set to peter out, Brimmer, one of Victory's best this season, brought it to life.
Davidson's corner was punched clear by Jamie Young but it bounced into the path of an unmarked Brimmer, who drove his effort over the goalkeeper's head.
United lifted their intensity but were unable to respond.
Victory were at full strength with Matthew Spiranovic returning and Brendan Hamill dropping to the bench.
Josh Risdon returned to United's starting XI with Tomoki Imai shifted centrally to cover the loss of Rene Khrin (hamstring).
10°C