Zimbabwe needed 10 runs off the last over to win as they chased down an imposing 299-run target on Friday, but Madushanka bowled Sikandar Raza for 92 with the first ball of the over and then had Brad Evans caught as he scooped the next ball to fine leg.
With his third ball of the over, Madushanka then bowled Richard Ngarava middle stump.
He finished off his brilliant job, conceding only two runs off the last three balls to ensure a narrow win as Zimbabwe finished on 8-291 in reply to Sri Lanka's 6-298.
Sri Lanka, who were put in to bat, piled on the runs near the end of their innings as Janith Liyanage and Kamindu Mendes led a recovery after they were reduced to 5-161 with 13 overs remaining.
Liyanage hit 70no off 47 balls and Kamindu 57 off 36 as they added 137 off 83 balls.
The home side's chase began disastrously as they lost their first two wickets in four balls, without scoring, but stand-in captain Sean Williams and Ben Curran put on 118 runs for the third wicket to give the home fans hope.
Curran scored 70 but it was a 128-run partnership for the sixth-wicket between Raza and Tony Munyonga (43no) that took Zimbabwe close before Madushanka's heroics with his 4-62.
Zimbabwe captain Craig Ervine was ruled out of the series after a calf tear on Thursday, with Williams taking over as skipper.
The last of the two-match ODI series is being played on Sunday followed by three Twenty20 internationals.
