Forest's dream season continued at the City Ground as they moved one step closer to a first trophy since 1990, while Ipswich now switch their attention to staying in the Premier League.
The result added to a season of woe for Australians Massimo Luongo and Cam Burgess.
Midfielder Luongo was subbed off after 67 minutes while in more concerning news for Socceroos boss Tony Popovic, central defender Burgess couldn't shake an ankle injury suffered in the first half and was replaced at halftime.
It leaves the Scottish-born 29-year-old in doubt for crucial World Cup qualifiers at home to Indonesia and away in China later this month.
Both sides were far from impressive on Monday night in front of England manager Thomas Tuchel in a first period that finished goalless at the break.
George Hirst's third goal of the season put the visitors on track for a first win over any Premier League side this calendar year until Ryan Yates' equalising header sent the match into extra time.
No goals came in the extra 30 minutes and Sels dived low to his left to stop Taylor from the spot to secure a 5-4 shoot-out success, setting up a quarter-final clash at Brighton & Hove Albion.
The best chance of the first period fell to Forest's Ibrahim Sangare, who glanced a header across the face of goal, before teammate Anthony Elanga had an effort from distance saved by Alex Palmer at the start of the second half.
Ipswich's Sam Morsy then got the ball out of his feet and decided to smash an audacious 35-yard effort towards goal but Sels got the finest of fingertips on it to push over the bar.
However, he could not stop the Tractor Boys from breaking the deadlock on 53 minutes from the resulting corner.
Ben Johnson's inswinging delivery picked out an unmarked Hirst at the far post and he nudged a header towards goal which bounced through Sels' legs.
Forest got themselves level in the 68th after substitute Elliot Anderson recycled the ball out to Elanga and he delivered a teasing ball into the area straight onto the head of Yates who nodded beyond Palmer.
Forest began to put their relegation-threatened visitors under pressure - Murillo's 25-yard bullet was stopped by Palmer before Callum Hudson-Odoi rattled the crossbar from similar range and neither team did enough in extra-time which then forced penalties.
And after nine perfect spot kicks, Taylor missed the only penalty as Sels guessed right to send Forest through to the last eight.
With AAP.
