The director said he had not met Epstein before he was among the guests at a dinner in December 2010, which included the Duke of York.
Allen, 89, told The Sunday Times a publicist invited him and his wife, Soon-Yi Previn, to the dinner with "one of those British royals" and other guests.
Epstein had just finished a prison term for soliciting an underage girl for prostitution.
"We didn't know Jeffrey at all then, but we see all these people there and they all embraced him, so we figured, 'OK, he's a substantial character'," Allen said.
"He told us he'd been in jail and that he had been ... falsely put in jail in some way.
"He told us he was trying to make up for it now by being philanthropic and giving money to cutting-edge scientists and universities. He couldn't have been nicer."
Allen said his wife, the adopted daughter of his ex-wife Mia Farrow and her previous husband Andre Previn, had wanted to meet the duke.
She told The Sunday Times she "could not stand" Andrew, describing him as "such a dullard".
Allen said he and Soon-Yi became regular guests at Epstein's house, where "there was always a table of illustrious people".
"We never, ever, saw Jeffrey with underage girls. He always had a girlfriend but never an underage girlfriend," he said.
He said Epstein's former girlfriend, Ghislaine Maxwell, who is serving a 20-year term for sex trafficking, was not at any of the dinner parties.
Epstein was found dead in his cell in a New York prison in 2019 after being arrested on sex trafficking charges.
The Duke of York stepped down from public life after the furore over his friendship with the paedophile financier, and paid millions to settle a civil sexual assault case with Virginia Giuffre, a woman he claimed never to have met.
1800 RESPECT (1800 737 732)
National Sexual Abuse and Redress Support Service 1800 211 028
14°C