Byatt's publisher Chatto & Windus said on Friday that the author died "peacefully at home surrounded by close family".
Byatt wrote two dozen books, starting with her first novel The Shadow of the Sun in 1964.
Possession, published in 1990, follows two modern-day academics investigating the lives of a pair of Victorian poets.
The novel, which skilfully layers a modern romance with mock-Victorian writing, was a huge best-seller and won the prestigious Booker Prize.
It was adapted as a 2002 film starring Gwyneth Paltrow and Aaron Eckhart.
Her other books include four novels set in 1950s and 60s Britain and known as the Frederica Quartet - The Virgin in the Garden, published in 1978, followed by Still Life, Babel Tower and A Whistling Woman - and Booker Prize-shortlisted historical novel The Children's Book, which won the James Tait Black Memorial Prize in 2009.
Her volumes of short stories include The Djinn in the Nightingale's Eye, the title story of which which won the 1995 Aga Khan Prize for Fiction and inspired the 2022 film Three Thousand Years of Longing by Mad Max director George Miller.
Another book of stories, Medusa's Ankles, was published in 2021.
Byatt's literary agent Zoe Waldie said the author "held readers spellbound" with writing that was "multi-layered, endlessly varied and deeply intellectual, threaded through with myths and metaphysics".
Clara Farmer, Byatt's publisher at Chatto & Windus, said the author's books - translated into 38 languages - were "the most wonderful jewel-boxes of stories and ideas".
"We mourn her loss but it's a comfort to know that her penetrating works will dazzle, shine and refract in the minds of readers for generations to come," Farmer said.
Born Antonia Drabble in Sheffield, northern England in 1936 - her sister is novelist Margaret Drabble - Byatt grew up in a Quaker family, attended Cambridge University and worked as a university lecturer
She married economist Ian Byatt in 1959 and they had a daughter and a son before divorcing.
In 1972 her 11-year-old son Charles was struck and killed by a car while he was walking home from school.
Charles died shortly after Byatt had taken a teaching post at University College London to pay for his private school fees.
After his death, she told The Guardian in 2009, she stayed in the job "as long as he had lived, which was 11 years".
In 1983 she quit to become a full-time writer.
Byatt lived in London with her second husband Peter Duffy, with whom she had two daughters.
7°C