The former Mr Universe will be sent back to the UK, two years after Home Affairs Minister Peter Dutton called for his deportation on character grounds.
Mr Lewer, 66, accepted he had a substantial criminal record including receiving a conviction and prison sentence on weapons charges more than 25 years ago.
He also has a 2004 conviction for selling steroids which was included in a police dossier that formed part of the basis for Mr Dutton's decision.
Mr Lewer argued there had been no substantial change in his character since a visa was last granted to him in 2017, but police said they believe he is involved in a sophisticated criminal drug importation and manufacturing enterprise.
He has never been convicted.
Victoria Police claimed that was because he "possesses sound knowledge of police investigation methodology, is extremely disciplined in his criminal conduct to avoid detection and therefore is an extremely difficult suspect to investigate".
Mr Lewer said their suspicion or belief was not evidence that could support Mr Dutton's assessment of him.
But Federal Court judge Justice Bernard Murphy rejected the argument and upheld Mr Lewer's visa cancellation.
Mr Lewer, a three-time world champion bodybuilder, left Britain at five and has never taken up Australian citizenship.
Friends have previously described the deportation calls as "unjust" and "a result of a long standing vendetta" by Victoria Police.
Mr Lewer is being held in immigration detention in Melbourne.
11°C