Group chairperson Frank Niglia said there were previous issues when locals who live near the prison felt they were being "harassed and questioned".
He said it was common for people living near the prison to be pulled up by prison security when they were doing frequent recreational activities near the prison including horse riding, running, jogging and bike riding.
But Mr Niglia said since speaking with the higher authorities at the prison and Corrections Victoria there has been a noticeable change.
“We had several meetings where we aired everything out,” Mr Niglia said.
“We as local people shouldn't be pulled up and asked what our businesses is.
“The committee pushed for more observation and less confrontation.
“It is starting to work a lot better, at this stage we're reasonably happy with everything."
Mr Niglia said the prison was also accommodating when informing the nearby locals of a prisoner transfer during the January bushfires, when 190 people were moved from the Beechworth facility to Dhurringile.
“Open conversation between us is the key for us to have a healthy conversation,” he said.
State Member for Northern Victoria Tania Maxwell said she had been in close contact with the government and Community Action Group regarding concerns.
She said there must be a transparent relationship between the two parties involved.
Ms Maxwell previously raised the issue in parliament, late last year.
The News approached Corrections Victoria for comment.