Former mayors want plans for more than 120 new homes close to Eaglehawk's town centre to be completely overhauled.
They believe it is too much public housing for the Virginia Hill block and have called on the state government to revamp the blueprints.
Willi Carney and Jill Williams are part of a new group questioning the wisdom of so many units being built on the land off of Sailors Gully Road.
"We weren't letterboxed," Ms Carney said.
The site is currently fenced off and the 64 ageing public housing units that once stood there have demolished.
Construction crews expect to build a mix of social and affordable homes using government funds earmarked through the Big Housing Build.
Eaglehawk real estate agent Maureen Hosking questioned why so many units would rise at the three-hectare site.
"I thought this idea of housing under-privileged people all together had gone," she said.
Ms Carney feared repeats of issues that emerged in places like Eaglehawk and Long Gully in the 20th century, when public housing created challenges for whole neighbourhoods.
Many of the vulnerable people who were moved in did not feel safe with their new neighbours and some suburbs developed negative reputations.
Ms Carney recalls one decisionmaker around the year 2000 concluding it was better to spread social housing out through Bendigo, rather than clumping it together.
It helped older people and women fleeing domestic violence feel safer, for one, she said.
Ms Williams lived close to Virginia Hill for decades and said there had been challenges around crime and safety through that period.
Those issues had unfairly stigmatised all who lived in Virginia Hill's houses, and those in surrounding areas, she said.
"People do need public housing. We've got a great shortage of it. But how about we set this up to succeed, rather than fail?" Ms Williams said.
Housing Victoria is leading the redevelopment work and did not directly answer a question about bundling homes together.
It did confirm people were being supported after homes in Virginia Hill were demolished and that a residents consultative committee had been enlisted, which included renters and members of the community.
Housing Victoria published a masterplan for the site in March 2023 and depicted it as a "framework" which would need extra community consultation as time went on.
"For each stage of delivery, planning approval will be required and will provide further opportunities for community engagement," the group said in the masterplan.
That plan envisioned a new Virginia Hill that was "welcoming, inclusive and family friendly", including more accessible space for people with disabilities.
The masterplan's authors wanted a mix of townhouses, duplexes and apartments for single people, couples and families, the masterplan stated.
A Housing Victoria spokesperson said there had been consultations over several years.
"We have consulted with Virginia Hill residents on this project since 2019 and have welcomed feedback from more than 100 community members on the masterplan," they said.
The spokesperson said homes within 150 metres of Virginia Hill were letterboxed in March about the masterplan when it was released in March, while posters and signage were displaced on site.
They expected further consultation in late-2024 on design and community spaces.
Sign up for our newsletter to stay up to date.