About 5000 masked protesters marched in Melbourne's CBD on Australia Day, moving in staggered COVID-safe units from Parliament House toward Flinders Street.

They chanted "Always was, always will be Aboriginal land" and "No pride in genocide" while holding placards and Aboriginal flags.

Lidia Thorpe, a Gunnai Gunditjmara and Djab Wurrung woman elected in September, said Australia needed a treaty to "resolve the war that was declared on our people almost 250 years ago".

"No more bulls*** symbolic gestures," the federal Greens senator told the crowd.

"No more change the date, no more Uluru Statement, no more Constitutional recognition, no more 'Survival Day'. This is Invasion Day and we need a treaty."

The protest organisers have demanded the abolition of Australia Day.

"There is nothing to celebrate on Australia Day," they said.

"Aboriginal communities continue to be oppressed, policed, discriminated against and disenfranchised."

Protesters were told to organise in groups of 100 and stay 10 metres apart, wear masks and bring hand sanitiser at Tuesday's COVID-safe rally, which was mostly peaceful.

A man wearing a T-shirt representing a far-right group, known as Proud Boys, was removed by police from the rally about 1.30pm after he was involved in a brief scuffle with protesters.

"F*** off fascist, f*** off," the crowd yelled as he was led away.

Others were seen wearing Proud Boys clothing and draped in Australian flags as about 100 people marched down St Kilda Road in a "freedom" rally later on Tuesday. Police helicopters circled above the demonstration.

Proud Boys was earlier this week labelled a "terrorist entity" by Canada's parliament.

Victoria Police confirmed the rally antagoniser was one of two men briefly detained for breaching the peace before being released without charge.

No arrests were made at either of the planned demonstrations in the CBD.

"Police were pleased with the behaviour of those that attended," a spokeswoman told AAP.

Earlier in the day, Wunundjeri elder Aunty Joy Murphy said she was "very disappointed" in Scott Morrison's comments about Australian cricket's stance on January 26.

The prime minister sparked a backlash last week when he criticised Cricket Australia for dropping "Australia Day" from its marketing of three Big Bash League matches to be played on Tuesday.

Aunty Joy was a speaker at Tuesday's Victorian flag raising service, along with Victorian Governor Linda Dessau and Premier Daniel Andrews.

"We're always talking about going forwards. In some way, for some reason, our community seems always to have to go backwards," Aunty Joy said at the Government House ceremony.

"We talk about let there be trust, let there be confidence and all we hear is about being patient. We have almost lost our patience."

Victoria's outdoor gathering limit of 100 people meant Melbourne's traditional Australia Day parade was cancelled.

Australian Associated Press