ACTU secretary Sally McManus has urged rogue union boss John Setka to resign, saying the Morrison government would use the controversy around him "for their own ends".
But Mr Setka, the Victorian secretary of the CFMMEU, insists only his construction union members will determine his future.
Mr Setka met with Ms McManus in Melbourne on Thursday, where she told him he should resign.
However, he firmly denies claims he criticised anti-domestic violence campaigner Rosie Batty and tweeted he would not be resigning.
He says he is the victim of false allegations being made for political gain.
Ms McManus said her decision was based on a range of issues involving Mr Setka, not the alleged comments about Ms Batty.
"For us, it is the whole situation (of his behaviour) ... it is difficult for me because all these matters are before the courts and I am bound by the laws around that," she said after the meeting.
"He said he is going to plead guilty to at least one of them. That is concerning. Having said that, the court process is still happening.
"Our point is this - we don't condone domestic violence and not only do we not condone it, we will stand against it."
Mr Setka is due in court on June 26 when he is expected to plead guilty to two criminal charges following an incident at a Melbourne home late last year.
He faces a charge of breaching a court order, along with another charge, details of which cannot be revealed for legal reasons.
Ms McManus said Mr Setka had caused the union movement damage.
"I believe that at the moment what he's said and possibly done has caused us damage and it would be the right thing to do, even if he's innocent, to stand down from his position," she said.
The ACTU boss said it was possible the Morrison government could use the scandal against the union movement and introduce tough new laws allowing the easier deregistration of unions and banning of union officials.
"It is clear they (the government) will try and exploit this situation for their own ends," she said.
Mr Setka told reporters as he left the meeting it had been a "nice open and frank discussion".
Asked about his future he told reporters: "I'm going back to the office. My members are the ones that ask me ... They're the ones that pay my wages.
"That's who I will be taking advice off."
Earlier on Thursday, a monthly meeting of Victorian construction union shop stewards endorsed Mr Setka's leadership and called for the national CFMMEU office to follow suit, The Age reported.
Mr Setka has been suspended from the Labor Party and faces likely expulsion after reportedly telling colleagues at a meeting that Ms Batty's advocacy had led to men having fewer rights.
He says a union member at the meeting had leaked the "false allegations" for political gain.
Maritime Union of Australia's WA secretary Christy Cain was at the meeting and says he supports Mr Setka's account.
Labor leader Anthony Albanese says he spoke with other people at the meeting who insist Mr Setka's comments were clear, whether they were misinterpreted or not.
But Mr Albanese also said his issue with Mr Setka stretched beyond the alleged comments about Ms Batty.
"There is a whole history of comments Mr Setka has made, not just the one at that meeting but ones that I've been critical of in the past," Mr Albanese told ABC Radio National.
Former Labor prime minister Kevin Rudd, who Mr Setka previously described as a "maggot" anda "dog", said action against the union boss was warranted.
Australian Associated Press