AP &

A judge handling James Brown's contested fortune refused Friday to remove himself from the case despite an argument from the late soul singer's former trustees that he forced them to resign.




"I would never force anyone to do anything, that's not me," Circuit Judge Jack Early said.




Buddy Dallas and Alford Bradley, who both spent years handling Brown's financial affairs, resigned in November. A few weeks later, they claimed the judge had pressured them to resign.




On Friday, an attorney for the trustees tried to retract the argument that the judge be removed, but Early wouldn't allow it and then ruled he wouldn't step aside.




Administrators looking into the handling of Brown's estate told the judge in September that another former trustee, David Cannon, may have misappropriated up to $7 million. And the singer's six adult children have been at odds with the trustees, claiming money has been mismanaged.




Dallas and Bradley have denied claims they violated Brown's trust.




Attorneys for Brown's adult children and a woman who claims to be his fourth wife argued the judge should stay.




Brown died in Atlanta on Christmas Day 2006, throwing into turmoil the future of his trust, which is said to contain most of his primary assets, including music rights and his 60-acre Beech Island home in South Carolina. The total value is still unclear.




Since he died, several people, including some claiming to be Brown's unacknowledged children, have come forward wanting a piece of his estate.




Tomi Rae Hynie, a former backup singer who claims to be Brown's fourth wife and the mother of his child, has said the singer wanted her to have his home in the western part of South Carolina, near the Georgia state line. She is vying for half of Brown's estate.