FORT WORTH (CBSDFW.COM) – An inmate at the Federal Correctional Institution (FCI) in Fort Worth was a suspected criminal, but the charges he faced involved taxes. Now the 71-year-old is accused of being the mastermind behind a murder-for-hire plot against a federal judge.

Federal officials say Phillip Monroe Ballard was willing to pay six figures to have U.S. District Judge John McBryde killed.

READ MORE: FBI Dallas Identifies Colleyville Synagogue Hostage Taker As Malik Faisal Akram

While locked up on the tax charges, Ballard allegedly laid out detailed plans on how the hitman was to a high-powered rifle and scope from Burnett Plaza, across the street from the federal courthouse, to kill Judge McBryde.

Today’s court hearing concerning the tax charges was cancelled on September 28, after authorities learned of the murder plot. Judge McBryde also immediately recused himself from the case.

After allegedly saying he feared Judge McBryde would sentence him to more than 20 years in prison, Ballard allegedly approached a fellow inmate about killing the judge for $100,000. The would-be hitman was actually a federal informant.

READ MORE: North Texas Rabbi Says He's 'Grateful To Be Alive' After Being Held Hostage 10 Hours Saturday

“These things fortunately usually come unraveled, as this one did,” former U.S. Attorney for Northern Texas Paul Coggins said of the plot. “You talk about turning a very bad situation for a defendant into a much worse situation. This is a much more serious crime.”

Ballard was apparently working on the rationale that his case would have to be transferred to another judge if McBryde was dead. According to federal officials, Ballard even had a contingency plan, of planting a bomb in the judge’s vehicle.

Coggins said these cases are rare and extremely serious.

“This one looks like, based upon the complaint that was filed, the inmate took a number of steps to actually see that it got carried out.”

MORE NEWS: All Hostages Are Out Alive And Safe At Congregation Beth Israel In Colleyville, Suspect Dead

If convicted Ballard faces up to 20 years in prison and a $250,000 fine.