The Court of Appeals decision was posted on its Web site, but the judges didn't offer a written ruling.
Muhammad and Lee Boyd Malvo were sentenced to life in prison in Maryland for the October 2002 sniper shootings that killed 10 people and wounded three in that state, Virginia and the District of Columbia.
Both men are in prison in Virginia. Muhammad has been sentenced to death there and Malvo is serving life without parole.
In arguing for a new trial, Muhammad claimed he was wrongly allowed to represent himself and was wrongly deemed mentally fit to stand trial. He also said he had been wrongly prevented from continuing a certain line of questioning of Malvo, who testified against him.
J. Wyndal Gordon, a Baltimore lawyer who acted as "standby counsel" during Muhammad's trial, said he was disappointed "that no one decided to take a closer look at this case." He said he thought there were several issues of fundamental rights relating to questioning witnesses that were not "addressed appropriately."
Last year, Maryland's appellate court rejected Muhammad's request in a sharply worded decision that compared him to Jack the Ripper, saying he terrorized the Washington region in a fashion similar to the notorious Victorian serial killer.