James Walker was born in England in 1813 and immigrated to the United States sometime before 1840. He and his wife Eliza (born in 1822 in Tennessee) had four children: Thomas (born in 1840 in Mississippi), William (born in 1841 in Mississippi), Mollie (born in 1847 in Missouri), and Jesse (born in 1860 in Texas). In 1860 Walker reported his profession as a doctor, and the family lived near Hallettsville, Lavaca County, Texas.
Walker formed a cavalry company at Hallettsville, which entered Confederate service on May 23, 1861, as Company D, Second Mounted Texas Rifles. Walker and his company were responsible for capturing forty-eight prisoners at Fort Stanton, New Mexico, in August 1861. In October 1862 the regiment elected him lieutenant colonel. In June 1863 the regiment was operating in the District of Western Louisiana under Gen. Richard Taylor. During Taylor's operations to disrupt Union supply lines, Walker was wounded in action at Lafourche Crossing, Louisiana, on June 21, 1863. This wound continued to trouble him throughout the war, and he was frequently on sick leave in 1864–65. He was paroled on June 26, 1865, at Columbus, Texas.
After the war, Walker returned to Hallettsville where he was operating a dry goods store by 1870. By 1880 he was widowed and living with his son William in Austin, where he died in 1886. He was honored in the naming of the Colonel Walker Camp No. 248 of the United Confederate Veterans in Hallettsville.