James Madison Brown, lawman and horseman, son of John Humphrey and Jane Ann Brown, was born in Alabama in 1839. The Browns, with several other families, settled permanently in what is now San Saba County, Texas, where John Brown helped establish the town of San Saba. James, the eldest son, grew up to be an excellent judge of horses and in his later years became wealthy from winning on the turf. During the Civil War he served as a minuteman under the command of W. R. Wood and later under his father's command. In 1869 he was probably one of the trail hands that drove a large herd from the San Saba region to Roswell, New Mexico Territory. In 1870 he resided in Washington County, where he had already established a reputation as a gambler. In 1872–73 he served as a private in the state police force. He joined Washington County Volunteer Militia Company A, commanded by Capt. Leander H. McNelly, and served from April 30 to November 30, 1875. During this time he also was establishing a reputation as a horseman; he raced successfully in Gonzales County as well as Travis County.
Lee County, separated from Washington County in 1874, was still a frontier area and needed strong law enforcement officials. James McKeown was the new county's first sheriff, and on February 15, 1876, Brown was elected to take his place. He served in that office during a very lawless era and established a reputation for being an energetic sheriff. But his years were not without controversy. He was involved in several personal feuds as well as several killings. His most notable act in office was the legal hanging of noted outlaw William P. Longley.
After his career as sheriff, Brown developed his racing stables and raced on tracks in cities outside of Texas, including St. Louis, Nashville, and Chicago. On September 6, 1892, while at the Garfield Park in Chicago, police attempted to arrest him on an old murder charge from Texas. Brown resisted, and in the ensuing gunfight he was killed, along with two members of the Chicago police force. Chicago newspapers described him as a millionaire but also alleged that he may have murdered as many as fourteen men while he was sheriff. It was surmised that his position and the fear he engendered among his contemporaries protected him from prosecution. Brown was buried in Fort Worth. He was survived by his wife and five children.