Joseph Henry Polley, rancher, son of Jonathan Polley, was born in Whitehall, Washington County, New York, on December 28, 1795. He joined Capt. Samuel Brown's company of New York militia as a teamster in the War of 1812, then moved to Missouri. He traveled with Moses Austin to Texas in 1820 and, as a member of Stephen F. Austin's expedition the following year, settled on a headright in the Brazoria District. He married Mary Augusta Bailey in a civil ceremony in 1823, followed by a Protestant union on October 24, 1826, and a Catholic convalidation on July 12, 1831, which allowed the Polley children a headright. The couple moved to San Felipe but returned to the Brazoria District and settled on land purchased from Mrs. Polley's father, James Briton Bailey, in 1832. They opened their home, Whitehall, as one of the earliest public houses. Polley became the first sheriff of the Austin colony. Several early revolutionary councils met at Whitehall, and Polley was appointed to escort fugitives during the Runaway Scrape.
While living by the Brazos River he accrued land and slaves through homesteading, purchase, and his wife's inheritance. He tried to grow cotton, but uncertain prices drove him to focus his efforts on cattle. His herds grew quickly, and in 1847 he purchased a new homesite in what is now Wilson County from his son-in-law John James. The move gave him room for expanding his operation and isolated his children from disease on the coast; they were often ill, and his first son died in 1834. The family settled east of Cibolo Creek in 1847, away from marshy ground but near the warm sulfur springs later advertised by Dr. John Sutherland as a cure for cholera and other ailments, including neuralgia, from which Polley suffered. By 1860 Polley owned land on perennial streams in an arc from Corpus Christi north to Llano County and west to Medina County, and his herds were among the largest in Texas. He marketed cattle primarily through San Antonio but invested in the San Antonio and Mexican Gulf Railway in an attempt to develop Port Lavaca. His slaves, who numbered twenty by 1861, built a new Whitehall, finished in the early 1850s, of stone, fitted with trim and furnishings from New York. Polley sponsored schools in Seguin and Sutherland Springs, which his nine surviving children attended, and his home became a social center for the community. Officers from the San Antonio garrison, including Robert E. Lee and , John B. Hood were frequent guests at the Polley house. Polley reluctantly supported secession after Lincoln's election; two of his sons fought for the Confederacy.
He continued to acquire land throughout the Civil War, but closed markets and burdensome taxes left him without ready funds. He also grew cotton, which he sold for specie in Mexico, along with cattle. He abandoned the effort when emancipation came, however, and divided management of his stock with his son-in-law Walker K. Baylor and Walker's cousin, John R. Baylor. Polley received a pardon from President Andrew Johnson and with the assistance of several former family slaves, who lived nearby on land donated by him and his neighbors, retained most of his property. He died on March 26, 1869, and was buried in the family cemetery at Whitehall, near the intersection of U.S. Highway 87 and Farm Road 539 in Wilson County.