Fort Phantom Hill was one of the second line of forts laid out in the early 1850s to protect the westward-moving frontier of Texas settlement. In 1849 the federal government sent Capt. Randolph B. Marcy to explore and mark the best route through the Comanchería, the vast region to the north and west of Austin inhabited by the warlike Comanche Indians. This was meant to give safer passage to immigrants headed for the California gold fields. The advanced cordon of forts, including Fort Phantom Hill, was established as a result of Marcy's recommendations.
Acting on orders from Gen. Persifor F. Smith, Lt. Col. John J. Abercrombie arrived at the Clear Fork of the Brazos in the area of present Jones County with five companies of the Fifth Infantry on November 14, 1851. Smith had recently taken command of the newly organized Texas (Eighth Military) Department from the ailing Gen. William G. Belknap, who had been supervising construction of the fort on the upper Brazos that was named for him. Originally, Belknap's orders had been to build a second fort on Pecan Bayou, at a site now in Coleman County. Smith, who was unfamiliar with this area, changed the locale to the Clear Fork near its junction with Elm Creek.
This unreasoned alteration affected the post's future, for the lack of an adequate water supply and the scarcity of building timbers added greatly to the hardships of the garrison. Though Lt. Clinton W. Lear, writing to his wife at Fort Washita, deemed the Clear Fork valley beautiful and abundant with game, he felt that it was never intended "for White man to occupy such a barren waste." Nevertheless, the troops dutifully began work on the new fort. A suitable stone quarry was located on Elm Creek about two miles south. Blackjack oak logs for the officers' quarters and hospital had to be brought in by ox wagon from as far away as forty miles. The company quarters and other buildings were of jacal construction. All of the buildings had stone chimneys, but only the magazine, guardhouse, and commissary storehouse were built entirely of stone.
Oddly enough, Fort Phantom Hill was never officially named; military records usually refer to it as the "Post on the Clear Fork of the Brazos." Although there are several legends about the origin of the unofficial designation Phantom Hill, the name probably derives from the fact that from a distance the hill on which it was built rises sharply from the plains but seems to level out as it is approached, vanishing like a phantom.
Life at the fort was difficult. Elm Creek was often dry, and the waters of the Clear Fork were brackish. At one time an eighty-foot-deep, walk-in well was dug near the guardhouse, but even it was not always reliable. More often than not, it was necessary to haul barrels of water in wagons from a spring about four miles upriver from the post.
Although the isolated fort was vulnerable to attacks, its garrison had only peaceful encounters with the Indians. Certainly, it would have been a tactical blunder to match infantry against the Comanche horsemen of the plains. A band of Penateka Comanches led by Buffalo Hump occasionally came calling, as did groups of Lipans, Wichitas, Kiowas, and Kickapoos. Mrs. Emma Johnson Elkins, who as a child lived with her parents at the fort, recalled a ritual held by a group of friendly Delawares in preparation for a hunt. Jim Shaw and Black Beaver were among the noted Delaware scouts employed by the garrison as interpreters and guides.
Colonel Abercrombie turned command of the post over to Lt. Col. Carlos A. Waite on April 27, 1852. In turn, Waite was succeeded by Maj. Henry H. Sibley on September 24, 1853. By this time four of the five companies had been withdrawn, and the remaining company was reinforced by Company I of the Second Dragoons. Lt. Newton C. Givens assumed command of the post on March 26, 1854, and was commander at the time it was first abandoned twelve days later, on April 6.
The decline in rank of its commanders shows the fort's decline in importance. Its initial occupation had been relatively uneventful. At the time of its evacuation, the purpose of curbing the Indian menace had momentarily been attained by the establishment of the reservations on the upper Brazos and the Clear Fork to the northeast near Fort Belknap. Shortly after the soldiers left Phantom Hill, the fort buildings mysteriously burned to the ground, an event variously attributed to everyone from an irate officer's wife to Indians and later to Union sympathizers. The scanty evidence points inconclusively to members of the departing garrison as the arsonists.
In 1858 the remaining structures of the fort were repaired and utilized as Way Station Number 54 by the Southern Overland (Butterfield) Mail. The station was managed by a man named Burlington, whose wife prepared meals for the stage passengers. Most travelers agreed with New York Herald correspondent Waterman L. Ormsby that Phantom Hill was "the cheapest and best new station on the route." During the Civil War, when the frontier was patrolled by Ranger companies and subsequently by the Frontier Battalion, Col. James B. (Buck) Barry or some of the units under his command used Fort Phantom Hill as a base of field operations. Beginning in 1871, the post served as a subpost of Fort Griffin, near the site of present Albany. Gen. William T. Sherman made an overnight stop here during his inspection tour of the Department of Texas. Capt. Theodore Schwan led one column of Col. Ranald S. Mackenzie's raiders from the post in the first series of Indian campaigns into West Texas between January and March 1872.
After the Indian wars subsided, a town grew up around the fort ruins. In 1876–77, it was a buying and shipping point for buffalo hides taken during the slaughter of the Southern Plains herds.The first post office in the region was established at Phantom Hill on July 1, 1879. Thomas Fletcher Scott was named postmaster. In anticipation of railroad construction, a hotel was built and managed by a man named Smith. By 1880 the community had a population of 546; it was made Jones county seat in May 1881 but lost that distinction to Anson on November 14, 1881, thirty years after the establishment of the post. The Texas and Pacific Railway routed its tracks through Abilene, fourteen miles to the south. A letter written to the San Antonio Express in 1892 commented that Fort Phantom contained nothing but "one hotel, one saloon, one general store, one blacksmith shop, and 10,000 prairie dogs."
Fort Phantom Hill is on private land. The owners have improved the fort and made it available to the public. At the site, three stone buildings and more than a dozen chimneys and foundations remain. Only two miles south of the fort is a manmade reservoir, Fort Phantom Hill Reservoir, which supplies the water for about 100,000 people in Taylor and Jones counties. The remains of Fort Phantom Hill have been celebrated in verse by the cowboy poet William L. Chittenden and in the prose of other western writers.