Fort Brown, originally called Fort Texas, was established when Zachary Taylor and the United States forces of occupation arrived on the Rio Grande on March 26, 1846, to establish the river as the southern boundary of Texas. In April 1846 Taylor built an earthen fort of 800 yards perimeter, with six bastions, walls more than nine feet high, a parapet of fifteen feet, and the whole surrounded by a ditch fifteen feet deep and twenty feet wide. Armament was four eighteen-pound guns. The Seventh Infantry, with Company I of the Second Artillery and Company E, Third Artillery, commanded by Maj. Jacob Brown, garrisoned the fort. Mexican troops led by Mariano Arista intercepted United States troops as they brought supplies from Fort Polk at Point Isabel to Fort Brown, leading to the opening battles of the war, Palo Alto and Resaca de la Palma, fought on May 8 and 9, 1846. On May 9 Major Brown died from injuries received during the bombardment of the fort by Mexican forces in Matamoros. Shortly after his death he was buried within the fortifications, and the post was named in his honor. The fort was held by a strong force during the Mexican War. In 1848 quarters for officers and enlisted men and a permanent post were built a quarter mile north of the first site. The land was purchased from heirs of José Salvador de la Garza, recipient of the Potrero del Espíritu Santo land grant and one of José de Escandón's colonists.
A brick wall, called the Quarter Master's Fence, divided the fort and the community that became the city of Brownsville. Rumors that the fort was to be abandoned started circulating in 1848; however, Indian raids in 1852 made it a necessary fortification for the town, and from one to four companies of troops were stationed there for protection from Indians and as a show of force against potential incursions into Texas by Mexico. In 1859 Juan Nepomuceno Cortina occupied the fort as a refuge. In 1860 Robert E. Lee was stationed at Fort Brown on assignment to quell border disturbances. The post was abandoned by United States troops in March 1861 and occupied by troops of the state of Texas. Fort Brown was garrisoned by a small force of Confederates until November 1863. On the approach of Union forces, the Confederates retreated and burned all stored cotton and the fort buildings. Union forces occupied the fort until July 1864, when it was again taken by Confederates forces, who held it until November 1865. During Confederate occupation the fort was described as a field work of six bastion fronts with a defense line of 950 yards garrisoned by 2,000 men with an armament of guns of different calibers. The fort was reoccupied by federal forces after the Civil War, again as a protection against Mexican invasion. Fort Brown was rebuilt with brick buildings starting in 1869. Among the buildings were the post hospital, an administration building, a large officers' quarters, and the chapel. In 1882 Dr. William Crawford Gorgas was sent to the post in response to the last and worst yellow fever epidemic at Fort Brown.
The Brownsville raid of 1906 occurred on August 13, when a group of black soldiers from the fort allegedly attacked the city. The consequences included a federal investigation, the demand of the removal of black troops from the fort by Brownsville citizens, the court-martial of the black troops, and the transfer of the fort from the War Department to the Department of the Interior by President Theodore Roosevelt. This department converted the fort into an experimental garden for spineless cacti. In 1914, because of increased racial and economic tensions in the Lower Rio Grande valley, Fort Brown was reactivated and made headquarters for the Brownsville Military District under Gen. James Parker. Within a few months 50,000 state guards were mobilized in the district. The first wireless station was established about 1916. From World War I to February 1941 Fort Brown was headquarters for the Twelfth Cavalry. That unit was replaced by the 124th Cavalry, which trained there until May 1944, when the fort was deactivated. Official deactivation was declared by the War Department in 1945, at which time the fort was turned over to the army engineers; it was certified to the War Assets Administration for disposal on May 15, 1946, and assigned to the Federal Works Agency on July 7. The agency then turned the land over to the Federal Land Bank for farming purposes. On July 22, 1948, the front 162 acres of Fort Brown was deeded to the city of Brownsville, and the old post hospital was granted to the Brownsville schools for the use of Texas Southmost College. The fort buildings were sold or donated to various organizations and schools in the Brownsville area. In the early 1990s much of what was left of Fort Brown buildings and land were used by the University of Texas-Pan American at Brownsville and Texas Southmost College.