Clifton College was organized on May 6, 1896, as the Lutheran College of Clifton, Texas. The school's charter was approved by the state government on May 26, 1896. The first building was dedicated on October 14, 1897, on land donated by N. Jacob Nelson and T. T. Hogevold; it was built by volunteer donations and labor. The school, which was financially supported by the Norwegian Lutheran Churches of America located in Texas, opened under the name Clifton High School on October 28, 1897. From 1907 to 1914 the Missouri Synod of German Lutherans helped support the college. The institution offered college courses for the first time in 1922. It was accredited in 1924 as a two-year college. In 1938 the high school was discontinued. The school's name changed from Lutheran College of Clifton, Texas, to Clifton Junior College on October 25, 1945, and to Clifton College on February 22, 1952. In 1954 the college merged with Texas Lutheran College in Seguin.
After the move the Clifton College administration building was sold to Carl Olsen, Sr., for his Gearwrench Manufacturing Company. In 1981 the building was dedicated as the Carl E. Olsen Fine Arts Building of the Bosque County Conservatory of Fine Arts. In 1982 a historical marker was placed on the building in commemoration of Clifton College. The remaining facilities of Clifton College were acquired by the Clifton Lutheran Sunset Home. This included an artifact collection of Jacob Olsen, an early Norwegian settler, which had been contributed to Clifton College in 1923. The collection became the core of the Bosque Memorial Museum. A log cabin like the Olsens' home has been reconstructed on the museum grounds.