The school traces its origins to 1787, when French priest Father Arnold Garnault established a Malay-language school at Church Street at the invitation of Francis Light. In 1852, the administration was taken over by the De La Salle Brothers, making it the first Lasallian school in Southeast Asia. It was renamed St. Francis Xavier’s Free School and later moved to its current Farquhar Street location in 1857. During World War II, the original ornate building—described as resembling a "grandiose Baroque European palace"—was occupied by Japanese troops as a naval base. This made it a target for Allied bombing in 1941, which completely destroyed the structure. The school functioned in temporary wooden buildings until the present modernist building was completed in 1954. SXI is renowned for its academic excellence and fierce rivalry with the nearby Penang Free School.