Victoria Tower was built especially as a "deposit to test for the books and documents", as required by the tender to rebuild the Palace of  after the fire of October 16, 1834, which destroyed the building and almost all records of the House of Commons. Records of the House of Lords survived the fire because they were, at that time, stored in the Jewel Tower, which was at a distance from the main building still stands across the street from the Victoria Tower.

The design of Charles Barry of new Palace of Westminster offered a tower above the entrance of the Sovereign, 12 floors of which built records storage. Augustus Pugin produced most of the architectural designs and elevations of the project and interior design. The first stone of the tower was laid by Queen Victoria on December 22, 1843, and construction was completed in 1860.

The tower was originally called "King's Tower" because the fire of 1834 that destroyed the old Palace of Westminster came during the reign of King William IV.