The Merchant's Wharf Customs House was constructed between 1832 and 1836 to manage the booming import-export trade passing through what was then one of the busiest commercial ports on the eastern seaboard. Its architect, Elias Grimshaw, delivered a neoclassical palazzo in granite and marble that contemporaries called 'absurdly grand for a building that smells of fish.'

Inside, the customs hall's original mahogany counters, iron safes, and ceiling-high ledger shelves remain intact — along with the remarkable collection of confiscated contraband the customs officers accumulated over 140 years of operation.

The building remained an active customs facility until 1976, when port operations relocated to a modern container terminal. It has operated as a heritage museum of maritime trade and commerce since 1984.