The Clark Library has been called possibly "the most exquisite building on the Pacific Coast". It was designed after Italian and French sources by Robert Farquhar, and erected between 1924 and 1926.

The main entrance to the building (now used only for public events) opens onto a marble vestibule with an arched ceiling bearing allegorical paintings by Allyn Cox.

The vestibule opens at either side onto twin book rooms, which originally housed Clark's whole collection.

At the far end the vestibule opens onto a lofty hall running the width of the building. This drawing room, which Clark built for the performance of music, provides the venue for the various concerts, academic programs, and other events sponsored by the Center and the Clark. Life-size paintings by Allyn Cox, set into ornate wood paneling in the ceiling and upper wall, depict scenes from John Dryden's All for Love.

The collection soon outgrew the book rooms, and today the main business of the library is conducted in the basement. Subterranean annexes off the public reading room now house most of the books and manuscripts, along with offices for long-term readers and kitchen facilities.

The North Range, across the lawn from the library, houses administrative personnel. Next to it, the old gatehouse serves for garden and security staff. Clark's house was torn down in 1971 to make room for a spacious parking lot.