The Avery Architectural & Fine Arts Library collects books and periodicals in archaeology, architecture, art history, city planning , decorative arts, design, historic preservation, landscape architecture, painting, photography, real estate development, and sculpture. The open-stacks research collections consist of three main collections: The Avery Collection (architecture, city planning, decorative arts, design, and related topics), the Fine Arts Collection (art history, painting, sculpture, drawing, and related topics) and the Avery-LC Collection (all topics in Avery Library’s scope collected after 2011).  The Avery and Fine Arts Collections are in-library use only, but the majority of the Avery-LC collection circulates. The Library currently contains more than 625,000 volumes and receives approximately 1,300 periodicals.

The scope of the Avery collection in architecture is outstanding; it ranges from the first Western printed book on architecture, De re aedificatoria (1485), by Leone Battista Alberti, to the classics of modernism by Frank Lloyd Wright and Le Corbusier. Avery Drawings & Archives Collection holds approximately two million drawings and original records in more than 950 collections.

The Avery Library is named for Henry Ogden Avery, one of late nineteenth century New York's promising young architects and a friend of William Robert Ware, who founded the Department of Architecture at Columbia in 1881. A few weeks after Avery's premature death in 1890, his parents, Samuel Putnam Avery and Mary Ogden Avery, established the library as a memorial to their son. They offered 2,000 of his books, mostly in architecture, archaeology, and the decorative arts, many of his original drawings, funds to round out the book collection, and an endowment to assure the continuous growth of the library.

Within five years the collection had grown to 13,000 volumes and Edwin Robinson Smith, a sculptor, was named the first Avery Librarian. In 1895, Samuel Putnam Avery funded a 1,139-page printed catalog which has since been published in two editions (1958, 1968), with three supplements (1972, 1975, 1977). Since 1978, all cataloging has been computerized and since 1988, records are only available on-line. In 1985-1987 cataloging records for 32,000 titles from Avery architectural collections were converted to machine readable form and today we estimate that approximately 98% of Avery's titles can be searched through CLIO, Columbia's online catalog.

In 1897 Avery Library moved from an alcove in the 49th Street library building into a new wing in Low Memorial Library on the Morningside Heights campus. In 1912 it became the first library to abandon its quarters in Low for its own building, Avery Hall, a new Renaissance building by McKim, Mead, and White. This final home of Avery Library was the gift of Samuel Putnam Avery II, son of the original donor.

Throughout its history Avery has had a succession of eminent librarians. The archaeologist and historian William Bell Dinsmoor (1920-1926) established the separate Fine Arts Library. He discovered and acquired some of Avery's most important architectural drawings, including Sebastiano Serlio's unpublished manuscript on domestic architecture, dating from the 1540s, illustrated with Serlio's own drawings. Serlio planned this to be the sixth book of his seven volume treatise on architecture. The manuscript was finally published in 1978 by the Architectural History Foundation.

From 1934 to 1945, Talbot F. Hamlin, an architect and one of the country's leading architectural historians, was librarian. He was the first to solicit drawings from active firms and he began the systematic indexing that became the Avery Index to Architectural Periodicals. First begun in 1934, as a card file, the Avery Index to Architectural Periodicals was published in 1963 and 1973 with 13 annual supplements through 1997.  In 1983 the Getty Research Institute adopted the Avery Index as an operating program, based at Avery Library. In 2009 The J. Paul Getty Trust returned ownership of the Avery Index to Columbia University. CD-ROMs were issued annually from 1997-2001.  The Index was available online from the Research Libraries Group (RLG) from 1979 to 2007 and is now available from EBSCO and ProQuest.  In 2000, the Avery Obituary Index was added to the online database.  The addition of records from the 10-volume Burnham Index to Architectural Literature in 2004 increased retrospective coverage.  As of Winter 2019, there are almost 800,000 records in the database.  The Index provides citations to articles in approximately 300 current and over 1,000 retrospective architectural and related periodicals, with primary emphasis on architectural design and history, urban planning and design, historic preservation -- as well as archaeology, environmental studies, garden history, interior design, landscape architecture, and real estate development.

After the leadership of James G. Van Derpool (1946-1960), who emphasized the collection of rare books published prior to 1500, the library came under the direction of art historian and librarian Adolf K. Placzek (1960-1980). During these twenty years, the library acquired 122 sketches and drawings by Louis Sullivan from the Frank Lloyd Wright Foundation, 23 drawings showing Giovanni Battista Piranesi's 1764 plan for rebuilding the sanctuary of San Giovanni in Laterano, and a superb Frank Lloyd Wright collection.

Under such a series of distinguished leaders, the library expanded beyond all expectation. In addition, unlike many European libraries, Avery suffered no losses and no gap in its development as a result of World War II. Consequently, a shortage of space developed that was exacerbated by the revival of the department of art history under the chairmanship of Rudolf Wittkower and the development of both the Division of Urban Planning in the 1960s and the Historic Preservation Program under James Marston Fitch. In the fall of 1974, construction began on a new underground extension designed by Alexander Kouzmanoff, chairman of design at the School of Architecture. With the opening of the extension Avery Library also accommodated the Fine Arts and Ware collections. With the extension, the Library housed a total of 200,000 books, 50,000 drawings, and 25,000 letters and manuscripts.

Avery's collection is an unrivaled printed record of architectural thinking, including Alberti's De Re Aedificatoria (1485), one of the most complete collections of the writings of Vitruvius, Palladio, and Vignola, and Francesco Colonna's Hypnerotomachia of 1499. Volumes of engravings of buildings and guide books form a very important part of the collection. Avery owns the majority of the books published in the field up to 1800 and is extremely strong in European works published after 1800. The Modern movement is particularly well-represented, with virtually complete coverage of Frank Lloyd Wright and Le Corbusier.

The American collection is one of the most extensive in existence. It begins with the first pertinent book to be published in the colonies, Abraham Swan's British Architect (Philadelphia, 1775), and includes a large number of titles listed in H.R. Hitchcock's basic bibliography, American Architectural Books. In the seventies and eighties the scope of the American collection was expanded to include printed source materials not previously collected. These include early trade catalogs from the manufacturers of building products (1840-1950) and city "view books" (1870- 1930), which provide extensive pictorial documentation.

The drawings and archives collection was formed around the nucleus of Henry Ogden Avery's drawings and now contains approximately two million drawings, photographs, letters, and manuscripts relating to architecture and architects. The focus of the collection is American architecture, with a strong emphasis on New York City and its architectural history. The growth of the collection has been particularly noteworthy since the 1970's, a time when the importance of original architectural drawings as prime historical source materials and as superior works of art in themselves has been generally realized. Included in this collection are several important archives: Richard Upjohn, Alexander Jackson Davis, Greene & Greene, Warren & Wetmore, Harold van Buren Magonigle, Stanford White, Wallace K. Harrison, Gordon Bunshaft, Philip Johnson, Frank Lloyd Wright and the archives of the Guastavino Fireproof Construction Company. In 2012, Avery’s existing holdings of Wright related archives was monumentally expanded by the co-acquisition with The Museum of Modern Art of the Frank Lloyd Wright Foundation Archives. Avery was a pioneer in the computerization of cataloging of architectural drawings. In 1992, project AVIADOR (Avery Videodisc Index of Architectural Drawings on RLIN) was completed and its guidelines for Cataloging Architectural Drawings were published by the Art Libraries Society of North America.

The Fine Arts collection has books on art history, art theory, painting, sculpture and prints. Specifically, it includes catalogs of exhibitions and private holdings, all standard histories, a rich collection of critical works, books on artistic iconography, monographs on special subjects, and complete or nearly complete holdings of major international art periodicals. In the decorative arts, the Library has an outstanding collection on architectural ornament, textile design, mosaics, stained glass, metal work, and ancient pottery.