DO WEBSITE OWNERS HAVE TO CHANGE OR ALTER WEBSITES TO BE INCLUDED IN THE CRAWLS?

No, website owners do not have to change the content, structure, or appearance of their websites to be included in the crawls. Website design can affect the completeness and accuracy of the archival captures, however; following these Guidelines for Preservable Websites helps ensure successful web archiving. 

WILL YOUR CRAWLING INTERFERE WITH ACCESS TO OUR WEBSITE?

We crawl websites at a polite rate so as not to interfere with access to your website. Crawls will generally be run quarterly or semi-annually for actively updated websites, and last for a few days. Once a crawl is complete, the crawler no longer interacts with your server. If you encounter any issues or have any additional questions, please contact us at webarchiving@library.columbia.edu.

ARE YOU ABLE TO CAPTURE MEDIA, AUDIO AND VIDEO FILES?

Yes, downloadable media, audio and video files can usually be captured, although videos hosted on third-party services like YouTube or Vimeo can be challenging. Our crawler follows links in order to discover and capture content, so links to content must exist on a website in order for that content to be included in the archive. We can’t capture files that are not linked and have to be retrieved from a database via user query. (For example, a publications database that requires one to execute a search in order to access publications.) Streaming audio and video can’t be captured at all by the current generation of web crawlers.

HOW CAN I VIEW THE WEBSITES THAT HAVE BEEN ARCHIVED? WILL ACCESS ALWAYS BE FREE?

Archived websites will remain freely accessible to the public. Websites can be viewed by date of capture via our Internet Archive partner page. Additional means of viewing archived websites will be explored by program staff.

WHY DO THE ARCHIVED VERSIONS OF SOME WEBSITES APPEAR TO BE INCOMPLETE?

There are several reasons why an archived website may appear to be incomplete. Some types of content are challenging or impossible to capture and/or reproduce, including JavaScript-driven navigation menus, streaming audio and video, and dynamic form and database-driven content. We can’t capture files that are not linked and have to be retrieved from a database via user query.  (For example, a publications database that requires one to execute a search in order to access publications.) Also, portions of a website may be restricted or password-protected. We will only collect public content, so password protected material will not be crawled. Owners wishing to optimize their site design to allow full archiving may be interested in these Guidelines for Preservable Websites.

I WOULD LIKE MY ORGANIZATION’S WEBSITE TO BE REMOVED FROM THE WEB RESOURCES COLLECTION. WHO DO I CONTACT?

We will honor requests to remove archived content. Please contact webarchiving@library.columbia.edu.