Sunday, September 4, 2011

1940 U.S. Census Contractors Questions and NARA Answers

In Comments on the 1940 U.S. Census RFQ and SOW, I linked to the Federal Business Opportunities website with the Request For Quote (RFQ) and Statement Of Work (SOW) for the National Archives (NARA) pending contract to host the 1940 United States Census images, and provide links to census pages in each Enumeration District.

The due date for responses to the RFQ was extended to 8 September at 4 p.m., and now the NARA answers to contractors' questions have been posted.  The link to the Q&A is here (under Amendment 2 on the right-hand side).

Questions and answers that I found interesting include:

How should we interpret the following condition found on page 3 for the SOW? "The contractor may take no action to develop or create a name index or any other product prior to April 2, 2012."
Does this only refer to indexing fields that would be considered Personally Identifiable Information (PII) - as listed on page 2 of NAMA-11-Q-0081 or any of the fields on the images?
The contractor may not create a name index using names or other information found on the Census schedules.
What is the scope of 'or any other product' in this context?
The contractor may not create or develop any product created using the information found on the Census schedules until after April 2, 2012.
Would this clause prohibit [XX] from creating collection specific functionality for our own site prior to April?
The Contractor could develop functionality on their own site but could not use the 1940 Census schedules until April 2, 2012.

This very clearly answers, several times, that the contractor cannot do anything to create an index until after 2 April 2012.  The contract is host the images and link to them.  After 2 April, the contractor can create an index and link to the images.

Will a user be denied access based on his location? Example: user coming from a hostile country.
At this time, NARA does not intend to restrict access to the census data.

That is clear also - anyone anywhere can view, and capture, the census images from the contractor's site.

Given the rapidly approaching deadline of April 2, 2012, what date does NARA expect to award the project?
NARA expects to award the contract by early October.

So the contractor will have about six months to complete the work required to allow thousands of users to view the 1940 census on 2 April 2012 within the specified requirements set out by the Statement of Work.

What specific browsers and versions are to be supported by the website?
The browsers and versions required to be supported are Internet Explorer, Netscape Navigator, Mozilla Firefox, backward-version (versions 4.x and up), and cross-platform (Windows and Mac) compatible

I think that the NARA response here is unclear - shouldn't they have specified specific versions for each browser?

Is there any additional detail available on the requirements for sharing images through social media tools?

A short answer for a specific question.  Entrepreneurs, start your social media engines!

Given NARA’s intent to offer the complete Census for sale in digital format beginning April 2, 2012, what anti-crawling policies should be established and/or what other methods are expected to be employed to prevent parties from simply downloading all images from every Enumeration District (as enabled by specifications in the RFQ) thus circumventing the process of ordering and paying for the digital images?
The National Archives Trust Fund is selling Census data in digital as well as microfilm format as a convenience. We do not wish to limit downloading in any manner.

This seems to say that you can buy it (every page for $200,000) digitally or on microfilm, or download it ED by ED, it's your choice.  I hope you have enough terabytes.

Please clarify if the image presented on the website for viewing in the browser needs to be a 4 MB file or if the file can be compressed and optimized for online viewing, zoom, and pan? It appears the 4MB file size only needs to be available for actual download to save to a user’s computer.
The files can be compressed and optimized for online viewing, zoom, and pan.  We would like the file size to remain at the full compressed size (approximately 4MB) for customer downloads.

Can a link be provided from the search results on the website to the Contractor’s name-index search?
No, however the search interface and search result displays include a link to the Contractor’s web site.

NARA says that the Contractor cannot put a link to the Contractor's name-index search site on the website, only to the Contractor's website. 

Para 9:  Will the contractor be involved with the sale of data or will only NARA sell the data? 
The National Archives Trust Fund will offer the census information in digital form through their online store, and the contractor will not be involved.

Some Contractor was looking to monetize their involvement here.  Um, no.

There are 32 questions and answers in the document. 

Hopefully, some Contractor will be awarded this contract in early October, and will flawlessly host the 1940 census page images, permit their viewing, and permit the ability to download one page, or a whole Enumeration District, on 2 April 2012. 

The challenge for the selected Contractor, and any other interested party, will be to create a useful name-index to the census as quickly as possible.  Who will those parties be?  We'll have to wait to find out.

The challenge for genealogists will be to have patience when the 1940 U.S. Census is released.  Each of us can try to find the EDs where our family members lived in 1940, using the process described in How to Access the 1940 Census in One Step (by Steve Morse, PhD, Joel Weintraub, PhD, and David R. Kehs, PhD).

More patience will be required waiting for the overload on the Contractor's servers on 2 April 2012 to reduce so that we can actually see the page images and download them, and then to wait for the name-index search capabilities to be developed after 2 April 2012.  There are about 130 million names to index, along with their personal identity information.  It's going to take 6 to 24 months to complete, I think. 

No comments: