One of the email correspondents was a person who has Ancestry and IT experience, and offered knowledge, experience and wisdom for users of Ancestry.com, especially on the trees and the search engines. The comments are detailed and ring true to me based on my experience (and that of others) working on Ancestry.com.
In this post, I want to concentrate on the indexing problems with Ancestry Member Trees because that seems to be where some of the problems and inconsistency of results seems to occur.
My correspondent offered this TL:DR summary (since the detailed comments are extensive), which I appreciate:
- "Trees are not indexed regularly because the process is time consuming, somewhat brittle, and running on 'old code'
- Not all tree profiles are indexed because the old system can only handle 4 billion indexed profiles.
1) Ancestry Member Trees are not indexed regularly.
"The Ancestry Member Trees are huge. Between the private and public member trees, there are over 6 billion records. [Several] years ago, Public Member Trees was approaching 4 billion profiles (records) and was adding 100-150 million items each month. Getting these profiles into the search index was a multi-week, multi-team effort. If everything goes flawlessly after the extract from the trees system, the indexing process would take 2-4 days. If something went wrong, the indexing process had to start over. Because of resource usage, both computationally and people to monitor the process, a monthly update was all the team would commit to back-in-the-day. And if the index failed, sometimes the process had to wait until the next month to allow other collections through the production pipeline.
2) Ancestry Member Trees do not index profiles without sources
"As mentioned, the Ancestry trees [several] years ago were approaching 4 billion profiles. By now, they have probably exceeded the number [Note by Randy: the corporate information page claims 6 billion now]. Unfortunately, the indexing system for Member Trees is based on a 32-bit architecture. Every record in a collection must have a unique ID. In a 32-bit system, that means that there are only 4,294,967,296 total records allowed. We were running out of space.
"The obvious solution is to move the search engine to a 64-bit architecture. That allows 18,446,744,073,709,551,616 records in an index, and is what current software / data architectures are built on. But that requires significant investment in development - not just for the search engine, but also for the tools that feed the engine. Process and people costs were too high [several] years ago...and the plan was to move everything from the home-grown search engine to Solr/Lucene, which already supported 64-bit. So, we punted. We'd get Solr spun up on some test projects, then migrate everything over to it and not have to pay the price for upgrading the home-server.
"We looked for ways to reduce the index size. Were there "trash" nodes in the tree that didn't belong? Yes. In doing analysis, we found rogue profiles that had 10,000+ events associated with them. Those were obvious candidates to drop, but that wasn't enough. What about empty profiles? If there is only a name and no other information, is there any value in adding it to the index? If it is the only reference to that person, then it is potentially of value to someone. But if it doesn't have a source to back it up, then it is of little value - it is just conjecture. That pulled the projected index size down into something we could manage for a while.
"It appears the rules have been expanded to 'anything without an attached Ancestry source.' In their defense, the number of users who actually create a non-Ancestry source is small, so they probably thought this wouldn't affect many profiles. But those users who take the time to do so are usually quite good and there could be benefit to others, so that defense doesn't go too far.
"The short-term solution - prune the index until Solr was ready - seemed reasonable [several] years ago. If you check the message boards, there was some understandable feedback when the cuts accidentally went too deep. The next roll of the index corrected that, but the index continued to grow and the Solr solution wasn't coming along as fast as hoped...so some different rules were apparently applied later."
3) Randy's opinion: What my correspondent describes above is really "Ancestry is a victim of its own success, and they didn't react fast enough." Like Topsy, the company grew quickly, a customer base grew, and then stuff happened. They started with a home-grown search system, and then had to react to the eventual limitations of the search system. I worked for 40 years in aerospace engineering and lived through several cycles of this type of business and technology challenges, but without a giant customer base.
Over about 20 years, this company, their products, and customer base grew significantly year-over-year, and resources were strained. Computer technology, hardware and software improved exponentially at the same time, and that continues unabated. The company was challenged by the need to improve their products, incorporate new technology, fight obsolescence, add capable staff and management, and be profitable, all while satisfying customer, corporate and investor expectations, in a dynamic, and competitive, technology and business environment.
In my opinion, more information is preferred when problems like the Tree Indexing issue occur, rather than less information. Customers don't always understand why things happen, and then complain about the perceived problems (this blogger included). I am more than willing to publish information from Ancestry.com, without my editorializing, to discuss these comments from my correspondent if they desire.
In the next post, my correspondent will discuss the issues relating to the search process for records on Ancestry.com.
=============================================
Disclosure: I have had a paid subscription to Ancestry.com since 2000, and use the site every day. I have received material considerations from Ancestry.com in years past, but that does not affect my objectivity in writing about their products and services.
Copyright (c) 2019, Randall J. Seaver
Please comment on this post on the website by clicking the URL above and then the "Comments" link at the bottom of each post. Share it on Twitter, Facebook, Google+ or Pinterest using the icons below. Or contact me by email at randy.seaver@gmail.com.
Please comment on this post on the website by clicking the URL above and then the "Comments" link at the bottom of each post. Share it on Twitter, Facebook, Google+ or Pinterest using the icons below. Or contact me by email at randy.seaver@gmail.com.
