Accuracy of Large-Scale Genome Scanning Services
Although the genome scanning services offered by companies such as 23andMe, deCODEme, and SeqWright have been front and center in the press the last few weeks, I’m sure that the following information will not be included in any of the reports.
Comparisons
Two different sources have concluded that the scanning service offered by 23andMe and deCODEme, who use different types of Illumina SNP Chips, are highly reproducible. In January 2008, Ann Turner compared the results of testing at deCODEme and 23andMe, and concluded that of the 560,163 SNPs that overlapped and had a “call” (meaning there was a measurable result), they agreed on 560,128 and disagreed on 35. Ann wrote in January:
In all of [the disagreed calls], one company would make a homozygous call while the other company made a heterozygous call – there were no cases where they made a completely discordant call. All in all, I’d say that is pretty impressive.
The second analysis comes from Antonio C B Oliveira at Longa Vista, a new blog that appears to have been created to present these results and related information. Oliveira obtained results from 23andMe and deCODEme and compared the results, which are available here. He concluded that of the 560,299 SNPs that overlapped and had a call, the two scans agreed on 560,276 and disagreed on 23. The 23 disagreed upon SNPs are listed by chromosome. Oliveira writes:
This error rate seems to me to be quite acceptable and I wonder if this is the rate expected in scientific studies using the same technology.
Program to Compare Your Results
Interestingly, Oliveira created a computer program to analyze the results for him, and he has graciously made that program available “as a Windows executable and the source code is provided under the GNU General Public License.”
Conclusions/Thoughts
Note that Oliveira’s results contained 136 more overlapping results, presumably because of fewer no-calls in the data. Is Illumina able to produce more calls as they gain experience with the process, or is this an expected amount of variation from person to person? I would be interested to see more results and comparisons to determine the answer to this question.
HT: Genetic Future. If you’re interested in genome sequencing or personalized genomics, you should be reading Genetic Future. I highly recommend adding the feed to your reader. Genetic Future gave a hat tip about this information to Kevin Kelly at The Quantified Self. There, Kelly points out that none of the SNPs in Oliviera’s analysis are currently associated with any physical phenotype or disease. I hope Kelly plans to do a comparative analysis of his results, as that would be an interesting addition to the information provided by Turner and Oliviera.
May 6th, 2008 at 8:44 pm
disagreements are fewer than the no call differences, which certainly seems like an acceptable rate to me. I wonder what the disagreements would be if someone were to have themselves tested twice, using two independent samples, say 6 months apart? TheGenetic Genealogisthas a nice summary. ShareThis
May 25th, 2008 at 4:07 am
The Genetic Genealogist How reproducible are the results of genome scanning services? Blaine Bettinger evaluates theAccuracy of Large-Scale Genome Scanning Services. He follows the article up by assessing the Reproducibility of SNP Testing.
May 5th, 2008 at 12:06 pm
[...] Genetic Genealogist reports that two different people have performed comparative studies of 23andMe and deCODEme’s services, finding that they both offer very low error rates and are [...]
May 5th, 2008 at 8:32 pm
In conflict with the title, “Accuracy of Large-Scale Genome Scanning Services”, I don’t see evidence that 560,000+ calls were indeed correct. Instead, I acknowleging that the results correlate well. This may seem like a small point, but as both methods are similar, concluding they are indeed accurate because they give common results is faulty and misleading.
May 5th, 2008 at 8:34 pm
Sorry, “acknowledging” should have been “acknowledge” [Note to self: must read full text before hitting "Post Comment" button]
May 5th, 2008 at 9:31 pm
JHealey, you are absolutely right that there is a very big difference between the concepts of accuracy and the reproducibility. As you’ll note however, the article itself clearly discusses the reproducibility and does not make any assumption or conclusion about accuracy (i.e. whether or not these results reflect the individual’s actual genetic code).
Regarding the title, I would argue that reproducibility is one of the factors involved in ultimately determining accuracy, and that this data begins to shed some light on the topic.
May 6th, 2008 at 4:23 pm
Thanks Blaine, no need to argue
I would only add that the title should reflect the primary content or purpose of the text. The title clearly points to accuracy and the text, as you noted, “does not make any assumption or conclusion about accuracy”. If I am incorrect, please allow me to send you a copy of the Bible retitled as “The Evolution of the Species”!
May 9th, 2008 at 4:41 pm
[...] by 23andMe and deCODEme using Illumina SNP chips (see the Quantified Self’s post and my post). In that post, it was revealed that two comparisons of the 560,000 overlapping SNP results from [...]
July 4th, 2008 at 7:03 am
[...] reproducible are the results of genome scanning services? Blaine Bettinger evaluates the Accuracy of Large-Scale Genome Scanning Services. He follows the article up by assessing the Reproducibility of SNP [...]
July 4th, 2008 at 7:03 am
[...] reproducible are the results of genome scanning services? Blaine Bettinger evaluates the Accuracy of Large-Scale Genome Scanning Services. He follows the article up by assessing the Reproducibility of SNP [...]
October 8th, 2008 at 12:25 pm
[...] Some comparisons of the type you’re considering have already been done. http://www.thegeneticgenealogist.com/2008/05/02/accuracy-of-large-scale-genome-scanning-services/ [...]
November 13th, 2008 at 2:17 pm
[...] of SNP testing by 23andMe and deCODEme and found that the results were almost identical (see here). Many genetic genealogists, especially those associated with the GENEALOGY-DNA mailing list, are [...]
May 31st, 2009 at 8:08 am
[...] article also briefly discussed the accuracy of DTC testing (something I’ve covered here before – Accuracy of Large-Scale Genome Scanning Services): “[The author] asked Princeton University molecular biologist Lee Silver via email how accurate [...]
January 22nd, 2011 at 10:08 pm
[...] one in every 14,000 data points: that’s far better than most routine clinical tests.” Antonio CB Oliveira wrote a computer program to compare his personal results from two companies, 23andMe and deCODEme [...]