June gwern.net newsletter
June 2020 gwern.net newsletter with 3 new pages/essays, and links on CRISPR, population screening, AI scaling, politics, and technological unemployment.
This is the June 2020 edition of the
gwern.net newsletter; previous, May 2020 (archives). This is a summary of the revision-history RSS feed, overlapping with my Changelog & /r/gwern; brought to you by my donors on Patreon.
Substack readers: This issue may be cut off due to length, or the auto-text version unreadable. I can fix neither; Substack has been informed but has no estimate of timelines for fixes. If you run into any problems, please read the web version, which is always the canonical version.
Everything Is Heritable:
“Efficient polygenic risk scores for biobank scale data by exploiting phenotypes from inferred relatives”, Truong et al 2020 (PGSes are not optimal for pure predictive power; BLUP/mixed-models greatly outperform them; see Gianola & Rosa 2015)
Sequence everyone: “Whole-genome sequencing of rare disease patients in a national healthcare system”, Ouwehand et al 2020; “Genomic analyses implicate noncoding de novo variants in congenital heart disease”, Richter et al 2020; “Genetic ancestry analysis on >93,000 individuals undergoing expanded carrier screening reveals limitations of ethnicity-based medical guidelines”, Kaseniit 2020; “An integrated polygenic and clinical risk tool enhances coronary artery disease prediction”, Aguilera et al 2020
“Preimplantation Genetic Testing for Polygenic Disease Relative Risk Reduction: Evaluation of Genomic Index Performance in 11,883 Adult Sibling Pairs”, Treff et al 2020 (previously: Karavani et al 2019 & Treff et al 2019; while I have concluded that transhumanists need not be interested in embryo selection because it will not be causing major societal changes in the next decade or two compared to the much more rapid progress in AI & it increasingly looks like more powerful paradigms will arrive too late to matter, it is still happening)
“Image Augmentations for GAN Training”, Zhao et al 2020b; Tran et al 2020; Karras et al 2020; Zhao et al 2020c^1 (Commentary—correct data augmentation for GANs: surprisingly simple & effective, and invented simultaneously 4 times in the past month; see also Zhao et al 2020a.)
“SBR: Learning to Play No-Press Diplomacy with Best Response Policy Iteration”, Anthony et al 2020 (natural language Diplomacy agents surely can’t be too much more difficult given NLM progress…)
Matters Of Scale:
“GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding”, Lepikhin et al 2020
Training a 600b-parameter NN translation model for 100 languages; +13.5 BLEU; training 1t-parameter models already tested. Note that while 600b is impressive, as a mixture of experts (MoE), it is more like thousands of much smaller highly-redundant models pasted together—this has great performance advantages in training & querying, but means it is more analogous to a regular NMT model with, say, 60b-parameters rather than 600b.
Literally GPT-2 but for 64px images—simply train a big Transformer to predict the next pixel, and it catches up with the fancier semi-supervised methods like SimCLR. This eschews any metric learning or bootstrapping or non-quadratic attention mechanisms like Sparse Transformers, but with enough compute still works… “Attention is all you need”? The new efficient attentions may break the quadratic bottleneck limiting current uses.
“How big should my language model be?”, Huggingface 2020
“Accelerating dynamics of collective attention”, Lorenz-Spreen et al 2019 (the issue-attention cycle speeds up? See also “The universal decay of collective memory and attention”, Candia et al 2019)
“Breast-Feeding of Animals by Women: Its Socio-Cultural Context and Geographic Occurrence”, Simoons & Baldwin 1982 (from “Did Breast-Feeding Play A Role In the Evolution of Pets?” via “Animal Ethics and Evolutionary Psychology—10 ideas”)
“Incest Laws and Absent Taboos in Roman Egypt”, Strong 2005
“The Curse of Konzo: In 1981, an international group of doctors identified the devastating disease behind a perplexing outbreak of paralysis in northern Mozambique” (the investigation of cyanide poisoning by, among others, Hans Rosling)
“Why Humans Totally Freak Out When They Get Lost: People really do circle past the same tree over and over again—it doesn’t just happen in movies” (why redirected walking works—we’re not good at this)
“Increased weight loading reduces body weight and body fat in obese subjects—A proof of concept randomized clinical trial”, Ohlsson et al 2020 (the rodent ‘gravitostat’ replicates in humans with weight vests? previously: Jansson et al 2018/Ohlsson et al 2018/Ohlsson & Jansson 2018/Palsdottir et al 2019)
“Fast food outlets, physical activity facilities, and obesity among adults: a nationwide longitudinal study from Sweden”, Okuyama et al 2020; “Association Between Lottery Prize Size and Self-reported Health Habits in Swedish Lottery Players”, Östling et al 2020 (population registries: where correlations go to die—note that the lottery gets much more precise & near-zero causal effect estimates than previous work, exemplifying the iron & stainless steel laws)
“Health Recommendations and Selection in Health Behaviors”, Oster 2020 (self-fulfilling prophecies in nutrition/diet)
“On the Potential of Silicon as a Building Block for Life”, Petkowski et al 2020 (generally infeasible except possibly in a sulfuric acid environment)
“A Group Is Its Own Worst Enemy”, Clay Shirky 2003/2005
“The Monotype 4-Line System for Setting Mathematics”, Rhatigan 2007 (the Monotype system brought the black art of boiling lead & mathematical typesetting to perfection just in time to be obsoleted by computers—famously spurring Knuth to invent TeX, eventually giving academia the omnipresent look of LaTeX; see also The Printing of Mathematics, Chaundy et al 1954)
“The Incredible Story of the U.S. Army’s Earth-Shaking, Off-Road Land Trains” (the 174-meter-long TC-497 Overland Train Mark II, designed for building the Arctic DEW, could haul 150 tons >640km; its predecessor, the VC-22 Sno-Freighter was designed & built in a month)
“Automation As Colonization Wave”, Robin Hanson (on Scholl & Hanson 2019: why is automation/productivity growth so slow & “you can see the computer age everywhere but the statistics”, especially when AI has become so powerful? Extreme inertia & path-dependence: per David 1989/1990 & Brynjolfsson & Hitt 1998/2000/2003 & Gordon 2012, existing companies & processes are so hardwired to use humans as the basic building block that they must be reconceived to exploit new possibilities. Otherwise, you get absurdities like robotic process automation. This ‘overhang’ also explains why startup ideas fail repeatedly before succeeding & crises can lead to abrupt increases in existing technologies: the “rising water” was held back by levees of local optimums.)
“Schrödinger’s Zombie: Adam Brown” (the quantum bomb tester can ‘photograph’ objects & ‘run’ computers to compute counterfactually without any interaction; what if instead you use a simulation or a real brain? Keith Bowden suggests it could test whether you have a quantum soul.)
The Printing of Mathematics, Chaundy et al 1954