March 2021 Gwern.net Newsletter

2 major new site features: 'popins' and recursive Wikipedia popups

March 2021’s Gwern.net newsletter is now out; previous, February 2021 (archives). This is a summary of the revision-history RSS feed, overlapping with my Changelog & /r/gwern; brought to you by my donors on Patreon.

1 Writings

• Gwern.net: mobile “popins” are finally enabled! (example); new Wikipedia popups (this 7th implementation enables recursive WP popups)

2 Links

2.1 AI

• “Multimodal Neurons in Artificial Neural Networks”, Goh et al 2021 (dissecting CLIP concepts, discovering typographical classification ‘attacks’^1^ and a Stroop effect! Is there anything CLIP can’t do?)

• “Evolving Reinforcement Learning Algorithms”, Co-Reyes et al 2021 (evolving eg TD-learning)

• “Waymo Simulated Driving Behavior in Reconstructed Fatal Crashes within an Autonomous Vehicle Operating Domain”, Scanlon et al 2021 (blog; hard negative mining—self-driving cars, being inhuman, can learn not just from their mistakes but humans’ mistakes too)

• “Debugging Reinforcement Learning Systems Without The Agonizing Pain”, Andy L. Jones; “My Reinforcement Learning Learnings”, Clemens Winter

Matters Of Scale:

• “SEER: Self-supervised Pretraining of Visual Features in the Wild”, Goyal et al 2021 (blog; near-SOTA by training 1b-param CNN on 1b unfiltered unlabeled Internet images—another reminder that unsupervised learning is really working!); “‘Learning From Videos’ to understand the world” (rapid FB expansion of self-supervised training to millions of photos/videos/hours-of-speech); “Contrasting Contrastive Self-Supervised Representation Learning Models”, Kotar et al 2021 (Supervised learning from ImageNet is now obsolete for transfer learning, and ImageNet just a contaminated validation set)

• “Understanding Robustness of Transformers for Image Classification”, Bhojanapalli et al 2021 (Vision Transformers gain robustness faster than CNNs as dataset size increases)

• “Artificial Intelligence Index Report 2021”: technical performance and cost (Ding questions whether this shows China catching up on AI at all, as we are incessantly told it is doing; one question to ask: ignoring fast-following, what, out of the thousands upon thousands of publications flooding out these days, are the last 3 major novel AI breakthroughs coming out of all pure-Chinese labs combined which could be plausibly equated in importance with, say, just OpenAI’s recent output of GPT-3/DALL·E/CLIP?)

• OA GPT-3 API: >300 apps, >10k developers, >4.5b words per day

• “A mathematical theory of semantic development in deep neural networks”, Saxe et al 2019 (are jumps in NN capabilities to be expected when scaling? see also Viering & Loog 2021’s discussion of phase transitions & averaging of exponentials giving power-laws)

• “An early cell shape transition drives evolutionary expansion of the human forebrain”, Benito-Kwiecinski et al 2021 (media; a simple switch for the scaling up of the primate brain)

  • “Crows possess higher intelligence long thought primarily human” (the remarkable, yet not extraordinary, crow/raven brain as scaled-up bird brain)

2.2 Genetics

Everything Is Heritable:

• “GWAS in almost 195,000 individuals identifies 50 previously unidentified genetic loci for eye color”, Simcoe et al 2021

• “Why Do Wealthy Parents Have Wealthy Children?”, Fagereng et al 2021 (I’m always impressed just how difficult it is for rich people to pass on wealth—“shirtsleeves to shirtsleeves in 3 generations” etc)

Evolution:

• “Nothing in evolution makes sense except in the light of parasites”, Hickinbotham et al 2021

Engineering:

• “The Demise and Potential Revival of the American Chestnut”

2.3 Statistics/Meta-Science

• “Broad cross-national public support for accelerated COVID-19 vaccine trial designs”, Broockman et al 2021 (“we can’t do challenge trials with volunteers in February 2020 to save countless thousands of lives because ordinary people might think it unethical”—have you tried asking them, or was that irrelevant because it was just another noble lie?)

• “This is the story of how I found what I believe to be scientific misconduct and what happened when I reported it”, Joe Hilgard

• “The Revolution in Classic Tetris: How a younger generation used the Internet to master the falling blocks” (how achieving classic Tetris maximum-scores, first done in 2010, became routine thanks to YouTube & online competition for excellence)

2.4 Politics/Religion

• “Magic, Explanations, and Evil: The Origins and Design of Witches and Sorcerers”, Singh 2021 (doubtless even cavemen were all “Og: sus.”)

• “Self-blinding citizen science to explore psychedelic microdosing”, Szigeti et al 2021 (related to Kaertner et al 2021; a self-blinding study, similar to my old self-blinding protocols, confirms that microdosing is just placebo effect, as I said in 2012, and I’m reminded of DNB studies like Foroughi et al 2016)

• The 2019–2020 vaping moral panic over adulterated black-market THC products (depressing to see how irresponsibly reported & alarmist this was, and how everyone attempted to frame nicotine for it2. Naturally, no one involved has apologized or admitted fault—after all, their intentions were good, “won’t someone think of the children”‽ The incompetence and/or dishonesty here emphasizes how 2020–2021 was business as usual, and the only unusual part is that reality happened so fast we saw some of the unseen.)

• Mark Hofmann

• Alexandra David-Néel (one of those 1800–1900s biographies)

• John Harvey Kellogg

2.5 Psychology/Biology

• “Can You Ever Be Too Smart for Your Own Good? Comparing Linear and Nonlinear Effects of Cognitive Ability on Life Outcomes”, Brown et al 2021

• “The pandemic fallacy: Inaccuracy of social scientists’ and lay judgments about COVID-19’s societal consequences in America”, Hutcherson et al 2021 (highly-inaccurate even retrospectively, typically grossly overestimating)

• “Training Working Memory for Two Years—No Evidence of Latent Transfer to Intelligence”, Watrin et al 2021 (fade-out of expectancy/placebo effects)

• “Real-time dialogue between experimenters and dreamers during REM sleep”, Konkoly et al 2021

• “Leroy’s elusive little people: A systematic review on lilliputian hallucinations”, Blom 2021 (Alice in Wonderland syndrome)

• “A Group of Orca Outcasts Is Now Dominating an Entire Sea: ‘Transient’ killer whales that feast on seals and hunt in small packs are thriving while their widely beloved ‘Resident’ siblings are dying out” (I wonder how the third orca type, ‘offshore’, are doing?)

• “Estimation of the total saliva volume produced per day in 5-year-old children”, Watanabe et al 1995

2.6 Technology

• “The Aesthetic-Usability Effect”, Moran 2017 (“They Might Never Tell You It’s Broken” if it’s pretty enough; see also “The Third User”)

• “Cameras and Lenses”, Bartosz Ciechanowski (explorable; followup to “Lights and Shadows”)

• “Large Batch Simulation for Deep Reinforcement Learning”, Shacklett et al 2021 (your computer is faster than you think)

• “The incredible boxes of Hock Wah Yeo” (unusual video game packaging design)

• “Stone Walls That Stay Built: A master waller shares how to dry-lay stone walls that hold their ground for centuries”, Post 2017

• Automated storage and retrieval system

• Visual cryptography

2.7 Economics

• “The Use and Misuse of Income Data and Extreme Poverty in the United States”, Meyer et al 2021 (measurement error in non-registry surveys of population extremes—not quite “lizardman” but similar problem)

• “Is economics performative? Option theory and the construction of derivatives markets”, Mackenzie 2006 (the mechanics of how the Black-Scholes model changed markets: Black ran a service printing “paper” estimating optimal prices for all options which traders could consult & use with simple heuristics to try to arbitrage the market)

• “Whitewood under Siege: On the front lines of the pallet wars” (the competition between the two ecosystems of shipping pallets: ‘whitewood’ & ‘blue pallet’)

• Mautam

2.8 Philosophy

• “Coping with mortality: responses of monkeys and great apes to collapsed, inanimate and dead conspecifics”, De Marco et al 2021

• Braitenberg vehicle

2.9 Fiction

• “Reply of the Zaporozhian Cossacks”

2.10 Miscellaneous

• America’s top ace, Major Dick Bong

3 Film/TV

Live-action:

• North by Northwest (Hitchcock 1959; for such a extremely respected movie, it felt oddly formless and like it was bouncing through genres as more of a comedic B-movie romp than a serious auteur’s effort—since James Bond started in 1953, with a TV adaptation in 1954, NbN comes off as almost a satire. I mean, really, monkeying around in Presidential noses!)

---

1. While interesting, these are ‘attacks’ only in the most generous interpretation possible (since it does know the difference), and the fact that CLIP can read text in images to note the semantic similarity, is to considerable credit. As the CLIP authors note, some queries benefit from ensembling, more context than a single word class name such as prefixing “A photograph of a”, and class names can be highly ambiguous: in ImageNet, the class name “crane” could refer to the bird or construction equipment; and the Oxford-IIIT Pet dataset labels one class “boxer”. (CLIP is still vulnerable to regular adversarial examples, of course.)↩

2. It couldn’t’ve been nicotine because people had been vaping for a decade and a half without widespread near-instantaneous lung-related fatalities! It had to be a new adulterant, and as soon as the first few black-market THC links surfaced, that meant the problem had to be THC-products-only because how would the same adulterant simultaneously get into the different supply chains? And yet, every article, health official, and activist did their paternalist best to suggest otherwise to pin the blame on regular vaping, no matter how many tests turned up clean, and it was the nicotine vaping products which got summarily banned…. One must assume many of those laws are still on the books, inasmuch as the shipping bans keep expanding.↩