Jonathon Shlens
I am a principal scientist and research director at Google DeepMind interested in vision, language and learning. I lead organizations interested a basic and applied research focused on machine learning, computer vision and basic science research. Some accomplishments included the invention of TensorFlow, the deployment of many production systems, and several large collaborations with Waymo.
To learn more about my work, please see a list of my former interns as well as my publications and tutorials.
Publications
2024
Capabilities of Gemini Models in MedicineKhaled Saab, Tao Tu, Wei-Hung Weng, Ryutaro Tanno, David Stutz, Ellery Wulczyn, Fan Zhang, Tim Strother, Chunjong Park, Elahe Vedadi, Juanma Zambrano Chaves, Szu-Yeu Hu, Mike Schaekermann, Aishwarya Kamath, Yong Cheng, David G.T. Barrett, Cathy Cheung, Basil Mustafa, Anil Palepu, Daniel McDuff, Le Hou, Tomer Golany, Luyang Liu, Jean-baptiste Alayrac, Neil Houlsby, Nenad Tomasev, Jan Freyberg, Charles Lau, Jonas Kemp, Jeremy Lai, Shekoofeh Azizi, Kimberly Kanada, SiWai Man, Kavita Kulkarni, Ruoxi Sun, Siamak Shakeri, Luheng He, Ben Caine, Albert Webson, Natasha Latysheva, Melvin Johnson, Philip Mansfield, Jian Lu, Ehud Rivlin, Jesper Anderson, Bradley Green, Renee Wong, Jonathan Krause, Jonathon Shlens, Ewa Dominowska, S. M. Ali Eslami, Katherine Chou, Claire Cui, Oriol Vinyals, Koray Kavukcuoglu, James Manyika, Jeff Dean, Demis Hassabis, Yossi Matias, Dale Webster, Joelle Barral, Greg Corrado, Christopher Semturs, S. Sara Mahdavi, Juraj Gottweis, Alan Karthikesalingam, Vivek Natarajan
In submission MOFI: Learning Image Representations from Noisy Entity Annotated Images
Wentao Wu, Aleksei Timofeev, Chen Chen, Bowen Zhang, Kun Duan, Shuangning Liu, Yantao Zheng, Jonathon Shlens, Xianzhi Du, Zhe Gan, Yinfei Yang
International Conference on Learning Representations (ICLR)
2023
Perceptual Grouping in Contrastive Vision-Language ModelsKanchana Ranasinghe, Brandon McKinzie, Sachin Ravi, Yinfei Yang, Alexander Toshev, Jonathon Shlens
Proceedings of the IEEE International Conference on Computer Vision (ICCV) STAIR: Learning Sparse Text and Image Representation in Grounded Tokens
Chen Chen, Bowen Zhang, Liangliang Cao, Jiguang Shen, Tom Gunter, Albin Madappally Jose, Alexander Toshev, Jonathon Shlens, Ruoming Pang, Yinfei Yang
Empirical Methods in Natural Language Processing (EMNLP) On Robustness in Multimodal Learning
Brandon McKinzie, Joseph Cheng, Vaishaal Shankar, Yinfei Yang, Jonathon Shlens, Alexander Toshev
Proceedings of the 40th International Conference of Machine Learning (ICML) Subtle adversarial image manipulations influence both human and machine perception
Vijay Veerabadran, Josh Goldman, Shreya Shankar, Brian Cheung, Nicolas Papernot, Alexey Kurakin, Ian Goodfellow, Jonathon Shlens, Jascha Sohl-Dickstein, Michael C. Mozer, Gamaleldin F. Elsayed
Nature Communications
2022
Subtle adversarial image manipulations influence both human and machine perceptionVijay Veerabadran, Josh Goldman, Shreya Shankar, Brian Cheung, Nicolas Papernot, Alexey Kurakin, Ian Goodfellow, Jonathon Shlens, Jascha Sohl-Dickstein, Michael C. Mozer, Gamaleldin Elsayed
Nature Communications Scene Transformer: A unified architecture for predicting multiple agent trajectories
Jiquan Ngiam*, Benjamin Caine*, Vijay Vasudevan*, Zhengdong Zhang, Hao-Tien Lewis Chiang, Jeffrey Ling, Rebecca Roelofs, Alex Bewley, Chenxi Liu, Ashish Venugopal, David Weiss, Ben Sapp, Zhifeng Chen, Jonathon Shlens
International Conference on Learning Representations (ICLR) Mitigating Bias in Calibration Error Estimation
Rebecca Roelofs, Nicholas Cain, Jonathon Shlens, Michael C. Mozer
Artificial Intelligence and Statistics (AISTATS) Scalable Scene Flow from Point Clouds in the Real World
Philipp Jund, Chris Sweeney, Nichola Abdo, Zhifeng Chen, Jonathon Shlens
IEEE Robotics and Automation Letters PseudoAugment: Learning to Use Unlabeled Data for Data Augmentation in Point Clouds
Zhaoqi Leng, Shuyang Cheng, Benjamin Caine, Weiyue Wang, Xiao Zhang, Jonathon Shlens, Mingxing Tan, Dragomir Anguelov
European Conference on Computer Vision (ECCV)
2021
Individual variability of neural computations in the primate retinaNishal P. Shah, Nora Brackbill, Ryan Samarakoon, Colleen Rhoades, Alexandra Kling, Alexander Sher, Alan Litke, Yoram Singer, Jonathon Shlens, E.J. Chichilnisky
Neuron Pseudo-labeling for Scalable 3D Object Detection
Benjamin Caine*, Rebecca Roelofs*, Vijay Vasudevan, Jiquan Ngiam, Yuning Chai, Zhifeng Chen, Jonathon Shlens
Unreviewed Technical Report Revisiting ResNets: Improved Training and Scaling Strategies
Irwan Bello, William Fedus, Xianzhi Du, Ekin D. Cubuk, Aravind Srinivas, Tsung-Yi Lin, Jonathon Shlens, Barret Zoph
Neural Information Processing Systems (NeurIPS). Spotlight presentation Soft Calibration Objectives for Neural Networks
Archit Karandikar, Nicholas Cain, Dustin Tran, Balaji Lakshminarayanan, Jonathon Shlens, Michael C. Mozer, Becca Roelofs
Neural Information Processing Systems (NeurIPS) Large Scale Interactive Motion Forecasting for Autonomous Driving: The Waymo Open Motion Dataset
Scott Ettinger, Shuyang Cheng, Benjamin Caine, Chenxi Liu, Hang Zhao, Sabeek Pradhan, Yuning Chai, Ben Sapp, Charles Qi, Yin Zhou, Zoey Yang, Aurelien Chouard, Pei Sun, Jiquan Ngiam, Vijay Vasudevan, Alexander McCauley, Jonathon Shlens, Dragomir Anguelov
Proceedings of the IEEE International Conference on Computer Vision (ICCV). Oral presentation Scaling Local Self-Attention For Parameter Efficient Visual Backbones
Ashish Vaswani, Prajit Ramachandran, Aravind Srinivas, Niki Parmar, Blake Hechtman, Jonathon Shlens
Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Oral presentation Bottleneck Transformers for Visual Recognition
Aravind Srinivas, Tsung-Yi Lin, Niki Parmar, Jonathon Shlens, Pieter Abbeel, Ashish Vaswani
Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
2020
Leveraging Semi-Supervised Learning in Video Sequences for Urban Scene SegmentationLiang-Chieh Chen, Raphael Gontijo Lopes, Bowen Cheng, Maxwell D. Collins, Ekin D. Cubuk, Barret Zoph, Hartwig Adam, Jonathon Shlens
European Conference on Computer Vision (ECCV) Streaming Object Detection for 3-D Point Clouds
Wei Han, Zhengdong Zhang, Benjamin Caine, Brandon Yang, Christoph Sprunk, Ouais Alsharif, Jiquan Ngiam, Vijay Vasudevan, Jonathon Shlens, Zhifeng Chen
European Conference on Computer Vision (ECCV) Improving 3D Object Detection through Progressive Population Based Augmentation
Shuyang Cheng, Zhaoqi Leng, Ekin Dogus Cubuk, Barret Zoph, Chunyan Bai, Jiquan Ngiam, Yang Song, Benjamin Caine, Vijay Vasudevan, Congcong Li, Quoc V. Le, Jonathon Shlens, Dragomir Anguelov
European Conference on Computer Vision (ECCV) On the Importance of Data Augmentation for Object Detection
Barret Zoph*, Ekin Dogus Cubuk*, Golnaz Ghiasi, Tsung-Yi Lin, Jonathon Shlens, Quoc V. Le
European Conference on Computer Vision (ECCV) Revisiting Spatial Invariance with Low-Rank Local Connectivity
Gamaleldin F. Elsayed, Prajit Ramachandran, Jonathon Shlens, Simon Kornblith
Proceedings of the 37th International Conference of Machine Learning (ICML) Scalability in Perception for Autonomous Driving: Waymo Open Dataset
Pei Sun, Henrik Kretzschmar, Xerxes Dotiwalla, Aurélien Chouard, Vijaysai Patnaik, Paul Tsui, James Guo, Yin Zhou, Yuning Chai, Benjamin Caine, Vijay Vasudevan, Wei Han, Jiquan Ngiam, Hang Zhao, Scott Ettinger, Aleksei Timofeev, Maxim Krivokon, Amy Gao, Aditya Joshi, Yu Zhang, Jonathon Shlens, Zhifeng Chen, Dragomir Anguelov
Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Practical Automated Data Augmentation with a Reduced Search Space
Ekin Dogus Cubuk*, Barret Zoph*, Jonathon Shlens, Quoc V. Le
Neural Information Processing Systems (NeurIPS)
2019
StarNet: Targeted Computation for Object Detection in Point CloudsJiquan Ngiam*, Benjamin Caine*, Wei Han, Brandon Yang, Yuning Chai, Pei Sun, Yin Zhou, Xi Yi, Ouais Alsharif, Patrick Nguyen, Zhifeng Chen, Jonathon Shlens, Vijay Vasudevan
Workshop on Machine Learning for Autonomous Driving at NeurIPS Stand-Alone Self-Attention in Vision Models
Prajit Ramachandran*, Niki Parmar*, Ashish Vaswani*, Irwan Bello, Anselm Levskaya, Jonathon Shlens
Neural Information Processing Systems (NeurIPS) A Fourier Perspective on Model Robustness in Computer Vision
Dong Yin, Raphael Gontijo Lopes, Jonathon Shlens, Ekin D. Cubuk, Justin Gilmer
Neural Information Processing Systems (NeurIPS) A Learned Representation for Scalable Vector Graphics
Raphael Gontijo Lopes, David Ha, Douglas Eck, Jonathon Shlens
Proceedings of the IEEE International Conference on Computer Vision (ICCV) Attention Augmented Convolutional Networks
Irwan Bello, Barret Zoph, Ashish Vaswani, Jonathon Shlens, Quoc V. Le
Proceedings of the IEEE International Conference on Computer Vision (ICCV) Do Better ImageNet Models Transfer Better?
Simon Kornblith, Jonathon Shlens, Quoc V. Le
Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Oral presentation Using Videos to Evaluate Image Model Robustness
Keren Gu*, Brandon Yang*, Jiquan Ngiam, Quoc Le, Jonathon Shlens
Workshop on Safe Machine Learning: Specification, Robustness, and Assurance at ICLR Using Learned Optimizers to Make Models Robust to Input Noise
Luke Metz, Niru Maheswaranathan, Jonathon Shlens, Jascha Sohl-Dickstein, Ekin Dogus Cubuk
Workshop on Robustness and Uncertainty Estimation in Deep Learning at ICML
2018
Searching for Efficient Multi-Scale Architectures for Dense Image PredictionLiang-Chieh Chen, Maxwell D.Collins, Yukun Zhu, George Papandreou, Barret Zoph, Florian Schroff, Hartwig Adam, Jonathon Shlens
Neural Information Processing Systems (NeurIPS) Progressive Neural Architecture Search
Chenxi Liu, Barret Zoph, Maxim Neumann, Jonathon Shlens, Wei Hua, Li-Jia Li, Li Fei-Fei, Alan Yuille, Jonathan Huang, Kevin Murphy
European Conference on Computer Vision (ECCV). Oral presentation Accelerating Training in Deep Networks with a Standardization Loss
Jasmine Collins, Johannes Ballé, Jonathon Shlens
Technical report for Women in Machine Learning (WiML) A Dataset and Architecture for Visual Reasoning with a Working Memory
Guangyu Robert Yang, Igor Ganichev, Xiao-jing Wang, Jonathon Shlens, David Sussillo
European Conference on Computer Vision (ECCV) Recurrent Segmentation for Variable Computational Budgets
Lane McIntosh, Niru Maheswaranathan, David Sussillo, Jonathon Shlens
Workshop on Efficient Deep Learning for Computer Vision at CVPR Learning Transferable Architectures for Scalable Image Recognition
Barret Zoph, Vijay Vasudevan, Jonathon Shlens, Quoc V. Le
Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Spotlight presentation Learning a Neural Response Metric for Retinal Prosthesis
Nishal P Shah, Sasidhar Madugula, EJ Chichilnisky, Yoram Singer, Jonathon Shlens
International Conference on Learning Representations (ICLR). Conference track.
2017
Exploring the structure of a real-time, arbitrary neural artistic stylization networkGolnaz Ghiasi, Honglak Lee, Manjunath Kudlur, Vincent Dumoulin and Jonathon Shlens
Proceedings of the 28th British Machine Vision Conference (BMVC). Oral presentation. PixColor: Pixel recursive colorization
Sergio Guadarrama, Ryan Dahl, David Bieber, Mohammad Norouzi, Jonathon Shlens, Kevin Murphy
Proceedings of the 28th British Machine Vision Conference (BMVC). Oral presentation. Pixel recursive super resolution
Ryan Dahl, Mohammad Norouzi, Jonathon Shlens
Proceedings of the IEEE International Conference on Computer Vision (ICCV). Conditional Image Synthesis With Auxiliary Classifier GANs.
Augustus Odena, Christopher Olah and Jonathon Shlens
Proceedings of the 34th International Conference of Machine Learning (ICML) YouTube-BoundingBoxes: A Large High-Precision Human-Annotated Data Set for Object Detection in Video
Esteban Real, Jonathon Shlens, Stefano Mazzocchi, Xin Pan and Vincent Vanhoucke
Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR) A Learned Representation for Artistic Style
Vincent Dumoulin, Jonathon Shlens, Manjunath Kudlur
International Conference on Learning Representations (ICLR). Conference track.
2016
Net2Net: Accelerating Learning via Knowledge TransferTianqi Chen, Ian Goodfellow, Jonathon Shlens
International Conference on Learning Representations (ICLR). Oral presentation. Adversarial Autoencoders
Alireza Makhzani, Jonathon Shlens, Navdeep Jaitly, Ian Goodfellow
International Conference on Learning Representations (ICLR). Workshop track. Rethinking the Inception Architecture for Computer Vision
Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jonathon Shlens, Zbigniew Wojna
Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
2015
TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed SystemsMartın Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangqing Jia, Rafal Jozefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dan Mane, Rajat Monga, Sherry Moore, Derek Murray, Chris Olah, Mike Schuster, Jonathon Shlens, Benoit Steiner, Ilya Sutskever, Kunal Talwar, Paul Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda Viegas, Oriol Vinyals, Pete Warden, Martin Wattenberg, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng
Unreviewed technical report. Explaining and Harnessing Adversarial Examples
Ian Goodfellow, Jonathon Shlens, Christian Szegedy
International Conference on Learning Representations (ICLR). Conference track. Deep Networks With Large Output Spaces
Sudheendra Vijayanarasimhan, Jonathon Shlens, Rajat Monga, Jay Yagnik
International Conference on Learning Representations (ICLR). Workshop track.
2014
Zero-Shot Learning by Convex Combination of Semantic EmbeddingsMohammad Norouzi, Tomas Mikolov, Samy Bengio, Yoram Singer, Jonathon Shlens, Andrea Frome, Greg Corrado, Jeffrey Dean
International Conference on Learning Representations (ICLR). Oral presentation
2013
DeViSE: A Deep Visual-Semantic Embedding ModelAndrea Frome*, Greg Corrado*, Jonathon Shlens*, Samy Bengio, Jeff Dean, Marc'Aurelio Ranzato and Tomas Mikolov
Neural Information Processing Systems (NIPS)
* Equal contribution of authors. Fast, Accurate Detection of 100,000 Object Classes on a Single Machine
Thomas Dean, Mark Ruzon, Mark Segal, Jonathon Shlens, Sudheendra Vijayanarasimhan, Jay Yagnik
IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Best Paper Award. Three controversial hypotheses concerning computation in the primate cortex
Thomas Dean, Greg Corrado and Jonathon Shlens
Association for the Advancement of Artificial Intelligence (AAAI). A model-based spike sorting algorithm for removing correlation artifacts in multi-neuron recordings.
Jonathan W. Pillow*, Jonathon Shlens*, EJ Chichilnisky and Eero Simoncelli
Public Library of Science (PLoS) One 8: 5
* Equal contribution of authors.
2012
Efficient Coding of Spatial Information in the Primate RetinaEizaburo Doi, Jeffrey L. Gauthier, Greg D. Field, Jonathon Shlens, Alexander Sher, Martin Greschner, Timothy A. Machado, Lauren H. Jepson, Keith Mathieson, Deborah E. Gunning, Alan M. Litke, Liam Paninski, E. J. Chichilnisky and Eero P. Simoncelli
Journal of Neuroscience. 32:46 Modeling the impact of common noise inputs on the network activity of retinal ganglion cells.
Michael Vidne, Yashar Ahmadian, Jonathon Shlens, Jonathan W. Pillow, Jayant Kulkarni, Alan M. Litke, E. J. Chichilnisky, Eero Simoncelli, Liam Paninski
Journal of Computational Neuroscience. 33:1
2011
Correlated firing among major ganglion cell types in primate retinaMartin Greschner, Jonathon Shlens, Tina Bakolitsa, Greg Field , Jeff Gauthier, Lauren Jepson, Alexander Sher, Alan Litke and E.J. Chichilnisky
Journal of Physiology
2010
Functional connectivity in the retina at the resolution of photoreceptorsGreg Field , Jeff Gauthier, Alexander Sher, Martin Greschner, Timothy Machado, Lauren Jepson, Jonathon Shlens, Deborah Gunning, Keith Mathieson, Wladyslaw Dabrowski, Liam Paninski, Alan Litke and E.J. Chichilnisky
Nature. 467, 673-677
supplement, news & views
2009
The structure of large-scale synchronized firing in primate retinaJonathon Shlens, Greg Field , Jeff Gauthier, Martin Greschner, Alexander Sher, Alan Litke and E.J. Chichilnisky
Journal of Neuroscience. 29, 5022-5031 High sensitivity rod photoreceptor input to blue-yellow color opponent pathway in primate retina
Greg Field, Martin Greschner, Jeffrey Gauthier, Jonathon Shlens, Alexander Sher, Alan Litke and E.J. Chichilnisky
Nature Neuroscience. 12, 1150 Receptive fields in primate retina are coordinated to sample visual space more uniformly
Jeff Gauthier, Greg Field , Alexander Sher, Martin Greschner, Jonathon Shlens, Alan Litke and E.J. Chichilnisky
Public Library of Science (PLoS)) Biology. 7, e63. Uniform signal redundancy of parasol and midget ganglion cells in primate retina
Jeff Gauthier, Greg Field , Alexander Sher, Jonathon Shlens, Martin Greschner, Alan Litke and E.J. Chichilnisky
Journal of Neuroscience. 29, 4675-4680.
2008
Synchronized firing in the retinaJonathon Shlens, Fred Rieke, E.J. Chichilnisky
Current Opinions in Neurobiology. 16, 396-402. Spatiotemporal correlations and visual signaling in a complete neuronal population
Jonathan Pillow, Jonathon Shlens, Liam Paninski, Alexander Sher, Alan Litke, E.J. Chichilnisky and Eero Simoncelli
Nature. 454:995-999.
2007
Estimating information rates in neural spike trains with confidence intervalsJonathon Shlens, Matthew Kennel, Henry Abarbanel and E.J. Chichilnisky
Neural Computation. 19, 1683-1719. Spatial properties and functional organization of small bistratified cells in primate retina
Greg Field, Alexander Sher, Jeff Gauthier, Martin Greschner, Jonathon Shlens, Alan Litke and E.J. Chichilnisky
Journal of Neuroscience. 270, 13261 Identification and characterization of a Y-like primate retinal ganglion cell type
Dumitru Petrusca, Matthew Grivich, Alexander Sher, Greg Field, Jeff Gauthier, Jonathon Shlens, E.J. Chichilnisky and Alan Litke
Journal of Neuroscience. 27, 11019.
2006
The structure of multi-neuron firing patterns in primate retinaJonathon Shlens, Greg Field, Jeff Gauthier, Matthew Grivich, Dumitru Petrusca, Alexander Sher, Alan Litke and E.J. Chichilnisky
Journal of Neuroscience. 26, 8254-8266.
cover, commentary
2005
Estimating entropy rates with Bayesian confidence intervalsMatthew Kennel, Jonathon Shlens, Henry Abarbanel and E.J. Chichilnisky
Neural Computation. 17, 1531-1576
Tutorials
These tutorials provide a general introduction to topics I find quite interesting but often lack good explanations in textbooks or the online literature.Tutorial on Independent Component Analysis
A complete introduction and discussion of independent component analysis. Builds on previous tutorial on principal component analysis.
Version 1.0 Tutorial on Principal Component Analysis
A full introduction, description, derivation, and discussion of principal component analysis. Concrete examples for intuition building, the mathematical relation to SVD, and new extensions of this algorithm.
Version 3.02 A Light Discussion and Derivation of Entropy
A light discussion of the underlying assumptions behind entropy followed by a rigorous but simple derivation of the formula for entropy.
Version 1.01 Notes on Kullback-Leibler Divergence and Likelihood
An intuitive discussion about where Kullback-Leibler divergence arises and its relationship to likelihood theory.
Version 1.01 Notes on Generalized Linear Models of Neurons
An introduction to the application of GLMs to model neurons and networks of neurons. Brief discussion and derivation of primary equations pertaining to maximum likelihood estimation.
Version 1.51
Interns and Residents
Kanchana Ranasinghe, Graduate Student, Stony Brook UniversityPhilipp Jund, Software Engineer, Google Research
Noha Radwan, Research Engineer, Inceptive
Keren Gu, Software Engineer, OpenAI
Brandon Yang, Tech Lead Manager, Snorkel AI
Rapha Gontijo Lopes, Researcher, OpenAI
Benjamin Caine, Research Software Engineer, Google DeepMind
Jasmine Collins, Research Scientist, MosaicML
Simon Kornblith, Research Scientist, Anthropic
Nishal Shah, Post-Doctoral Researcher, Stanford University
Lane McIntosh, Senior Staff Research Scientist, Tesla Motors
Guangyu Robert Yang, Assistant Professor, MIT
Ryan Dahl, Startup company
Steve Mussmann, Assistant Professor, Georgia Tech
Augustus Odena, Co-Founder, Adept AI
Vincent Dumoulin, Research Scientist, Google DeepMind
Tianqi Chen, Assistant Professor, Carnegie-Mellon University
Alireza Makhzani, Research Scientist, Vector Institute
Jiang Wang, Software Engineer, Google Perception
Andrej Karpathy, Research Scientist, OpenAI