(T) Jeff Dean announced the vision for Google’s next generation model Pathways in a blog post in the fall of 2021. Another blog article and a paper was published about the Pathways Language Model (PaLM), in the Spring of 2022.

Pathways is Google vision for creating large scale, multi-task, multi-modal models supporting both large dense models and sparse architectures

PaLM was the first implementation of the Pathway vision. PaML is multimodal e.g. the model accepts multiple types of inputs (text, images, and sounds). It has a sparse architecture. And, it implements a chain-of-thoughts prompting.

Note that PaML is as well auto-regressive like GPT-3 and based on a dense decoder-only transformer architecture. PaML has been trained with 540-billion parameters.

Google designed for PaML, a new machine learning platform called the Pathways System, described in a paper in the Spring of 2022.

The Pathways System schedules distributed heterogeneous computations over thousands TPUs, in the case of PaLM 6,144 TPUv4 over two TPU pods. The key innovation of the Pathways System is that its control plane can parallelize the computations of the data planes, even if there are dependencies between the distributed data planes that might require data transfers:

One of the Google engineers working on the Pathway system, Aakanksha Chowdhery, gave recently a high level talk on it at the Stanford MLSys Seminars series:

Note: The picture above is a sunset in Half Moon Bay.

Copyright © 2005-2023 by Serge-Paul Carrasco. All rights reserved.
Contact Us: asvinsider at gmail dot com