AI research finds a ‘compute divide’ concentrates power and accelerates inequality in the era of deep learning

AI researchers from Virginia Tech and Western University have concluded that an unequal distribution of compute power in academia is furthering inequality in the era of deep learning. They also point to the impact on academia of people leaving prestigious universities for high-paying industry jobs.

The concentration of compute power at elite universities crowds out mid- to low-tier research organizations, according to analysis that draws on 171,394 papers from nearly 60 prestigious computer science conferences. The team reviewed papers accepted for publication at large AI conferences such as ACL, ICML, and NeurIPS in categories like computer vision, data mining, machine learning, and NLP.

“Exploiting the sudden rise of deep learning due to an unanticipated usage of GPUs since 2012, we find that AI is increasingly being shaped by a few actors, and these actors are mostly affiliated with either large technology firms or elite universities,” their paper reads. “To truly ‘democratize’ AI, a concerted effort by policymakers, academic institutions, and firm-level actors is needed to tackle the compute divide.”

Nur Ahmed and Muntasir Wahed summarized their findings and recommendations in a paper titled “The De-democratization of AI: Deep Learning and the Compute Divide in Artificial Intelligence Research.” The paper was published recently on arXiv and presented in late October at Strategic Management Society, a business research conference.

The fact that wealthier universities and companies have an advantage in deep learning is not surprising. Large modern networks like AlphaGo Zero and GPT-3 can require millions of dollars in compute for training. And a December 2019 analysis labeled Google, Stanford University, MIT, Carnegie Mellon University, UC Berkeley, and Microsoft as the top six contributors at leading AI research conferences.

At the same time, smaller schools often lack the financial resources to consider deep learning applications. This limitation can define the kinds of AI that researchers in academia explore or accelerate brain drain to Big Tech companies with plenty of money to compete for top AI talent.

Confirming this opportunity gap, the paper found that universities ranked 301-500 by U.S. News and World Report have published on average six fewer papers at AI research conferences — or 25% fewer than a counterfactual estimator — since the rise of deep learning. Fortune 500 companies, Big Tech leaders, and elite universities saw dramatically different trends.

“To the best of our knowledge, this is the first study that finds evidence that an increased need for specialized equipment can result in ‘haves and have-nots’ in a scientific field,” the paper reads. “We contend that the rise of deep learning increases the importance of compute and data drastically, which, in turn, heightens the barriers of entry by increasing the costs of knowledge production.”

The coauthors say their work demonstrates what they call the “compute divide” along a series of social fault lines. Elite universities tend to have more wealthy students and are typically less diverse than other schools. Big Tech firms also lack diversity, particularly among engineers, people who design products, and AI research. Since AI has become a general purpose technology impacting aspects of business, public services, and private lives, this demographic imbalance has widespread consequences.

In analyzing this trend, Ahmed and Wahed break the history of artificial intelligence into two eras. They define the first as stretching from the 1960s to about 2012, when general purpose hardware was used to train AI. In the second era, deep learning and specialized hardware like GPUs have defined the industry, since the two were found to be effective together in the ImageNet image classification competition to advance computer vision.

When it comes to solutions, the coauthors say their findings present “concrete evidence” of the need for a national AI research cloud. In June, major universities, tech companies, and members of the U.S. Senate backed the idea of a national AI research cloud. Shared public datasets that can help train and test AI models will be particularly beneficial for resource-constrained organizations.

The paper asserts that the U.S. government should help universities by extending shared public datasets and other resources. Groups like the Defense Innovation Board and National Security Commission on AI (NSCAI) have advised the Pentagon and Congress to increase public-private partnerships, government funding, and outreach to developers working remotely as a way to attract talent from nontraditional backgrounds.

We could see movement on these fronts in the months ahead. President-elect Joe Biden’s platform has committed to investing $300 billion in research and development in areas like 5G and artificial intelligence.

Ahmed and Wahed’s findings are backed up by other recent papers evaluating the AI ecosystem and the technology’s role in bringing academia and industry closer together. For example, a paper called “Artificial Intelligence, Human Capital, and Innovation” found that AI created an unprecedented brain drain from academia between 2004 and 2018, leading to more than 200 people leaving for industry positions. Published in fall 2019 and updated last month, the paper finds that top universities, Ph.D. students, and startups in deep learning are among those that benefit most from current AI talent shortages. The analysis also found that Carnegie Mellon University, MIT, and Stanford University rank highest among colleges whose alumni go on to launch AI startups.

Ahmed and Wahed’s paper also follows a survey of more than 200 computer science department chairs on the impact of industry on academia. Commissioned by the Computing Research Association (CRA) and released a few months ago, the study identifies both positive and negative results of close cooperation between academia and industry. These changes include a shift of computing research faculty to industry jobs.

“This shift has the potential for negative impacts on the kinds of research done, the quality of the research, the culture of computer science departments, and the training of undergraduates and graduate students. Particular attention needs to be focused on issues related to department culture, potential conflict of interest, intellectual property, and ensuring that students continue to have sufficient faculty mentoring and contact to prepare them for their career,” a white paper about the survey reads.