Large language model expands natural language understanding, moves beyond English

One of the primary use cases for artificial intelligence (AI) is to help organizations process text data.

It’s an area where natural language processing and natural language understanding (NLP/NLU) is a foundational technology. One such foundational large language model (LLM) technology comes from OpenAI rival, Cohere, which launched its commercial platform in 2021. 

The Toronto-based startup’s founders benefitted from machine learning (ML)-research efforts at the University of Toronto, as well as the Google Brain research effort in Toronto led by Geoffrey Hinton, which explored deep learning neural network approaches. Cohere’s goal is to go beyond research to bring the benefits of LLM to enterprise users.

“We had this vision of creating large language models and then giving access to businesses so that they could build cool stuff with this tech that they couldn’t build in-house,” Nick Frosst, cofounder at Cohere, told VentureBeat.

To date, Cohere’s models have been based on the English language, but that is now changing. Today, the company announced the release of a multilingual text-understanding LLM that can understand and work with more than 100 different languages.

It’s a multilingual world and now AI lives in it, too

Cohere is not the first LLM to venture beyond the confines of the English language to support multilingual capabilities.

BLOOM (which is an acronym for BigScience Large Open-science Open-access Multilingual Language Model) was officially launched in July. The BLOOM effort is backed by a series of organizations including HuggingFace and CNRS, the French National Research Agency. 

The Cohere multilingual approach is a bit different than BLOOM and is initially focused on understanding languages to help support different natural language use cases. Cohere’s model does not yet actually generate multilingual text like BLOOM, but that is a capability that Frosst said will be coming in the future.

Nils Reimers, director of machine learning at Cohere, explained to VentureBeat that among the core use cases for Cohere’s multilingual approach is enabling semantic search across languages. The model is also useful for enabling content moderation across languages and aggregating customer feedback.

“Cohere first focused on just English models, but we thought maybe it’s a bit boring just to focus on English models because a large majority of the population on the Earth is non-English speaking,” Reimers said. 

How Cohere’s LLM uses natural language understanding to become multilingual

Training an LLM to be multilingual is not a trivial task.

Reimers explained that first, Cohere built out a large corpus of question-and-answer pairs that included hundreds of millions of data points in English and non-English languages. The training looked to help determine when the same content was being presented in different languages.

For example, if there is a line of text in English, matching that same line in Arabic or any other language, then aligning that as a mathematical vector such that the ML system understands the two pieces of text are similar. 

As such, for a content moderation use case, a line of hateful text, for example, can be identified, regardless of the language it is written in. The natural language training also enables semantic search such that similar pieces of news written in different languages can be identified.

“Creating models like this takes a fair bit of compute, and it takes compute not only in processing all of the data, but also in training the model,” Frosst said.

Looking forward, the goal for Cohere is to continue to build out its capabilities to better understand increasingly larger volumes of text in any language.

“Generally, what’s next for Cohere at large is continuing to make amazing language models and make them accessible and useful to people,” Frosst said.

VentureBeat’s mission is to be a digital town square for technical decision-makers to gain knowledge about transformative enterprise technology and transact. Discover our Briefings.