What’s behind the huge amount of attention going to LLMs like ChatGPT, Llama and Gemini? Rich text representations that go way beyond “straight” encodings like ASCII and Unicode. Rich text representations capture the reality of knowing a word by the company it keeps. These representations are kind of the compressed context of a word or phrase. They are the key to LLM's abilities to produce meaningful language in response to prompts. Indeed, rich text encodings like word embeddings pull the representations directly from activations within a neural network trained on a recovery or prediction task. But, word embeddings (rich text representations) can have a wide variety of applications that may be much better suited to your task than wide-open LLM interfaces like ChatGPT.
Semantic Clustering
Why do we care about rich text representations and what do we mean by ‘semantics’ ? Equivalent meaning between words and phrases is something humans are good at, but computers have not been great at until recently. Human judgment about whether phrases or documents “mean” the “same”, are similar or dis-similar accounts for a huge amount of human effort in the workplace. Semantic clustering and related tools can significantly amplify the productivity of the humans doing work involving organization, categorization and consolidation. Highly paid financial analysts spend much of their time scouring earnings reports and press releases, ultimately in order to make a judgment or reconcile contradictory information. Analysis and judgment benefits from the experience and high-order pattern matching of the analyst. The reading and initial grouping and organization of areas of focus, however, (for example, new product prospects, supplier bottleneck risks, competitive landscape) could be greatly sped up and made much more accurate through semantic clustering tools.
How much do end-users need to know about the workings of these tools and methods? We would say, a rough understanding of how these encodings are derived (by learning context relationships from a dataset and a prediction or recovery deep-learning task) can be extremely helpful. Of course, you can make use of these tools without knowing anything about how they are derived, but even a conceptual understanding may allow for more creative use and application of the tools as well as their limitations.
“Today’s semantic understanding tools allow for post-analysis of existing data without making significant or even any changes to business processes.”
Business Intelligence of a few years ago understood the value of detailed analytics on customer feedback and interactions, but often required complex changes to business processes to ‘code’ and track interactions in a way that could be counted and trended over time. Very few organizations were successful in transforming their business processes to successfully enable this. Data drift and data pollution are extremely common, further lowering the value of these types of enterprise data. Today’s semantic understanding tools allow for (and may even get more value from) post-analysis of existing data without making significant or even any changes to business processes. Textual or recorded (audio) natural language data contains the real judgment and input of humans all along the value chain, from the ideation, production and eventual consumption of consumer goods & services. In a way, this data is much more “pure” than data which Business Intelligence deployers created through coded menus and drop-downs trying to capture anticipated categories of interactions. Not only does this constrain and distort what gets reported, the work of accurate coding on top of existing processes is often dehumanizing and boring! This exacerbates the problem of drift and pollution through “encouraged” laziness and shortcuts. (Imagine the Verizon customer support rep who has to help you fix your bill, and simultaneously choose the cause of your trouble from some long-winded drop-down menu so BI can drive marketing or product changes months from now…)
Semantic clustering can be used for a wide variety of text (or speech-to-text) data cleanup, categorization, classification and similarity tasks. Indeed, many language AI projects are best begun by first analyzing clusters of the text you ultimately want to work with (whether that’s building search, conversational or business intelligence systems). Narrow tasks like survey response analysis, customer feedback analysis, social media trending and customer support triage can all greatly benefit from an initial (and iterative) round of semantic clustering. And, there are great tools that can take advantage of the latest embedding models and clustering approaches.
If you’re wondering how semantic clustering can help your business or would like to signup for a free 45 mins consultation session, reach out to us at info@theblue.earth