Competitive intelligence as a service is made possible through advances in natural language processing and database technology.

Recent developments in natural language processing such as ChatGPT have engendered much excitement. Clearly, ChatGPT and large language model capabilities have a role to play in competitive intelligence as a service.

Conversational agents represent one component of the overall system, an important, highly visible component certainly, but not the only component.

• External data collection. Natural language processing capabilities are key to processing the large quantities of unstructured data or text that continuously flow into a knowledge base. We gather information from the public domain in an automated fashion. We use Web crawlers and scrapers to assist in this process. We draw on text parsers to prepare data for input into the knowledge base. While storing text in the database, we ensure that documents are indexed in a meaningful way, so they can be quickly located in response to user questions.

• Internal data collection. Large language models like ChatGPT draw on publicly available data. But much of the information about a company is internal information, proprietary information. Competitive intelligence as a service, if given access to internal information, can build on that information in providing answers to management.

• Named entity recognition. We utilize tools of named entity recognition to provide additional indexing and organization of documents. We identify names of people, places, and things, paying special attention to the essentials of competitive intelligence, including names of competitors and products, prices, and market shares.

• Sentiment analysis. Customer reviews and logged service requests enter the system as text data, along with data from social media and formal publications. It is important to know what people are saying about products and brands. We employ natural language processing to conduct sentiment analysis.

• Topic modeling. What are the major ideas and themes mentioned in a collection of documents? How shall we organize documents by topic? These questions are addressed by topic modeling, another tool of natural language processing.

• Conversation. This is where ChatGPT and ChatGPT-like tools shine. Decision makers ask questions as they would when talking to an expert or researcher. Decision makers get answers in words they understand. Conversational agents or chatbots provide a capable, human-like interface to the knowledge base. Think of chat as the face or voice of the system, with the knowledge base being the heart.

• Embeddings-facilitated search. A byproduct of large language models is technology for generating numerical vectors or embeddings that characterize documents as well as user queries. These can be added to a knowledge base for competitive intelligence. The benefit is more efficient and more thorough searching across the knowledge base. This, too, is a component of competitive intelligence as a service.

Back to In the News