Transforming Knowledge Management with ChatGPT and Azure OpenAI

Introduction

ChatGPT, OpenAI’s text-generating AI chatbot, has recently generated a lot of buzz. It is based on GPT (Generative Pre-trained Transformer), a large language model (LLM) developed by OpenAI. This deep learning model is pre-trained on extensive amounts of text data and can be fine-tuned for various natural language processing tasks, such as text classification, sentiment analysis, and question answering. The GPT model utilizes transformer architecture to process long text sequences efficiently.

Over 90% of Fortune 500 companies are leveraging ChatGPT for diverse applications. At Microland, we embarked on our own ChatGPT journey by developing an innovative Knowledge Management Solution for Enterprise Data. This use case highlights the potential of ChatGPT in addressing real-world challenges.

Problem statement

In today’s digital age, a vast amount of data is available in various forms and types. Effective knowledge management is crucial for organizations to properly handle their knowledge and expertise. For many, this involves managing enterprise data scattered across multiple repositories, with SharePoint sites commonly storing relevant data in numerous folders and subfolders.  

Consider the scenario of needing to find information in a knowledge base document: you must first locate the correct folder, select the document from many, and search within that document for the specific data. This process can be quite laborious and time-consuming.

Solution approach

To tackle these issues, we pondered,

"How can we build something like ChatGPT within our organization to interact with our own internal data?”

We embarked on a journey to explore various solutions and approaches to develop an innovative solution leveraging Generative AI, a ChatGPT-like experience with enterprise data using the Retrieval Augmented Generation (RAG) pattern. This solution promises to revolutionize how enterprises manage their knowledge assets by leveraging the power of Azure OpenAI Service to access a GPT model (GPT-35-turbo) and Azure AI Search for data indexing and retrieval.

This whitepaper explores our Generative AI-based Knowledge Management solution's architecture, implementation, and real-time use cases, offering insights into its transformative potential for businesses. 

Architecture

We have deployed a solution in which ChatGPT interacts with a dataset comprising knowledge artifacts such as SOPs, QRGs, RCAs, configuration documents, RFPs, SOWs, and more, using Azure Cognitive Search. The dataset will be stored in an Azure Blob Storage container and indexed using Azure AI Search. ChatGPT will query this index to retrieve relevant information.

• We start by asking a question through a web application interface
• The question is then forwarded to the Azure AI Search as a search query
• This query is sent to Azure AI Search, which fetches relevant data from our datasets
• These search results are brought back into the GPT 3.5 Turbo Completions model, which then generates an answer based on the initial question and search results
• We then receive a response to our question in the WebApp.

ChatGPT, Azure OpenAI, and Azure AI Search are powerful technologies that, when integrated, can form a sophisticated natural language processing system capable of interacting with and searching through enterprise data. Here is an overview of a few important components:

• OpenAI GPT: GPT (Generative Pre-trained Transformer) is a type of LLM (Large Language Model) developed by OpenAI. It is a deep learning model that is pre-trained on large amounts of text data and can be fine-tuned for various natural language processing tasks, such as text classification, sentiment analysis, and question answering. GPT is based on the transformer architecture, allowing it to process long text sequences efficiently.
• Azure OpenAI: Azure OpenAI is a service provided by Microsoft Azure that allows you to integrate OpenAI models, including GPT-based models, into your applications and services. This component provides access to the GPT model, specifically the GPT-3.5 architecture (in this case, the "GPT-35-turbo" variant). In this architecture, Azure OpenAI Service acts as a gateway to access and utilize the GPT model for generating responses and text based on user inputs.
• Azure AI Search: Azure AI Search is a fully managed search-as-a-service in Microsoft Azure. This component is used for data indexing and retrieval. Azure AI Search allows you to create a search index over your own data, making it easy to search and retrieve relevant information. In the context of creating ChatGPT-like experiences, Azure AI Search is utilized to index and organize the data with which the GPT model will interact. This indexed data serves as the knowledge base from which the GPT model can retrieve information to generate responses.
• RAG (Retrieval Augmented Generation): Retrieval augmented generation is a technique used in natural language processing that combines two steps: retrieval and generation. In this architecture, the retrieval aspect is handled by Azure AI Search, which retrieves relevant information from the indexed data based on user queries. The retrieved information is then used as context or input for the GPT model, which generates responses or text based on this context. By combining retrieval and generation techniques, this pattern aims to produce more accurate and contextually relevant responses compared to purely generative models. This method saves time and effort by eliminating the need to retrain the underlying LLM model to obtain responses from the organization's own data.

Overall, the architecture leverages Azure OpenAI Service for accessing the GPT model, Azure AI Search for data indexing and retrieval, and combines retrieval and generation techniques to create ChatGPT-like experiences over enterprise data.

Is the solution secure?

Our solution uses Microsoft Azure services like Azure Open AI and Azure AI Search, and it's built to keep the data safe and secure. User data is guarded against unauthorized access, ensuring privacy and security in line with Microsoft's data protection standards.

Here are some key security features:

• Access control: Access to Azure ChatGPT and enterprise data is controlled through MS Entra ID, which provides RBAC and MFA.  
• Data Privacy: Enterprise data resides within the Azure Data Centers, ensuring that it is segregated from other customers' data. We get a private endpoint to the LLM model, also hosted in Microsoft’s datacenters. Microsoft also guarantees that no customer data will be used to train the underlying LLM model.

As with any other Azure Cloud service, the security works on the principle of shared responsibility.

For example, Microsoft ensures the following:

• Protecting the Azure infrastructure
• Ensuring that the Azure OpenAI, Azure AI Search, and other services are secure by default
• Providing identity and access management capabilities

The customer is responsible for:

• Setting appropriate access controls and permissions for their use of Azure OpenAI & Azure AI Search
• Protecting our Azure credentials and managing access to Azure subscription(s)
• Document level access control over the data of different customers that is ingested to a single Azure AI Search for data indexing and retrieval
• Ensuring that the security measures around our applications are applicable

This shared responsibility ensures that Microsoft provides a basic level of security while we can customize additional security controls based on our data's specific needs and sensitivity.

Use-cases

Now, let’s look at a few use cases for this solution.

Knowledge Retrieval + Augmentation
The AI chatbot can answer questions based on our data while enhancing responses with its training data. We can create guardrails on the System Prompt to allow this or limit the responses only to our own data.

Exec Summary & Trend Analysis

Members of the Senior Leadership Team can access insights on the go. They may receive a summary of ticket data, updates about a customer or outage, a snapshot of the monthly operations review, and more.

Content Creation

The solution can assist in generating high-quality, personalized content for IT engineers, tech documents, SOPs, marketing campaigns, social media posts, and sales materials. It can quickly produce engaging descriptions, product pitches, and responses to customer inquiries, saving time and resources for the team.

Technical Evaluation

Azure Open AI and ChatGPT offer immediate support to IT engineers by responding to queries, offering suggestions, and helping in decision-making processes. This functionality boosts efficiency, enabling engineers to swiftly address technical evaluations and root causes of issues.

 

Process Specific Questions

After observing the various applications, it becomes apparent that this solution produces reliable answers. Here are some industry-specific use cases: 

• Customer support: An organization can use this solution to organize its knowledge base and customer support tickets so customers can quickly find answers to their questions. With Azure OpenAI ChatGPT, the search results can offer personalized and natural responses to customer questions.
• Product analysis and suggestions: Organizations can enhance their product recommendation systems by leveraging Generative AI technologies like ChatGPT alongside cloud-native solutions such as Azure OpenAI and Azure AI Search. By using Azure AI Search to sort product lists and customer details, organizations can create personalized product suggestions for each customer. ChatGPT can then describe these products in simple terms and answer any questions from customers, enhancing their overall experience.
• Marketing and sales: The Knowledge Management solution can significantly assist the Marketing & Sales team internally in several ways
• Insights Generation: By leveraging Generative AI and Azure AI Search, the solution can analyze vast amounts of internal data to uncover valuable insights about customer preferences, market trends, and competitor strategies. This enables the Marketing & Sales team to make more informed decisions and tailor their approach accordingly.
• Customer Engagement: With Azure AI Search, the solution can efficiently organize and retrieve relevant customer data, including past interactions, preferences, and feedback. This allows the Marketing & Sales team to personalize their communications and offers, improving customer engagement and satisfaction.
• Competitive Analysis: By utilizing Azure AI Search to gather and analyze market data, the solution can help the Marketing & Sales team stay updated on industry trends, competitor activities, and emerging opportunities. This facilitates strategic planning and enables the team to stay ahead in the market.

The strategy of retrieval-augmented generation starts with basic methods and then progresses to more sophisticated techniques as the situation demands. It relies on one's ingenuity to derive the necessary responses. Numerous strategies exist for crafting prompts, phrasing questions to efficiently retrieve information from the knowledge repository, and managing the dialogue between OpenAI and the Knowledge base. 

Conclusion

The combination of ChatGPT, Azure OpenAI, and AI Search represents a significant advancement in enterprise Knowledge Management. By combining the power of generative AI alongside cloud-native technologies, organizations can unlock valuable insights from their data repositories while maintaining cost efficiency and governance standards. This integration offers a transformative approach to knowledge management, empowering businesses to make better-informed decisions and drive innovation. Embracing ChatGPT, Azure OpenAI, and AI Search opens the door to more intelligent and efficient knowledge management practices within enterprises.

Our solution helps organizations manage and extract value from their enterprise data by providing powerful search features and personalized, natural language interactions with the data.

About the Author:

Narendra Mahendrakar - Manager - Cloud

Narendra has over 15 years of experience in IT Infrastructure and Cloud space. He's well versed in Consulting/Delivery, Hybrid Multi-Cloud Platforms, Cloud Native Transformation, Platform Automation, IAM & IaC.