For many organizations, however, these efforts have failed to yield the anticipated results proportional to their investments. According to Gartner’s recently published “Hype Cycle for Artificial Intelligence, 2024”, AI has entered the “trough of disillusionment.” 

We’re witnessing this firsthand as organizations are hiring EK to address AI projects that have stalled due to content and data challenges. Many are still grappling with how to ensure quality and diversity of their AI products  – with the biggest hurdle being the lack of institutional and domain knowledge that AI requires to deliver meaningful results tailored to a specific organization. 

The reality is that algorithms trained in one company or public data sets may not work well on organization and domain-specific problems, especially in domains where industry preferences are relevant. Thus, organizational knowledge is a prerequisite for success, not just for generative AI, but for all applications of enterprise AI and data science solutions. This is where experience and best practices in knowledge and data management are lending the AI space effective and proven approaches to how domain and institutional knowledge can be shared effectively. Below, I have picked the top three ways we have been working to embed domain and organizational knowledge and empower enterprise AI solutions.

1. Semantic Layer

Without context, raw data or text is often messy, outdated, redundant, and unstructured, making it difficult for AI algorithms to extract meaningful use. The key step to addressing this AI problem involves the ability to connect all types of organizational knowledge assets, i.e., using shared language, involving experts, related data, content, videos, best practices, and operational insights from across the organization. In other words, to fully benefit from an organization’s knowledge and information, both structured and unstructured information as well as expertise knowledge must be represented and understood by machines.

A Semantic Layer provides AI with a programmatic framework to make organizational context, content, and domain knowledge machine readable. Techniques such as data labeling, taxonomy development, business glossaries, ontology and knowledge graph creation make up the semantic layer to facilitate this process.  

How a Semantic Layer Provides Context for Structured Data

• Contextual Metadata & Business Glossaries: A semantic layer provides a framework to add contextual metadata and glossaries to structured data through definitions, usage examples, data lineage, category tags, etc. This enrichment aids analytics and AI teams in understanding organizational nomenclature, signaling the importance of one dataset compared to another, and improves their ability to align on using the right data for their analytics, metrics, and AI models. 
• Hierarchical Structures (Taxonomies): Implementing hierarchical structures within the semantic layer allows for categorization and sub-categorization of data. This structure helps AI models identify broader/narrower relationships and dependencies with the data, making it easier for AI algorithms to derive and understand organizational frameworks. For instance, product or service categories allow AI models to analyze and understand relationships between data points related in those domains and  recommend similar or related data or services. This allows data teams to understand and incorporate implicit business concepts for AI as well as discover new information they would have otherwise not looked for or knew existed.
• Encoding Business Logic (Ontologies): Using standardized data modeling schemas, such as ontologies, a semantic layer allows for programmatically applying business rules and logic that govern data relationships, entities, and constraints. By incorporating this logic, AI models gain a deeper understanding of the operational context in which the data exists, leading to more relevant and actionable insights. For example, our clients in the pharmaceutical industry use ontologies to explicitly define their domains and connect information about drugs, diseases, and biological pathways. This enables AI models to identify potential drug targets, predict drug interactions or adverse effects, and accelerate their drug discovery process.
• Data Aggregation & Semantic Mapping (Knowledge Graphs): A knowledge graph aggregates data from multiple structured sources (like databases, data warehouses, CRM systems, etc.) into a unified view without the need to physically move or migrate data. In so doing, it provides a comprehensive view of organizational knowledge assets for AI models, enabling AI to draw knowledge and insights from broader sources. Furthermore, knowledge graphs allow organizations to create semantic mappings between different data schemas  (e.g., mapping “customer ID” in one system to “client ID” in another)  and helps AI understand the meaning and relationships of data fields ensuing models interpret data consistently while normalizing data quality across various sources.

How a Semantic Layer Extracts Knowledge from Unstructured Content 

• Natural Language Processing (NLP): Natural language based models help analyze unstructured text to identify entities, concepts, and relationships from a large corpus of unstructured content – extracting key phrases or sentiments from documents and text and enabling AI to understand context and meaning. For instance, at a global policy research institute, we are leveraging LLMs for NLP by monitoring industry news and social media and extracting information about news trends to build taxonomy structures and inform a recommendation engine that is providing targeted policy recommendations. 
• Named Entity Recognition (NER) and Classification: Organizations are augmenting knowledge model development by automatically identifying and classifying entities such as people, places, and things within unstructured content to create enterprise taxonomies and knowledge graphs. For example, by extracting and classifying entities like patient names, diagnoses, medications, and procedures from unstructured medical records, healthcare providers are connecting and applying the knowledge associated with these entities for clinical research and improving patient care outcomes. The structured representation of entities within text allows AI to leverage information for more precise responses and analysis.
• Taxonomy, Ontology, and Graph Construction: By defining relationships between concepts, domain-specific ontologies that organize unstructured content into a structured framework enable AI solutions to understand and infer knowledge from the content more effectively. A semantic layer is able to build knowledge graphs from unstructured data, linking entities (extracted using NLP/NERs) and their attributes. This map-like representation of information helps AI systems navigate complex relationships and generate insights by making the knowledge explicit about how data is interconnected.

2. Domain Knowledge Capture and Expertise 

AI systems need to learn from explicit content and data as well as the insights and intuition of human experts. This is where knowledge management (KM) and AI are becoming increasingly intertwined. 

On the one hand, the traditional challenges of capturing, sharing, and transferring knowledge are becoming more pronounced, as many organizations struggle with a retiring workforce, high turnover rates, slow upskilling processes, and the limitations of AI systems that often fall short of expectations. Knowledge Capture and Transfer are becoming even more integral for organizations to enable knowledge flow among experts, and the ability to capture, disseminate, and use its institutional knowledge.

On the other hand, the expanding landscape of AI is opening up new possibilities for KM, especially in automating knowledge capture and transfer approaches. For instance, in many of our projects, experienced domain experts and AI engineers are collaborating to define rules and heuristics that reflect organizational wisdom – by creating decision trees, developing rule-based solutions, or using NLP and semantics to extract and infer expert knowledge from documents and conversations. Specific approaches that enable this process include: 

• Mining Expert Libraries: Mining repositories of case studies and use cases, including extraction of knowledge from images and videos, that illustrate domain expertise in action by building a structured repository of facts and relationships, enable AI to learn from real-world applications and scenarios. 
• Expert Annotations: Engaging subject matter experts to annotate datasets with contextual information and business logic or operational concepts (using metadata, taxonomies, ontologies) enhances understanding for AI models by making tacit knowledge explicit. 
• Automated Knowledge Capture: Advanced applications of the expert annotation approach also include using AI-powered knowledge discovery tools that automatically analyze and extract knowledge from text or voice using NLP techniques, augmenting the development of knowledge graphs. This approach allows for the discovery of knowledge that is tacit in relationships between content in order to systematically provide it for AI training.
• Embedded Feedback Loops: To ensure alignment with domain knowledge, many organizations and KM/AI solutions should incorporate a feedback loop. This involves providing domain experts with an embedded process and tools to review AI outputs and provide corrections or enhancements. That feedback can be used to refine models based on real-world applications and organizational changes.

3. Retrieval Augmented Generation (RAG) 

Retrieval Augmented Generation (RAG) architecture is a mechanism to provide LLMs with relevant organizational information and knowledge. However, LLMs trained on outdated content have resulted in mistakes in decision-making that have real consequences to an organization’s bottom line. 

Several RAG architectures are used for domain-specific knowledge transfer within the organizations we work with. The table below provides a comparison of these approaches and ideal scenarios for effective applications or use.

Many organizations are seeing better results from employing hybrid approaches to cater to specific use cases and solutions. For example, one of the top tax and financial services firm we are working with is leveraging “Semantic Routing” techniques in order to respond with the most accurate and specific information for their search solutions by evaluating users’ query and determining the best route to take from the above three approaches to fetch, combine, and deliver a response to user queries.  

Conclusion

Successfully injecting organizational knowledge into AI is not just a technical challenge but also a strategic organizational decision that requires a shift in mindset and collaboration across knowledge, content, data, and AI teams and solutions. 

• Organizational experts know how to interpret data and how to handle missing values and outliers. They are crucial for identifying relevant data sources, interpreting information with contextual nuances, and helping in addressing data quality and bias issues. 
• KM and content teams need AI literacy for effective collaboration – to provide expertise in knowledge retrieval and ensure content readiness and quality for AI solutions. 
• Data and AI teams need to have a deep understanding of the organization’s domain knowledge, business objectives, and access to reliable data regardless of its type and location. 

The semantic layer and the knowledge extraction/application approaches discussed above facilitate this integration and ensure that AI can operate not just as a tool, but as an intelligent organizational partner that understands the unique nuances of an organization and enables knowledge intelligence.

Is your AI initiative stalled? Does it lack the knowledge necessary to make it trustworthy and valuable? Contact us to learn how to put your knowledge at the center of your AI.

The post How to Inject Organizational Knowledge in AI: 3 Proven Strategies to Achieve Knowledge Intelligence appeared first on Enterprise Knowledge.

As a technology SME in the KM space, I am excited about the changes happening in the semantic software industry. Just two years ago, in my book, I provided a complete analysis of the leading providers of taxonomy and ontology management systems, as well as graph providers, auto-tagging systems, and more. While the software products I evaluated are still around, most of them have new owners. The amount of change that has happened in just two years is incredible.

We recognized the importance of these products early on at EK. Enterprise-Scale Knowledge Management cannot work without technology solutions that capture, align, and make information discoverable. We partnered with organizations like The Semantic Web Company, Synaptica, OntoText, TopQuadrant, and Neo4j to help some of the world’s most well-known companies solve some of the most complex knowledge management problems.

Now the rest of the industry is realizing the importance of semantics and the semantic layer. Well-funded software companies are acquiring many of the independent software vendors in this space so that they can offer a more comprehensive semantic layer solution to their customers. MarkLogic, an enterprise NoSQL database, bought Semaphore (a taxonomy/ontology management platform) and both were later acquired by Progress Software. Squirro (an Artificial Intelligence (AI) enabled search platform) bought Synaptica (taxonomy/ontology management software) and has also purchased meetsynthia.ai (prompt engineering solution for AI). Fluree (a graph data management platform) bought Mondeca (a taxonomy/ontology management software). In this same timeframe, Samsung Electronics acquired Oxford Semantic (an RDF graph database). In each case, these vendors are looking to offer their customers a single integrated solution for the semantic layer.

Organizations that purchased these products suddenly have risks to their software investment. The new product owner could choose to take the software in a different direction that does not support your use case. You could have new points of contact that do not understand your organization’s needs. Senior leadership in your organization may want to minimize the investment in a tool that was recently purchased out of fear. While less likely in this case, the vendor could also choose to wind down support for the product. On a more positive note, these larger, more well-funded companies might add compelling new features or they might integrate it with their existing tools to provide a comprehensive solution to what you would have had to integrate yourself.

There are two primary drivers behind all of this industry change. The first is the explosion of generative AI. Companies are trying to implement generative AI projects, and they are failing. According to Gartner, 85% of AI projects fail. Data quality and a proper understanding of AI tools and capabilities are two of the most common causes of these failures. The semantic tools we are talking about address these issues. Taxonomy and ontology management systems along with graph databases organize and curate information so that data quality problems are minimized. In addition, many of these tools are now offering frameworks for generative AI solutions. Think of them as a configurable engine on which generative AI solutions can be built. Every software vendor is looking for a way to provide AI solutions to their customers. Our favorite semantic software tools are being bought up so that the vendors can provide a single integrated AI solution to their clients.

The second major driver is the semantic layer. As data continues to grow exponentially, the need to map data in a way that the business understands has become even more critical. One of our retail clients had 12 different point of sale systems. Answering a simple question like “What is an average sales transaction?” was incredibly complicated. A knowledge graph can map each of these data elements to a transaction and a transaction amount in a machine-readable way. Business leaders can ask for this information in a way that makes sense to them and the knowledge graph automatically generates the answers from the data where it sits. As more organizations understand the power of a semantic layer, the need for semantic tools continues to grow. Data vendors see this opportunity and are purchasing semantic tools that they can integrate into their current solution stack.

Given all of the momentum in this area, we will continue to see more acquisitions of semantic software solutions. Our team at EK is watching this industry closely to guide our customers so that they will have the best vendors with the best solutions during this changing time. If you are thinking about purchasing a semantic software product we have a proprietary matrix of semantic solutions that was developed over 10 years and has over 200 requirements for semantic capabilities. If you are concerned because your product was purchased, we know all of the players in the industry and can guide you to the best possible answer for moving forward. Contact us so that we can give you the right guidance both now and in the long term.

The post Consolidation in the Semantic Software Industry appeared first on Enterprise Knowledge.

The post Semantic Use Cases appeared first on Enterprise Knowledge.

• Manage content more effectively;
• Maximize findability and discoverability of information;
• Increase the reuse of “hidden” and unknown information;
• Elevate SEO for public sites; and
• Create relationships between disparate and distributed information items.

EK’s Semantic Web Starter Kit is an easy and efficient way for organizations to get started with their first Semantic solution. The starter kit will help your organization to:

• Get introduced to semantic technologies;
• Take up to 28 hours of web-based training on ontologies and the semantic web;
• Develop an initial ontology that can grow over time;
• Validate the ontology using real content and information; and
• Implement a pilot version of PoolParty to manage your ontology.

The Semantic Web Starter Kit begins with a customized workshop where we introduce semantic technologies to your business and technology teams, align with the organization’s content relationships management goals, and develop measurable success criteria for the pilot.

We utilize the industry leading taxonomy, thesaurus, and ontology management tool PoolParty, developed by our partners Semantic Web Company. PoolParty allows us to help our clients model their domain knowledge, utilize available Linked Data resources, and integrate disparate internal and external data sources with both structured and unstructured data to provide seamless integration across the enterprise.

As part of the Semantic Web Starter Kit, we work with your knowledge management and IT specialists to define an initial ontology or augment your existing ones. With the starter ontology completed, we validate it using your existing content from various sources and iteratively adjust it to best model your domain knowledge and provide high-quality auto-tagging suggestions, concept identification, and knowledge linking.   

We implement PoolParty in a reusable and scalable proof of concept instance to help you achieve next generation knowledge integration across your organization.

Benefits & Outcomes

	Streamlined introduction to Semantic Web concepts and best practices.
	Quick, working, and reusable proof of concept installation of PoolParty to manage ontologies for your organization.
	Completed starter ontology validated against your existing content.
	Auto-tagging suggestions based on your content.
	Clear, practical, and tailored next-steps plan for enriching and expanding the starter ontology to the rest of your organization.
	Training and transfer of knowledge to your knowledge managers and SMEs, including web-based certification courses in Semantic Web concepts and PoolParty administration.

Semantic Web Starter Kit Process

The Semantic Web Starter Kit is a 10 to 20-day engagement, depending on the defined scope and complexity of the proof of concept. We break down this project in the following tasks:

	Kick-off Workshop Vision and objectives for the Starter Kit project Presentation of ontology primer High-level review of existing taxonomies, thesauri, and/or ontologies
	Prototype specification Analysis of existing taxonomies/ontologies and content Interviews and focus groups with Subject Matter Experts as necessary Technical and functional description of targeted functionality
	Test environment   Installation and configuration of PoolParty at customer site or cloud server with a 90-day free license
	Data analysis and processing Iterative analysis and development of initial ontology/taxonomy Integration of predetermined internal and external data sources Validation and tuning of initial ontology/taxonomy
	Your prototype Functional prototype/proof of concept of a PoolParty instance utilizing a predetermined subset of your taxonomy/ontology and data sources Starter taxonomy/ontology based on your domain
	Roadmap Guidance on how to continue enriching your ontology Best practices for PoolParty administration and integration
	Access to PoolParty Academy online training and certification for 2 project team members for 3 months covering: Taxonomy, ontology, linked data, and semantic web concepts PoolParty configuration and administration to allow you to manage your taxonomies and/or ontologies and integrate additional data sources into your knowledge graph

Contact us at info@enterprise-knowledge.com to get your Semantic Web Starter Kit.

The post Semantic Web Starter Kit appeared first on Enterprise Knowledge.