This presentation delved into the importance of the semantic layer and detailed four real-world applications. Hilger and Nash explored how a robust semantic layer architecture optimizes user journeys across diverse organizational needs, including data consistency and usability, search and discovery, reporting and insights, and data modernization. Practical use cases explore a variety of industries such as biotechnology, financial services, and global retail. 

Check out this presentation to gain answers to the following questions:

• What is a semantic layer? 
• What problems does a semantic layer solve?
• What does a semantic layer enable? 
• What are the key components of a semantic layer?
• What are top enterprise use cases for a semantic layer?

The post Building a Semantic Layer of your Data Platform appeared first on Enterprise Knowledge.

To expand on the definition of a Semantic Layer, and moreover to eliminate the noise and confusion around them, I present to you what it is not. Debunking these misconceptions steers organizations from common detours or roadblocks when building semantic layers, such as: 

• Vendor lock – expecting a silver bullet solution to provide the connectivity and interoperability that a componentized semantic layer offers; 
• Lack of adoption – building out semantic solutions in a technology-driven silo with little interaction or ownership by the business; and 
• Limited enterprise reach – restricting the domain to single departments or use cases.

While you should be wary of the misconceptions about Semantic Layers, their value as tangible solutions that are driving enterprise AI, reporting, search, and personalization initiatives couldn’t be more real. Starting small with a Semantic LLM Accelerator or a 2-day Semantic Architecture Workshop can help to show value quickly while building a foundation for future scale. If you’ve got foundational components and are looking to see how other organizations have achieved the Semantic Layer, check out our case studies or contact us with questions. 

The post What Isn’t a Semantic Layer? appeared first on Enterprise Knowledge.

• Assess entity and relationship extraction readiness according to EK’s Extraction Maturity Spectrum and Relationship Extraction Maturity Spectrum.
• Utilize knowledge extraction from content to translate important insights into organizational data.
• Extract knowledge with three approaches:
  • RedEx Rule
  • Auto-Classification Rule
  • Custom ML Model
• Examine key factors such as how to leverage SMEs, iterate AI processes, define use cases, and invest in establishing robust AI models.

The post Scaling Knowledge Graph Architectures with AI appeared first on Enterprise Knowledge.

Introduction

In today’s data-driven world, effective data management is crucial for businesses to remain competitive. A modern approach to data management is the use of data fabric architecture. A data fabric is a data management solution that connects and manages data in a federated way, employing a logical data architecture that captures connections relevant to the business. Data fabrics help businesses make sense of their data by organizing it in a domain-centric way without physically moving data from source systems. What makes this possible is a shift in focus on metadata as opposed to data itself. At a high level, a semantic data fabric leverages a knowledge graph as an abstraction architecture layer to provide connectivity between diverse metadata. The knowledge graph enriches metadata by aggregating, connecting, and storing relationships between unstructured and structured data in a standardized, domain-centric format. Using a graph-based data structure helps the business embed their business data with context, drive information discovery and inference, and lay a foundation for scale.

Unlike a monolithic solution, a data fabric facilitates the alignment of different toolsets to enable domain-centric, integrated data as a service to multiple downstream applications. A data fabric architecture consists of five main components:

1. A data/metadata model
2. Entity extraction
3. Relationship extraction
4. Data pipeline orchestration
5. Persistent graph data storage. 

While there are a number of approaches to designing all of these components, there are best practices to ensure the quality and scalability of a data fabric. This blog post will enumerate the approaches for each architectural component, discuss how to achieve a data fabric implementation from a technical approach and tooling perspective that suits a wide variety of business needs, and ultimately detail how data fabrics support the development of artificial intelligence (AI).

Data/Metadata Model

Data models – specifically, ontologies and taxonomies – play a vital role in building a data fabric architecture. An ontology is a central aspect of a data fabric that defines concepts, attributes, and relationships in a domain that is encoded in a machine and human-readable graph format. Similarly, a taxonomy is essential for metadata management in a data fabric, storing extracted entities and defining controlled vocabularies for core business domains like products, business lines, services, and skills. By creating relationships between domains and data, businesses can help users find insights and discover content more easily. Therefore, to effectively manage taxonomies and ontologies, business owners need a Taxonomy/Ontology Management System (TOMS) that provides a user-friendly platform and interface. A good TOMS should:

• Help users build data models that follow common standards like RDF (Resource Description Framework), OWL (Web Ontology Language), and SKOS (Simple Knowledge Organization System); 
• Let users configure the main components of a data model such as classes, relationships, attributes, and labels that define the concepts, connections, and properties in the domain; 
• Add metadata about the data model itself through annotations, such as its name, description, version, creator, etc.;
• Support automated governance, supporting quality checks for errors;
• Allow for easy portability of the data model in different ways, serving multiple enterprise use cases; and
• Allow users to link to external data models that already exist and can be reused.

Organizations that do not place their data modeling and management at the forefront of their data fabric introduce the risk of scalability issues, limited user-friendly schema views, and hampered utilization of linked open data. Furthermore, the absence of formal metadata management poses a risk of inadequate alignment with business needs and hinders flexible information discovery within the data fabric. There are different ways of creating and using data models with a TOMS to avoid these risks. One way is to use code or scripts to generate and validate the data model based on the rules and requirements of the domain. Using subject matter expertise input helps to further validate the data model and confirm that it aligns with business needs. 

Entity Extraction 

One of the functions of building your data fabric is to perform entity extraction. This is the process of identifying and categorizing named entities in both structured and unstructured data, such as person names, locations, organizations, dates, etc. Entity extraction enriches the data with additional information and enables semantic analysis. Identifying Named Entity Recognition (NER) tools and performing text preprocessing (e.g., tokenization, stop words elimination, coreference resolution) is recommended before determining an entity extraction approach, of which there are several: rule-based, machine learning-based, or a hybrid of both. 

• Rule-based approaches rely on predefined rules that use syntactic and lexical cues to extract entities. They require domain expertise to develop and maintain, and may not adapt well to new or evolving data. 
• Machine learning-based approaches use deep learning models that can learn complex patterns in the data and extrapolate to unseen cases. However, they may require large amounts of labeled data and computational resources to train and deploy. 
• Hybrid approaches (Best Practice) combine rule-based and machine learning-based methods to leverage the strengths of both. Hybrid approaches are recommended for businesses that foresee expanding their data fabric solutions. 

Relationship Extraction 

Relationship extraction is the process of identifying and categorizing semantic relationships that occur between two entities in text, such as who works for whom, what business line sells which product, what is located in what place, and so on. Relationship extraction helps construct a knowledge graph that represents the connections and interactions among entities, which enables semantic analysis and reasoning. However, relationship extraction can be challenging due to the diversity and complexity of natural language.There are again multiple approaches, including rule-based, machine learning-based, or hybrid. 

• Rule-based approaches rely on predefined rules that use word-sequence patterns and dependency paths in sentences to extract relationships. They require domain expertise to develop and maintain, and they may not capture all the possible variations and nuances of natural language. 
• One machine learning approach is to use an n-ary classifier that assigns a probability score to each possible relationship between two entities and selects the highest one. This supports capturing the variations and nuances of natural language and handling complex and ambiguous cases. However, machine learning approaches may require large amounts of labeled data and computational resources to train and deploy. 
• Hybrid approaches (Best Practice) employ a combination of ontology-driven relationship extraction and machine learning approaches. Ontology-driven relationship extraction uses a predefined set of relationships that are relevant to the domain and the task. This helps avoid generating a sparse relationship matrix that results in a non-traversable knowledge graph.

Data Pipeline Orchestration

Data pipeline orchestration is the driving force of data fabric creation that brings all the components together. This is the process of integrating data sources with two or more applications or services to populate the knowledge graph initially and update it regularly thereafter. It involves coordinating and scheduling various tasks, such as data extraction, transformation, loading, validation, and analysis, and helps ensure data quality, consistency, and availability across the knowledge graph. Data pipeline orchestration can be performed using different approaches, such as a manual implementation, an open source orchestration engine, or using a vendor-specific orchestration engine / cloud service provider. 

• A manual approach involves executing each step of the workflow manually, which is time-consuming, error-prone, and costly. 
• An open source orchestration engine approach involves managing ETL pipelines as directed acyclic graphs (DAGs) that define the dependencies and order of execution of each task. This helps automate and monitor the workflow and handle failures and retries. Open source orchestration engines may require installation and configuration, and businesses need to take into account the required features and integrations before opting to use one.
• Third-party vendors or cloud service providers can leverage the existing infrastructure and services and provide scalability and reliability. However, vendor specific orchestration engines / cloud service providers may have limitations in terms of customization and portability.

Persistent Graph Data Storage 

One of the central ideas behind a data fabric is the ability to store metadata and core relationships centrally while connecting to source data in a federated way. This manage-in-place approach enables data discovery and integration without moving or copying data. Persistent graph data storage is the glue that brings all the components together, storing extracted entities and relationships according to the ontology, and persisting the connected data for use in any downstream applications. A graph database helps preserve the semantic relationships among the data and enable efficient querying and analysis. However, not all graph databases are created equal. When selecting a graph database, there are 4 key characteristics to consider. Graph databases should be: standards-based, ACID compliant, widely-used, editable, and explorable via a UI. 

• Standards-based involves making sure the graph database follows a common standard, such as RDF (Resource Description Framework), to ensure interoperability so that it is easier to transition from one tool to another. 
• ACID compliant means the graph database ensures Atomicity, Consistency, Isolation, and Durability of the data transactions, which protects the data from infrastructure failures. 
• Strong user and community support ensures that developers will have access to good documentation and feedback.
• Explorable via a UI supports verification by experts to ensure data quality and alignment with domain and use case needs. 

Some of the common approaches for graph databases are using RDF-based graph databases, labeled property graphs, or custom implementations. 

• RDF-based graph databases use RDF which is the standard model for representing and querying data. 
• Labeled property graph databases use nodes, edges, and properties as the basic elements for storing and querying data. 

Data Fabric Architecture for AI

A mature data fabric architecture built upon the preceding standards plays a pivotal role in supporting the development of artificial intelligence (AI) for businesses by providing a solid foundation for harnessing the power of data. The data fabric’s support for data exploration, data preparation, and seamless integration empowers businesses to harness the transformative and generative power of AI.

By leveraging an existing data fabric architecture, businesses can seamlessly integrate structured and unstructured data, capturing the relationships between them within a standardized, domain-centric format. With the help of the knowledge graph at the core of the data fabric, businesses can empower AI algorithms to discover patterns, make informed decisions, and generate valuable insights by traversing and navigating the graph. This capability allows AI models to uncover valuable insights that are not immediately apparent in isolated data silos. 

Furthermore, data fabrics facilitate the process of data preparation and feature engineering, which are crucial steps in AI development. The logical architecture of the data fabric allows for efficient data transformation, aggregation, and enrichment. By streamlining the data preparation pipeline, AI practitioners can focus more on modeling and algorithm development, accelerating the overall AI development process. AI models often need continuous refinement and adaptation, and data fabrics enable seamless integration of new data sources and updates to ensure that AI models have the most up-to-date information.

Conclusion

A data fabric is a modern approach to data management that is crucial for businesses to remain competitive in a data-driven world. However, a data fabric is not a monolithic solution and the supporting architecture and technical approach can vary based on the state of sources, supporting use cases, and existing tooling at an organization. It’s important to prove out the value of the solutions before investing in costly tool procurement. We recommend starting small and iterating, beginning with a targeted domain in mind and sample source systems to lay a foundation for an enterprise data fabric. Once a data fabric has been established, businesses can unlock the full potential of their data assets, enabling AI algorithms to make intelligent predictions, discover hidden insights, and drive valuable business outcomes. Looking for a kick-start to get your solutions off the ground? Contact us to get started.

The post What is A Data Fabric Architecture and What Are The Design Considerations? appeared first on Enterprise Knowledge.

Tesfaye and Nash drew on lessons learned from several knowledge graph development efforts to define how to diagnose a bad use case and outlined their impact on initiatives – including strained relationships with stakeholders, time spent reworking priorities, and team turnover. They also share guidance on how to navigate these scenarios and provide a checklist to assess a strong use case.

The post Knowledge Graph Use Cases are Priceless appeared first on Enterprise Knowledge.

The technical expertise required for implementing a knowledge graph can be solved in various ways – through hiring, upskilling, or in-house consulting. Organizations often focus on their core, domain-specific capabilities, and these may not include skill sets in knowledge engineering. EK can enhance their capabilities through collaborative delivery or in-house consulting, working closely with domain experts to build and maintain knowledge graph services. We also offer a Knowledge Graph University to build the knowledge graph design and development competencies on your team. Depending on the nature of your organization, your team structure may vary. However, we find that the following roles and skill groups provide the core capabilities for many organizations to create a solution and program that is user-centered, standards-based, and well-integrated into your enterprise architecture.

High-Level Teams for Knowledge Graph Development

What Are the Teams and Skill Sets You Need?

Product Success and Coordination

This team ensures that the knowledge graph aligns with business needs and technical requirements. These individuals provide leadership and decision-making to the delivery team while communicating outcomes to stakeholders.

It’s important to have a Business/Product Lead and Technical Lead that have experience with enterprise data solutions and can both guide internal development teams and be a point of contact for organizational leadership and other stakeholders. They are responsible for scoping and executing knowledge graph initiatives and should leverage product management best practices to be successful.

This team will make sure that your solution is providing tangible value by translating business challenges into actionable use cases and guiding the knowledge graph in a sustainable and relevant fashion.  

Knowledge Modeling and Data Preparation

This team designs, maintains, and grows the ontology and taxonomy models that are the foundation for the knowledge graph. These key team members ensure the alignment and readiness of integrating source data with the knowledge model. 

The knowledge modeling should be led by experienced ontologists, information architects, and taxonomists. These facilitators should interact with SMEs to design taxonomy and ontology models while applying semantic web standards (RDF, RDF*, SKOS, OWL) to ensure applicability and interoperability of the models and schemas. They collaborate closely with technical analysts who guide the data inventorying and define how data should be transformed and integrated according to the foundational schemas.

This team is central to successful knowledge graph development, making sure that use cases in data standardization, artificial intelligence, search, and more can be achieved through defined data concepts and relationships. This team is successful when knowledge models embed business concepts in a machine readable manner and when source data relevant to the use cases can be ingested, transformed, and stored according to the model. 

Data and Software Engineering

These roles are responsible for implementing the pipelines and algorithms required for populating the graph solution, as well as building the infrastructure and connectivity between the graph and downstream applications.

Semantic data engineers who have programming ability in data extraction/transformation and experience in data-centric applications are key to successful knowledge graph development. It’s important they have a skillset in querying and data manipulation languages, like SQL and Python, alongside experience working with graph-based querying languages and data types, like SPARQL, XML, RDF, JSON, and OWL. 

Having an experienced data engineering team will accelerate your knowledge graph development, ensuring that technical solutions are high quality and well integrated into the enterprise architecture. One of the primary advantages of graph-based solutions is their flexibility and extensibility, and this team makes that tangible for the organization. 

Establishing Your Team

There is not a one-size-fits-all model for establishing a knowledge graph development team. The team structure and needs will differ depending on relevant use cases, where the knowledge graph fits in the enterprise architecture, and the complexity of the domain area. Fulfilling roles can happen in multiple ways. It is possible that a single person may fulfill multiple roles or that each role may be fulfilled by one or more people. Regardless of the team structure, it’s critical to have the right sets of skills represented to successfully achieve your knowledge graph use cases.

At EK, we support many organizations in building out their capabilities to design and implement knowledge graph solutions, closely partnering with teams to close skill and experience gaps. If you’d like to work with us through in-house consulting, advising, training, and coaching, reach out to info@enterprise-knowledge.com. 

The post What Team Do You Need for Successful Knowledge Graph Development? appeared first on Enterprise Knowledge.

The post How to Quickly Prototype a Scalable Graph Architecture: A Framework for Rapid Knowledge Graph Implementation appeared first on Enterprise Knowledge.

Graph Database as Primary Storage: All data and ontology is stored within the graph database, ingesting all relevant source data and enabling inference and reasoning capabilities.

Graph Database as Relationship Management and Taxonomy Integration: Relationships between core concepts and content metadata (like taxonomy tags on documents) are stored within the graph, but actual content and descriptive metadata are stored within other systems and are connected to the graph via virtualization.

Graph Data Model (Ontology) as Relational Data Schema: The ontology model is an Enterprise Relational Diagram (ERD) that sets the “vision” for how to connect and leverage data stored in a relational database.

Organizations can see the value of capturing relationships in a machine-readable way, even when not all of the data relevant to the use case is captured in a graph database. The model that makes sense for your organization and your use case is dependent on factors including: 

• Restrictiveness of source data systems; 
• Volume and scale of data;
• Enterprise architecture maturity; 
• Inference and reasoning needs; and
• Integration needs with downstream systems.

At EK, we design graph-based architectures in a way that leverages your organization’s specifications and conventions, while introducing best practices and standards from the industry. Looking to get started? Contact us.

The post Where Does a Knowledge Graph Fit Within the Enterprise? appeared first on Enterprise Knowledge.

The post Presentation: Introduction to Knowledge Graphs appeared first on Enterprise Knowledge.

Causes: Why is Content Siloed?

Organizations introduce new content management systems as a quick fix to their problems when their existing systems can’t support new functionality. They think that it’s easier to add a new technology rather than confront the root issues. There are a variety of use cases that prompt companies to add a CMS:

• Expanding Content Types: Legacy systems do not have the capability to accommodate new types of content. Enter new platforms to manage data, videos, and social media.
• Collaboration: Companies are seeing the benefit of collaborating on documents in real-time rather than exchanging edits over email. Enter new collaboration tool.
• Permissions Control: Not everyone at a company should have access to particularly sensitive documents. Enter new legal document management system.
• New Ways to Present Content: People are increasingly consuming content on their phones and watches, not just through print and the web. Enter new a new CMS to publish to each channel. 

Problems: Implications of Content Silos

Disconnected, siloed systems make it much more difficult to get value from your content. There are a number of implications to content silos, including:

• Duplication: Companies want to be able to reuse their content, but they can’t think of how to do that if there are multiple systems. A common quick fix for this is creating copies of content in each CMS. If one of these pieces of content needs to be updated or removed, there’s no way to access every version across disconnected systems.
• Findability: Users have a hard time finding content when it is stored in multiple systems. Not only do they have to alter their search behavior per system, but they also spend extra time searching in each system separately.
• Lost Relationships: When content is managed in silos, organizations cannot capture relationships between content, like a blog sharing the same topic as a video or a legal document substantiating a claim in a deliverable. These relationships create important context that increases the usability of content.

Solutions: Breaking Down the Silos  

There are ways to address content silos other than trying to force all of your content into a single traditional CMS. Content migration to a monolithic CMS is typically a lengthy and costly process, and still only fulfills a limited set of requirements. Fortunately, there are alternatives:

Introduce a Headless CMS: A Headless CMS takes a content-first approach to give you flexibility in the expanding ways that your organization will use its content. Going headless centralizes and normalizes the way that content is modeled across the enterprise, but its feasibility depends on reliance on existing systems. There is a short-term tradeoff of cost, time, and disruption, but in the long-term it reduces the risk of future silos and accommodates a wide set of requirements. 

Develop a Metadata Hub: Without moving content from existing systems, organizations can aggregate metadata, or the descriptive information associated with content. This solution is beneficial if workflows and logic are highly embedded into existing systems and cannot be decoupled. The best way to implement this solution is through a graph database that contextualizes content by pulling the associated metadata from siloed systems. This is foundational for creating a streamlined search experience and even allows organizations to capture the relationships between content required for recommendation engines.

Implement Enterprise Search: Introducing an enterprise-wide search layer on content silos is an effective way to uncover and address issues with duplication and findability. This solution allows organizations to maintain their current systems, while aggregating the content inside of them. Enterprise Search is less disruptive than the above solutions because it does not require migration or database management, but it does not allow users to manage content across systems. However, creating a streamlined and intuitive search experience can be exactly what an organization needs to increase productivity and reveal inconsistencies. 

Design a Business Taxonomy: A Business Taxonomy will standardize the way that content is described and organized across siloed systems. This enables users to have a consistent experience in different CMS and is foundational for improving findability and productivity. This does not address challenges with duplicative content or lost relationships, but it significantly lowers the barriers that organizations face with content management. Implementing a Business Taxonomy is also one of the best ways to see improvements quickly. 

Breaking down content silos streamlines the ways that users access content to increase productivity and reduce the risk of having duplicate content across systems. Introducing a solution to address your organization’s content silos immediately adds value to your content and is critical to anticipate future uses. Not sure how to get started? We’re here to help! Reach out to info@enterprise-knowledge.com.

The post Content Silos: Causes, Problems, and Solutions appeared first on Enterprise Knowledge.