The Solution

To unlock data insights across trials in one accessible location, EK designed and developed a Power BI dashboard to visualize data from multiple trials in one centralized location. To begin developing dashboards, EK met with the client to confirm the business questions the dashboards would answer, ensuring the dashboards would visually display the patient and trial information needed to answer them. To remedy the varying data formats sent by vendors, EK mapped data values from trial reports to each other, normalizing and enriching the data with metadata and lineage. With structure and standardization added to the data, the dashboards could display robust data insights into patient status with filterable trial-specific information for the clinical imaging team.

EK also worked to transform the company’s data management environment—developing a medallion architecture structure to handle historical files and enforcing data cleaning and standardization on raw data inputs—to ensure dashboard insights were accurate and scalable to the inclusion of future trials. Implementing these data quality pre-processing steps and architecture considerations prepared the company for future applications and uses of reliable data, including the development of data products or the creation of a single view into the company-wide data landscape.

The EK Difference

To support the usage, maintenance, and future expansion of the data environment and data visualization tooling, EK developed knowledge transfer materials. These proprietary materials included setting up a semantic modeling foundation via a data dictionary to explain and define dashboard fields and features, a proposed future medallion architecture, and materials to socialize and expand the usage of visualization tools to additional sections of the company that could benefit from them.

Dashboard Knowledge Transfer Framework
To ensure the longevity of the dashboard, especially with the future inclusion of additional trial data, it was essential to develop materials for future dashboard users and developers. The knowledge transfer framework designed by EK outlined a repeatable process for dashboard development with enough detail and information that someone unfamiliar with the dashboards can understand the background, use cases, data inputs, visualization outputs, and the overall purpose of the dashboarding effort. Instructions for dashboard upkeep, including how to update and add data to the dashboard as business needs evolve, were also provided.

Semantic Model Foundations: Data Dictionary
To semantically enhance the dashboards, all dashboard fields and features were cataloged and defined by EK experts in semantics and data analysis. In addition to definitions, the dictionary included purpose statements and calculation rules for each dashboard concept (where applicable). This data dictionary was created to prepare the client to process all trial information moving forward and serve as a reference for the data transformation process.

Proposed Future Architecture
To optimize data storage in the future, EK proposed a medallion architecture strategy consisting of Bronze, Silver, and Gold layers to preserve historical data and pave the way for matured logging techniques. At the time EK engaged the client, there was no proper data storage. EK’s architecture strategy detailed storage preparation considerations for each layer, including workspace creation, file retention policies, and options for ingesting and storing data. EK leveraged technical expertise and a rich background in architecture strategies to provide expert advisory on the client’s future architecture.

Roadshow Materials
EK developed materials that summarized the mission and value of the clinical imaging dashboards. These materials included a high-level overview of the dashboard ecosystem so all audiences could comprehend the dashboard’s purpose and execution. With a KM-angled focus, the overall purpose of the materials was to gain organizational buy-in for the dashboard and build awareness of the clinical imaging team and the importance of the work they do. The roadshow materials also sought to promote dashboard adoption and future expansion of dashboarding into other areas of the company.

The Solution

EK partnered with the professional association to establish the foundational content strategy, semantic models, and solution architecture to enable their goals for content discovery and analytics. First, EK conducted a current state analysis and target state definition, as well as a semantic maturity assessment. This helped EK understand the factors that could be leveraged to help the association realize its goals. EK subsequently completed three parallel workstreams:

1. Content Assessment: EK audited a sample of assets on priority web properties to understand the condition of the association’s content and semantic practices. EK identified recommendations for how to enhance the performance, governance, and discoverability of content. Based on these recommendations, EK provided step-by-step procedures to support the association in completing a comprehensive audit to enhance their content quality and aid in future findability enhancement and content personalization efforts.
2. Taxonomy and Ontology Development: EK developed an enterprise taxonomy and ontology framework for the association—to provide a standardized vocabulary for use across the association’s systems, and increase the maturity of the association’s semantic models. The enterprise taxonomy included 12 facets to support 12 metadata fields, with a cumulative total of over 900 concepts. An ontology identified key relationships between the different taxonomy facets, establishing a foundation for identifying related content and supporting auto-tagging.
3. Semantic Layer Architecture: EK provided recommendations for maturing the association’s tooling and integrations in support of their goals. Specifically, EK developed a solution architecture to integrate taxonomy, ontology, and auto-tagging across content, asset, and learning management systems, in order to inform a variety of content analytics, discovery, recommendation, and assembly applications. This architecture was designed to form the basis of a semantic layer that the association could later use to connect and relate content enterprise-wide. The architecture included the addition of a taxonomy and ontology management system (TOMS) to centralize semantic model management and to introduce auto-tagging capabilities. Alongside years of experience in tool evaluation, EK leveraged their proprietary TOMS evaluation matrix to score candidate vendors and TOMS solutions, supporting the association in selecting a tool that was the best fit for their needs.
4. Auto-Tagging Proof of Concept: Building on these efforts, EK conducted an auto-tagging proof of concept (PoC), to support the association in applying the taxonomy to their content. The PoC automatically tagged all content assets in 2 priority CMSes with concepts from 2 prioritized topic taxonomy facets. The EK team prepared the processing pipeline for the auto-tagging effort, including pre-processing the content and conducting analysis of the tags to gauge quality and improvement over time.

To determine the exact level of improvement, EK worked with subject matter experts to establish a gold standard set of expected tags for a sample of content assets. The tags produced by the auto-tagger were compared to the expected tag set, to generate measures of recall, precision, and accuracy. EK used the analytics to inform adjustments to the taxonomy facets and to fine-tune and improve the auto-tagger’s performance over successive rounds.

To support the association in continuing to grow and leverage their semantic maturity, EK provided a detailed semantic maturity implementation roadmap. The roadmap identified five target outcomes for semantic enrichment, including: enhancing analytics to provide insights into content use and content gaps; and recommending content by using content tags to suggest related resources. For each outcome, EK detailed the requisite goals, business value, tasks, and dependencies—providing the association with the guidance they needed to realize each outcome and further advance their semantic maturity.

The EK Difference

EK was uniquely positioned to help the association improve their semantic maturity. As thought leaders in the semantic space, EK had the expertise and experience to assess the association’s semantic maturity, identify opportunities for growth, and define a vision and roadmap to help the association realize its business priorities. Further, EK has a deep understanding of the semantic technology landscape. This positioned EK to deliver tailored solutions that reflect the specific needs of the association, ensuring the solutions contribute to the association’s long-term technology roadmap.

EK leveraged a holistic approach to assessing and advancing the association’s semantic maturity. EK’s proprietary semantic maturity assessment accounts for the varied factors that influence an organization’s semantic maturity, including considerations for people, process, content, models, and technology. This positions the association to develop the capabilities required for semantic maturity across all contributing factors. Building off of the semantic maturity assessment, EK delivered end-to-end services that supported the entire semantic lifecycle, from strategy through design, implementation, and governance. This provided the association with the semantic infrastructure to realize near-term value; for instance, developing an enterprise taxonomy and applying it to their content assets using auto-tagging. By using proprietary, industry-leading approaches, EK was able to deliver these end-to-end services with tangible results within 4 months.

The Solution

Leveraging both industry and organizational taxonomies for clinical topics and conditions, EK architected a search solution that could take both the technical terms preferred by clinicians and the more conversational language used by consumers and match them to conditions and relevant medical information. 

To improve search while maintaining a user-friendly experience, EK worked to:

1. Enhance keyword search through metadata enrichment;
2. Enable natural language search using large language models (LLMs) and vector search techniques, and;
3. Introduce advanced search features post-initial search, allowing users to refine results with various facets.

The core components of EK’s semantic search advisory and implementation included:

1. Search Solution Vision: EK collaborated with client stakeholders to determine and implement business and technical requirements with associated search metrics. This would allow the client to effectively evaluate LLM-powered search performance and measure levels of improvement. This approach focused on making the experience faster for clinicians searching for information and for consumers seeking to connect with a doctor. This work supported the long-term goal of improving the overall experience for consumers using the search platform. The choice of LLM and associated embeddings played a key role: by selecting the right embeddings, EK could improve the association of search terms, enabling more accurate and efficient connections, which proved especially critical during crisis situations. 
2. Future State Roadmap: As part of the strategy portion of this engagement, EK worked with the client to create a roadmap for deploying the knowledge panel to the consumer-facing website in production. This roadmap involved deploying and hosting the content recommender, further expanding the clinical taxonomy, adding additional filters to the knowledge panel (such as insurance networks and location data), and search features such as autocomplete and type-ahead search. Setting future goals after implementation, EK suggested the client use machine learning methods to classify consumer queries based on language and predict their intent, as well as establish a way to personalize the user experience based on collected behavioral data/characteristics.
3. Keyword and Natural Language Search Enhancement: EK developed a gold standard template for client experts in the medical domain to provide the ideal expected search results for particular clinician queries. This gold standard served as the foundation for validating the accuracy of the search solution in pointing clinicians to the right topics. Additionally, EK used semantic clustering and synonym analysis in order to identify further search terms to add as synonyms into the client’s enterprise taxonomy. Enriching the taxonomy with more clinician-specific language used when searching for concepts with natural language improved the retrieval of more relevant search results.
4. Semantic Search Architecture Design and LLM Integration: EK designed and implemented a semantic search architecture to support the solution’s search features, EK connecting the client’s existing taxonomy and ontology management system (TOMS), the client’s search engine, and a new LLM. Leveraging the taxonomy stored in the TOMS and using the LLM to match search terms and taxonomy concepts based on similarity enriched the accuracy and contextualization of search results. EK also wrote custom scripts to evaluate the LLM’s understanding of medical terminology and generate evaluation metrics, allowing for performance monitoring and continuous improvement to keep the client’s search solution at the forefront of LLM technology. Finally, EK created a bespoke, reusable benchmark for LLM scores, evaluating how well a certain model matched natural language queries to clinical search terms and allowing the client to select the highest-performing model for consumer use.
5. Semantic Knowledge Panel: To demonstrate the value this technology would bring to consumers, EK developed a clickable, action-oriented knowledge panel that showcased the envisioned future-state experience. Designed to support consumer health journeys, the knowledge panel guides users through a seamless journey – from conversational search (e.g. “I think I broke my ankle”), to surfacing relevant contextual information (such as web content related to terms and definitions drawn from the taxonomy), to connecting users to recommended clinicians and their scheduling pages based on their ability to treat the condition being searched (e.g. An orthopedist for a broken ankle). EK’s prototype leveraged a taxonomy of tagged keywords and provider expertise, with a scoring algorithm that assessed how many, and how well, those tags matched the user’s query. This scoring informed a sorted display of provider results, enabling users to take direct action (e.g. scheduling an appointment with an orthopedist) without leaving the search experience.

The EK Difference

EK’s expertise in semantic layer, solution architecture, artificial intelligence, and enterprise search came together to deliver a bespoke and unified solution that returned more accurate, context-aware information for clinicians and consumers. By collaborating with key medical experts to enrich the client’s enterprise taxonomy, EK’s semantic experts were able to share unique insights and knowledge on LLMs, combined with their experience with applying taxonomy and semantic similarity in natural language search use cases, to place the client in the best position to enable accurate search. EK also was able to upskill the client’s technical team on semantic capabilities and the architecture of the knowledge panel through knowledge transfers and paired programming, so that they could continue to maintain and enhance the solution in the future.

Additionally, EK’s solution architects, possessing deep knowledge of enterprise search and artificial intelligence technologies, were uniquely positioned to provide recommendations on the most advantageous method to seamlessly integrate the client’s TOMS and existing search engine with an LLM specifically developed for information retrieval. While a standard-purpose LLM could perform these tasks to some extent, EK helped design a purpose-built semantic search solution leveraging a specialized LLM that better identified and disambiguated user terms and phrases. 

Finally, EK’s search experts were able to define and monitor key search metrics with the client’s team, enabling them to closely monitor improvement over time, identifying trends and suggesting improvements to match. These search improvements resulted in a solution the client could be confident in and trust to be accurate.