One key step in preparing content is the application of metadata and organizational context to pieces of content through tagging. There are several tagging approaches an organization can take to enrich pre-existing content with metadata and organizational context, including manual tagging, automated tagging capabilities from a taxonomy and ontology management system (TOMS), using apps and features directly from a content management solution, and various hybrid approaches. While many of these approaches, in particular acquiring a TOMS, are recommended as a long-term auto-tagging solution, EK has recommended and implemented Large Language Model (LLM)-based auto-tagging capabilities across several recent engagements. Due to LLM-based tagging’s lower initial investment compared to a TOMS and its greater efficiency than manual tagging, these auto-tagging solutions have been able to provide immediate value and jumpstart the process of re-tagging existing content. This blog will dive deeper into how LLM tagging works, the value of semantics, technical considerations, and next steps for implementing an LLM-based tagging solution.

Overview of LLM-Based Auto-Tagging Process

Similar to existing auto-tagging approaches, the LLM suggests a tag by parsing through a piece of content, processing and identifying key phrases, terms, or structure that gives the document context. Through prompt engineering, the LLM is then asked to compare the similarity of key semantic components (e.g., named entities, key phrases) with various term lists, returning a set of terms that could be used to categorize the piece of content. These responses can be adjusted in the tagging workflow to only return terms meeting a specific similarity score. These tagging results are then exported to a data store and applied to the content source. Many factors, including the particular LLM used, the knowledge an LLM is working with, and the source location of content, can greatly impact the tagging effectiveness and accuracy. In addition, adjusting parameters, taxonomies/term lists, and/or prompts to improve precision and recall can ensure tagging results align with an organization’s needs. The final step is the auto-tagging itself and the application of the tags in the source system. This could look like a script or workflow that applies the stored tags to pieces of content.

Figure 1: High-level steps for LLM content enrichment

EK has put these steps into practice, for example, when engaging with a trade association on a content modernization project to migrate and auto-tag content into a new content management system (CMS). The organization had been struggling with content findability, standardization, and governance, in particular, the language used to describe the diverse areas of work the trade association covers. As part of this engagement, EK first worked with the organization’s subject matter experts (SMEs) to develop new enterprise-wide taxonomies and controlled vocabularies integrated across multiple platforms to be utilized by both external and internal end-users. To operationalize and apply these common vocabularies, EK developed an LLM-based auto-tagging workflow utilizing the four high-level steps above to auto-tag metadata fields and identify content types. This content modernization effort set up the organization for document workflows, search solutions, and generative AI projects, all of which are able to leverage the added metadata on documents. 

Value of Semantics with LLM-Based Auto-Tagging

Semantic models such as taxonomies, metadata models, ontologies, and content types can all be valuable inputs to guide an LLM on how to effectively categorize a piece of content. When considering how an LLM is trained for auto-tagging content, a greater emphasis needs to be put on organization-specific context. If using a taxonomy as a training input, organizational context can be added through weighting specific terms, increasing the number of synonyms/alternative labels, and providing organization-specific definitions. For example, by providing organizational context through a taxonomy or business glossary that the term “Green Account” refers to accounts that have met a specific environmental standard, the LLM would not accidentally tag content related to the color green or an account that is financially successful.

Another benefit of an LLM-based approach is the ability to evolve both the semantic model and LLM as tagging results are received. As sets of tags are generated for an initial set of content, the taxonomies and content models being used to train the LLM can be refined to better fit the specific organizational context. This could look like adding additional alternative labels, adjusting the definition of terms, or adjusting the taxonomy hierarchy. Similarly, additional tools and techniques, such as weighting and prompt engineering, can tune the results provided by the LLM and evolve the results generated to achieve a higher recall (rate the LLM is including the correct term) and precision (rate the LLM is selecting only the correct term) when recommending terms. One example of this is  adding weighting from 0 to 10 for all taxonomy terms and assigning a higher score for terms the organization prefers to use. The workflow developed alongside the LLM can use this context to include or exclude a particular term.

Implementation Considerations for LLM-Based Auto-Tagging 

Several factors, such as the timeframe, volume of information, necessary accuracy, types of content management systems, and desired capabilities, inform the complexity and resources needed for LLM-based content enrichment. The following considerations expand upon the factors an organization must consider for effective LLM content enrichment. 

Tagging Accuracy

The accuracy of tags from an LLM directly impacts end-users and systems (e.g., search instances or dashboards) that are utilizing the tags. Safeguards need to be implemented to ensure end-users can trust the accuracy of the tagged content they are using. These help ensure that a user is not mistakenly accessing or using a particular document, or that they are frustrated by the results they get. To mitigate both of these concerns, a high recall and precision score with the LLM tagging improves the overall accuracy and lowers the chance for miscategorization. This can be done by investing further into human test-tagging and input from SMEs to create a gold-standard set of tagged content as training data for the LLM. The gold-standard set can then be used to adjust how the LLM weights or prioritizes terms, based on the organizational context in the gold-standard set. These practices will help to avoid hallucinations (factually incorrect or misleading content) that could appear in applications utilizing the auto-tagged set of content.

Content Repositories

One factor that greatly adds technical complexity is accessing the various types of content repositories that an LLM solution, or any auto-tagging solution, needs to read from. The best content management practice for auto-tagging is to read content in its source location, limiting the risk of duplication and the effort needed to download and then read content. When developing a custom solution, each content repository often needs a distinctive approach to read and apply tags. A content or document repository like SharePoint, for example, has a robust API for reading content and seamlessly applying tags, while a less widely adopted platform may not have the same level of support. It is important to account for the unique needs of each system in order to limit the disruption end-users may experience when embarking on a tagging effort.

Knowledge Assets

When considering the scalability of the auto-tagging effort, it is also important to evaluate the breadth of knowledge asset types being analyzed. While the ability of LLMs to process several types of knowledge assets has been growing, each step of additional complexity, particularly evaluating multiple types, can result in additional resources and time needed to read and tag documents. A PDF document with 2-3 pages of content will take far fewer tokens and resources for an LLM to read its content than a long visual or audio asset. Going from a tagging workflow of structured knowledge assets to tagging unstructured content will increase the overall time, resources, and custom development needed to run a tagging workflow. 

Data Security & Entitlements

When utilizing an LLM, it is recommended that an organization invest in a private or an in-house LLM to complete analysis, rather than leveraging a publicly available model. In particular, an LLM does not need to be ‘on-premises’, as several providers have options for LLMs in your company’s own environment. This ensures a higher level of document security and additional features for customization. Particularly when tackling use cases with higher levels of personal information and access controls, a robust mapping of content and an understanding of what needs to be tagged is imperative. As an example, if a publicly facing LLM was reading confidential documents on how to develop a company-specific product, this information could then be leveraged in other public queries and has a higher likelihood of being accessed outside of the organization. In an enterprise data ecosystem, running an LLM-based auto-tagging solution can raise red flags around data access, controls, and compliance. These challenges can be addressed through a Unified Entitlements System (UES) that creates a centralized policy management system for both end users and LLM solutions being deployed.

Next Steps:

One major consideration with an LLM tagging solution is maintenance and governance over time. For some organizations, after completing an initial enrichment of content by the LLM, a combination of manual tagging and forms within each CMS helps them maintain tagging standards over time. However, a more mature organization that is dealing with several content repositories and systems may want to either operationalize the content enrichment solution for continued use or invest in a TOMS. With either approach, completing an initial LLM enrichment of content is a key method to prove the value of semantics and metadata to decision-makers in an organization. 
Many technical solutions and initiatives that excite both technical and business stakeholders can be actualized by an LLM content enrichment effort. By having content that is tagged and adhering to semantic standards, solutions like knowledge graphs, knowledge portals, and semantic search engines, or even an enterprise-wide LLM Solution, are upgraded even further to show organizational value.

If your organization is interested in upgrading your content and developing new KM solutions, contact us!

The post How to Leverage LLMs for Auto-tagging & Content Enrichment appeared first on Enterprise Knowledge.

Organizational Implementation Considerations:

Before starting on the technical journey from “RAGs to Riches”, there are several considerations for an organization before, during, and after creating a production solution. By taking into account each of these considerations, a production LLM solution has a much higher chance of success.

Before: Aligning Business Outcomes

Prior to building out a production LLM solution, a team will have developed a PoC LLM solution that is able to answer a limited set of use cases. Before the start of production development, it is imperative that business outcomes and the priorities of key stakeholders are aligned with project goals. This often looks like mapping business outcomes – such as enhanced customer interactions, operational efficiency, or reduced compliance risk to quantifiable outcomes such as shorter response times and findability of information. It is important to ensure these business goals translate from development to production and adoption by customers. Besides meeting technical functionality, setting up clear customer and organizational goals will help to ensure the production LLM solution continues to have organizational support throughout its entire lifecycle.

During: Training Talent and Proving Solutions

Building out a production LLM solution will require a team with specialized skills in natural language processing (NLP), prompt engineering, semantic integration, and embedding strategies. In addition, EK recommends investing resources into content strategists and SMEs who understand the state of their organization’s data and/or content. These roles in particular are critical to help prepare content for AI solutions, ensuring the LLM solution has comprehensive and semantically meaningful content. Organizations that EK has worked with have successfully launched and maintained production LLM solutions by proactively investing in these skills for organizational staff. This helps organizations build resilience in the overall solution, driving success in LLM solution development.

After: Infrastructure Planning and Roadmapping

To maintain a production LLM solution after it has been deployed to end-users, organizations must account for the infrastructure investments and operational costs needed, as well as necessary content and data maintenance. Some of these resources might include enterprise licensing, additional software infrastructure, and ongoing support costs. While many of these additional costs can be mitigated by effectively aligning business outcomes and training organizational talent, there still needs to be a roadmap and investment into the future infrastructure (both systematically and content-wise) of the LLM production solution.

Technical Criteria for Evaluating LLM PoCs:

In parallel with the organizational implementation considerations, and from EK’s depth of experience in developing LLM MVPs, designing enterprise AI architecture, and implementing more advanced LLM solutions such as Semantic RAG, EK has developed 7 key dimensions that can be used to evaluate the effectiveness of an LLM PoC:

Figure 1: Dimensions for Evaluating an LLM Solution

1. Depth of Interaction: refers to how deeply and dynamically users can engage with the LLM solution. At a lower level, interaction might simply involve asking questions and receiving direct answers, while at the highest level, intelligent agents act on behalf of the user autonomously to leverage multiple tools and execute tasks.

2. Freshness of Information: describes how frequently the content and data behind the semantic search solution are updated and how quickly users receive these updates. While lower freshness implies data updated infrequently, at higher freshness levels, data is updated frequently or even continuously which helps to ensure users are always interacting with the most current, accurate, and updated information available.

3. Level of Explanation: refers to how transparently the LLM solution communicates the rationale behind its responses. At a lower level of explanation, users simply are receiving answers without clear reasoning. In contrast, a high level of explanation would include evidence, citations, audit trails, and a clear path on how information was retrieved. 

4. Personalization, Access & Entitlements Requirements: describes how specifically content and data are tailored and made accessible based on user identity, roles, behavior, or needs. At lower levels, content is available to all users without personalization or adaptations. At higher levels, content personalization is integrated with user profiles, entitlements, and explicit access controls, ensuring users only see highly relevant, permissioned content. 

5. Accuracy of Information: refers to how reliably and correctly the LLM solution can answer user queries. At lower levels, users receive reasonable answers that may have minor ambiguities or occasional inaccuracies. At the highest accuracy level, each response is traced back to original source materials and are cross-validated with authoritative sources. 

6. Enterprise Agentic Support: describes how the LLM solution interacts with the broader enterprise AI ecosystem, and coordinates with other AI agents. At the lowest level, the solution acts independently without any coordination with external AI agents. At the highest level, the solution seamlessly integrates as a consumer and provider within an ecosystem of other intelligent agents.

7. Enterprise Embedding Strategy: refers to how the LLM solution converts information into vector representations (embeddings) to support retrieval. At a lower level embeddings are simple vector representations with minimal or no structured metadata. At the highest levels, embeddings include robust metadata and are integrated with enterprise context through semantic interpretation and ontology-based linkages. 

For an organization, each of the technical criteria will be weighed differently based on the unique use cases and requirements of the LLM solution. For example, an organization that is working on a content generation use case could have a greater emphasis on Level of Explanation and Freshness of Information while an organization that is working on an information retrieval use case may care more about Personalization, Access, & Entitlements Requirements. This is an integral part of the evaluation process, with an organization coming to agreement on the level of proficiency needed within each factor. Leveraging this standard, EK has worked with organizations across various industries and diverse LLM use cases to optimize their solutions.

Additionally, EK recommends that an organization undergoing an LLM PoC evaluation also conduct an in-depth analysis of content relevant to their selected use case(s). This enables them to gain a more comprehensive understanding of its quality – including factors like completeness, relevancy, and currency – and can help unearth gaps in what the LLM may be able to answer. All of this informs the testing phase by guiding the creation of each test, as well as the expected outcomes, and can be generally categorized across three main areas of remediation:

• Content Quality – The content regarding a certain topic doesn’t explicitly exist and is not standardized – this may necessitate creating dummy data to enable certain types of tests.
• Content Structure – The way certain content is structured varies – we can likely posit that one particular structure will give more accurate results than another. This may necessitate creating headings to indicate clear hierarchy on pages, and templates to consistently structure content. 
• Content Metadata – Contextual information that may be useful to users is missing from content. This may necessitate establishing a taxonomy to tag with a controlled vocabulary, or an ontology to establish relationships between concepts. 

Technical Evaluation of LLM PoCs In Practice:

Putting the organizational implementation and technical considerations into practice, EK recently completed an engagement with a leading semiconductor manufacturer, employing the standard process for evaluating their PoC LLM search solution. The organization had developed a PoC search solution that was being utilized for answering questions against a series of user-selected PDFs relating to the company’s technical onboarding documentation. EK worked with the organization  to align on key functional requirements via a capability assessment for a production LLM solution based on the 7 dimensions EK has identified. Additionally, EK completed a simultaneous analysis of in-scope content for the use case. The results of this content evaluation informed which content components should be prioritized and candidates for the testing plan.

After aligning on priority requirements, in this case, accuracy and freshness of information, EK developed and conducted a testing plan for parts of the PoC LLM. To operationalize the testing plan, EK created a four-phase RAG Evaluation & Optimization Workflow to turn the testing plan into actionable insights.This workflow helped produce a present-state snapshot of the LLM solution, a target-state benchmark, and a bridging roadmap that prioritizes retriever tuning, prompt adjustments, and content enrichment. Based on the workflow results, stakeholders at the organization were able to easily interpret how improved semantics, content quality, structure, and metadata would improve the results of their LLM search solution.

In the following blogs of the “RAGs to Riches” series, EK will be explaining the process for developing a capability assessment and testing plan for LLM based PoCs. These blogs will expand further on how each of the technical criteria can be measured as well as how to develop long-term strategy for production solutions.

Conclusion

Moving an LLM solution from proof-of-concept to enterprise production is no small feat. It requires careful attention to organizational alignment, strong business cases, technical planning, compliance readiness, content optimization, and a commitment to ongoing talent development. Addressing these dimensions systematically will ensure that your organization will be well positioned to turn AI innovation into a durable competitive advantage.

If you are interested in having EK evaluate your LLM-based solution, and help build out an enterprise-grade implementation contact us here. 

The post LLM Solutions PoC to Production: From RAGs to Riches (Part 1) appeared first on Enterprise Knowledge.