Why KPIs?

KPIs are evaluation metrics that are specific, measurable, and goal-oriented. Good KPIs are easy to communicate and translate directly into business value. They are used to support trends over time and can be combined with other measures to tell a story.  They are focused on the  activity at hand, and thus the specific metrics are tied to the use-case, business drivers, and the overall goals of any given initiative. 

The broad objectives frequently associated with semantic tools, such as increased efficiency or more cohesive understanding across the organization can be particularly difficult to quantify and track.  KPIs serve as a bridge, creating qualitative metrics to measure against and serving as course correctors to keep projects focused and on track.  Without KPIs, lofty objectives can hang like rain clouds, hovering with ambiguity over otherwise productive efforts. At EK we have seen the impact of murky objectives or a lack of KPIs manifest through time errantly devoted to the wrong efforts, misunderstandings of the intended outcomes, or even an impediment to generating buy-in from leadership. KPIs quantify ROI and provide clear metrics to measure impact before, during, and after a semantic integration. These ways to see progress throughout a project lifecycle serve to unify and create momentum for businesses. 

Creating KPIs that assess the impact of a semantic integration is both a challenge and a skill. Since KPIs are inherently tied to business objectives and specific use-cases it is hard to provide a standard set of KPIs for the semantic layer. The matrix below provides some example objectives and potential KPIs in 3 key areas: Scale, Content, and Time. More details on each focus area is elaborated on following the graphic.  

Scale

Measurements of scale can be used to understand the sheer size of the semantic layer and to compare multiple iterations to demonstrate growth, such as the number of entities in a knowledge graph, or terms in a taxonomy. Conversely, the reduction in size of the content or asset library can also be an important measure, for instance when performing deduplication. Reducing the scale of asset libraries can also have a marked impact on additional business motivations such as a reduction in the cost and/or carbon footprint required to support oversized data servers. Additionally, a smaller content library results in a lower cognitive load for end users and less redundant, or incorrect information. In one project, an investment and insurance company was struggling with inefficient search and users were frustrated because the content that was returned to them was frequently out of date or inaccessible. Content auditing, a standardized taxonomy, and auto-tagging workflows were employed and EK identified that about 45% of the content in their existing library was obsolete or outdated. By cleaning up this content prior to migrating to a new system the scale of the library was reduced. This reduction in size paired with a newly standardized enterprise taxonomy resulted in more accurate content descriptions and a more relevant library for users. This occurrence is far from uncommon and emphasizes the critical nature of content cleanup and maintaining a “right-size” content library. 

Content Management and Fit

Common measures of accuracy for taxonomies include an understanding of precision (is the returned information correct?) as well as recall (is all the expected information included?). Assessing content fit is notoriously difficult, generally reliant on Subject Matter Experts and qualitative data, and thus is very tricky to quantify. Focused KPIs can help to quantify this data, for example, if the percentage of correctly auto tagged terms is 75% at the first evaluation mark and 90% at the next point of measure, then the system is 15% more accurate. Measuring accuracy is a critical step in the iterative development of taxonomy and other semantic solutions. Without an understanding of what is working, it is impossible to understand what changes need to be made. Being able to find and access accurate and up-to-date information is a common pain point that we see among businesses who come to EK. After reducing the size of the content library through content clean up, the business in our earlier example was able to apply a newly formed enterprise taxonomy to their content with an auto-tagging workflow. Analysis of this effort found that the one time auto-tagging process had a 86-99% success rate depending on the content type. This effort greatly increased the accuracy of the metadata describing the content resulting in a better semantic search experience and increased findability across the enterprise. 

Time

One of the major changes that a well tuned semantic layer can bring about is a measurable amount of time saved. From auto-tagging and other automations to reducing duplication and incorrect information. Time savings increase user satisfaction as well as create a tangible benefit for the company in reduction of “human-hours” spent on any given task. This reduction in hours is an easily measurable reference point, since it can generally translate directly into the labor cost of any given employee. If a semantic solution reduces errors in the data, then measuring the number of errors corrected within the new system can be used to understand the time-savings now that errors do not need to be manually corrected. Similarly, the number of documents auto-tagged can be used to tell a story of time not spent manually tagging content. In the example cited in the two previous sections, after cleaning up content and applying the new taxonomy through auto-tagging EK estimated that the necessary human effort related to the content migration was reduced by nearly 80%. This is truly a remarkable improvement that had both immediate and long term benefits. 

Conclusion

Throughout our engagements EK has learned critical lessons about the importance of creating and using KPIs. Anecdotal evidence can be a powerful tool, but even more powerful are anecdotes backed by quantified measures. Learning how to set and track KPIs at the outset of a project provides the opportunity to understand and quantify the impact of semantic integrations. Across a variety of projects, KPIs have been core data points that help keep projects focused, and keep efforts directed towards the most important areas of a project.  When faced with murky or unclear objectives and a lack of KPIs our teams have seen projects sit with the wheels spinning, have needed to re-do work, and have seen focus shift onto unnecessary components of a project, to the detriment of other components. On the flip side, well crafted KPIs have been utilized to prove business value, spot inefficiencies, and improve semantic solutions over and over again. KPIs can be utilized over time to tell a story that will help you and your business better understand and track goals met through the use of your taxonomy, ontology, or knowledge graph. The team at EK is well versed in developing both the semantic solutions you need as well as the KPIs necessary to evaluate, understand, and promote these solutions. To learn more about how we can help, contact EK today. 

The post Measuring the Value of your Semantic Layer: KPIs for Taxonomies, Ontologies, and Knowledge Graphs appeared first on Enterprise Knowledge.

4. Build to Scale

It’s not uncommon for organizations to implement a new tool or process that benefits them in the short term, only to reinvest in an entirely different solution years down the line. Large organizations are often subject to rapid growth, even after scaling from a start-up or scale-up into a large corporation. Understanding the size and complexity of a given organization and its anticipated growth is essential in implementing any type of technical or non-technical KM change. 

In non-technical KM engagements, any new processes, models, or overall organizational changes must be in accordance with an organization’s strategic vision for the coming years. For example, if a content governance team is designed and established within an organization, this model should not be stagnant but instead include predetermined measures for adding new members in the event of growth or expansion, such as onboarding and training materials, or publishing and management processes for new types of content. Designing and implementing change management processes before a change happens can help to align employees prior to any actual transformation, streamlining the transition and providing employees with mitigation strategies for potential risks. 

In highly technical KM engagements, any new system or tool that is being implemented within an organization must be able to scale accordingly with any growth the organization might face, particularly when the system is intended for the entire enterprise. This begins with the selection of the tool itself, as many tools have technical limitations and/or significantly increase costs when the number of users rises, so it is imperative to conduct thorough vendor reviews and market research before selecting. Beyond the research and selection of a tool, it is also key to design and implement any technical solution with consensus alignment on what functionalities and integrations are essential in both the short and long term. To avoid having to further invest in another system or make adjustments to an existing one due to a tool or strategy becoming obsolete, it’s crucial to understand the direction of your organization and what capabilities are needed in the immediate future, as well as what capabilities may be needed in the coming years. 

Last year, I worked with an international development organization that was seeking to implement a new content management system. The organization was using an outdated application that was set to expire within the next couple of years, so selecting a new system and migrating the existing content quickly was a high priority. EK conducted a thorough analysis of content management systems, taking into account the organization’s priorities, use cases, required functionalities, budget, number of users, and future plans to formulate a cost-benefit analysis on potential options. This enabled the organization to thoroughly understand what options exist, what capabilities they have, and what the best fit for them could potentially be, before connecting with the identified top vendors. EK also provided the organization with critical information on their current technical environment along with recommendations on how to integrate and optimize it with the new system. This streamlined the negotiation process with the chosen vendor, and enabled the organization to efficiently identify and implement the ideal solution for their use case and budget without overspending or omitting essential functionalities.

5. Engage the Enterprise

Some initiatives can encompass an entire enterprise, whereas others may be focused on a single department or business unit. Regardless of the scale of a project, it is important to engage with a diverse group of stakeholders from various departments, teams, tenures, and roles in order to garner a comprehensive understanding of what issues may or may not be present within an organization. Without doing so, there is a risk of overlooking or disregarding important perspectives that could help formulate an optimal solution for all stakeholders. 

EK is often engaged by organizations to identify or solve an overarching problem or series of problems. Engaging with a diverse array of perspectives provides the organization with a better understanding of how different stakeholders perceive these problems and what their views on potential solutions are. For example, if you were to exclusively interview executives or middle managers about a given challenge an organization is facing, you’d likely receive a strong understanding of where the problem lies and how it is affecting the organization, but you may be missing out on critical day-to-day perspectives and interactions from grassroots stakeholders. Equally, if you were to only engage with daily users, you may be missing out on key perspectives from leadership and management about how the problem inhibits the organization from reaching its goals in a business and operational context. 

Recently, I delivered a KM Strategy for a Fortune 500 financial institution’s call center. Prior to the engagement, the organization already had a strong understanding of what targeted problems they were trying to solve, and EK had completed prior work with this organization, so there was pre-existing context on the organization and the call center. Despite this, we still actively requested to engage with stakeholders of all levels and roles throughout the organization to ensure we were inclusive of all relevant individuals. At one point during the discovery phase of the project, one group stated that a specific team within the call center should be owning the management and governance of training and KM content. When EK engaged with that team, they gave a conflicting opinion and stated it was another team’s responsibility to own that material. To address this disparity, EK also engaged with senior management stakeholders to surface these statements and unilaterally identify the best possible use case for content ownership. Engaging with a diverse array of stakeholders not only gives a more complete understanding of an organization, but it may also help the organization uncover different perspectives on previously identified issues, or potentially even discover something the organization was unaware of. 

Furthermore, it is important to not only diversify the types of stakeholders that you’re engaging with, but also the discovery techniques used to obtain and analyze information. A few of EK’s tried and trusted methods can be found below: 

• • Interviews: Typically 1:1 interviews with participants varying in role, tenure, expertise, or department about a predetermined topic. This method is effective for garnering unbiased insights from key individuals. 
  • Focus Groups: Often an existing team or a series of individuals with similar and/or related roles across divisions (ex. Directors Focus Group or B2B Sales Focus Group) conducted as a facilitated group interview. Focus groups enable open group discussion on a specific topic from different perspectives. 
  • Workshops: A facilitated presentation with interactive activities targeted towards a larger group audience. Workshops engage participants using a variety of methods which can be effective in ensuring equal stakeholder representation and exposing participants to high-level training and best practices. 
  • Technical Demonstrations or ‘Dry-Runs’: A technical interview with IT or system owners that provides a visual walkthrough of a key system and its core functionalities/workflows. Technical demos are very effective for understanding how a system is used, what can be improved, and details around technical infrastructure for possible integrations or enhancements. This is an essential method for technical strategy and solution implementation projects.
  • Enterprise Surveys: A method of engaging with a wider audience using targeted questions that are co-created with key project stakeholders from the organization. A 5-20 minute survey can be an efficient way to get statistically significant results at a large scale without seriously disrupting workflows for employees.

6. Educate to Maintain

Just as it’s common for an organization to implement a solution and reinvest in another a few years down the line, it’s also normal for organizations to implement a new technical or non-technical solution to solve an existing problem but struggle with the adoption or long-term maintenance associated with that solution. It can be hard to garner buy-in for a new process or workflow change, and technical solutions can be equally if not more challenging to optimally use and maintain without the right integrations, resources, and skills in place to do so.

Some engagements have a longer duration than others, and it’s not irregular for EK to support organizations with the maintenance of solutions post-implementation or even stay on to provide expert advisory services for an indefinite period. Regardless of the length of a project, it is imperative to provide stakeholders with the knowledge and resources they need to maintain both technical and non-technical solutions in the long term. 

Recently, I served as the Project Manager for a Taxonomy Management System (TMS) Implementation and Integration project for another consulting firm. A month into building the API integrations for the TMS and the identified systems, IT stakeholders from the firm expressed concerns with maintaining the integrations beyond the scope of the project due to a relative unfamiliarity with the system/process and already being overloaded with system maintenance. As a result, EK agreed to augment the system handoff process to not only act as temporary solution support, but also help the IT team upskill on TMS maintenance processes, best practices, and common risks to be aware of so that they can eventually take on system ownership when they are better positioned to do so. While it was important that EK served as temporary help desk support for the solution, the most significant aspect of EK’s delivery was the provision of skills, resources, and knowledge necessary to support the TMS in the long term. Without this, the organization may have had to permanently outsource support for the solution, which can be very expensive, or take on the many risks associated with maintaining an unfamiliar system internally. 

A strategic example of effective knowledge transfer came during a recent KM Strategy engagement for a multinational company. To conclude the engagement, we delivered a 2-year transformative roadmap on how the organization could achieve its desired target state, including high-level recommendation areas broken down by task, estimated Full-Time Employees (FTEs), critical challenges addressed, and the strategic and business impacts of the workstream if implemented. Each individual task was also represented as its own slide, including a detailed approach with thoroughly defined sub-tasks, dependencies, estimated duration, business impacts, and measurable success criteria. This granular level of detail provided the organization with a clear, tangible path forward on how to reach their goals without necessitating further investment. If the organization were to begin executing the roadmap immediately, they would know exactly what to do and how to do it, even without additional external support. EK also encourages a live delivery of the report in order to verbally communicate to key stakeholders how to most effectively utilize the roadmap as well as answer any questions. Similarly to the previous example, the most important aspect of the roadmap was the level of detail around how to execute the recommended solutions and ensure that the organization had the skills and resources in place to do so.

Conclusion

Engaging in good KM can be the key to realizing sustained success and efficiency at large enterprises that must be able to adapt to volatility and constant change. Whether a technical transformation or a non-technical one, these best practices may make the difference in the effectiveness and longevity of your organization, division, or team. It is very challenging to manage all of the information, data, and knowledge within any organization, and this is especially true at complex, globally dispersed, and rapidly scaling organizations. Whether you’re looking for strategy, design, implementation, or maintenance, EK’s services run the gamut in all things KM as a true end-to-end consulting and solutions firm. If any of this sounds relevant to you or your organization and you’re interested in seeking external support, don’t hesitate to reach out and contact us at Enterprise Knowledge. We’re here to help with any and all of your knowledge, information, and technology needs. 

The post Applied Knowledge Management Series: Addressing Challenges at Large Enterprises, Part 2 appeared first on Enterprise Knowledge.

If you are exploring ways your own enterprise can benefit from implementing AI capabilities, we can help! EK has deep experience in designing and implementing solutions that optimizes the way you use your knowledge, data, and information, and can produce actionable and personalized recommendations for you. Please feel free to contact us for more information.

The post What is the Roadmap to Enterprise AI? appeared first on Enterprise Knowledge.

Applying any of the principles identified in the above infographic can upgrade your search strategy to a future-ready, semantic experience. Whether you think your search needs a simple update or is ready for a serious upgrade, we can help. EK offers a range of search-specific services that will produce actionable recommendations. Please feel free to contact us for more information.

The post The 5 Key Components of a Semantic Search Experience appeared first on Enterprise Knowledge.