Discover Insights from Einstein Generative AI Audit and Feedback Data

Learning Objectives

After completing this unit, you’ll be able to:

Explain how to enable, collect, and store generative AI audit and feedback data for analytics
Get insights from the data with custom calculated insights objects (CIOs)

How Do You Collect and Store Generative AI Data for AI-Generated Search Answers?

You can collect and store generative AI audit and feedback data with the Einstein Generative AI platform. Use this data to enhance your generative AI applications with analytic dashboards, LLM tuning, or prompt adjustments. When you turn on generative AI data collection and storage, the platform collects and stores audit and feedback data in Data Cloud DMOs. You can turn on Einstein Generative AI data collection and storage in Einstein Setup. To understand the data model schema for generative AI audit and feedback data, see Data Model for Generative AI Audit and Feedback.

Join and aggregate the data stored in the audit and feedback DMOs and Search Analytics DMOs to get additional insights on applications such as AI-Generated Search Answers. Create custom Data Cloud calculated insights objects for these expressions and use them to monitor performance.

To understand the types of audit and feedback data collected and stored in Data Cloud DMOs, review Generative AI Audit and Feedback Data.

To turn on Einstein generative AI data collection and storage, ensure that Data Cloud is provisioned in your Salesforce org and that you’ve turned on Einstein. For detailed requirements and instructions, reference the help article, Set Up Einstein Generative AI Data Collection and Storage.

Build Custom CIOs and Dashboards

Remember that Get Cloudy Consulting executives are interested in the thumbs-up and thumbs-down feedback their agents provide. In the previous unit, you saw that Search Analytics provides an out-of-the-box interactions dashboard for different interaction types.

Awhina Oahu has permissions to access all functionality in the Data Cloud org where the audit and feedback DMOs and Search Analytics DMOs exist. Data Cloud CIOs allow storing complex calculation formulas in structured query language (SQL) format. Use SQL to create calculated insights on your data and store them in custom CIOs. Awhina Oahu can create a CIO to filter only by thumbs-up and thumbs-down reactions and store this as a custom CIO.

Let’s look at some examples of custom CIOs to discover additional insights.

Calculated Insight Object to View Agent Feedback by Thumbs-Up and Thumbs-Down Interaction

This custom CIO joins the knowledge object (ingested from a Salesforce CRM org), Search Analytics DMO, and Generative AI Feedback DMOs to calculate thumbs-up and thumbs-down interaction type grouped by different dimensions.

SELECT count(sasearchinteractions__dlm.interactiontype__c) AS numinteractions__c,
       genaigeneration__dlm.responsetext__c                AS responsetext__c,
       sasearchinteractions__dlm.interactiontype__c        AS interactiontype__c,
       sasearchqueries__dlm.querydate__c                   AS querydate__c,
       sasearchqueries__dlm.querytext__c                   AS querytext__c,
       sasearchresults__dlm.queryid__c                     AS queryid__c,
       sasearchresults__dlm.recordid__c                    AS recordid__c
FROM   sasearchqueries__dlm
       JOIN genaigatewayrequest__dlm ON
( sasearchqueries__dlm.queryid__c = genaigatewayrequest__dlm.gatewayrequestid__c
             AND sasearchqueries__dlm.queryid__c = genaigatewayrequest__dlm.kq_gatewayrequestid__c)
       LEFT OUTER JOIN sasearchresults__dlm ON
( sasearchqueries__dlm.queryid__c = sasearchresults__dlm.queryid__c )
       LEFT OUTER JOIN sasearchinteractions__dlm ON
( sasearchresults__dlm.resultid__c = sasearchinteractions__dlm.resultid__c )
       LEFT OUTER JOIN genaigatewayresponse__dlm ON
( genaigatewayrequest__dlm.gatewayrequestid__c =
       genaigatewayresponse__dlm.generationrequestid__c )
       JOIN genaigeneration__dlm
         ON ( genaigatewayresponse__dlm.generationresponseid__c =
                     genaigeneration__dlm.generationresponseid__c
              AND genaigatewayresponse__dlm.kq_generationresponseid__c =
                  genaigeneration__dlm.kq_generationresponseid__c )
       FULL OUTER JOIN ssot__knowledgearticle__dlm
                    ON ( sasearchresults__dlm.recordid__c =
                         ssot__knowledgearticle__dlm.ssot__id__c )
WHERE  sasearchresults__dlm.typeofsearchresult__c = 'SEARCH_ANSWER' AND sasearchinteractions__dlm.interactiontype__c IN ('thumbs_up', 'thumbs_down')
GROUP  BY
          genaigeneration__dlm.responsetext__c,
          sasearchinteractions__dlm.interactiontype__c,
          sasearchqueries__dlm.querydate__c,
          sasearchqueries__dlm.querytext__c,
          sasearchresults__dlm.queryid__c,
          sasearchresults__dlm.recordid__c

Beyond what the executives are looking at, Awhina Oahu is interested in viewing the hydrated prompt text - fully expanded prompt instruction, and the LLM response text for a given day. She wants to see how well the LLMs perform and if they can provide their agents with guidance on prompts and queries.

Custom Calculated Insight Object to View Hydrated Prompt Texts for Search Answers Queries in a Day

SELECT
    SUM(SASearchInteractions__dlm.NumberOfInteractions__c) AS sum_interactions__c,
    GenAIGeneration__dlm.responseText__c AS reponse_text__c,
    GenAIGatewayRequest__dlm.userId__c AS uid__c,
    GenAIGatewayRequest__dlm.feature__c AS feature__c,
    GenAIGatewayRequest__dlm.maskedPrompt__c AS masked_prompt__c,
    SASearchQueries__dlm.QueryId__c AS qid__c,
    SASearchInteractions__dlm.InteractionType__c AS interaction_type__c,
    SASearchQueries__dlm.QueryText__c AS query_text__c,
    SASearchQueries__dlm.QueryDate__c AS query_date__c,
    SASearchResults__dlm.RecordId__c AS record_id__c
FROM GenAIGatewayRequest__dlm
JOIN GenAIGatewayResponse__dlm ON (
    GenAIGatewayRequest__dlm.gatewayRequestId__c = GenAIGatewayResponse__dlm.generationRequestId__c
    AND GenAIGatewayRequest__dlm.KQ_gatewayRequestId__c = GenAIGatewayResponse__dlm.KQ_generationRequestId__c
)
JOIN GenAIGeneration__dlm ON (
    GenAIGatewayResponse__dlm.generationResponseId__c = GenAIGeneration__dlm.generationResponseId__c
    AND GenAIGatewayResponse__dlm.KQ_generationResponseId__c = GenAIGeneration__dlm.KQ_generationResponseId__c
)
LEFT OUTER JOIN SASearchQueries__dlm ON (
    GenAIGatewayResponse__dlm.generationRequestId__c = SASearchQueries__dlm.QueryId__c
)
LEFT OUTER JOIN SASearchResults__dlm ON (
    SASearchQueries__dlm.QueryId__c = SASearchResults__dlm.QueryId__c
)
LEFT OUTER JOIN SASearchInteractions__dlm ON (
    SASearchResults__dlm.ResultId__c = SASearchInteractions__dlm.ResultId__c
)
WHERE GenAIGatewayRequest__dlm.feature__c = 'SearchAnswersGenerativeAI'
GROUP BY
    reponse_text__c,
    uid__c,
    feature__c,
    masked_prompt__c,
    qid__c,
    interaction_type__c,
    query_text__c,
    query_date__c,
    record_id__c

Custom Calculated Insight Object to View LLM Response Texts for Search Answers Queries in a Day

SELECT
    SUM(SASearchInteractions__dlm.NumberOfInteractions__c) AS sum_interactions__c,
    GenAIGeneration__dlm.responseText__c AS reponse_text__c,
    GenAIGatewayRequest__dlm.userId__c AS uid__c,
    GenAIGatewayRequest__dlm.feature__c AS feature__c,
    SASearchQueries__dlm.QueryId__c AS qid__c,
    SASearchInteractions__dlm.InteractionType__c AS interaction_type__c,
    SASearchQueries__dlm.QueryText__c AS query_text__c,
    SASearchQueries__dlm.QueryDate__c AS query_date__c,
    SASearchResults__dlm.RecordId__c AS record_id__c
FROM GenAIGatewayRequest__dlm
JOIN GenAIGatewayResponse__dlm ON (
    GenAIGatewayRequest__dlm.gatewayRequestId__c = GenAIGatewayResponse__dlm.generationRequestId__c
    AND GenAIGatewayRequest__dlm.KQ_gatewayRequestId__c = GenAIGatewayResponse__dlm.KQ_generationRequestId__c
)
JOIN GenAIGeneration__dlm ON (
    GenAIGatewayResponse__dlm.generationResponseId__c = GenAIGeneration__dlm.generationResponseId__c
    AND GenAIGatewayResponse__dlm.KQ_generationResponseId__c = GenAIGeneration__dlm.KQ_generationResponseId__c
)
LEFT OUTER JOIN SASearchQueries__dlm ON (
    GenAIGatewayResponse__dlm.generationRequestId__c = SASearchQueries__dlm.QueryId__c
)
LEFT OUTER JOIN SASearchResults__dlm ON (
    SASearchQueries__dlm.QueryId__c = SASearchResults__dlm.QueryId__c
)
LEFT OUTER JOIN SASearchInteractions__dlm ON (
    SASearchResults__dlm.ResultId__c = SASearchInteractions__dlm.ResultId__c
)
WHERE
    GenAIGatewayRequest__dlm.feature__c = 'SearchAnswersGenerativeAI'
GROUP BY
    reponse_text__c,
    uid__c,
    feature__c,
    qid__c,
    interaction_type__c,
    query_text__c,
    query_date__c,
    record_id__c

We reviewed three examples of custom CIOs that help get additional insights on search data and generative AI audit and feedback data for Einstein AI-Generated Search Answers service.

Now, create dashboards in Analytics Builder to visualize the data stored in these CIOs.

In the Data Cloud org, from Setup, select Analytics Studio.
Create a new dashboard, and use Salesforce Data Cloud as the data source.
Select the CIOs you created and create a dashboard.
Save the dashboard.

Summary

Use these custom dashboards to monitor LLM performance. Fine-tune and improve results by testing your queries and responses. Additionally, check out some of the best practices we have documented to improve search results with AI-Generated Search Answers in our help documentation, Optimize Knowledge Articles for Search Answers. You can now gain deeper insights into your AI-Generated Search Answers and optimize its performance.

Estimación de tiempo

Temas

¿Necesita ayuda?

Recursos de CRM Analytics

Recursos de Data Cloud