I am testing my first attempt at agentforce/AI in a developer org and am looking to build a "chat summary" feature which summarises an ongoing enhanced chat/messaging session into a field on the messaging session record. This summary is done at the point of transfer between either Bot->Agent or Agent-> agent (will save the agent time reading through the whole transcript when receiving chats)
(I do undertand this feature is available via a paid add-on..... but this is a simple POC I am trying out to test the viability)
Currently - I am triggering the whole piece via a record triggered flow which runs at the point a Messaging Session changes ownership. At this time, my flow does the following:
1) Use the current MessagingSession ID to get the messaging session record
2) Call a piece of apex code which gathers all conversation items relating to the messaging session, orders them from old to new, then outputs them as a single piece of text.
3) Use this single piece of text as an input to Prompt Builder. Ask the agent to summarise the text in a concise manner as to what has been spoken about so far in the conversation and offer the human agent suggested follow up questions. Output this summary as another piece of long text
4) Back in the flow, assign the output from the prompt builder to a field on the Messaging Session then update the messaging session
The whole flow runs successfully, and when I manually debug the flow by entering a messaging session ID as a debug input, it works like a dream. The conversation in the messaging session is summarized exactly as I would expect.
However - when I run this "real time" - the flow runs successfully and I can see (by running a debug log) that the input to the prompt builder is exactly as I expect (ie a long text field of the whole conversation to date). However, at this point, the output from the prompt builder seems to return a complete hallucination - providing an output which is a summary or something, but has no relationship at all to my inputs. It consistently mentions fields like <ORDER_NO> and <CUSTOMER_ID> - even though these are items for which I have never referenced in any of my inputs, and which do not exist in my prompt builder prompt.
It is like it is summarising messaging sessions from another org!? The thing is, I have also made an output to stick the current Messaging Session ID onto the messaging session after the output, so I know it is using this correct ID to lookup the converation.
My Question:
Is there any way for me to "see" what is happening inside prompt builder during a real-time run. Because when I do this using the debug in the prompt builder UI - it works perfectly! The issue seems to only happen during "production" end-to-end runs.
Example Input:
OUTPUT provided using Debug Session:
OUTPUT provided using real-time end to end flow:
#Prompt Builder #Messaging Session #Agentforce #Developer Edition Org #Salesforce Developer
Yes. Turn on the Einstein generative AI audit data. You can then see the exact prompt that was sent to the LLM, and the exact response, for every call, including the ones fired from your record-triggered flow.
How to see it
- Setup > Einstein Setup: turn on Einstein Generative AI data collection and storage (audit and feedback data). Data Cloud must be provisioned in the org.
- Do one real end-to-end transfer again.
- In Data Cloud > Query Editor (or a Data Cloud report), look at GenAIGatewayRequest__dlm. The maskedPrompt__c field holds the fully resolved prompt that actually went to the model (after Trust Layer masking), and userId__c shows which user the call ran as. The model's answer is in GenAIGeneration__dlm.responseText__c. New records can take a little while to show up.
What to look for
- Compare the prompt from a debug run with the prompt from a live run. Placeholders like <ORDER_NO> and <CUSTOMER_ID> are a classic sign that the model received your instructions but not your transcript, so it wrote a generic sample summary. The audit record will show whether the transcript was really in the live prompt.
- Check userId__c on the live request. When you debug, everything runs as you. In a live transfer, the ownership change is made by routing (the accepting agent, or an automated/system user), so the flow and the prompt template run as that user. Make sure that user has the Prompt Template User permission set and access to everything the template and your Apex read.
- Check that the prompt action's input in the flow is mapped to your Apex output on the path that runs live, not to a variable that only gets populated when you debug.
If you found this answer helpful, please mark it as helpful.