Context
We use a Custom AI Assistant (Breeze), integrated with HubSpot, to support our customer service teams in their daily work — answering questions, guiding procedures, and helping to standardize customer responses. The assistant has already been validated internally and is currently in the expansion phase of usage.
Problem
Currently, the only performance information available for a Custom Assistant is a simple dashboard showing the total number of triggers. This is insufficient to evaluate the tool’s actual value, identify content issues, or support improvements. Without more comprehensive metrics, it is difficult to:
- Measure the assistant’s real impact and return on the operation;
- Identify gaps in the knowledge base that feeds the assistant;
- Evaluate the quality of the responses delivered;
- Track the adoption of the tool by the teams using it;
- Support, with data, the maintenance and adjustment cycles of the assistant.
Suggestion
We would like to suggest the creation of a more comprehensive metrics dashboard for Custom Assistants, covering at least the categories below:
1. Volume and usage
- Total interactions (by day, week, and month, with period filter)
- Interactions per assistant (when there is more than one Custom Assistant active in the account)
- Active users vs. users with access (adoption rate)
- Peak usage hours and days
2. Quality and resolution
- Resolution rate (conversations completed without the need for additional manual search)
- Gap/fallback rate (how many times the assistant did not find sufficient information)
- Rephrasing rate (user had to rephrase the question)
- User feedback per response (e.g.,
/
or rating)
3. Content and sources consulted
- Most consulted/cited articles by the assistant
- Queries without satisfactory results (indicating possible content gaps)
- Searches with no results in the knowledge base
4. Operational efficiency
- Average response time
- Average number of investigative questions per conversation
- Average conversation length (number of exchanges until resolution)
5. Trends and comparisons
- Month-over-month usage evolution
- Most recurring topics/categories in questions
- Performance comparison between different Custom Assistants, when more than one is active in the account
Complementary alternative
If a complete native dashboard is not feasible in the short term, an alternative would be to provide the export of raw interaction data (CSV or access via API), so that each company can build its own analyses.
Expected benefit
With these metrics, it would be possible to objectively measure the value generated by the assistant, prioritize content and configuration improvements based on real data, and continuously track tool adoption — today this depends on manual processes outside of HubSpot.
Who else uses Custom Assistants and misses metrics like these? 