AI chatbots can answer a member's question in seconds, with total confidence. But confident and correct aren't the same thing. I've seen chatbots on association websites recommending events that had already been and gone.
So what actually makes a better AI answer? And how can you tell a good one from a bad one?
Why do answers matter more for associations?
Associations do a lot with less. Small teams hold deep knowledge across standards, guidance, webinars and years of archive, sometimes spread across different platforms. Questions about renewals, CPD and events tend to arrive when teams are most stretched.
A wrong answer also has far more impact on a paying member than on someone just visiting your website. That's why a good answer needs to pass four tests.
1. Does it know everything it should?
Many assistants only know what you remember to tell them: FAQs and a few web pages, kept in a separate store that falls out of date. A good answer draws on everything the assistant is meant to know, kept in sync as you publish. That starts with how your content goes in, which means your AI should:
- Read documents with their layout. A good AI understands how a PDF is laid out, including headings, tables and charts, right down to the values plotted on a graph.
- Transcribe webinars, videos and podcasts. Recordings should be broken into sections at logical points and timestamped, so a 30 or 40-minute recording becomes something members can get answers from.
- Work out missing metadata. I recently worked with an association that had hundreds of podcasts but no record of the guests. ReadyIntelligence extracted guests’ details from the transcripts themselves, so members can now ask which show featured a particular guest.
2. Does it work the question?
Many chatbots run a single search and display the closest answers back. Nuanced questions get thin, generic replies. A good answer comes from working the question. Your AI assistant should break apart acronyms and terminology, search on each, check whether it has enough and go back for more. In the AI assistants I set up, that's generally up to about 20 searches, guided by rules on which knowledge to prioritise.
3. Does it know who's asking?
Many assistants don't know who they're dealing with, so every member gets the same reply.
A good answer is about the person asking. When a member signs in, their membership level, interests and renewal date should be passed to the assistant as context.
In my recent webinar, AI assistants for answers your members can trust, I signed in as a member on a test site and asked an assistant, "Do you know about me?" It came back with my membership, my interests, and the fact that my auto-renewal was off.
I'm working with an organisation that has a four-day conference with 400 sessions. There, suggesting sessions matched to each member becomes a really useful tool. A good assistant also learns each member's acronyms and language, so they're not having to repeat themselves.
4. Can you see where the answer came from?
A good answer shows its source, so members can check it for themselves. Every answer should be cited. The National Association of Local Councils, for example, has given over 10,000 users access to a member agent whose answers are all cited and referenced across years of knowledge and data.
Good answer or bad answer? How does your chatbot measure up?
| A good answer | A bad answer | Ask your chatbot this… | |
|---|---|---|---|
| What it knows | A good answerKept in sync, with new content added automatically, including PDFs, charts, video and audio | A bad answerOnly knows what you remembered to tell it, from a separate store someone has to keep up to date | Ask your chatbot this…“Which was the show with [guest name]?” |
| How it searches | A good answerBreaks apart acronyms, searches, checks whether it has enough and goes back for more | A bad answerRuns a single search and displays the closest answers back | Ask your chatbot this…A question using your own acronyms and terminology |
| Who it’s for | A good answerKnows the member’s level, interests and renewal date | A bad answerDoesn’t know who it’s dealing with | Ask your chatbot this…“Do you know about me?” |
| Can you see it? | A good answerEvery answer is cited and shows its source, so members can check it | A bad answerNo source shown, so there’s no way to check it | Ask your chatbot this…“Where did that answer come from?” |
Want to see what a good AI answer looks like?
As I've explored, a good AI answer draws on everything it should know, works the question, knows who's asking and shows its source. Get those four right, and members get answers they can trust.
My advice is not to bite off too much at first. Start with a clear use case, build confidence as the answers come back, then scale up.
See how AI built for better answers works in practice. Join me for an AI workshop on Wednesday 25th November in London and create an AI assistant that works with your membership knowledge in just a couple of hours. Book your place here.
Comparing assistants? See how ReadyIntelligence checks every answer.
