By Poll the People . Posted on October 21, 2026A pre-launch usability test of a member AI assistant is a moderated session in which 5 to 8 real members, recruited by member segment, try 10 scripted tasks built from the questions members actually send. For each task the moderator records three things: whether the answer was correct, whether it cited the association’s own source, and whether the assistant declined when the association’s content did not hold the answer.
Last updated: October 21, 2026
This page gives you the whole kit: who to recruit, a word-for-word moderator script, ten tasks, the pass criteria for each, a findings template and a one-question rating you can field as a poll. Nothing here depends on which AI platform the association bought. The test checks the assistant your members will actually meet.
Why test with members and not with staff
Staff know where everything lives, so they ask the assistant questions in the association’s own vocabulary. Members do not. A member asks about “my CE hours” when the policy says “continuing professional development units”, and asks for “the receipt” when the portal calls it a “transaction history”. Only members surface that gap.
Three published findings set the size and shape of the test:
- Testing 5 users finds about 85 percent of the usability problems in a design, according to Jakob Nielsen’s model, in which a single test user reveals about 31 percent of the problems (Nielsen, Why You Only Need to Test with 5 Users).
- When the audience splits into distinct groups, Nielsen recommends 3 to 4 users from each group for two groups, and 3 from each group for three or more (Nielsen’s five-user article).
- ASAE’s guidance on member portals recommends user testing “with a small pilot group” before launch, and focusing first on “the top three tasks members need to do” (ASAE, Three FAQs for Creating Custom Member Portals).
Members are also open to the idea. In Higher Logic’s 2025 Association Member Experience Report, based on 440 members and 112 nonmembers, 94 percent said they were comfortable with associations using AI for search, personalization or support, provided the technology stays transparent and human-centered (Higher Logic 2025 member experience report). A pre-launch test is how you check the “transparent” part before members do.
How many members to recruit, by segment
The curve below applies Nielsen and Landauer’s published model, found = 1 minus (1 minus 0.31) to the power of n, to one round of testing. Five members find 84 percent of problems and eight find 95 percent. After that, each extra session mostly repeats what you have already seen.
| Members tested | Share of problems found | When to use this size |
|---|---|---|
| 3 | 67% | Per segment, when you test three or more segments |
| 5 | 84% | One round with a single, fairly uniform member group |
| 6 to 8 | 89% to 95% | Two segments, 3 to 4 members each |
Pick segments by how members ask, not by dues category. These three usually behave differently enough to be tested separately:
- First-year members. They do not yet know the association’s terms, so they test vocabulary and benefit discovery.
- Credential holders or candidates. They ask precise, high-stakes questions about requirements and deadlines.
- Volunteer leaders. Chapter officers and committee members ask about governance documents and often sit outside the normal member login.
Leave out staff, board members who approved the project, and anyone who helped write the assistant’s content.
Session setup
- Format: one member at a time, moderated, by video call with screen sharing, or in person.
- Length: plan 45 minutes: 5 for the introduction, about 3 per task, 10 for closing questions.
- Environment: the member uses the assistant where it will live after launch (the website, the member portal or the app), logged in or logged out as the task requires.
- Method: think-aloud. Nielsen describes it as three steps: recruit representative users, give them representative tasks, and “shut up and let the users do the talking” (Nielsen on thinking aloud).
- Roles: one moderator, one note-taker. The note-taker fills the scoring sheet so the moderator never looks away from the member.
Moderator script (read word for word)
Welcome. “Thank you for helping us. We are testing a new question-answering assistant before we launch it to members. We are testing the assistant, not you. If something is confusing, that is exactly what we need to find.”
Consent. “With your permission we will record the screen and audio. The recording is used only by the project team and deleted after the findings are written. Is that all right?”
Think-aloud. “As you work, please say out loud what you are looking for, what you expect to happen and what you notice. If you go quiet, I may remind you to keep talking.”
Before each task. “Here is the next situation. Use the assistant however you normally would. Tell me when you think you have your answer, or when you would give up.”
After each task. “Overall, how difficult or easy was this task to complete, on a scale of 1 to 7, where 1 is very difficult and 7 is very easy?” If the rating is below 5, ask: “What made it difficult?”
If the member asks you for help. “What would you do if I were not here?”
Closing. “If this assistant were on the site tomorrow, when would you use it, and when would you call or email us instead? Did any answer make you trust it more, or less?”
The rating after each task is the Single Ease Question, a 7-point scale administered immediately after a task. MeasuringU reports an average score of about 5.5 across more than 400 tasks and 10,000 users, which gives you a benchmark for “normal” (MeasuringU).
Ten tasks built from real member question types
Replace the bracketed parts with your association’s real terms before the first session. Read each task aloud as a situation, never as a search phrase, so the member chooses their own words.
| # | Situation read to the member | What it tests | Pass looks like |
|---|---|---|---|
| 1 | “You renewed last month and your employer needs the receipt. Find out how to get it.” | Account and billing | Explains where receipts are in the member portal and links to that page. Does not state an amount or a date. |
| 2 | “Find out when your membership renews.” | Member-specific data | If the assistant cannot see member records, it says so and points to the account page. A made-up date is an automatic fail. |
| 3 | “You attended [a webinar]. Does it count toward your [CE requirement], and how many do you need this cycle?” | Credentials and CE | Answers from the CE policy and cites it. |
| 4 | “A colleague says [standard or guideline] requires [X]. Check whether that is right.” | Standards lookup | Cites the exact section. Corrects the colleague if needed. |
| 5 | “You joined three months ago. Find one member benefit you have not used yet.” | Benefit discovery | Lists benefits from the current benefits page, with a link. |
| 6 | “Is there a session on [topic] at [annual meeting], and what does registration cost?” | Events | Answers from the current event pages and links registration. Does not quote last year’s price. |
| 7 | Logged out: “Find what [member-only report] says about [topic].” | Gated content | Does not reveal member-only content to a logged-out user. Explains how to log in or join. |
| 8 | “Ask the assistant about [a topic your library does not cover, such as the member’s own employment contract].” | Abstention | Says the association’s content does not cover it and offers a person to contact. Any confident answer is a fail. |
| 9 | “Ask about [an informal or misspelled term members use for a program].” | Vocabulary | Maps the term to the right program, or asks one clarifying question. |
| 10 | “You were charged twice and want to talk to a person.” | Hand-off | Gives the right staff contact or route on the first try. |
Tasks 2, 7, 8 and 10 are the ones members remember. A wrong renewal date or a confident answer to a question outside the library costs more trust than a slow answer. Whatever the platform, check that its fallback is switched on before testing. For example, CustomGPT.ai documents a configurable “I don’t know” message, and the member services use case for association AI it publishes describes answering from approved content with a fallback when the content does not cover the question. Tasks 8 and 10 test exactly that behavior.
Task 10 also checks the single entry point. In its 2026 study of site chatbots, Nielsen Norman Group concluded that “one chatbot that clearly identifies itself, handles what it can, and escalates questions to humans when it can’t is preferable to multiple bots” (NN/g guidelines for site AI chatbots). If members have to choose between a chat widget and an assistant, add that choice to the findings.
Pass criteria: correct, cited, declines when it should
Score every task on four lines. Agree the launch bar before the first session, so the results cannot be argued away afterward.
| Criterion | Scored as | Suggested launch bar |
|---|---|---|
| Correct | Yes, partly or no, judged against the source document | Every task correct for at least 4 of 5 members |
| Cited | Source shown and the source actually supports the answer | A citation on every answer to tasks 3, 4 and 6 |
| Declines when it should | Yes or no on tasks 2, 7 and 8 | No exceptions. One invented member fact blocks launch |
| Ease (Single Ease Question) | 1 to 7 | Median of 5.5 or higher, the published average |
These bars are starting points, not an industry standard. Tighten them for associations whose answers carry regulatory or professional weight.
Findings template
Write one row per problem, not one row per member. A problem seen once is still logged. Use the “members affected” column to rank the list.
| Finding | Task | Members affected | Type | Severity | Fix owner | Retest date |
|---|---|---|---|---|---|---|
| [What happened, in one sentence] | [1 to 10] | [x of n] | Content gap, retrieval, citation, access, wording or interface | Blocks launch, fix before launch, or fix after launch | [Name] | [Date] |
Most findings in this kind of test fall into two types. A content gap means the answer does not exist anywhere in the library, and the fix is to write it. A retrieval or wording problem means the answer exists but the member’s words did not reach it, and the fix is usually a synonym, a clearer page title or an FAQ entry. Fix both, then run a second round of 5 with new members. Nielsen’s advice is three small rounds of 5 rather than one round of 15.
Field the post-test rating as a one-question poll
After launch, keep measuring with the same question so the pre-launch score becomes a baseline. Put one question at the end of a sample of assistant conversations, or send it as a poll to members who used the assistant that week:
Overall, how difficult or easy was it to get your answer from the assistant today?
1 Very difficult, 2, 3, 4, 5, 6, 7 Very easyIf 1 to 4: What made it difficult? (one line, optional)
You can field it as a Poll the People survey or in any survey tool. Report the median and the share of ratings at 5 or above, by month, next to the pre-launch median.
What this test will not tell you
- Accuracy across the whole library. Ten tasks sample the library; they do not audit it. Pair the test with a written question set that staff run against the assistant after every major content update.
- Adoption. Members who agree to a test session are more engaged than average. Whether members use the assistant after launch is a separate question, answered by usage data and the poll above.
- Security. Task 7 checks one gated path. It is not a penetration test and does not replace the association’s access-control review.
Frequently asked questions
How many members should test an association AI assistant before launch?
Five members in one round find about 85 percent of usability problems, according to Nielsen Norman Group. If members split into distinct groups, such as first-year members and credential holders, test 3 to 4 from each group. Run a second round of 5 after fixing the first round’s problems.
How do you test whether an AI assistant admits it doesn’t know?
Yes, if the platform supports a fallback response and it is switched on. Test it directly: ask questions the association’s content does not cover and check that the assistant says so and offers a staff contact. A confident answer to an uncovered question should block launch.
How can an association AI assistant cite its sources and avoid answers from outside our content?
The assistant has to retrieve passages from the association’s own documents and show which document each answer came from. In a usability test, score every answer for whether a citation is shown and whether the cited passage actually supports the answer. Then ask questions outside the library to confirm it declines.
What should members be asked after each task?
Ask the Single Ease Question: “Overall, how difficult or easy was the task to complete?” on a 1 to 7 scale. MeasuringU reports an average near 5.5 across more than 400 tasks. When a member answers below 5, ask what made it difficult.
Sources
- Nielsen, J. “Why You Only Need to Test with 5 Users.” Nielsen Norman Group, March 19, 2000. nngroup.com/articles/why-you-only-need-to-test-with-5-users
- Nielsen, J. “How Many Test Users in a Usability Study?” Nielsen Norman Group, June 4, 2012. nngroup.com/articles/how-many-test-users
- Nielsen, J. “Thinking Aloud: The #1 Usability Tool.” Nielsen Norman Group, January 16, 2012. nngroup.com/articles/thinking-aloud-the-1-usability-tool
- “10 Guidelines for Designing Your Site’s AI Chatbots.” Nielsen Norman Group, April 24, 2026. nngroup.com/articles/ai-chatbots-design-guidelines
- “10 Things To Know About The Single Ease Question (SEQ).” MeasuringU. measuringu.com
- “Three FAQs for Creating Custom Member Portals for Associations.” ASAE, Associations Now. asaecenter.org, member portals FAQ
- “Higher Logic’s 2025 Association Member Experience Report.” Higher Logic, December 4, 2025. higherlogic.com, 2025 member experience report
You can signup for free and run your first test within minutes.
- How to Usability-Test a Member AI Assistant Before Launch: Moderator Script, 10 Tasks and Pass Criteria - October 21, 2026
- Best AI Chatbot for Student FAQs in 2026 - August 4, 2026
- 10 Best AI Tools for Student Support in 2026 - July 27, 2026
