Proposed customer support assistant: findings and roadmap (ENG-D004)

The three findings that matter most

1. F-001. The assistant could issue refunds of any amount, with no approval step (high). Under the plan the assistant can refund any amount by itself. Today, refunds above the agents' discretion go to a team lead, and nobody has worked out how that would work with the assistant. Sorting out refunds is what the owner wants the assistant for, so this is where a mistake costs most directly.
2. F-005. The assistant would answer and act from three policy sources that disagree and have no owner (high). The help-center page says 30 days and the wiki says 45. The plan loads both into the assistant and does not say which one wins. Agents already give customers different return windows for the same kind of return. The assistant would repeat that problem in every website chat, and it would also act on it.
3. F-002. There is no test step, no pilot and no test material (high). The release plan switches the assistant on for all website chats on launch day, before the holiday peak. Customers would find the errors first, in every chat at once, during the busiest period.

The other three high findings are just as urgent before launch. They are untrusted chat text reaching a model that can act (F-012), no named person or mechanism to switch the assistant off (F-003), and customer data going to the API under terms whose customer-data provisions nobody has read (F-004).

Recommended first step. Clara Schumann, VP of Customer Service, owns the release decision. The first step is for Schumann to agree now that the assistant goes live on no customer chat until a release gate is met (F-002). The gate has three parts:

• a test set built from the cases in deliverables/01-requirements.md section 8 meets the targets the client agrees;
• the off switch has been tried once (F-003);
• refunds above $100 (proposed; the client's agent owner confirms) are held in the system for a team lead (F-001).

Release then starts with a pilot smaller than all website chats. The draft control for this gate is C-05.

Cost of waiting. The evidence gives no launch date, only "before the holiday peak". The pre-launch fixes are estimated at 1-3 days to 1-2 weeks of work each. Counsel's review of the vendor terms is estimated at 1-3 weeks elapsed. If work starts late, either launch slips or the assistant goes live with these gaps open. Which is more likely depends on the launch date: unknown (to ask Bizet).

• the assistant as described in the plan (evidence/agent-proposal.md);
• the three policy sources (evidence/policy-sources.md);
• the support process (evidence/process-notes.md);
• the position on the vendor terms (evidence/vendor-terms.md);
• the fields in the chat export (evidence/transcript-schema.md);
• four interviews.

The help-desk "suggested replies" feature (AI-002) came up in the evidence and is covered where findings rest on that evidence.

Out of scope. Legal advice; audits or certifications; changes to client systems; PHI; any build of the assistant.

Access. No access to client systems was granted. The evidence arrived only as documents and notes. Nothing in this report was tested against a live system.

Interviews held (October 2026):

Johannes Brahms, CEO, was not interviewed. The client's counsel was not interviewed; legal questions go to counsel.

Evidence reviewed. The full list is in Appendix A.

Method.

1. Each document and interview was reviewed against Touchstone Data's readiness criteria for an assistant that takes actions.
2. The work produced, in order: the requirements (deliverables/01-requirements.md); an inventory of AI systems with a preliminary risk classification (deliverables/02-risk-classification.md); a target architecture (deliverables/03-architecture.md); a proposed decision record on human approval steps and the list of permitted tools (deliverables/04-adr-001-human-gates-and-tool-allowlist.md); and a draft controls design (deliverables/controls-design.md).
3. Each gap is written as a finding. Every finding cites at least one item in the evidence index.
4. Severity reflects the consequence for Bizet & Brahms Home Goods now. The launch date is not in the evidence. No severity was raised or lowered because of a date. Findings that depend on the launch should be re-rated once the date is known.

3.1 High

F-001. The assistant could issue refunds with no amount limit and no approval step

• Severity: high
• System: AI-001 (proposed support assistant)
• Evidence:
  • evidence/agent-proposal.md: "The plan states no amount limit on issue_refund and no approval step for any action."
  • evidence/process-notes.md: "Refunds up to $100 are at the agent's discretion. Larger refunds go to a team lead"
  • interviews/2026-10-19-dvorak-support-team-lead.md: "I approve refunds over $100. Nobody has asked me how that works with the bot."
  • interviews/2026-10-15-schumann-vp-customer-service.md: "Capping it at $100 defeats the purpose."
• Risk:
  • Under the plan, the assistant can issue a refund of any amount by itself. Today, refunds above the agents' discretion go to a team lead. Nobody has worked out how that approval would work with the assistant.
  • The VP of Customer Service has said a cap at that level "defeats the purpose". The limit is therefore a decision for Schumann and the client, not for Touchstone Data.
  • The assistant is not built yet, so no harm has happened. Once it is live, nothing caps what one conversation can refund.
  • This is rated on the plan as it stands. Re-rate it when the launch date is known.
• Recommendation:
  • Before launch, Bizet & Brahms Home Goods sets an amount limit on issue_refund: $100 (proposed; the client's agent owner confirms). Schumann, as the agent owner, confirms the value.
  • Refunds above the limit are held for a team lead, as happens today. The team lead's route with the assistant is defined.
  • The limit is enforced in the system, not in the model's instructions.
  • Start with approval before each action, within the limit. Give the assistant more freedom only when the evidence supports it.
  • Requirements FR-14 and G-2 in deliverables/01-requirements.md.
• Draft controls that close it: C-01 (refund limit held as a parameter in the tool gateway), C-02 (team-lead approval at the refund gate), C-03 (permitted tools, identity check and flag block in the gateway). C-09 keeps the record.
• Effort: 1-2 weeks of design and build work as part of building the assistant (estimate; the client confirms).
• Regulatory mapping: Preliminary; for review by the client's counsel. None confirmed; for counsel.

F-002. Release plan has no test step, no pilot and no test material

• Severity: high
• System: AI-001
• Evidence:
  • evidence/agent-proposal.md: "Release plan: switch it on for all website chats on launch day, targeted before the holiday peak. The plan has no test step, no pilot"
  • interviews/2026-10-15-schumann-vp-customer-service.md: "We'll test it in production; that's where the real questions are."
  • interviews/2026-10-19-dvorak-support-team-lead.md: "Nobody has asked my team for example chats to test it with."
• Risk:
  • The assistant answers policy questions and takes refund, return and address actions. The plan would switch it on for every website chat at once.
  • There is no evidence of how it behaves on known hard cases: return windows that conflict, false claims that a refund was already approved, and failed identity checks.
  • Customers would find the errors first, across all chats, in the busiest period.
• Recommendation:
  • Hold the release until a test set drawn from the cases in deliverables/01-requirements.md section 8 meets the agreed targets.
  • Ask Dvořák's team for example chats with customer details masked.
  • Agree a pilot that covers fewer than all website chats.
  • Schumann, as owner, decides the release.
• Draft controls that close it: C-05 (release gate on acceptance tests), C-06 (retest on every change), C-17 (review of the assistant's chats after launch).
• Effort: 1-2 weeks to put the cases together and run the first evaluation (estimate; the client confirms).
• Regulatory mapping: Preliminary; for review by the client's counsel. None confirmed; for counsel.

F-003. No named person or mechanism to switch the assistant off

• Severity: high
• System: AI-001
• Evidence:
  • evidence/agent-proposal.md: "no named person or mechanism to switch the assistant off."
  • interviews/2026-10-15-schumann-vp-customer-service.md: "If it misbehaves we'd raise a ticket with IT, I suppose."
• Risk:
  • The assistant might start issuing wrong refunds or quoting wrong terms while it runs on every website chat.
  • The only way described to stop it is an IT ticket, and there is no stated time for that to take effect.
  • The longer it takes, the more customers are affected.
• Recommendation:
  • Name a person and a backup who can switch the assistant off for all chats at once without an IT ticket. Chats then go back to the shared inbox.
  • Try the switch once before launch (NFR-4, NFR-5).
  • This fix is separate from F-001. It needs a switch and a person with the right to use it. It does not change how the assistant behaves.
• Draft controls that close it: C-07 (off switch with a named owner, tried in a drill), C-08 (incident runbook).
• Effort: 1-3 days, including the drill (estimate; the client confirms).
• Regulatory mapping: Preliminary; for review by the client's counsel. None confirmed; for counsel.

F-004. Customer data would go to the API each turn; the terms' customer-data provisions are unread and no DPA is on file; for counsel

• Severity: high
• System: AI-001
• Evidence:
  • evidence/vendor-terms.md: "There is no record that anyone has read the terms' provisions on customer data, retention or training, no data processing addendum on file, and no decision on retention settings."
  • evidence/vendor-terms.md: "each chat turn would send the customer's name, email, shipping address, order history and order status to the API."
  • interviews/2026-10-16-mendelssohn-governance-lead.md: "I haven't seen the vendor terms."
  • evidence/process-notes.md: "ships across the U.S. (including to Maine and Utah) and to customers in the EU"
• Risk:
  • On every chat turn, customers' names, emails, addresses and order history would go to Anthropic. That includes customers in the EU.
  • The only vendor terms in evidence are Anthropic's Commercial Terms of Service, accepted online. Nobody has read their provisions on customer data, retention and training.
  • No data has been sent yet, because the assistant is not built. Nothing shows a conflict with a contract or policy today. What the terms allow is unknown.
• Recommendation:
  • Before launch, counsel reads the terms' provisions on customer data, retention and training.
  • Counsel says whether a data processing addendum is needed (Q-C2), and whether sending EU customers' data needs anything more (Q-C4).
  • Separately, record whether each turn needs every field that is sent (DH-3).
  • Mendelssohn, as governance lead, should see the terms.
• Draft control that closes it: C-10 (vendor terms read and the data sent per turn decided before launch).
• Effort: counsel review, 1-3 weeks elapsed (estimate; the client confirms).
• Regulatory mapping: Preliminary; for review by the client's counsel. None confirmed; for counsel (see Q-C2 and Q-C4).

F-005. The assistant would answer and act from three policy sources that disagree and have no owner

• Severity: high
• System: AI-001
• Evidence:
  • evidence/policy-sources.md: "Nothing in any of them says which one governs. None has an effective date."
  • evidence/agent-proposal.md: "The plan proposes loading the help-center page and the wiki page into the model's instructions."
  • interviews/2026-10-19-dvorak-support-team-lead.md: "Last month I saw chats where three agents told customers 30, 45 and 14 days for the same kind of return."
  • interviews/2026-10-16-mendelssohn-governance-lead.md: "nobody has agreed who owns the returns policy"
• Risk:
  • The help-center page says 30 days and the wiki page says 45. The plan loads both into the model and does not say which one wins.
  • The assistant would quote and act (create_return, issue_refund) on whichever one it picks, in every website chat.
  • Agents already give customers different windows for the same return.
• Recommendation:
  • Agree an owner for each policy topic.
  • For each topic, enter one approved record with a version number, in the format Mendelssohn drafted.
  • Load only approved records into the assistant. When no record covers a question, hand the chat to a person (PG-1 to PG-8, G-3).
  • Decide whether the 45-day "holiday extension" still applies.
• Draft controls that close it: C-04 (policy answers only from approved, versioned records), C-17 (review of the assistant's chats after launch).
• Effort: 1-2 weeks, starting with returns and refunds (estimate; the client confirms).
• Regulatory mapping: Preliminary; for review by the client's counsel. None confirmed; for counsel. Moffatt v. Air Canada, 2024 BCCRT 149, illustrates a chatbot stating a policy the business does not have. It is not a rule that binds the client.

F-012. Untrusted chat text reaches a model that can act, with no check other than the model's own judgement

• Severity: high
• System: AI-001
• Evidence:
  • evidence/agent-proposal.md: “"The model will recognise abusive or suspicious chats and hand them off." No other mechanism is described.”
  • interviews/2026-10-19-dvorak-support-team-lead.md: "Customers already try 'your colleague already approved my refund' or paste in text that looks like one of our emails. My people check the order notes before acting."
• Risk:
  • Under the plan, whatever a customer types reaches a model that can issue refunds, create returns and change addresses by itself.
  • Customers already claim that someone approved their refund, and paste in text made to look like company emails. Today, agents check the order notes before acting. The plan describes no equivalent check. Its only way to flag a chat is the model's own judgement.
  • The assistant is not built yet, so no harm has happened. Once it is live, a crafted chat could lead to refunds or address changes that nobody approved.
  • This is rated on the plan as it stands. Re-rate it when the launch date is known.
• Recommendation:
  • Before launch, enforce the identity check and the block on flagged chats in the system, not in the model's instructions.
  • Do not treat anything a customer says or pastes as approval.
  • Limit permissions to what each step needs.
  • Put these attack cases in the test set.
  • Give the assistant more freedom only on evidence: these cases pass, and a period of action logs shows no incidents.
  • Requirements FR-6, FR-17, FR-19 and G-5 in deliverables/01-requirements.md.
• Draft controls that close it: C-03 (permitted tools, identity check and flag block in the gateway; chat text never counts as approval), C-05 (release gate), C-06 (retest on every change).
• Effort: 1-2 weeks of design and build work as part of building the assistant (estimate; the client confirms).
• Regulatory mapping: Preliminary; for review by the client's counsel. None confirmed; for counsel.

3.2 Medium

F-006. A live AI feature drafts customer replies with no owner, no approval record and no risk-register entry

• Severity: medium
• System: AI-002 (help-desk "suggested replies")
• Evidence:
  • evidence/process-notes.md: “The help-desk tool has had a vendor "suggested replies" feature switched on since spring 2026. It drafts replies from past tickets; agents can send them with one click. No one recorded who turned it on.”
  • interviews/2026-10-16-mendelssohn-governance-lead.md: "The risk register has no AI entries. Not the assistant, not the help-desk suggestions."
  • evidence/transcript-schema.md: "customer_email, order_number, handled_by (agent ID, or assistant once live), messages[]"
• Risk:
  • For months, an AI feature has drafted replies to customers from past tickets. Those tickets hold customer emails and message text. An agent can send a draft with one click, and nobody is accountable for the feature.
  • Several things are unknown: the feature's vendor terms, the model behind it, and whether its drafts repeat the conflicting return windows found in past tickets.
  • No harm has been shown, but nobody is in a position to notice one.
  • Re-rate this when the feature's vendor terms and data flow are known.
• Recommendation:
  • Name an owner.
  • Record the feature in the risk register and in the AI inventory.
  • Get the help-desk vendor's terms for the feature: training on ticket data, retention, location and DPA.
  • Decide who may switch vendor AI features on or off, and set an approval route for future features.
  • The feature stays on or is switched off by the owner's decision, not by default.
• Draft controls that close it: C-15 (owner and approval route for vendor AI features), C-14 (AI inventory and risk register entries).
• Effort: 2-5 days, not counting the time the vendor takes to respond (estimate; the client confirms).
• Regulatory mapping: Preliminary; for review by the client's counsel. None confirmed; for counsel.

F-007. No AI system has been recorded or classified in the risk register

• Severity: medium
• Systems: AI-001, AI-002
• Evidence:
  • interviews/2026-10-16-mendelssohn-governance-lead.md: "The risk register has no AI entries. Not the assistant, not the help-desk suggestions."
• Risk:
  • Bizet & Brahms Home Goods cannot show which AI systems it runs, what data they receive or how they are classified.
  • Neither system has been classified, although AI-001 would decide refunds and returns for customers.
  • Touchstone Data's draft inventory changes nothing until the client adopts it.
  • Rated medium because F-001 to F-006 cover the specific gaps in each system. This finding is the gap in the overall program behind them.
• Recommendation:
  • Adopt an AI inventory. The draft in deliverables/02-risk-classification.md is a starting point.
  • Name the inventory's owner. Mendelssohn owns the risk register.
  • Enter both systems in the risk register.
  • Review the inventory whenever a vendor AI feature is switched on or a model changes.
  • To find AI that is not yet listed, ask for invoices, SaaS consoles, and counts of SSO apps and browser extensions.
• Draft control that closes it: C-14 (AI inventory and risk register entries).
• Effort: 2-3 days, then a review every quarter (estimate; the client confirms).
• Regulatory mapping: Preliminary; for review by the client's counsel. None confirmed; for counsel.

F-008. Disclosure that customers are talking to an assistant is undecided, and the owner prefers it not repeat; for counsel

• Severity: medium
• System: AI-001
• Evidence:
  • evidence/agent-proposal.md: "Wording, placement and frequency: to be decided."
  • interviews/2026-10-15-schumann-vp-customer-service.md: "I don't want 'I am a bot' on every message; Rosie should feel like one of our people."
  • interviews/2026-10-16-mendelssohn-governance-lead.md: "Some of our customers are in the EU and some U.S. states; I don't know what disclosure we owe them. That's for counsel."
• Risk:
  • The plan commits to telling customers they are talking to an assistant, but not to the wording. The owner wants the assistant to "feel like one of our people".
  • Counsel may find that the disclosure falls short of what is owed in the EU or a U.S. state. If so, the fix comes after launch.
  • Rated medium because the plan already includes a disclosure step and the wording is still open.
• Recommendation:
  • Ask counsel what disclosure is owed (Q-C1) before settling the wording.
  • Then decide the wording, placement and frequency, and include the disclosure in the test set.
• Draft control that closes it: C-13 (fixed, approved disclosure text).
• Effort: 1 day after counsel answers (estimate; the client confirms).
• Regulatory mapping: Preliminary; for review by the client's counsel. Possible; for the client's counsel to confirm. Questions for counsel:
  • What do EU AI Act Article 50(1) and 50(5) require here? These articles cover telling people they are interacting with an AI system, clearly and at the latest at the first interaction.
  • Is Bizet & Brahms Home Goods the provider, the deployer or both?
  • Under Article 113, the original text applies Article 50 from 2 August 2026. Counsel to confirm the date for this client.
  • U.S.: what do Maine 10 M.R.S. § 1500-DD and Utah Code §§ 13-77-103 and 13-77-104 require?

F-009. Chat transcripts are kept indefinitely with no stated reason, and the owner plans to use them as training data; for counsel

• Severity: medium
• Systems: AI-001, AI-002
• Evidence:
  • evidence/transcript-schema.md: "Transcripts are kept until someone deletes them; nobody does."
  • interviews/2026-10-15-schumann-vp-customer-service.md: "Keep every transcript forever; it's training data for next year."
  • interviews/2026-10-16-mendelssohn-governance-lead.md: "There is no retention schedule for chats. I'd propose 90 days, but it isn't adopted."
• Risk:
  • Customer transcripts pile up with no retention period and no stated reason.
  • The owner and the governance lead take opposite positions (keep forever versus 90 days), and nobody is named to decide.
  • Using transcripts as training data is proposed. Nothing shows it has happened.
• Recommendation:
  • Decide who sets chat retention.
  • Adopt a retention schedule with a stated reason for the period.
  • Before any training use, ask counsel whether transcripts may be kept indefinitely and used as training data (Q-C3).
  • Do not delete or purge existing transcripts until counsel confirms that no legal hold applies.
• Draft control that closes it: C-12 (transcript retention schedule and hold on training use).
• Effort: 1 week to agree and adopt (estimate; the client confirms).
• Regulatory mapping: Preliminary; for review by the client's counsel. None confirmed; for counsel.

F-010. No decision on the API account's retention settings

• Severity: medium
• System: AI-001
• Evidence:
  • evidence/vendor-terms.md: "no decision on retention settings"
  • interviews/2026-10-16-mendelssohn-governance-lead.md: "I haven't seen the vendor terms."
• Risk:
  • Nobody has decided how long the vendor keeps customer data that is sent to the API.
  • This is kept separate from F-004 because it is a settings decision for IT and Mendelssohn, not contract work for counsel.
• Recommendation: Record which retention options the account offers, and choose one before launch. The choice should be consistent with the chat retention decision in F-009 (TR-4).
• Draft control that closes it: C-11 (vendor-side retention setting decided and checked).
• Effort: under a day once the options are known (estimate; the client confirms).
• Regulatory mapping: Preliminary; for review by the client's counsel. None confirmed; for counsel.

F-011. The chat log would not record tools called, policy quoted or approvals

• Severity: medium
• System: AI-001
• Evidence:
  • evidence/transcript-schema.md: "There is no field for the policy text quoted, the tools called, or approval decisions."
  • evidence/agent-proposal.md: "Chats are saved in the help-desk tool like today."
• Risk:
  • After a wrong refund or a disputed answer, Bizet & Brahms Home Goods could not work out which tool the assistant called, with what inputs, which policy it relied on, or who approved it.
  • The only fallback is the chat text itself.
• Recommendation:
  • Before launch, add the fields in NFR-2: the policy cited and its version, each tool call and its result, each approval decision and the approver, the outcome of the identity check, and the flag status.
  • Keep these fields for the same period as the transcript.
• Draft control that closes it: C-09 (action log and reconstruction check).
• Effort: 2-4 days as part of building the assistant (estimate; the client confirms).
• Regulatory mapping: Preliminary; for review by the client's counsel. None confirmed; for counsel.

F-013. One-click send; no check of policy terms before sending is on record

• Severity: medium
• System: AI-002
• Evidence:
  • evidence/process-notes.md: "It drafts replies from past tickets; agents can send them with one click. No one recorded who turned it on."
  • evidence/process-notes.md: "Support agents answer from whichever policy page they find first. There is no rule about which source wins."
  • interviews/2026-10-19-dvorak-support-team-lead.md: "Last month I saw chats where three agents told customers 30, 45 and 14 days for the same kind of return."
  • evidence/policy-sources.md: "Nothing in any of them says which one governs. None has an effective date."
• Risk:
  • For suggested replies, the only human check is the agent who clicks send. By default, one click sends a draft to the customer.
  • The drafts come from past tickets, and past answers have given customers different return windows for the same kind of return.
  • The evidence does not show whether drafts state policy terms, whether agents read or edit drafts, or whether the help-desk tool records which replies were drafted by AI.
  • No harm has been shown. The one-click review step is not shown to catch what the customer is told.
  • This is separate from F-006, which covers ownership and terms, not the review step.
• Recommendation:
  • The owner of AI-002 (once named under F-006) decides whether one-click send stays as it is.
  • Agents get short guidance: a draft can repeat an outdated return window, so check it against the approved policy record before sending.
  • Review a sample of sent AI-drafted replies and record the results. The client sets the schedule and the sample size.
  • Whether the help-desk tool can mark AI-drafted replies or require an edit before sending: unknown (to ask Mendelssohn).
• Draft control that closes it: C-16 (review at the point of send for suggested replies).
• Effort: a short guidance note and a sampling routine. The client sizes the recurring review time (estimate; the client confirms).
• Regulatory mapping: Preliminary; for review by the client's counsel. None confirmed; for counsel.

3.3 Low

F-014. No model version is recorded for the assistant, and no retest is planned for a change of model

• Severity: low
• System: AI-001
• Evidence:
  • evidence/vendor-terms.md: "IT created the account in October 2026 and accepted Anthropic's Commercial Terms of Service online; the procurement file holds the terms' URL and the acceptance date, and nothing else."
  • evidence/agent-proposal.md: "The plan has no test step, no pilot, and no named person or mechanism to switch the assistant off."
  • interviews/2026-10-15-schumann-vp-customer-service.md: "We'll test it in production; that's where the real questions are."
• Risk:
  • The plan names the model only as Claude through Anthropic's API. The version is not recorded.
  • Nobody is named to watch for the vendor's model changes. No retest is planned for when the model, the instructions, the list of permitted tools or a policy record changes.
  • In the target design, the approval steps sit in code. A model change would then mainly affect the wording of answers and the policies cited, not which actions can run.
  • The assistant is not built yet, so no harm has happened.
  • Rated low because the gateway limits the effect. Re-rate it if the approval steps are not built as designed.
• Recommendation:
  • Record the model version in the AI inventory, and fix the version where the vendor allows.
  • Name a person to watch the vendor's notices about changes and retired models.
  • Rerun the acceptance test set (deliverables/01-requirements.md section 8) before any of these changes goes live: the model, the instructions, the list of permitted tools, the refund limit or a policy record. The owner approves the result.
• Draft controls that close it: C-06 (retest on every change, including a change of model), C-14 (AI inventory).
• Effort: small once the acceptance test set exists, then a retest for each change (estimate; the client confirms).
• Regulatory mapping: Preliminary; for review by the client's counsel. None confirmed; for counsel.

First 30 days: decisions, questions to counsel, and the start of the build

Days 30 to 60: build and test before release

Days 60 to 90: pilot

Beyond 90 days

The source is evidence/index.md. Dates received are as recorded there.

Appendix B. Related documents

• deliverables/01-requirements.md: requirements and acceptance tests (the IDs FR, G, PG, DH, TR, NFR, AT and Q-C).
• deliverables/02-risk-classification.md: AI system inventory and preliminary risk classification.
• deliverables/03-architecture.md: target architecture.
• deliverables/04-adr-001-human-gates-and-tool-allowlist.md: proposed decision record on human approval steps and permitted tools (status: Proposed).
• deliverables/controls-design.md: draft controls C-01 to C-17, with a trace from each finding to its controls.

Appendix C. Glossary