Question KB Retrieval Prompt LLM / CLI PM Answer

AI Feature Consulting Agent

Your AI-Powered Teams PM Assistant

Copilot Studio · Teams SaaS CLI Mode · Claude Opus 4.6 + MCP LLM Mode · GPT-5.4 Web UI
365Q&A Records
12Feature Areas
68%Troubleshooting
4Channels

liuhanlin  |  Mentor: Fei Zuo  |  FY26 Vendor Student Program  |  Microsoft Teams CMD

The Problem — Why We Built This

Teams PM team (CMD SOX) receives a high volume of feature queries daily via Chat, Email, IcM, Office Hours — covering transcription, recording, Copilot, Graph API, compliance policies, and more.

PM Pain Points

Context Switching
5+ apps: Teams, Wiki, ADO, Learn, Email
Repetitive Work
Same questions re-searched and re-answered
Knowledge Loss
Quality answers scattered in chat, not reusable

Field Pain Points

Long Wait Times
Hours or even days for a PM response
Don't Know Who to Ask
Questions bounce across channels, no clear owner
Inconsistent Answers
Different PMs, different answers — no single source
365
Q&A Records
12
Feature Areas
68%
Troubleshooting
4
Consulting Channels

The Solution — Three Modes

Copilot Studio
GPT-5.4 · Teams SaaS
~15s response, zero-code deployment
1:1 direct chat in Teams
Knowledge: Docs + Web + Graph Connector
Best for quick Field questions
Limitation: no local KB
CLI Mode
Claude Opus 4.6 · Local Terminal
MCP connection to SharePoint / Teams / Outlook
Accesses full personal KB
Deepest accuracy & retrieval
Best for complex & high-stakes questions
LLM Mode
GPT-5.4 · Web UI / API
8-step pipeline · REST API interface
Ideal for batch evaluation & CI automation
Best for systematic testing at scale
Copilot Studio
Copilot Studio Agent in Teams
LLM Mode
LLM Mode Web UI

How It Works — Pipeline

Question Field / Support / IcM KB Retrieval 365 Records · 12 Areas Prompt Engineering Context + Instructions LLM / CLI Processing 3 Modes Available PM Answer Forwardable Reply
Input Layer
Questions from Teams Chat, Email, IcM, Office Hours — auto-classified by topic area
Knowledge Base
365 Q&A records across 12 feature areas, keyword-indexed, with gold standard PM answers for evaluation
Prompt Engineering
Structured system prompt with role definition, KB context injection, output format, and confidence calibration
Multi-Model
GPT-5.4 (Studio & LLM), Claude Opus 4.6 (CLI) — each optimized for its deployment context

Results — Evaluation Scores

πŸ† Top Score β€” CLI Mode
4.62
surpasses Gold PM 4.58
πŸ’‘ Full KB + MCP = deepest retrieval

Weighted Overall Scores

Gold PM
4.58
CLI Mode
4.62
Studio
4.07
LLM Mode
3.69

Four-Dimension Evaluation

🎯
Accuracy (40%)
Factual consistency with Gold Standard
πŸ“
Fitness (25%)
Depth & breadth match complexity
⚑
Actionability (20%)
Actionable steps PM can use directly
πŸ”—
Traceability (15%)
Cites KB records & documentation
Judge: Claude Opus 4.6 | 0.5 step | 20 Gold Standard

Per-Dimension Comparison

DimensionStudioLLMCLI
Accuracy (40%)4.173.834.80
Fitness (25%)3.773.624.55
Actionability (20%)3.983.774.72
Traceability (15%)3.353.304.12
Weighted4.073.694.62

Case Studies — Comparative Analysis

πŸ’¬ A customer wants to file a DCR to enable transcript download for all attendees by default. What is the DCR process?
πŸ–₯️ CLI Mode 0
πŸ“‹ [Policy Settings / Transcription] — DCR process for enabling transcript download
Confidence: βœ… HIGH

═══ FORWARDABLE ANSWER ═══

A similar DCR was previously evaluated and rejected by the product team. Rejection based on: (1) transcript lives in organizer’s OneDrive — broad download bypasses ODSP permission model, (2) tenant admins need data exfiltration control, (3) organizer-as-owner model aligns with M365 governance.1βœ“ Cites specific historical DCR rejection with 3 detailed reasons The customer is welcome to file a new DCR.

How to file: Through IcM under Teams DCR queue (Team ID: 108657, Service: Microsoft Teams Support). Complete Business Impact Statement within 3 days.2βœ“ Exact IcM queue + Team ID — PM can act immediately

Required: (1) What customer requests, (2) Current behavior gap, (3) Business impact, (4) Affected users (Low/Med/High), (5) Workaround availability

Refs: KB: TD-018, TD-005, SIM-086
βœ“ Processβœ“ Precedentβœ“ Templateβœ“ KB refs
πŸ’¬ Studio 0
πŸ“‹ [Policy / Transcript Access] — DCR request for attendee transcript download
Confidence: ⚠️ MEDIUM

═══ FORWARDABLE ANSWER ═══

The closest matching ask was rejected when framed as β€œAllow all participants to download transcripts via Graph.”3~ Mentions rejection but lacks the 3 specific reasons Reason: compliance, governance, and ownership model — transcripts stored in organizer’s OneDrive/SharePoint.

For a new DCR, submit as a tenant-admin-controlled policy change request4~ Good advice but missing IcM queue details (Team ID). Describe: business scenario, compliance reason, scope (internal vs external), why current model blocks workflow.

Refs: KB: TD-018, TD-005
βœ“ Process~ Precedent~ Templateβœ“ KB refs
🌐 LLM 0
πŸ“‹ [Transcription] - DCR request for default transcript download access
Confidence: ⚠️ MEDIUM

═══ FORWARDABLE ANSWER ═══

There is no documented DCR intake process in the available transcription guidance5βœ— Factually wrong — DCR process exists via IcM (Team 108657) for changing transcript download to be enabled by default. Download depends on admin policy and organizer β€œAllow download” setting6βœ— Over-hedged — missed the entire IcM/DCR workflow.

Minimum package: exact pain point, affected users, business impact, workarounds.
βœ— Processβœ— Precedent~ Templateβœ— KB refs
⚑
Key Difference: CLI cited historical DCR rejection with 3 reasons + exact IcM queue (Team 108657) — LLM said β€œno documented process.”
πŸ’¬ Retrieve meeting transcripts via Graph API — exact endpoints, permissions, VTT format? Does this work for adhoc 1:1 calls?
πŸ–₯️ CLI Mode 0
πŸ“‹ Transcription — Graph API Endpoints, Permissions & VTT (incl. Adhoc)
Confidence: βœ… HIGH

═══ FORWARDABLE ANSWER ═══

Yes — Graph API supports transcript retrieval for both scheduled meetings and 1:1 adhoc calls1βœ“ Explicitly confirms adhoc support with prerequisites.

Endpoints: /callTranscripts, /transcripts2βœ“ Both exact endpoint paths provided
Permissions: OnlineMeetingTranscript.Read.All
VTT: $format=text/vtt on content endpoint
Adhoc: Supported — transcription must be started during call, call must be recorded.3βœ“ Specific adhoc prerequisites documented

Refs: KB: SIM-009, SIM-042, TD-012
βœ“ Endpointsβœ“ Permissionsβœ“ VTTβœ“ Adhoc
πŸ’¬ Studio 0
πŸ“‹ [Graph API] — Retrieve transcripts, VTT, ad hoc support
Confidence: βœ… HIGH

═══ FORWARDABLE ANSWER ═══

Retrieve Teams meeting transcripts and ad hoc call transcripts via Graph, including VTT format via $format=text/vtt on the content endpoint.

Adhoc call conditions imprecise4~ Mentions adhoc but lacks specific prerequisites — VTT path details slightly incomplete.

Refs: KB: SIM-009, TX-010, TX-015
βœ“ Endpointsβœ“ Permissionsβœ“ VTT~ Adhoc
🌐 LLM 0
πŸ“‹ [graph_api] - Microsoft Graph transcript retrieval
Confidence: ⚠️ MEDIUM

═══ FORWARDABLE ANSWER ═══

Retrieve transcripts through Microsoft Graph in WebVTT format by listing transcript resource then downloading content.

For 1:1 adhoc calls: β€œnot fully confirmed”5βœ— Over-hedged — adhoc IS supported (CLI confirms). Basic info covered but weak traceability.

Refs: KB: TC-025, SIM-086
~ Endpoints~ Permissionsβœ“ VTTβœ— Adhoc
⚑
Key Difference: CLI provided exact endpoints + adhoc prerequisites — LLM hedged with β€œnot fully confirmed.”
πŸ’¬ BoA wants R/T enabled but hide AI summary/Recap. Is this real? Do you have the Comcast demo?
πŸ–₯️ CLI Mode 0
πŸ“‹ [Recap / AI Toggle] — BoA scenario: R/T on, hide Recap
Confidence: βœ… HIGH

═══ FORWARDABLE ANSWER ═══

Vincent — yes, I’m aware of BoA’s scenario. The ask: enable R/T but hide AI summary (Recap), resolved by turning off the all-up AI toggle1βœ“ Gives the specific solution — AI toggle. AI artifacts governed: Intelligent Recap, Facilitator Live Notes, Podcast Recap, Video Recap.

This is NOT the same as β€œFacilitator during meeting but no Recap.”2βœ“ Precisely differentiates the two scenario types Conversations with BNP Paribas and UBS show the inverse: no R/T but keep Recap3βœ“ Cross-customer intelligence — 3 scenarios distinguished.

Comcast demo (3/26): transcript-less Copilot — AI summary persists without persistent transcript.

Refs: KB: TX-003, TX-006, TX-010, TX-007
βœ“ Solutionβœ“ Differentiationβœ“ Cross-customerβœ“ Demo ref
πŸ’¬ Studio 0
πŸ“‹ [Recap / Copilot] — Comcast demo for β€œR/T on, hide Recap”
Confidence: ⚠️ MEDIUM

═══ FORWARDABLE ANSWER ═══

Comcast scenario is the inverse pattern: enable Copilot/AI without saving R/T4~ Identifies inverse but doesn’t clearly separate 3 scenarios.

For BoA-style ask — that is a real customer scenario. Demand for R/T + no AI recap exposure.

Strongest control: disable transcription entirely5~ Blunt approach — misses the AI toggle solution. Admins can set Copilot Off/During/After via meeting policy.

Refs: KB: TX-010, TX-011, TX-015
~ Solution~ Differentiationβœ— Cross-customer~ Demo ref
🌐 LLM 0
πŸ“‹ [Recap] - R/T enabled while hiding AI summary/Recap
Confidence: ⚠️ MEDIUM

═══ FORWARDABLE ANSWER ═══

Yes — real customer scenario. Intelligent Recap depends on transcript data; if transcription enabled + Teams Premium, users get AI summaries.

Practical control: β€œaccess and generation prerequisites”6βœ— Vague — no specific toggle or admin path given rather than a clean toggle. Moving toward separate retention policies — future direction only.

Refs: KB: TC-025, SIM-086, TX-015, TX-017
βœ— Solutionβœ— Differentiationβœ— Cross-customerβœ— Demo ref
⚑
Key Difference: CLI differentiated 3 customer scenarios (BoA vs BNP/UBS vs Comcast) + gave AI toggle solution — LLM gave a vague generic answer.
πŸ† CLI wins every case — deep KB + MCP real-time data
βœ“ Strong ~ Partial βœ— Missing

Summary & Outlook

What I Learned

RAG System Design
Full RAG pipeline end-to-end: data collection, KB indexing, retrieval strategy, prompt engineering, LLM orchestration
LLM Evaluation
4-dimension weighted scoring framework — systematically measure and compare LLM output quality
MCP & Tool Integration
MCP protocol connecting real-time enterprise data (SharePoint, Teams, Outlook) — bridging AI with live data
PM Collaboration
Worked with PMs to understand real pain points, collected gold-standard data, translated user needs into features

Next Steps

Semantic Retrieval
Keyword β†’ Semantic Search
Eliminate retrieval misses
Hybrid Mode
Studio instant draft
CLI deep verification
PM Feedback Loop
Thumbs up/down
Continuously optimize prompts
Faster Response
Optimize MCP call chain
Target <60s

Thank You — Questions Welcome

liuhanlin · Mentor: Fei Zuo · CMD SOX · Meeting Intelligence

× Fullscreen