Microsoft 365 Copilot and Azure
StackJack can read Microsoft 365 Copilot use and the logs of Azure OpenAI, Foundry and API Management. You do this on one page, Microsoft 365. You connect your directory once. Then you add the sources…
Written By Christopher Scaminaci
Last updated 13 days ago
StackJack can read Microsoft 365 Copilot use and the logs of Azure OpenAI, Foundry and API Management. You do this on one page, Microsoft 365. You connect your directory once. Then you add the sources you want. This page covers what to check in Microsoft so the sources have something to read.
At a glance
- Plan: every plan. Scanning Copilot conversations with rules needs Pro.
- Who: a governance administrator. A Global Administrator or Privileged Role Administrator of your Microsoft directory gives the consent.
- Where you set it up: AI Governance → Connected accounts → Microsoft 365. The page title is "Microsoft 365 and Azure".
- Where you see the data: the sources on Connected accounts → All sources, facts in Activity → Activity explorer.
Connect your directory
- Open AI Governance → Connected accounts → Microsoft 365.
- Select Connect Microsoft 365.
- You go to Microsoft to approve read-only permissions. Then you sign in once more so StackJack can confirm you are a Global Administrator or Privileged Role Administrator of that directory. Both steps happen on Microsoft's own pages. StackJack never sees the administrator's password.
The permissions let StackJack read the Microsoft 365 audit feed, read Copilot conversations and read Copilot usage reports.
For a client organization, the client's own Global Administrator or Privileged Role Administrator completes both steps, signing in with an account of the client's directory. An account from your own directory, even with delegated (GDAP) access to the client, is not enough.
To stop, select Disconnect. The Microsoft 365 sources are switched off and StackJack stops reading your directory. What was already collected stays under your retention rules. You can connect again at any time. If consent is removed in Microsoft, the page says so. Select Reconnect Microsoft 365 to restore access. Your sources keep their place meanwhile.
Add Copilot sources
After you connect, add one or more of these. Each one is a source with its own capture level.
Check these in Microsoft 365 for Copilot
- Copilot licences. Copilot conversations and usage exist only for people who hold a Microsoft 365 Copilot licence. For live updates across your whole directory, all six Copilot service plans must be active in it: Microsoft 365 Copilot connectors in Microsoft 365 Copilot, Intelligent search, Microsoft 365 Copilot in Microsoft Teams, Microsoft 365 Copilot in productivity apps, Microsoft 365 Copilot Chat, and Power Platform connectors in Microsoft 365 Copilot. Without all six, StackJack still reads the conversations once an hour instead.
- Show user names in reports. In the Microsoft 365 admin center go to Settings, Org settings, Reports, and turn off "Display concealed user, group, and site names in all reports". While names are concealed Microsoft hides who was active, so the usage source cannot tell people apart and writes nothing.
- Let StackJack receive live updates. In the Microsoft Entra admin center open Enterprise applications, then "StackJack AI Governance", then Properties, and set "Assignment required" to No. Without this Microsoft cannot encrypt the live updates for StackJack.
- Auditing. Copilot audit records come from Microsoft Purview, so auditing must be turned on for your organization.
Azure OpenAI, Foundry and API Management
These sources read an Event Hub in your own Azure subscription.
- Create the hub. In your own Azure subscription create an Event Hubs namespace and an event hub in it.
- Create a consumer group. On the hub, add a consumer group named
stackjack-governanceso StackJack reads without disturbing anything else that reads the hub. - Create a read-only policy on the hub. On the hub itself, not on the namespace, add a shared access policy with only Listen ticked. Copy its connection string.
- Send Azure OpenAI and Foundry logs to the hub. On each Azure OpenAI or Foundry resource add a diagnostic setting that sends the
RequestResponseandAzureOpenAIRequestUsagecategories to the hub. - Send API Management gateway logs to the hub. On API Management add a diagnostic setting for the "Generative AI gateway" log category. Turn on message logging as well if you want prompts and answers scanned.
- Add the source in StackJack. On the Microsoft 365 page, add an Azure Event Hub source. Enter a Name, the Connection string and the Consumer group. Connect your Microsoft 365 directory first.
- What the logs hold. Azure OpenAI diagnostic logs never contain prompt text. Those sources hold which model was called, when, by which application and how many tokens, and nothing more. Only API Management message logging can carry text.
- Foundry tracing. Prompts and answers from Foundry tracing arrive through a telemetry source instead. Point the project's tracing at a source you add on Connected accounts → All sources. See Send telemetry from coding tools.
The page checks the connection string. It must be the whole string of a policy on the hub itself, ending in .servicebus.windows.net.
What you will see
- Your connection with its status, and the permissions it holds.
- The Copilot and Event Hub sources on Connected accounts → All sources, each with a Health page.
- Copilot use as facts in Activity → Activity explorer, and Copilot tools in AI inventory → Inventory.
Troubleshooting
More help is in Troubleshooting and FAQ, by symptom.
Next
Was this helpful?
More in Connect your AI
Data sourcesSend telemetry from coding toolsSettings for staff machinesDelivering the settings through your RMMStill need help? Ask the team