Skip to main content
Connect your AI

Microsoft 365 Copilot and Azure

StackJack can read Microsoft 365 Copilot use and the logs of Azure OpenAI, Foundry and API Management. You do this on one page, Microsoft 365. You connect your directory once. Then you add the sources…

Written By Christopher Scaminaci

Last updated 13 days ago

StackJack can read Microsoft 365 Copilot use and the logs of Azure OpenAI, Foundry and API Management. You do this on one page, Microsoft 365. You connect your directory once. Then you add the sources you want. This page covers what to check in Microsoft so the sources have something to read.

At a glance

  • Plan: every plan. Scanning Copilot conversations with rules needs Pro.
  • Who: a governance administrator. A Global Administrator or Privileged Role Administrator of your Microsoft directory gives the consent.
  • Where you set it up: AI Governance → Connected accounts → Microsoft 365. The page title is "Microsoft 365 and Azure".
  • Where you see the data: the sources on Connected accounts → All sources, facts in Activity → Activity explorer.

Connect your directory

  1. Open AI Governance → Connected accounts → Microsoft 365.
  2. Select Connect Microsoft 365.
  3. You go to Microsoft to approve read-only permissions. Then you sign in once more so StackJack can confirm you are a Global Administrator or Privileged Role Administrator of that directory. Both steps happen on Microsoft's own pages. StackJack never sees the administrator's password.

The permissions let StackJack read the Microsoft 365 audit feed, read Copilot conversations and read Copilot usage reports.

For a client organization, the client's own Global Administrator or Privileged Role Administrator completes both steps, signing in with an account of the client's directory. An account from your own directory, even with delegated (GDAP) access to the client, is not enough.

To stop, select Disconnect. The Microsoft 365 sources are switched off and StackJack stops reading your directory. What was already collected stays under your retention rules. You can connect again at any time. If consent is removed in Microsoft, the page says so. Select Reconnect Microsoft 365 to restore access. Your sources keep their place meanwhile.

Add Copilot sources

After you connect, add one or more of these. Each one is a source with its own capture level.

SourceWhat it readsText
Copilot audit recordsWho used Copilot, in which app, on which model, and which labelled files it touched. File names and agent names are never kept.Facts only
Copilot conversations, scanned for DLPThe prompts and answers of every licensed person, checked by your data-loss rules.You choose how much text StackJack keeps
Copilot usage by personWhich Copilot apps each person was active in, day by day.Facts only

Check these in Microsoft 365 for Copilot

  1. Copilot licences. Copilot conversations and usage exist only for people who hold a Microsoft 365 Copilot licence. For live updates across your whole directory, all six Copilot service plans must be active in it: Microsoft 365 Copilot connectors in Microsoft 365 Copilot, Intelligent search, Microsoft 365 Copilot in Microsoft Teams, Microsoft 365 Copilot in productivity apps, Microsoft 365 Copilot Chat, and Power Platform connectors in Microsoft 365 Copilot. Without all six, StackJack still reads the conversations once an hour instead.
  2. Show user names in reports. In the Microsoft 365 admin center go to Settings, Org settings, Reports, and turn off "Display concealed user, group, and site names in all reports". While names are concealed Microsoft hides who was active, so the usage source cannot tell people apart and writes nothing.
  3. Let StackJack receive live updates. In the Microsoft Entra admin center open Enterprise applications, then "StackJack AI Governance", then Properties, and set "Assignment required" to No. Without this Microsoft cannot encrypt the live updates for StackJack.
  4. Auditing. Copilot audit records come from Microsoft Purview, so auditing must be turned on for your organization.

Azure OpenAI, Foundry and API Management

These sources read an Event Hub in your own Azure subscription.

  1. Create the hub. In your own Azure subscription create an Event Hubs namespace and an event hub in it.
  2. Create a consumer group. On the hub, add a consumer group named stackjack-governance so StackJack reads without disturbing anything else that reads the hub.
  3. Create a read-only policy on the hub. On the hub itself, not on the namespace, add a shared access policy with only Listen ticked. Copy its connection string.
  4. Send Azure OpenAI and Foundry logs to the hub. On each Azure OpenAI or Foundry resource add a diagnostic setting that sends the RequestResponse and AzureOpenAIRequestUsage categories to the hub.
  5. Send API Management gateway logs to the hub. On API Management add a diagnostic setting for the "Generative AI gateway" log category. Turn on message logging as well if you want prompts and answers scanned.
  6. Add the source in StackJack. On the Microsoft 365 page, add an Azure Event Hub source. Enter a Name, the Connection string and the Consumer group. Connect your Microsoft 365 directory first.
  7. What the logs hold. Azure OpenAI diagnostic logs never contain prompt text. Those sources hold which model was called, when, by which application and how many tokens, and nothing more. Only API Management message logging can carry text.
  8. Foundry tracing. Prompts and answers from Foundry tracing arrive through a telemetry source instead. Point the project's tracing at a source you add on Connected accounts → All sources. See Send telemetry from coding tools.

The page checks the connection string. It must be the whole string of a policy on the hub itself, ending in .servicebus.windows.net.

What you will see

  • Your connection with its status, and the permissions it holds.
  • The Copilot and Event Hub sources on Connected accounts → All sources, each with a Health page.
  • Copilot use as facts in Activity → Activity explorer, and Copilot tools in AI inventory → Inventory.

Troubleshooting

What you seeWhat to check
"The permissions were not granted. Ask a Global Administrator or Privileged Role Administrator to approve them, then try again."Ask the right administrator to approve.
"The account you signed in with is not a Global Administrator or Privileged Role Administrator of that Microsoft 365 directory. Sign in with one that is."Sign in with another account.
"This organization is already connected to a different Microsoft 365 directory. Disconnect it first, then connect the other one."Disconnect first.
"That Microsoft 365 directory is already connected to another StackJack organization."One directory connects to one organization. Contact support if that is wrong.
"Waiting for consent in Microsoft."The consent is not finished. Start again with Connect Microsoft 365.
The usage source writes nothingMicrosoft is concealing user names in reports. See step 2 above.
"StackJack could not connect to that Event Hub with the connection string."Check that the policy has the Listen right, the key is current and the hub exists.
Copilot conversations arrive hourly, not liveNot all six Copilot service plans are active. See step 1 above.

More help is in Troubleshooting and FAQ, by symptom.

Next