Skip to main content
For MSPs

AI exposure scan

The exposure scan shows how exposed an organization already is to AI, with no software installed on anyone's computer: Microsoft 365 Copilot licenses and use, the AI apps connected to the company, AI…

Written By Christopher Scaminaci

Last updated 13 days ago

The exposure scan shows how exposed an organization already is to AI, with no software installed on anyone's computer: Microsoft 365 Copilot licenses and use, the AI apps connected to the company, AI agents, the data AI can reach, the controls around it, and what nobody can see. Run it for your own organization, for a client organization, or for a company you hope to win (see Prospects).

At a glance

  • What it does: reads settings, licenses and counts, then scores the organization and writes a report under your own name.
  • Plan: your own organization on every plan. A client organization or a prospect needs Pro or above.
  • Who: an Administrator runs it and presents it. An Administrator or an Auditor reads it and downloads the report.
  • Where: AI Governance → Organizations: My organization, or open a client organization or a prospect, then AI exposure scan.

What a scan reads

A scan reads only what the organization allowed, and only settings, licenses and counts:

AreaWhat the scan reads
Microsoft 365 Copilot and Microsoft AICopilot seats bought and assigned, how many people used Copilot and in which apps, how many prompts they sent, and from the Microsoft 365 audit log how many people used Copilot and free Copilot Chat, whether labeled content was touched and whether a prompt injection was detected.
AI apps connected to your companyThe AI apps people or an administrator approved in Microsoft 365, what each can reach, whether it works with nobody signed in, how many people signed in to it with a work account, AI apps Microsoft Defender for Cloud Apps saw on the network, the AI sites staff reached through Microsoft Entra Internet Access (Global Secure Access), and the AI keys and accounts in the vendor connections you choose.
The data AI can reachWhether files can be shared with anyone, how many files and sites there are, guests, public groups and teams, and whether any sensitivity label exists.
Identity and access controlsWho may connect apps, the admin consent workflow, Conditional Access, security defaults, MFA registration, how many Global Administrators there are, and Secure Score.
AI agents and automationAI agents with their own identity in the directory, and the agents people used. A directory agent is shown by its kind (for example "Agent identity in the directory"), because the scan reads no name for it. An agent that people used is shown by the name Microsoft's audit log records for it, or by its ID when Microsoft records no name.
Company computers and desktop AIAI desktop apps Microsoft Intune found, the number of managed computers, and the AI services and apps Microsoft Defender saw on company computers.
What nobody can seeThe Microsoft products the organization does not own, and what that hides from it and from you.

For Anthropic and OpenAI connections it also lists API keys, service accounts and seats, counts the accounts, and reads what the organization spent over the last 30 days when the key can read it. For GitHub Copilot and Google Workspace you choose a connection you already added on the Connectors page.

What a scan never reads

A scan never reads message text, prompts, answers, file contents or mail. Prompt text is not in Microsoft's audit log, so no report can show what a person typed. It stores no secret from a vendor. It changes nothing in Microsoft 365 or in any vendor account.

The report names names

A finding you cannot back with a who and a what does not hold up in a meeting, so the report prints the people and things behind each finding. They appear under the finding in the Portal, in the presentation view, in the PDF and in what your AI tools read.

FindingWhat it names
Copilot read a hidden instruction, or someone tried to break its rulesEach detection: what was detected, when, the person using Copilot, where in Copilot, the agent, the resource and its location, what Copilot did, and the client IP address
Administrators with no MFA methodEach administrator's name and account
Too many Global AdministratorsEach Global Administrator's name and account
AI apps and outside appsWho connected each app: the people who agreed for themselves, or an administrator for everyone
Copilot read labeled contentEach resource, its location and label, how many reads, and who was using Copilot
Copilot in useEach person, how much, when last, and where
Copilot with no paid seat, and seats nobody usesThe people on each list
AI agentsEach agent that people used, by the name Microsoft's audit log records for it (its ID when there is no name), with its kind, how many times it was used and who used it
Guests and public groupsEach guest or group, its address and the date it was made
AI app sign-insEach person, the app, how many sign-ins, and the last one
AI desktop appsEach computer, its person, and the app

The person named beside a detection was using Copilot when it read that resource. They are the one exposed, not necessarily the attacker.

Your name policy decides. An organization whose name policy shows names to its administrators gets all of this. Under the pseudonymous name policy, the scan stores no person: no name, account or client IP address, and it reads no seat list and no computer list. The tables say "Hidden by this organization's name setting" where the person would be, and keep the rest of the row, such as the resource and its location. The scan reads the policy when it runs, so a changed policy applies to the next scan.

Limits. A scan stores at most 500 rows for each list, and 5,000 for the Copilot people and the Copilot seats, so it can compare those two lists for a large organization. Past 5,000 the report does not name anyone in those comparisons. A table shows at most 500 rows, and its title says how many there were in all. The PDF prints every detection and at most 100 rows of any other list, then says how many more the Portal holds. These lists never change a score.

A slow sign-in log. The scan reads the 30 days of sign-ins in pieces of 6 days, for at most 20 minutes. If Microsoft stops answering for a piece, or the time runs out, the scan keeps what it read. The report then says "At least" beside the numbers, and the table says there can be more people.

The scan does leave one thing behind. It runs one search of the organization's Microsoft 365 audit log, named StackJack exposure scan with its date, and that search is visible in the organization's Microsoft Purview audit search list. Microsoft keeps a finished search for 30 days.

Before you run a scan

  1. Open AI Governance → Organizations. Choose My organization, or open a client organization or a prospect.
  2. Under Scan setup, choose what the scan reads:
    • Microsoft 365. Select Connect Microsoft 365. A Global Administrator or Privileged Role Administrator of the organization's own Microsoft 365 completes it. For a company outside your account, send its administrator a consent link instead (see The consent link).
    • AI vendor connections. Tick the connections the scan may read. You can add an Anthropic or OpenAI key for this scan only; it is used for the scan and removed when a prospect is closed.
  3. The organization's staff notice must be in place before anything is read. If it is not, the scan stops and says so.

Run a scan

Select Run exposure scan. It usually starts within a minute. A small organization usually finishes in a few minutes, and a large one can take up to about two hours. You can leave the page while it runs. If StackJack restarts for an update while a scan runs, the scan starts over on its own. Only one scan of an organization is open at a time: asking again while one is waiting or running shows that scan.

A part that cannot be read does not stop the scan. For example, if one vendor key is missing or Microsoft is too busy to answer one question, that part says so and the rest is still read.

Copilot activity arrives a little later. The audit log search runs on its own, usually for under an hour, and in a large organization it can take a few hours. While it runs, the page and the report say that Copilot activity is still being collected. You can make the report at any time; make it again later to include that activity.

An AI assistant can also ask for a scan (see Use AI Governance from your AI). A scan reads data from Microsoft 365 and from your AI vendors, so an asked-for scan waits until an administrator approves it in the Portal. Nothing is read until then, and a request that nobody approves lapses after seven days.

Read the results

The results page shows:

  • Two scores, never one. The exposure score, 0 to 100 where higher is worse, comes only from what was found: Low, Guarded, Elevated, High or Severe. The visibility score says how much of the environment the scan could see. A company that approves less gets a lower visibility score, and never a different exposure score.
  • The headline, for example "At least 41 of 87 people already use AI at work". The number is the largest single source, and it says which. Sources are never added together, because one person can appear in several, and work-account sign-ins cannot show personal AI accounts, so the real number can only be higher. When the only people counted hold a seat or an account with an AI vendor, the headline says they hold AI seats or accounts instead, because that shows access and not use.
  • The key numbers: Copilot seats, AI apps connected, people signing in to outside AI, AI agents, files AI can reach, and what Microsoft Defender saw when the organization has it. A figure that was not read says Not checked, never zero. A figure from a list the scan could only read in part says At least, and a zero from such a list says Not fully read.
  • One card per area, with its score and its findings. Each finding says what the scan found with its numbers, what could happen and how to fix it, with the evidence for that finding. For example, a prompt injection finding shows how many attempts Microsoft detected and where in Copilot. A finding built on a list the scan could only read in part says so, and a finding that says something is missing appears only when the whole list was read.
  • What we could not check and why, per area.
  • What was read, and every item the scan found, which you can filter by risk.

An AI check decides which apps count as AI products: the scan sends each outside app's name, publisher, web host names and permission names to TypeSafe AI, and nothing else: no person's name, no file and no message. It also looks for AI in an app's name and web address. If the AI check does not run, for example because your organization has not opted in, the report says the apps were matched by name only. An outside app that the scan did not find to be AI but that holds broad access to mail, files or chat gets its own finding, because the scan cannot tell what that app does.

Severity comes from fixed rules, not from a model: Critical, High, Medium, Low and Info. A finding that combines several facts is Critical only when every one of them was read, so a part the scan could not read can never make a finding worse. Unused paid Copilot licenses are shown as Cost, with the price they are worked out from, and never as a risk.

What we could not check, and why it matters

  • The organization does not own the product. For example, sign-in logs need Microsoft Entra ID P1. This is a real risk, and the report says so: nobody can see that data, neither the company nor you.
  • An approval is missing. The read needs one more approval from the organization's administrator. This is a gap in the scan, not a risk at the company, and it only lowers the visibility score. Select Reconnect Microsoft 365, or send a new consent link.
  • Microsoft offers no way to read it. The report says so plainly.
  • It could not be read this time. The report says so; run the scan again.

The report

Download the report makes a detailed PDF to send to the company:

  1. The first page is the one to lead with, and it is always exactly one page: your logo and colors, the two scores, the headline, the three worst findings, the key numbers and your contact details.
  2. One or more pages per area, each finding with its evidence, what could happen, how to fix it, and what could not be checked and why.
  3. What changed since the baseline, when there is one.
  4. How you close these gaps with StackJack, area by area.
  5. How the scan works: what was read, the permissions used, and the privacy lines.

Set your logo, colors and contact details under Report branding. The report is made when you ask and is not kept: StackJack stores no copy. Because a key or a service account often carries a person's name in the vendor's console, the report names it only by its vendor and kind (for example "An OpenAI API key"), and apps that do not look like AI products are counted, not listed.

Present it in a meeting

Present to the client opens the same report full screen, page by page, in your branding, for a screen-shared meeting. Use the arrow keys or the Back and Next buttons. Only a signed-in Administrator of that organization can open it. It is never a public link, so the company's data stays in your session.

Changes since the baseline

The first scan that read everything it was asked to read becomes the baseline, so a scan that could read a list only in part cannot become it. A baseline from before the current scan (it read only the connected apps) gives way to the next scan that reads everything; the older scan stays in the list. Later scans are compared with it: what was added, what was removed and what changed, including how a figure or a setting moved (for example "Copilot active people: 12 to 21"). A part that was not read completely in both scans is listed as not compared, so a failed read is never shown as something removed. A part a scan made before this report existed never read is listed the same way.

If an approval is missing

The scan asks for a set of read permissions. Each one is shown on the approval screen (see The consent link). An organization that approved StackJack before the scan asked for a permission must approve once more for that part to be read: select Reconnect Microsoft 365, or send a new consent link. The other parts of the scan still work.

Good to know

  • Prospects are deleted. A prospect and everything its scans found are deleted 30 days after its last scan, 30 days after it was added if it was never scanned, or when you close it. Your report branding is yours and is kept.
  • A scan is a snapshot. It shows what existed when it ran. Run it again to see what changed.
  • It adds nothing to your inventory. Scan findings are shown on the scan pages only.