Pulumi Cloud OIDC from Azure DevOps¶
How Azure DevOps pipelines authenticate to Pulumi Cloud and to Azure without a long-lived
PULUMI_ACCESS_TOKEN or client secret anywhere, and how the whole trust is created and
managed as code from this repo.
Status: proven and self-managed
The handshake was proven end-to-end by a standalone CI-persona smoke-test pipeline (build 180769: an Azure DevOps pipeline authenticates to Pulumi Cloud and lists ESC environments with no Pulumi access token anywhere), then hardened into a least-privilege, two-persona model (builds 180845–181029, see Narrowing the identity).
Both sides of the trust are now created by infra/pulumi-config itself — the Entra
app registrations, the Azure DevOps service connections, the Pulumi Cloud teams/roles, and
the OIDC issuer's auth policies. Nothing here depends on a Terraform PR in another repo.
Why¶
Storing a Pulumi access token — or an Azure client secret — in a variable group means a long-lived credential that never expires on its own, is copyable, and has to be rotated by hand. Azure DevOps already mints short-lived OIDC id_tokens for Workload Identity Federation service connections; both Entra ID and Pulumi Cloud can be registered as relying parties for them. Every pipeline run gets a fresh, scoped token instead of a standing secret.
How it's built¶
Everything is created by one Pulumi program, infra/pulumi-config (Pulumi project
cloud-foundations, stack pulumi-config). There is no external repo in the loop and no
manual Azure DevOps or Entra portal step for the steady-state path — only the original,
long-since-completed registration of the Pulumi Cloud OIDC issuer (see
The issuer).
For each Azure environment in Pulumi.pulumi-config.yaml (development-sandbox,
shared-services, platformdev, test, qa, uat, prod), the EnvironmentIdentity
component (infra/pulumi-config/Environments/EnvironmentIdentity.cs) creates:
- an Entra app registration
iac-<env>and its service principal, granted Owner on that environment's subscription - a Pulumi ESC environment
azure/<env>that logs the app in via OIDC (fn::open::azure-login) — no client secret, noPULUMI_ACCESS_TOKEN - a federated credential on the app trusting Pulumi ESC's own OIDC issuer, scoped to that one
environment (
pulumi:environments:org:saif-corp:env:azure/<env>) — this is what lets the ESC environment mint Azure tokens for the app in the first place - one Azure DevOps service connection per project, named
iac-<env>, in every project listed undercloud-foundations:azureDevOps.projects— anAzureRMconnection usingWorkloadIdentityFederation, backed by the same app - an all-pipelines
azuredevops.PipelineAuthorizationon each of those connections, so no pipeline needs the manual, click-through authorization Azure DevOps otherwise demands the first time it references the connection - a matching federated credential on the app for each of those connections, with its
issuer/subject read back from the connection resource's own exported
workload_identity_federation_issuer/_subjectattributes rather than hand-typed, so they cannot drift from what Azure DevOps actually presents
Those resources are grouped into three child components, split by blast radius rather than by resource type:
| Component | Holds |
|---|---|
WorkloadIdentity |
the Entra app and principal, Owner on the subscription, two directory roles, two Microsoft Graph permissions |
EscEnvironmentAccess |
the credential trusting Pulumi Cloud, the azure/<env> ESC environment, and the two team grants |
AzureDevOpsAccess |
one service connection per Azure DevOps project, each with the credential that trusts it and an all-pipelines authorization |
The two access components are independent ways of obtaining a token for the same identity, so revoking one leaves the other intact.
Tenant and organization singletons live under Organization/ instead, because they are shared
by every environment and so belong to none of them:
| Unit | Holds | Shape |
|---|---|---|
MachineTeam ×2 |
a Pulumi Cloud team, its custom organization role, and the assignment binding them — plus any standing grants with no local resource to sit beside | component |
AzureBaseEnvironment |
the shared azure/base ESC environment and both personas' open grants on it |
component |
AdminAccess |
the iac-platform-root service connection and the credential that trusts it |
component |
DirectoryRoles |
the two tenant-singleton directory role activations | loose resources |
OidcTrust |
the OIDC issuer and its two auth policies | loose resource |
The last two are deliberately not components. The rule this repo applies is that a component
earns its node when its resources share an invariant — remove one and the rest are broken or
meaningless. A Team without its TeamRoleAssignment has no organization access level, so
every pipeline token fails at exchange rather than at deploy; that is an invariant.
DirectoryRoles' two activations need nothing from each other, and OidcTrust declares a
single resource whose policies are inline properties, so neither has one. Grouping them anyway
would be filing by "declared in the same place", which is what a component is not for.
Note also that grouping by provider would be the wrong axis here, however tempting the
Organization/ folder's provider mix makes it look. This program's whole job is federation
between providers: AzureDevOpsAccess reads a service connection's exported issuer and
subject straight into an Entra credential, and EscEnvironmentAccess derives a credential's
subject from an ESC environment's path. Splitting either by provider severs the dependency the
pair exists for. The components are named for what trust they establish — which is why a
future OktaAccess would slot in beside EscEnvironmentAccess as a fourth child of
EnvironmentIdentity, not into an Okta/ folder.
Separately, the AdminAccess component
(infra/pulumi-config/Organization/AdminAccess.cs)
creates one more connection, iac-platform-root — the identity that deploys
infra/pulumi-config itself. It reuses the pre-existing iac-platform-root app (looked up by
display name, not created here) rather than minting a new one — which is why it is named for
the access path rather than the identity, exactly parallel to AzureDevOpsAccess.
\"platform\" means the platform, not the Platform project
The name comes from cloud-foundations:admin.identityName and refers to the platform as a
whole — it is the root IaC identity for the estate. It is not derived from the Azure
DevOps project that happens to share the name. Which project hosts the connection is decided
independently, by the pulumi-config-admin tag in azureDevOps.projects (today: Platform).
Those two were briefly conflated: the name used to be built as iac-{adoProjectName}-root.
That encoded a dependency that does not exist — the Entra app is manually maintained and
always called iac-platform-root — so moving the admin tag to another project would have
renamed the lookup and failed the deploy outright. Move the tag freely; the identity keeps
its name.
flowchart LR
subgraph "Per environment × per ADO project"
ADO[ADO pipeline<br/>AzureCLI@2]
SC["Service connection<br/>iac-<env>"]
end
ADMIN["Service connection<br/>iac-platform-root<br/>(Platform project only)"]
EID[Entra ID<br/>login.microsoftonline.com]
PC[Pulumi Cloud<br/>org saif-corp]
ADO -->|requests id_token| SC
SC -->|federated cred| EID
EID -->|Entra-issued id_token| ADO
ADO -->|az login / az account set| ADO
ADO -->|pulumi login --oidc-token| PC
PC -->|short-lived Pulumi token| ADO
ADMIN -->|deploys pulumi-config itself| PC
AzureCLI@2 also performs a real az login / az account set before running your script. If
the backing service principal cannot see the connection's subscription, the task fails there —
before any Pulumi command runs. infra/shared-services/permissions.tf documents this
constraint for other pipelines.
The mechanism is addSpnToEnvironment: true, which exposes the OIDC id_token to the inline
script as $env:idToken. That token is then handed to pulumi login.
The config file is the source of truth¶
infra/pulumi-config/Pulumi.pulumi-config.yaml drives the whole fan-out. Four stack-config
sections:
config:
cloud-foundations:admin:
identityName: iac-platform-root
environmentName: shared-services
cloud-foundations:azure:
tenantId: <Entra tenant guid>
environments:
- name: development-sandbox
subscriptionId: <guid>
- name: shared-services
subscriptionId: <guid>
displayName: Shared Services
- name: platformdev
subscriptionId: <guid>
displayName: Platform Dev
# test / qa / uat / prod likewise
cloud-foundations:azureDevOps:
organizationId: <Azure DevOps org guid>
organizationName: <Azure DevOps org display name>
projects:
# Every project lists azure-devops-ci, which is what gets it the per-environment
# iac-<env> connections. Exactly one project also lists pulumi-config-admin, which
# is what gets it the one-off iac-platform-root connection.
- name: Platform
pulumiTeams: [azure-devops-ci, pulumi-config-admin]
- name: Architecture
pulumiTeams: [azure-devops-ci]
# Case-Management / Claims / Corporate / Customer / Documents / Policy /
# Regulatory / SAIF likewise
cloud-foundations:pulumiOidc:
displayName: pulumi-esc-oidc
issuer: https://api.pulumi.com/oidc
audience: azure:saif-corp
subjectPrefix: "pulumi:environments:org:"
Adding an environment (a new subscription) or an Azure DevOps project is a YAML edit and a
pulumi up — no external PR, no manual portal step, no new hardcoded array. Every environment
gets the identical treatment above — including development-sandbox (ad hoc manual testing)
and shared-services (shared resources other pipelines reach into) — since all of them create
service connections and none is special-cased.
Projects are cheap, but not unlimited — 9 slots remain
Entra caps an application at 20 federated identity credentials, and the iac-<env>
applications are per environment, not per project. So each project added to projects:
costs one credential on every application at once, not one in total.
Today that is 10 projects plus the Pulumi ESC credential — 11 of 20 on each of the
seven applications. Nine more projects fit. The tenth fails during pulumi up, on
whichever environment Pulumi happens to reach first, so the error will name an arbitrary
environment rather than the project you just added.
Should that ceiling ever bind, the fix is to stop federating every project against the same per-environment application — either an application per project/environment pair, or a single shared connection projects are granted access to.
Only one ESC import remains: bootstrap/azure
Earlier revisions of this stack imported azure/<env> back into itself to read
{env}:subscriptionId — circular, since this stack is what creates those environments.
Subscription IDs now live as plain stack config instead (above), and this stack is their
source of truth, not a consumer of it. The stack also no longer imports platform/all
(which existed only to pull in platform/pulumi's long-lived PULUMI_ACCESS_TOKEN) —
Pulumi Cloud auth for deploying this very stack now comes from the iac-platform-root
service connection's OIDC exchange, via a federated credential scoped to
pulumi:environments:org:saif-corp:env:bootstrap/azure.
This environment is manually maintained and deliberately not managed by this stack —
it is what the stack authenticates with, so it cannot be something the stack creates.
It was previously platform/iac-platform-root, then bootstrap/pulumi-config. The
bootstrap/ project namespace is reserved for exactly one category: credentials a
human enters by hand because no stack can create them. Third-party vendor credentials
belong here too, one environment per vendor and tier — bootstrap/okta-nonprod,
bootstrap/okta-prod, bootstrap/dynatrace — so each keeps its own grant boundary while
Pulumi Cloud's project grouping still presents them as a single place to look.
Naming environments after what the credential is for, not what consumes it, is what
makes that namespace scan consistently: bootstrap/azure sits beside bootstrap/okta-prod
and reads the same way. It also means the environment holds only credentials. Non-secret
settings that merely happened to be convenient there belong in stack config instead — the
Azure DevOps organization URL moved out for exactly this reason, and is now derived from
cloud-foundations:azureDevOps.organizationName.
bootstrap/azure is the one asymmetric member of the namespace: its path is baked into a
federated credential subject, so it cannot be renamed or deleted casually (see below). The
vendor environments carry no such constraint.
Consume vendor credentials with implicit imports, not imports:
An explicit imports: entry merges the entire imported environment into the
consumer's resolved output, so every value in bootstrap/okta-prod would be readable
by anyone who can open the importing environment. An
implicit import exposes only the
value actually referenced:
# in a consuming environment — only clientSecret crosses the boundary
values:
okta:
clientSecret: ${environments.bootstrap.okta-prod.okta.clientSecret}
Never import vendor credentials into azure/base: AzureBaseEnvironment gives both
personas an open grant on it, so a secret placed there is readable by every CI pipeline
in every Azure DevOps project whether or not anything imports it.
Nothing imports azure/base today, so it propagates nothing on its own — the exposure
above is the grants, not inheritance. Whether to delete it or wire the azure/<env>
environments to genuinely import it is tracked in cloud-foundations#104.
Whether the reader additionally needs an open grant on the imported environment is
not documented and has not been verified here. Until it is, assume it does not and rely
on implicit imports to limit exposure. A granted/ungranted assertion pair is the natural
way to prove it; the retired CI smoke test carried one and is archived under
docs/_internal/ if you want the shape of it.
Renaming this environment is a two-step change, and cleanup is part of it
The environment path is load-bearing: it is half of the federated credential subject
(pulumi:environments:org:saif-corp:env:<project>/<environment>) that lets ESC mint the
iac-platform-root app's Azure tokens. Add the new subject as an additional federated
credential on that app before switching the environment: import, and only remove the
old one after a real pipeline run proves the new path. Switching the import first leaves
the stack unable to authenticate — including unable to deploy the fix.
Verify the new credential in isolation before switching anything:
That forces the azure-login exchange to actually execute while printing only a
non-secret client id. Expect a OIDC token validation failed with Azure error for the
first minute or so — Entra takes a short while to propagate a new credential, so retry
before concluding the subject is wrong.
Then delete the old environment and its now-orphaned federated credential. A
credential whose environment no longer exists is not inert: its subject is just a string,
so anyone able to create an ESC environment at that exact path can mint tokens for the
app — which here is Owner on seven subscriptions. Two such orphans accumulated in this
tenant from earlier renames (platform/iac-platform-root and platform/all, both 404)
before anyone noticed. Audit with:
az ad app federated-credential list --id <appId> --query "[].{name:name,subject:subject}" -o table
# then, for each pulumi:environments subject, confirm the environment still exists:
pulumi env get saif-corp/<project>/<environment> --definition
Entra also caps an application at 20 federated credentials, so orphans consume a finite budget as well as widening the trust surface.
Deleting a team or an ESC environment in the UI silently orphans its grants
TeamEnvironmentPermission, TeamStackPermission and TeamRoleAssignment are
children of the team and environment they bind. Deleting either side in the Pulumi
Cloud UI removes the grants with it — but Pulumi's state file still records them as
live, and pulumi preview compares desired state against state, not against the
server. The stack therefore reports itself clean while the CI identity has silently
lost access.
This has already happened once here: 13 environment grants, one role assignment and
one stack permission were destroyed out of band and went unnoticed across four
consecutive clean previews. The only thing that surfaces it is
pulumi refresh, which reconciles state against the provider:
pulumi refresh --stack pulumi-config # detects the vanished grants
pulumi up --stack pulumi-config # recreates them
Verify the repair against the API rather than trusting the resulting state — that is the failure mode being diagnosed:
curl -H "Authorization: token $PULUMI_ACCESS_TOKEN" \
https://api.pulumi.com/api/orgs/saif-corp/teams/azure-devops-ci
Prefer deleting environments and teams through this stack. When something must be
removed in the UI, run a refresh immediately afterwards.
Reference values¶
| Item | Value |
|---|---|
| Azure DevOps organization | SAIFCorporation |
| Azure DevOps org GUID | 7f80b586-0eee-43fa-8743-8d4025b72800 |
| Entra tenant | a86cb8ed-369b-4df5-ace5-43811f6e08cf |
Issuer (iss) |
https://login.microsoftonline.com/a86cb8ed-369b-4df5-ace5-43811f6e08cf/v2.0 |
Audience (aud) |
fb60f99c-7a34-4190-8149-302f77469936 — see below |
| Service connections | iac-<env>, one per environment × Azure DevOps project (config-driven, see above); iac-platform-root in Platform only |
| App registrations | iac-<env>, one per environment; iac-platform-root (admin) |
| Pulumi organization | saif-corp |
The registered audience and the emitted aud claim are different strings
Every federated credential reports its audience as api://AzureADTokenExchange:
$ az ad app federated-credential list --id <app-id> --query "[].audiences"
api://AzureADTokenExchange
But the token actually issued carries the magic GUID form:
Confirmed in build 180769. Pulumi matches the claim in the token, not the value in the
credential, so the policy rule must be the GUID. Using api://AzureADTokenExchange
fails with an audience mismatch. OidcTrust.AadTokenExchangeAudience is set to the GUID for
exactly this reason.
The subject is a canonical Entra identifier, not a readable path. Its grammar is:
| Segment | Encoding | Value here | Varies? |
|---|---|---|---|
eid1 |
literal | Entra subject scheme, v1 | no |
c/pub |
literal | cloud = public | no |
t/{tenant} |
base64url GUID | 7bhsqJs29U2s5UOBH24Izw → a86cb8ed-369b-4df5-ace5-43811f6e08cf |
no |
a/{app} |
base64url GUID | rISbSSETf0KqFyZ8ppdXmA → 499b84ac-1321-427f-aa17-267ca6975798 |
no |
sc |
literal | subject kind = service connection | no |
{adoOrg} |
plain GUID | 7f80b586-0eee-43fa-8743-8d4025b72800 |
no |
{serviceEndpointId} |
plain GUID | one per service connection | yes |
Note the mixed encoding: the tenant and app segments are base64url-encoded GUIDs, while the org and endpoint segments are plain dashed GUIDs.
{app} is Azure DevOps, not one of our app registrations
499b84ac-… is the well-known Azure DevOps first-party application ID — the same value you
pass to az account get-access-token --resource. It identifies Azure DevOps as a whole,
not any individual iac-<env> app.
Because this segment (and everything before it) never varies, the subject says nothing
about which app backs a connection — only that a token came from some Azure DevOps
Entra-federated service connection in this tenant, in this org, from this specific
endpoint. That is exactly what the subject prefix OidcTrust composes pins, and it is why the
CI policy's wildcard (see Choosing the subject rule) covers
every iac-<env> connection this stack creates without listing any of them.
To decode a segment yourself:
$s='rISbSSETf0KqFyZ8ppdXmA'; $p=$s.Replace('-','+').Replace('_','/')
switch($p.Length % 4){ 2 {$p+='=='} 3 {$p+='='} }
[guid]::new([Convert]::FromBase64String($p)).Guid
Verify the Entra discovery document if something looks wrong (confirmed reachable):
curl -s https://login.microsoftonline.com/a86cb8ed-369b-4df5-ace5-43811f6e08cf/v2.0/.well-known/openid-configuration
Why iss and aud are not enough¶
A reference article this work started from suggested iss + aud "will most likely be
sufficient", treating sub as optional. That does not hold here.
Every federated credential registered against this tenant carries the same audience —
api://AzureADTokenExchange as registered, emitted as fb60f99c-7a34-4190-8149-302f77469936.
It is the fixed audience for Entra workload identity federation, so as a rule it excludes
nothing. iss narrows only to the Entra tenant — the whole tenant, not just Azure DevOps or
just this Pulumi program's connections. So an iss + aud policy would accept any Entra
workload identity in the tenant that can obtain this audience, and sub is the only rule
doing real access control.
Both issuer forms exist in this tenant
Some older, unrelated app registrations still present vstoken.dev.azure.com/<orgId>
tokens. Every connection created by this repo uses the Entra form above. Don't assume from
a neighbouring connection which form applies — read the actual federated credential.
Choosing the subject rule¶
Pulumi policy claim values support wildcards: * matches zero or more characters, ? matches
zero or one, and . matches exactly one character — not a literal dot. Don't read these
rules as plain string equality.
The CI rule wildcards only the final segment:
That still pins the Entra tenant, that the token was issued via Azure DevOps rather than some
other workload identity, that it came from the SAIFCorporation organization, and that the
subject is a service connection. It is much tighter than omitting sub. What it does not
pin is which service connection — so any pipeline in any of the projects listed under
cloud-foundations:azureDevOps.projects, using any iac-<env> connection, can mint the CI
token.
Why not pin the endpoint GUID?¶
Considered and rejected, for the same reason it was rejected before this stack owned the
connections: Azure DevOps is the deployment mechanism for these environments, so a pipeline
that can deploy through one iac-<env> connection can deploy through any of them in that
project — pinning a GUID defends against nothing an attacker couldn't reach another way. It
would also multiply the maintenance cost: one rule per environment × project, forever, growing
every time a project or environment is added, versus the single wildcard rule today that covers
new connections automatically.
The one place a GUID is pinned is the elevated (admin) policy — see Two personas, one issuer — because that one connection genuinely should not be reachable from anywhere else.
The tripwire
The wildcard's residual exposure is a hand-made Entra-WIF service connection somewhere
in the tenant that isn't one this stack created. Such a connection would not automatically
gain anything here, since the CI policy's teamName requires the pipeline to explicitly
request azure-devops-ci — but it is worth periodically confirming that every Entra-WIF
Azure DevOps connection in the tenant is one this stack (or a known predecessor) created.
Remember deny always beats allow regardless of rule order or specificity — a deny policy is the reliable way to carve out an exception if a specific connection is ever compromised.
Verifying the handshake¶
The test pipelines were retired
Two standalone pipelines proved this handshake and exercised the two personas —
pulumi-oidc.yml (CI persona, definitions 5595 and 5597) and
pulumi-config-admin.yml (elevated persona, definition 5608). Both lived under
.azdo/experiments/ and have been deleted, since the shared
v2/deployments/pulumi-pulumicloud/ service in SAIF/pipeline-templates supersedes them.
Their findings, caveats and full source are archived at
docs/_internal/pulumi-oidc-experiment-pipelines.md in this repository. That page is
excluded from this site, so read it in the repo.
Until the shared deployment service ships, nothing re-verifies this chain on a pull
request. A break in infra/pulumi-config would not be caught automatically. To
re-verify by hand, restore the archived admin pipeline to a branch and run it.
What the elevated pipeline established, and what any replacement should still cover: minting
a pulumi-config-admin token through iac-platform-root and running pulumi preview on
infra/pulumi-config itself. That single command exercises the whole chain — stack read,
open on bootstrap/azure, ESC's federated credential minting Azure tokens, and the
azure/azuread/azuredevops providers authenticating with them — so any broken link fails
the run.
Why the pipeline asserted the token subject
Until build 181564 the issuer carried two rules allowing the admin team: one for
iac-platform-root and one for the old externally-managed connection. A successful login
therefore did not by itself prove the new connection worked — the old one may have satisfied
the policy. The pipeline's expectedEndpointId parameter closed that hole by asserting the
token's sub claim ends with the expected endpoint GUID, and failed loudly when it did not.
That assertion produced the evidence to retire the legacy rule. Carry it forward: it is also the tripwire that would catch the admin persona silently authenticating through some other connection.
Two behaviours worth carrying into any replacement:
- Plain
pulumi previewcompares against the state file, so grants deleted in the Pulumi Cloud UI stay invisible to it. Only--refreshreads the server, and it is the only way to catch that class of drift. pulumi uphas never been run from a pipeline here. Every green run is preview only, so the apply path is unproven.
Generated ESC YAML must be newline-pinned¶
Running this stack from a Linux agent for the first time (build 181555) previewed 8 to update where the same commit previewed 253 unchanged on a Windows workstation.
YamlDotNet writes line breaks through TextWriter.WriteLine(), which emits
Environment.NewLine — CRLF on Windows, LF on Linux. That rendered YAML is the
pulumiservice.Environment resource's input, so every azure/* environment's content was
platform-dependent: a local pulumi up and a build agent would each rewrite all eight back to
their own line endings, indefinitely, churning the very environments the OIDC login path
depends on.
All ESC YAML is therefore serialized through EscYaml, which pins .WithNewLine("\n"). Use
it for any new call site rather than constructing a SerializerBuilder inline.
Don't measure line endings through the PowerShell pipeline
pulumi env get … --definition | Out-String reports CRLF whatever is actually stored,
because PowerShell normalises line endings. Read raw bytes instead:
cmd /c "pulumi env get saif-corp/azure/base --definition > $env:TEMP\d.txt 2>nul"
$b = [IO.File]::ReadAllBytes("$env:TEMP\d.txt")
"CR=$(($b | ? { $_ -eq 13 }).Count) LF=$(($b | ? { $_ -eq 10 }).Count)"
The reliable signal is simpler: if a preview is clean on one OS and dirty on another, something in the program is platform-dependent.
What a successful run looks like¶
--- token claims (configure Pulumi Cloud from these) ---
iss: ***
aud: fb60f99c-7a34-4190-8149-302f77469936
sub: ***
-------------------------------------------------------
Logging in to Pulumi Cloud as org 'saif-corp'
Logged in to pulumi.com as service-account:991b80a7-…
--- pulumi whoami ---
User: service-account:991b80a7-…
Organizations: saif-corp
Token type: team: azure-devops-ci
Token name: Azure DevOps – SAIFCo-1785448625627678
The identity is an ephemeral Pulumi service account, minted per run and named after the OIDC issuer. Nothing is stored.
Environment names are visible to any authorized token
pulumi env ls returns every ESC environment in the organization regardless of how narrowly
the token is scoped — TeamEnvironmentPermission gates opening, not listing. Contents
stay protected; names and structure do not. See
Listing is not permission-gated.
Troubleshooting¶
Start from the printed claims, but expect masking
The smoke test decodes and prints iss, aud and sub. Azure DevOps redacts iss and
sub as *** because they derive from the service connection; aud comes through. For the
masked values, read Graph or the connection's federation details panel.
| Symptom | Likely cause |
|---|---|
unsupported protocol scheme "" |
The backend URL was omitted. Use pulumi login https://api.pulumi.com --oidc-token ..., not pulumi login --oidc-token .... |
No OIDC token was returned (our guard) |
The connection is not using Workload Identity Federation, or addSpnToEnvironment: true is missing. |
idToken is not a JWT (our guard) |
The task exposed something other than a JWT. Check the service connection's authentication scheme. |
Invalid issuer / issuer is not registered: … for org saif-corp |
The issuer registered in Pulumi Cloud does not exactly match the iss claim — it should be https://login.microsoftonline.com/a86cb8ed-…/v2.0, not a vstoken.dev.azure.com URL. Issuer URLs are immutable, so this usually means re-registering. |
| Audience mismatch | The aud rule is almost certainly set to api://AzureADTokenExchange. The emitted claim is the GUID fb60f99c-7a34-4190-8149-302f77469936 — use that. |
| Access denied / no matching policy | Either the sub rule does not match the organization GUID 7f80b586-…, the requested --oidc-team doesn't match any policy's teamName, or a deny policy is winning. |
OIDC token exchange failed: access_denied when requesting the admin team |
Expected from anything other than the iac-platform-root connection. The elevated policy is pinned to that one endpoint GUID — see Two personas, one issuer. |
| Requested token type unavailable | The Pulumi organization's edition does not grant that token type. saif-corp is Business Critical — personal, organization, team and deployment-runner are all available. |
Fails at az account set, before any Pulumi output |
An Entra problem, not a Pulumi one. The service principal cannot see the connection's subscription. |
Narrowing the identity¶
A broad organization / Member Allow policy is what proved the handshake first (build
180769) — it grants more than any deployment pipeline needs. infra/pulumi-config now models a
narrower identity in code — teams and roles in Organization/MachineTeam.cs, the issuer's
policies in Organization/OidcTrust.cs — so the model is iterated by editing C#, not by
clicking.
Two personas, one issuer¶
azure-devops-ci |
pulumi-config-admin |
|
|---|---|---|
| Deploys | app / infra pipelines via iac-<env> connections |
infra/pulumi-config itself, via iac-platform-root |
| Does | pulumi env run … -- terraform apply, pulumi up on its own stacks |
creates Entra apps, ESC environments, Azure Owner assignments, the OIDC issuer's own policies |
| ESC | Open on azure/<env> |
Open on every environment it manages (transitive closure — see below) |
| Stacks | creates and owns its own (stack:create + creator grant) |
Edit on cloud-foundations/pulumi-config |
| Org access | stack:create only |
stack:create, environment:create, team/role/OIDC-issuer/auth-policy management verbs |
The only part of sub that varies between connections is the endpoint GUID, and that alone
separates the two identities via a second policy on the same issuer:
| Policy | sub rule |
Mints |
|---|---|---|
| Elevated | pinned to the iac-platform-root connection's endpoint GUID (an Output, not a hardcoded constant) |
team pulumi-config-admin |
| CI | …/sc/7f80b586-…/* — any project, any iac-<env> connection |
team azure-devops-ci |
These do not race each other. The pipeline requests a team
(pulumi login --oidc-team azure-devops-ci), so each policy is an authorization check on a
(sub, tokenType, teamName) triple — proven in build 180845
(Token type: team: azure-devops-ci) and build 180854 (a non-Platform connection's admin
request refused at the exchange, while Platform's succeeds in build 180855). The two-persona
boundary holds in both directions.
The cutover rule is gone
OidcTrust.Create declares exactly two policies: the
iac-platform-root-pinned elevated rule above and the CI wildcard. A third rule pinned to
the old externally-managed Platform Azure-PulumiCloud endpoint (3ac7d649-…) existed
during cutover so the identity that deploys this stack couldn't lock itself out. Builds
181555 and 181560 proved the new connection mints the admin token, so it was removed and
build 181564 confirmed the admin persona still authenticates without it.
Where the grants live¶
Environment access for the CI persona is granted next to the environment it grants, inside
the EscEnvironmentAccess component — every azure/<env> this stack creates gets its CI
TeamEnvironmentPermission in the same commit, so an environment and its pipeline access cannot
drift apart. azure/base is granted once separately (a grant per azure/<env> would collide on
the same shared environment).
Neither team has a human member, which makes their grants a ceiling for machine tokens rather than something a person inherits.
Pulumi Cloud adds whoever first applies the stack as a team admin
Team membership is populated at create time and cannot be declined: whichever identity
runs the first pulumi up is added to both teams as an admin. That is normally the
organization service account, but a human running the initial apply is added instead — as
happened when this stack was rebuilt from empty. Remove them by hand afterwards:
curl -X PATCH -H "Authorization: token $PULUMI_ACCESS_TOKEN" \
-d '{"memberAction":"remove","member":"<github-login>"}' \
https://api.pulumi.com/api/orgs/<org>/teams/<team>
Removing the member produces no Pulumi diff, because the provider only acts on changes to the declared list — see below.
Check membership rather than assuming it, since GET /teams reports members: 0 for every
team; fetch each team individually. When a team genuinely has no members the response omits
the members key altogether rather than returning an empty array — so verify the key's
presence before counting, or a language that wraps null into a one-element array (as
PowerShell's @($x) does) will report one phantom member and send you chasing a removal
that already worked.
Declaring Members empty is intent, not enforcement — and IgnoreChanges makes it worse
An empty Members list does not remove whoever Pulumi Cloud added: the provider acts
only on changes to the declared list, so a member it never saw declared is left alone
indefinitely. Declaring it empty records intent and nothing more.
IgnoreChanges on Members is actively harmful here, and the reason is the reverse of the
intuition. It substitutes the live value into the desired inputs, so a declared [] and
a cloud-populated ['someone'] are guaranteed to disagree on every run — the option
causes a permanent ~members diff rather than suppressing one. It was removed; the
preview went clean at 274 unchanged with no update even attempted.
Two scope vocabularies exist and are easy to confuse. stack:read / environment:open are
entity scopes, granted per entity by TeamStackPermission / TeamEnvironmentPermission. A
global-resource-type OrganizationRole carries only the org-wide verbs:
| Operation | Needs | Granted by |
|---|---|---|
pulumi preview / up on an existing stack |
stack:read / stack:write |
TeamStackPermission, or the creator grant (see below) |
| Create a new stack | stack:create |
OrganizationRole (global) |
pulumi env run / pulumi env open |
environment:open |
TeamEnvironmentPermission |
| …on everything it imports, transitively | environment:open |
TeamEnvironmentPermission |
| Create a new ESC environment | environment:create |
OrganizationRole (global) |
The authoritative scope catalogue is the getOrganizationRoleScopes data source, or
GET /api/orgs/{org}/roles/scopes.
Pulumi Cloud silently drops team:list
It's in the catalogue but discarded on write — ask for it and read the role back, and it is
simply absent, no error. Worse, Pulumi's state keeps what you declared, so pulumi preview
stays clean and the gap never surfaces from the CLI. AdminGlobalScopes leaves it out so the
code matches what the server actually stores; confirm any future scope change with
GET /api/orgs/{org}/roles/{roleId}, not pulumi preview.
Create-if-absent, and what it costs¶
Pipelines create their own stacks on first deploy — stack:create is the only stack-level
scope either team's role carries, and no TeamStackPermission is provisioned ahead of a
pipeline's first run. This is a deliberate departure from the Terraform estate, which splits
bootstrap from deploy purely as a workaround for Terraform Cloud variable-set mapping; that
split is not being carried over.
What makes it work is Pulumi Cloud's creator grant: whoever creates a stack (or an ESC
environment) automatically gets full control over it, with no corresponding
TeamStackPermission needed. Build 181027 proved it for CI — holding stack:create and zero
stack grants:
The same mechanism explains why the admin persona opens every azure/<env> environment (and,
during cutover, every auth/azure-<env> this stack used to own) while holding no
TeamEnvironmentPermission on most of them: it created them.
The creator grant is full control, including delete
The team above deleted the stack it had just created, despite the role carrying no
stack:delete and the team holding Stack Admin nowhere. Under create-if-absent, CI can
delete its own production stacks, and no role in MachineTeam.cs can restrain that
— the grant is issued by Pulumi Cloud at creation time, not by anything declared here.
Mitigation is out of band: the org's Allow stack admins to delete stacks toggle, or
accepting it as the price of having no bootstrap step.
The elevated persona's environment grants are a transitive closure, plus creator access¶
A TeamStackPermission says nothing about environments, so the persona needs open grants of
its own — and each one must be the transitive import closure of what it consumes, not just
the environments it names, because Pulumi checks permission at every link of an import chain.
Missing a link fails misleadingly: a refused import resolves to
nothing rather than raising, so it surfaces as unknown property "subscriptionId" rather than a
permission error pointing at the actual missing grant.
Those grants live beside the resource they grant on, so most of them are not in
MachineTeam.cs at all: the per-environment grants sit in EscEnvironmentAccess and the
azure/base grants in AzureBaseEnvironment, next to the environments they open. What remains
on MachineTeam.CreateAdmin as ExternalEnvironments is only the closure with no local
resource to sit beside — today just bootstrap/azure, which this stack consumes but does not
create.
Do not read this persona's blast radius off its grant list
pulumi-config-admin opens every environment infra/pulumi-config manages — via the
creator grant, not its declared grants — while correctly being refused environments the
stack does not manage. When writing a negative assertion for this persona, target an
environment the stack doesn't own, or it will always open.
Effective permissions are a union, so the org-wide defaults matter more than the roles¶
Pulumi Cloud computes effective permissions as the union of a principal's own organization role and every role assigned to teams it belongs to — team roles add to the baseline, they don't replace it. Both teams here resolve to the org service account, which carries the built-in Member role, whose reach for stacks/environments is set org-wide by two UI-only dropdowns under Settings → Access management → Roles → organization-wide role settings (no REST route, no provider resource).
Build 181022 showed what that costs today: the CI team, holding zero stack grants, read an unrelated project's stack in full (metadata, full state export, latest update — all HTTP 200). Environments were correctly refused, pinning the asymmetry to Stack permissions ≥ Read, Environment permissions = None.
Stack scoping is decorative until that dropdown is None
No experiment run against this org can currently prove a stack-level scope is honoured,
because the Member baseline already supplies read regardless of what any role here declares.
Setting Stack permissions to None is a pending, org-level, UI-only change — tracked
separately, not blocking anything documented above.
A related, previously-wrong assumption: a global-resource-type role was thought to reject
entity scopes like stack:read / environment:open. It does not — POSTing one with all four
returns 200 and reads them back; only team:list is dropped, per the warning above. So a global
stack:read (or a tag-based ABAC rule) is a real option for granting StackReference access at
scale, once the dropdown above makes it testable.
The issuer¶
The Pulumi Cloud OIDC issuer is an ordinary managed resource in this stack — created by
pulumi up, not registered by hand. It was originally registered in the UI and adopted with a
one-time ImportId fed by a cloud-foundations:oidcIssuerId config value; both have since been
removed, and a later teardown/rebuild created the issuer from empty, which proved the create
path that adoption had until then kept untested.
Removing the config value retired a genuine footgun. It was load-bearing in a way its name did
not suggest: clearing it made Create return early, which dropped the issuer from the program,
and pulumi up would then delete the registered issuer along with both auth policies —
locking out every persona, including the one that deploys this stack. There is now no config
value to clear.
Never pin the issuer thumbprint
OidcIssuer accepts a Thumbprints list, and reading the live value back into the source
as a constant looks like sound practice. It is a trap: the thumbprint belongs to
Microsoft's TLS certificate, not to this configuration, so pinning it makes the resource
fail the moment that certificate rotates:
error creating oidc issuer: 400 Bad Request:
Calculated thumbprint 5cc383ad… does not match any of the provided.
Adoption hides this — an imported issuer never re-validates — so the failure surfaces only
on the first real create, potentially long after the pin was introduced. The rebuild
demonstrated the volatility directly: the value Pulumi calculated during the failure and the
value it stored minutes later on success were different again. Leave Thumbprints
unset, which is what registering through the UI does.
The issuer URL is still replace-on-change
Url and Organization are replace-on-change and the URL is immutable server-side, so
changing the tenant id re-registers the issuer and takes both policies with it — a lockout
reached a different way. A genuinely new stack creates its own issuer, which is the intended
behaviour; this stack must keep pointing at the tenant it already trusts.
To stop managing the issuer without destroying it, remove it from state with
pulumi state delete rather than deleting the resource from the program.
Policies aren't readable over REST
Every /oidc/issuers/{id}/… policy route 404s, so the live policy list can't be fetched
directly. Read them from pulumi stack export (they are stored as properties of the
OidcIssuer resource) or from pulumi preview --diff.
The provider's own permissions helper is unusable on the current .NET SDK
OrganizationRole.Permissions is a descriptor tree keyed by a __type discriminator, and
the provider ships BuildAllowPermissions to build it. The .NET SDK — not the helper —
strips __-prefixed keys while deserializing invoke results, in
Deserializer.DeserializeStruct. BuildAllowPermissionsResult.Permissions is an untyped
ImmutableDictionary<string, object>, so the discriminator is just a key inside it and gets
filtered out with the rest. The descriptor reaches the provider without it (failed to parse
permission descriptor: type '' not recognized). MachineTeam.cs hand-authors the literal
InputMap<object> instead, which works because the serializer has no matching filter on the
input path — __type is legal outbound and only unreadable inbound.
Still present in Pulumi 3.110.0, in 3.110.1-alpha, and on main; reported upstream as
pulumi/pulumi-dotnet#1102. pulumi/pulumi#22738 is the Python twin, accepted and fixed
there. Provider 1.3.0 renamed the discriminator to kind at the SDK boundary
(pulumi-pulumiservice#778), so the helper may round-trip cleanly now — confirm with a
real invoke before dropping the workaround.
Resource discovery with Pulumi Insights¶
Each environment also gets a Pulumi Insights account that scans its subscription and
indexes what it finds, so the estate can be searched for resources that no Pulumi stack
manages. Environments/InsightsDiscovery.cs creates one InsightsAccount per environment.
It adds no new credential path, which is the whole reason it is cheap. Pulumi's Azure
Insights recipe asks for an OIDC federated credential with audience azure:<org> and
subject pulumi:environments:org:<org>:env:<project>/<env>, plus an ESC environment
exposing fn::open::azure-login with oidc: true. The azure/<env> environments this
stack already creates satisfy that exactly — so the account selects an existing environment
by name rather than provisioning a parallel identity:
That is the console wizard's "Connect using existing ESC credentials" option, expressed in code.
The scanner inherits Owner
Reusing the ESC environment means the scanner authenticates as iac-<env>, which holds
Owner on the subscription. Insights needs only read. This is not a regression —
Pulumi Cloud could already open azure/<env> and mint that token, so attaching Insights
widens what Pulumi Cloud does with a credential it could already obtain, not what it
can obtain. It is still more privilege than a read-only scan warrants, and a natural
place to narrow later.
Two knobs, because a playground is not an inventory¶
The scan schedule has an estate-wide default and a per-environment override:
cloud-foundations:insights:
scanSchedule: daily # optional; daily when absent
cloud-foundations:azure:
environments:
- name: development-sandbox
insightsScanSchedule: none # a playground produces churn, not an inventory
Accepted values are none, 12h and daily. none genuinely disables scanning rather
than merely recording an intention — verified by reading scheduledScanEnabled back from
the API, which is false for the sandbox and true for the other six.
Accounts are named azure-<env>, with a hyphen¶
Insights reads / in an account name as hierarchy, where a scan or a delete on a parent
cascades to its children. The seven subscriptions are peers, not a tree, so azure/<env>
would misdescribe the relationship and make a delete far more dangerous than it looks.
DependsOn here is load-bearing
Environment is passed as a plain string, so Pulumi sees no data dependency on the ESC
environment. Account creation validates the credential immediately and 404s if the
environment does not exist yet, so InsightsDiscovery takes an explicit DependsOn on
it.
Verifying a scan actually ran¶
scanStatus on the account is always empty — do not read it
GET …/accounts/{name} returns a scanStatus object that stays blank
({id: "", status: "", startedAt: "0001-01-01…"}) no matter how many scans have
succeeded. Reading it suggests scanning is broken when it is working perfectly. The real
history is a separate sub-resource:
which returns scanStatuses[] with status, startedAt and finishedAt. All seven
accounts report succeeded, taking roughly two minutes each.
Discovered resources are not under the account — …/accounts/{name}/resources 404s.
They are in the org-wide resource search:
Finding undocumented Pulumi Cloud routes
Note the shape: preview leads the path and there is no orgs segment, so none of
these routes are guessable. Every attempt to infer them returned 404. They were recovered
by scanning the provider plugin binary's string table for route templates:
$exe = "$env:USERPROFILE\.pulumi\plugins\resource-pulumiservice-v1.3.0\pulumi-resource-pulumiservice.exe"
[Text.Encoding]::ASCII.GetString([IO.File]::ReadAllBytes($exe)) |
Select-String -AllMatches 'api/[a-z/{}]*insight[a-z/{}]*' |
ForEach-Object { $_.Matches.Value } | Sort-Object -Unique
The technique generalises to any Pulumi Cloud endpoint the docs do not cover: the provider must contain the literal it calls.
- Pulumi Cloud OIDC documentation
- Pulumi ESC – Configuring OIDC for Azure
- Using OIDC authentication between Azure DevOps and Pulumi Cloud
— useful background on the general pattern, but it describes the
vstoken.dev.azure.comissuer form and a readablesc://subject. Neither matches this tenant; see Reference values for what actually appears on the wire here.