{"id":"GHSA-rjr3-43j3-4q8g","summary":"Kubeflow Pipelines API server lacks authorization for shared pipeline uploads","details":"## Summary\n\nIn multi-user mode, the Kubeflow Pipelines API server's HTTP multipart upload endpoints (`POST /apis/v2beta1/pipelines/upload` and `POST /apis/v2beta1/pipelines/upload_version`) perform no authorization when the target namespace is empty. The authorization helper `canUploadVersionedPipeline` returns `nil`, skipping `SubjectAccessReview`, whenever the resolved namespace is the empty string (`backend/src/apiserver/server/pipeline_upload_server.go:401-403`). An empty namespace is how KFP addresses shared, cluster-wide pipelines and is the default whenever the caller omits the `namespace` query parameter, because `REQUIRE_NAMESPACE_FOR_PIPELINES` defaults to `false` and is unset in every shipped manifest. As a result, an arbitrary low-privilege tenant (or, when the API server is reachable, an unauthenticated caller) can create cluster-shared pipelines and inject new versions into existing shared pipelines, replacing the container image, command, and args that other tenants execute. The gRPC sibling for the same write, `canAccessPipeline`, instead remaps the empty namespace to the system namespace and authorizes, with a comment stating shared-pipeline writes \"must still be authorized\". That confirms this is an omission, not intended behavior.\n\n## Description\n\n`uploadPipeline` and `uploadPipelineVersion` both build a `ResourceAttributes{Verb: create}` and delegate the access decision to `canUploadVersionedPipeline`:\n\n```go\n// backend/src/apiserver/server/pipeline_upload_server.go:387-421\nfunc (s *PipelineUploadServer) canUploadVersionedPipeline(r *http.Request, pipelineID string, resourceAttributes *authorizationv1.ResourceAttributes) error {\n\tif !common.IsMultiUserMode() {\n\t\treturn nil\n\t}\n\tif len(pipelineID) \u003e 0 {\n\t\tnamespace, err := s.resourceManager.FetchNamespaceFromPipelineId(pipelineID)\n\t\tif err != nil {\n\t\t\treturn util.Wrap(err, \"Failed to authorize with the Pipeline ID\")\n\t\t}\n\t\tif len(resourceAttributes.Namespace) == 0 {\n\t\t\tresourceAttributes.Namespace = namespace\n\t\t}\n\t}\n\tif resourceAttributes.Namespace == \"\" {\n\t\treturn nil          // empty namespace = no SubjectAccessReview\n\t}\n\t// ... only here does IsAuthorized() run ...\n}\n```\n\nWhen the namespace is empty (a shared/cluster-wide pipeline, or simply an upload with no `namespace` parameter), the function returns at line 401-403 and `IsAuthorized` is never reached.\n\nReachability is the default. `ValidateNamespaceRequired` only rejects an empty namespace when `IsNamespaceRequiredForPipelines()` is true:\n\n```go\n// backend/src/apiserver/validation/namespace.go:25-30\nfunc ValidateNamespaceRequired(namespace string) error {\n\tif common.IsMultiUserMode() && common.IsNamespaceRequiredForPipelines() && namespace == \"\" {\n\t\treturn util.NewInvalidInputError(\"Namespace is required for pipeline operations\")\n\t}\n\treturn nil\n}\n// backend/src/apiserver/common/config.go:73-74  -\u003e default FALSE\nfunc IsNamespaceRequiredForPipelines() bool {\n\treturn GetBoolConfigWithDefault(RequireNamespaceForPipelines, false)\n}\n```\n\n`REQUIRE_NAMESPACE_FOR_PIPELINES` (`config.go:41`) defaults to `false` and is not set in any manifest under `manifests/`. So in a default multi-user install, empty namespaces pass validation and flow into the unguarded branch.\n\nThe gRPC sibling proves the intended behavior is to authorize empty-namespace writes:\n\n```go\n// backend/src/apiserver/server/pipeline_server.go:1290-1306 (canAccessPipeline)\n\t// Skip authorization for read-only operations on shared pipelines in multi-user mode.\n\t// Write operations (create, update, delete) on shared pipelines must still be authorized\n\t// against the KFP system namespace, since shared pipelines have no namespace of their own.\n\tif s.resourceManager.IsEmptyNamespace(resourceAttributes.Namespace) {\n\t\tif resourceAttributes.Verb == common.RbacResourceVerbGet || resourceAttributes.Verb == common.RbacResourceVerbList {\n\t\t\treturn nil\n\t\t}\n\t\tresourceAttributes.Namespace = common.GetPodNamespace()\n\t\t// ... IsAuthorized(...) ...\n\t\treturn nil\n\t}\n```\n\nReads of shared pipelines intentionally skip authorization; writes must be authorized against the system namespace. The upload endpoints only ever perform writes, yet `canUploadVersionedPipeline` applies the read behavior to them.\n\n## Proof of Concept\n\nReproduced live against the real API server binary (`ghcr.io/kubeflow/kfp-api-server@sha256:621fdae9bfcc93405eca299c24f715bc7f17dbb76f1b53e8b2b35852e4d7d423`, `MULTIUSER=true`, `REQUIRE_NAMESPACE_FOR_PIPELINES` unset). The API server SA holds `subjectaccessreviews/tokenreviews: [create]` (authz fully functional); identity `attacker@evil.com` has no RBAC anywhere; `victim-ns` exists with no grant to the attacker.\n\n1. Shared (empty-namespace) pipeline create as the unprivileged attacker (HTTP 200):\n   ```\n   curl -F uploadfile=@pipeline.yaml -H \"kubeflow-userid: attacker@evil.com\" \\\n     \"http://APISERVER:8888/apis/v2beta1/pipelines/upload?name=shared-pwn\"\n   -\u003e {\"pipeline_id\":\"80cd99df-...\",\"name\":\"shared-pwn\", ...}\n   ```\n2. Inject a new version into that shared pipeline, the high-impact primitive (HTTP 200):\n   ```\n   curl -F uploadfile=@pipeline2.yaml -H \"kubeflow-userid: attacker@evil.com\" \\\n     \"http://APISERVER:8888/apis/v2beta1/pipelines/upload_version?name=evilversion&pipelineid=80cd99df-...\"\n   -\u003e {\"pipeline_version_id\":\"32299f65-...\",\"name\":\"evilversion\",\"pipeline_spec\":{ attacker IR }}\n   ```\n3. Same create with no identity header at all: HTTP 200 (`anon-shared-pwn` created).\n4. Negative control, same attacker, `&namespace=victim-ns` (real namespace, no RBAC): HTTP 400, rejected at `pipeline_upload_server.go:135` (the `canUploadVersionedPipeline` error path). Authorization is live; only the empty-namespace branch bypasses it.\n5. Asymmetry, gRPC `POST /apis/v2beta1/pipelines` with empty namespace as the same attacker: HTTP 403 PermissionDenied (`ResourceAttributes{Namespace:kubeflow,Verb:create,...}`). The gRPC path remapped empty to the system namespace and authorized.\n\nAPI server logs corroborate: every empty-namespace upload logs `Upload pipeline called` with no following `Authorizing request`, while every namespaced upload and every gRPC create logs `Authorizing request` then `PermissionDenied`. Database state persists the attacker's `shared-pwn` and `anon-shared-pwn` pipelines and the injected `evilversion`; the denied `ns-pwn` created nothing.\n\nFull scaffolding (`run.sh`, `01-mysql.yaml` through `05-apiserver.yaml`, `pipeline.yaml`, `pipeline2.yaml`) and captured evidence are available on request.\n\n## Impact\n\nIn multi-user deployments that permit shared pipelines, a low-privilege tenant can create shared pipelines or add attacker-controlled versions to existing shared pipelines without the required write permission. Direct access to the API server without an authenticating proxy can also expose the bypass to unauthenticated callers; the CVSS vector below assesses the authenticated tenant scenario.\n\nAn attacker can upload a malicious version of an existing shared pipeline consumed by a recurring run configured to follow the latest version. On its next scheduled execution, the controller resolves the newly uploaded version and executes it in the consuming run's namespace using its configured service account, without further user interaction. Recurring runs pinned to a specific version or containing an embedded specification are not subject to this automatic substitution.\n\nThis automatic path requires an enabled, unpinned recurring run consuming the targeted shared pipeline and an uploaded specification compatible with the run's inputs. Latest-version selection queries the newest ready pipeline version; it does not depend on the `UpdatePipelineVersionByDefault` setting.\n\nThe automatic execution path was verified by source inspection, including the `2.17.2` tag: recurring-run creation preserves an unpinned pipeline reference, the scheduled-workflow controller submits it to `CreateRun`, and the API resolves the latest version before constructing the workflow. This is distinct from the live upload-bypass reproduction above; an end-to-end scheduled victim-execution reproduction has not been performed as part of this maintainer assessment.\n\nExecution occurs with the victim workload's permissions and access to its mounted data and credentials. It can compromise confidential data and execution integrity available to that workload. It does not inherently grant access to every secret in the victim namespace or cluster-admin privileges.\n\n### CVSS rationale\n\nThe retained CVSS v3.1 vector is **9.6 Critical — `CVSS:3.1/AV:N/AC:L/PR:L/UI:N/S:C/C:H/I:H/A:N`**.\n\n- **AV:N / AC:L / PR:L:** an authenticated low-privilege tenant can submit a normal multipart version upload targeting a shared pipeline, without shared-pipeline write permission.\n- **UI:N:** an existing unpinned recurring run can consume the injected version automatically; no further victim action is required.\n- **S:C:** the scored consequence extends from unauthorized modification of the shared pipeline catalog to code execution under a separate victim workload's Kubernetes namespace and service-account authority, not merely modification of another catalog entry.\n- **C:H / I:H:** attacker-controlled workload code can compromise the confidentiality and integrity of resources available under that victim execution authority. These impacts do not imply cluster-wide access.\n- **A:N:** this vector does not claim an independently established availability impact.\n\n## Patches / Remediation\n\nThe fix merged into `master` on September 9, 2026 in [kubeflow/pipelines#14352](https://github.com/kubeflow/pipelines/pull/14352), commit [62516d8c525f146b754cb515cf441dd0290ad841](https://github.com/kubeflow/pipelines/commit/62516d8c525f146b754cb515cf441dd0290ad841).\n\nThe upload authorization helper now uses `ResourceManager.IsEmptyNamespace` to recognize both the empty string and the legacy `model.NoNamespace` (`\"-\"`) representation of shared pipelines. It checks shared writes against the configured KFP system namespace through `IsAuthorized`, requiring `create` on the `pipelines` resource in the `pipelines.kubeflow.org` API group. This matches the gRPC shared-write policy without changing the stored pipeline namespace. Namespaced authorization, single-user behavior, and intentional latest-version scheduling remain unchanged.\n\nThe original fix by Vito Carolillo (@vcrll) was carried forward with its authorship and DCO sign-off preserved. Separate maintainer commits added regression coverage, sentinel handling, and test-harness compatibility corrections.\n\n### Validation\n\n- 32 endpoint regression cases cover both v1beta1 and v2beta1 pipeline and pipeline-version uploads: allowed and denied shared writes (empty and sentinel namespaces), namespaced writes, and single-user behavior. Tests verify exact SubjectAccessReview attributes and preservation of stored namespaces.\n- A negative control restoring the original early return caused the new shared-upload tests to fail, including denied requests returning HTTP 200 without an authorization check.\n- The full server unit-test package and changed-package lint passed during remediation.\n- E2E test clients were corrected to send their generated authentication token and target the tenant namespace for HTTP uploads. Kubernetes-native clients retain their constructor-configured namespace rather than receiving an unsupported upload parameter. Helper regression tests and E2E/API package compilation passed.\n\n### Release status and upgrade guidance\n\nThe first fixed release is **[2.18.0](https://github.com/kubeflow/pipelines/releases/tag/2.18.0)**. Releases through **2.17.3** remain affected; the affected-version upper bound is `\u003c= 2.17.3` and the patched version is `2.18.0`.\n\nThe 2.18 [RBAC and client migration guide](https://github.com/kubeflow/pipelines/blob/2.18.0/docs/operator-guides/rbac-migration-2.18.md) is included through [PR #14655](https://github.com/kubeflow/pipelines/pull/14655). Related trusted-scheduling changes do not replace upload authorization or remove previously uploaded versions. Roll out the matching server/manifests and review existing shared versions and schedules.\n\nIn multi-user deployments, callers that previously relied on unauthenticated or unauthorized shared uploads will now be rejected:\n\n- **UI:** choose **Private** (the default in the upload dialog) for tenant-owned pipelines. **Shared** uploads require an authorized shared-pipeline writer.\n- **Python SDK:** explicitly pass the tenant namespace, for example `client.upload_pipeline(\"pipeline.yaml\", namespace=\"my-namespace\")`. Setting `Client(namespace=...)` alone does not populate this upload argument.\n- **New versions:** inherit the existing pipeline's namespace. Existing shared pipelines must be maintained by an identity with the required system-namespace pipeline permission, or re-uploaded as new namespaced pipelines with consuming runs and schedules repointed.\n- **Kubernetes-native uploads:** retain their existing client-configured namespace contract.\n\nDo not grant blanket system-namespace access or restore the authorization bypass to preserve old upload scripts. Grant only the required pipeline-write permission to designated shared-pipeline maintainers.\n\nAs an interim hardening measure before deploying the fix, operators can set `REQUIRE_NAMESPACE_FOR_PIPELINES=true` to reject empty-namespace pipeline operations, with the corresponding compatibility impact on shared uploads.\n\nRestoring upload authorization does not remove previously uploaded malicious versions or migrate existing pipelines and runs. Operators should review shared-pipeline versions and consuming recurring runs for unauthorized changes; pinning to a reviewed version prevents subsequent latest-version substitution.\n\nThe visualization authorization issue is tracked separately in [GHSA-w9j6-6675-75m4](https://github.com/kubeflow/pipelines/security/advisories/GHSA-w9j6-6675-75m4) and is outside this fix's scope.\n\n## Resources\n\n- `backend/src/apiserver/server/pipeline_upload_server.go:387-421` (`canUploadVersionedPipeline`; bypass at `:401-403`), `:118-138`, `:290-312`\n- `backend/src/apiserver/server/pipeline_server.go:1290-1306` (`canAccessPipeline`; load-bearing comment `:1290-1292`)\n- `backend/src/apiserver/validation/namespace.go:25-30`\n- `backend/src/apiserver/common/config.go:41`, `:73-74`\n- `backend/src/apiserver/resource/resource_manager.go` (`CreateJob`, `fetchPipelineVersionFromPipelineSpec`, `CreateRun`)\n- `backend/src/crd/controller/scheduledworkflow/controller.go` (`submitNewWorkflowIfNotAlreadySubmitted`)\n- `backend/src/apiserver/storage/pipeline_store.go` (`GetLatestPipelineVersion`)\n- Distinct from CVE-2023-6570 and CVE-2023-6571 (both in `kubeflow/kubeflow`, different components and classes).\n\n\u003c!-- maintainer-family-cleanup:2026-09-04 --\u003e\n### Maintainer disposition\n\nThis is the canonical advisory for the **shared pipeline upload authorization** family. Related submissions closed as duplicates or superseded by this canonical:\n\n- [GHSA-87q5-j5jh-fhp8](https://github.com/kubeflow/pipelines/security/advisories/GHSA-87q5-j5jh-fhp8)\n\nEach linked submission retains its own reporter attribution. A cross-link does not transfer authorship or grant co-reporter credit on this advisory. The fix merged through [#14352](https://github.com/kubeflow/pipelines/pull/14352) on September 9, 2026. The fix is available in 2.18.0. This advisory remains a maintained draft pending publication; CVE-2026-77596 and the 9.6 CVSS score are unchanged.\n\n### Private patch disposition\n\nThe earlier [private PR #1](https://github.com/kubeflow/pipelines-ghsa-rjr3-43j3-4q8g/pull/1) is closed as superseded by [public PR #14352](https://github.com/kubeflow/pipelines/pull/14352) and the released follow-ups described above. The private branch and review history are retained; this closure does not dismiss the report or change reporter credit. The advisory remains a draft pending the publication review.","aliases":["CVE-2026-77596"],"modified":"2026-11-09T22:30:04.085079221Z","published":"2026-11-09T22:22:41Z","database_specific":{"github_reviewed_at":"2026-11-09T22:22:41Z","nvd_published_at":"2026-11-05T23:17:00Z","cwe_ids":["CWE-639","CWE-862"],"severity":"CRITICAL","github_reviewed":true},"references":[{"type":"WEB","url":"https://github.com/kubeflow/pipelines/security/advisories/GHSA-rjr3-43j3-4q8g"},{"type":"ADVISORY","url":"https://nvd.nist.gov/vuln/detail/CVE-2026-77596"},{"type":"WEB","url":"https://github.com/kubeflow/pipelines/pull/14352"},{"type":"WEB","url":"https://github.com/kubeflow/pipelines/commit/62516d8c525f146b754cb515cf441dd0290ad841"},{"type":"PACKAGE","url":"https://github.com/kubeflow/pipelines"},{"type":"WEB","url":"https://github.com/kubeflow/pipelines/releases/tag/2.18.0"}],"affected":[{"package":{"name":"github.com/kubeflow/pipelines","ecosystem":"Go","purl":"pkg:golang/github.com/kubeflow/pipelines"},"ranges":[{"type":"SEMVER","events":[{"introduced":"0"},{"fixed":"0.0.0-20260909234119-62516d8c525f"}]}],"database_specific":{"source":"https://github.com/github/advisory-database/blob/main/advisories/github-reviewed/2026/11/GHSA-rjr3-43j3-4q8g/GHSA-rjr3-43j3-4q8g.json"}}],"schema_version":"1.9.2","severity":[{"type":"CVSS_V3","score":"CVSS:3.1/AV:N/AC:L/PR:L/UI:N/S:C/C:H/I:H/A:N"}]}