Skip to content

feat(e2e): AGX1-331 — event route authz black-box suite#277

Closed
dm36 wants to merge 1 commit into
mainfrom
dhruv/agx1-331-e2e-event-authz
Closed

feat(e2e): AGX1-331 — event route authz black-box suite#277
dm36 wants to merge 1 commit into
mainfrom
dhruv/agx1-331-e2e-event-authz

Conversation

@dm36

@dm36 dm36 commented Jun 4, 2026

Copy link
Copy Markdown
Contributor

Summary

Establishes the spark-authz e2e test harness in scripts/spark-authz-e2e-tests/ and ships the first resource: events (AGX1-331). Subsequent PRs (api_keys / AGX1-325 — see #276 — and future resources) layer test files on top of this infrastructure.

Models the EGP suite shape from scaleapi#142983 — same clients, conftest pattern, cleanup-tracker, config.json layout.

Linear: AGX1-331.

What this PR ships

Scaffolding (reused by all future resource test files):

  • clients/agentex_client.py — HTTP client for scale-agentex routes (agents + api_keys + events). api_key methods land here ahead of AGX1-325 so the client doesn't grow in two places.
  • clients/spark_authz_client.py — direct SpiceDB-state client (HTTP-transcoded). Repo-agnostic; copied verbatim from the EGP suite.
  • conftest.py — config loader, identity credentials, two AgentexClient instances (user_a / user_b), SparkAuthzClient with graceful-skip semantics (authz_reachable probes /healthz, returns True only on 2xx), parent_agent fixture that falls back to a pre-existing agentex.agent_id when the test user lacks agent.create, function-scoped cleanup tracker.
  • helpers/{cleanup,factories}.py — LIFO teardown + unique-name generators.
  • Makefile / README.md — four-terminal local-run recipe + per-resource + logical-group targets.

AGX1-331 tests (tests/test_event_authz.py):

Test What it verifies
test_get_event_nonexistent_returns_404 Sanity: ItemDoesNotExist short-circuits before any authz fires
test_list_events_without_agent_view_returns_404 user_b lacks read on user_a's agent → DAuthorizedQuery on agent_id collapses denial to 404
test_list_events_denied_on_both_query_params_returns_404 Route is gated end-to-end (does not isolate which gate fires — would need SpiceDB-side grants)
test_get_event_with_view_returns_200 Skipped — no public POST /events; would need an event-seeding helper out of scope for an authz-focused PR

Verified against pubsec-dev

Ran against agentex.sgp-pubsec-dev.scale.com on 2026-06-04:

tests/test_event_authz.py::test_get_event_nonexistent_returns_404                       PASSED
tests/test_event_authz.py::test_list_events_without_agent_view_returns_404              PASSED
tests/test_event_authz.py::test_list_events_denied_on_both_query_params_returns_404     PASSED
tests/test_event_authz.py::test_get_event_with_view_returns_200                         SKIPPED
                                                                       3 passed, 1 skipped

Setup used:

  • config.json with a real pubsec-dev API key + the EGP account_id as x-selected-account-id.
  • agentex.agent_id set to a pre-existing agent (the user lacks agent.create on the tenant wildcard, so parent_agent falls back to the configured id).
  • spark_authz.host pointed at localhost:8090 via kubectl port-forward to ns/spark/svc/spark-authz. The event tests don't assert against SpiceDB directly, so they pass without it too; the port-forward is only required when api_key SpiceDB-asserting tests get added on top (see feat(e2e): AGX1-325 — agent_api_keys authz black-box suite #276).

Out of scope

Greptile Summary

Introduces a black-box FGAC test suite for the events routes (GET /events/{id} and GET /events), along with all the scaffolding — httpx clients, conftest fixtures, LIFO cleanup tracker, and a Makefile with per-resource and aggregate targets — needed to host future sub-resource test files.

  • Three denied-path tests are implemented and verified against pubsec-dev; the happy-path test is correctly skipped with an explicit reason (no public POST /events to seed test data), leaving a clear hook for whoever adds an event-seeding helper later.
  • The parent_agent fixture provides a fallback to a pre-existing agentex.agent_id from config for environments where the test user lacks agent.create, and the authz_client fixture degrades gracefully to a skip when spark-authz isn't reachable.

Confidence Score: 5/5

This PR adds only a new test script directory with no changes to production application code — it is safe to merge.

All changes are confined to a standalone e2e test suite under scripts/. The denied-path assertions are straightforward and were verified against pubsec-dev. No production code, migrations, or auth logic is touched.

No files require special attention.

Important Files Changed

Filename Overview
scripts/spark-authz-e2e-tests/tests/test_event_authz.py Four tests covering denied-path authz for GET /events/{id} and GET /events; happy-path test is correctly skipped with a clear reason (though the TODO is missing a Linear ticket ID).
scripts/spark-authz-e2e-tests/conftest.py Session-scoped clients, function-scoped cleanup tracker, parent_agent fallback — well-structured with correct fixture scopes and graceful skip semantics for unreachable authz server.
scripts/spark-authz-e2e-tests/clients/agentex_client.py Thin httpx wrapper covering agents, events (AGX1-331), and api_keys; returns raw responses for caller-controlled assertions.
scripts/spark-authz-e2e-tests/clients/spark_authz_client.py HTTP-transcoded gRPC client for SpiceDB via spark-authz REST; covers check, grant, revoke, create, delete, and lookup operations.
scripts/spark-authz-e2e-tests/config.json.example Template config missing the optional agentex.agent_id field used by the parent_agent fixture fallback path.
scripts/spark-authz-e2e-tests/helpers/cleanup.py LIFO cleanup tracker — swallows exceptions and logs failures so cleanup errors never mask test results.
scripts/spark-authz-e2e-tests/Makefile Organized Makefile with per-resource and aggregate grouping targets; extensible for future sub-resource test files.

Sequence Diagram

sequenceDiagram
    participant TB as Test (user_b)
    participant AX as scale-agentex
    participant AA as agentex-auth
    participant SDB as SpiceDB (via spark-authz)

    TB->>AX: "GET /events?agent_id=X&task_id=Y"
    AX->>AA: resolve principal (forward headers)
    AA-->>AX: "{user_id, account_id}"
    AX->>SDB: DAuthorizedQuery: check agent:X read for user_b
    SDB-->>AX: DENIED (no tuple)
    AX-->>TB: 404 (existence not leaked)

    Note over TB,SDB: GET /events/{nonexistent_id} short-circuits before authz
    TB->>AX: GET /events/00000000-…
    AX->>AX: repo lookup → ItemDoesNotExist
    AX-->>TB: 404 (no authz call)
Loading

Fix All in Cursor Fix All in Claude Code Fix All in Codex

Prompt To Fix All With AI
Fix the following 2 code review issues. Work through them one at a time, proposing concise fixes.

---

### Issue 1 of 2
scripts/spark-authz-e2e-tests/tests/test_event_authz.py:84
The `TODO follow-up` comment inside the skip reason doesn't reference a Linear ticket number. Per the team's rules, TODO comments should include a linked ticket ID so the work can be found and tracked (e.g., `cmd+f` on a ticket number). The placeholder text "Linear: TODO follow-up" won't survive a search once the next sprint is underway.

```suggestion
            "once a test-only seeding helper exists (Linear: AGX1-<ticket>)."
```

### Issue 2 of 2
scripts/spark-authz-e2e-tests/config.json.example:27-32
The `parent_agent` fixture in `conftest.py` looks up `config.get("agentex", {}).get("agent_id")` as a fallback for environments where the test user lacks `agent.create` permission (e.g., pubsec-dev). This key is documented in the PR description and README but is absent from the example config, which is the first reference most users will reach for. Adding a commented-out block prevents them from having to grep the source.

```suggestion
  "test_settings": {
    "request_timeout_seconds": 30,
    "cleanup_on_success": true,
    "cleanup_on_failure": true
  },
  "agentex": {
    "agent_id": ""
  }
}
```

Reviews (7): Last reviewed commit: "feat(e2e): AGX1-331 — event authz black-..." | Re-trigger Greptile

Context used:

  • Rule used - Include ticket numbers in TODO comments to make cl... (source)

Learned From
scaleapi/scaleapi#126926

  • Rule used - Create Linear tasks for TODO comments in code and ... (source)

Learned From
scaleapi/scaleapi#127117

dm36 added a commit that referenced this pull request Jun 4, 2026
Set up the conventions for future resources before scope grows:

- test-direct-resources: resources with their own SpiceDB type
  (currently api_key; agent and task slot in here later).
- test-sub-resources: resources that delegate authz to a parent
  (currently event; state, message, tracker, checkpoint slot in here).
- test-<resource>: all cases for one resource.
- test-<resource>-<case>: one case for one resource.

Aggregate groupings (DIRECT_RESOURCE_TESTS, SUB_RESOURCE_TESTS) are
declared at the top so future PRs add a single line per resource rather
than churning the target list.

Added `make help` listing the convention. README's Run section rewritten
to match.

This commit lays the groundwork; AGX1-331 (PR #277) will pick up the
new shape on rebase — its `test-events` target becomes `test-event`
aligned with the singular-resource naming convention.
@dm36 dm36 force-pushed the dhruv/agx1-331-e2e-event-authz branch 4 times, most recently from 14d1221 to 430b1e8 Compare June 4, 2026 22:07
@dm36 dm36 marked this pull request as ready for review June 4, 2026 22:15
@dm36 dm36 requested a review from a team as a code owner June 4, 2026 22:15
Comment thread scripts/spark-authz-e2e-tests/tests/test_event_authz.py Outdated
dm36 added a commit that referenced this pull request Jun 4, 2026
Per Greptile review on #277: the test's previous name + docstring claimed
to verify "BOTH query params are gated" by passing zero-UUIDs for both
task_id and agent_id, but FastAPI evaluates the ``task_id`` Depends first
and short-circuits before the agent_id gate runs. The test would still
pass if the agent_id gate were removed entirely — it doesn't isolate
either gate.

Rename and rewrite the docstring to be honest about what it actually
verifies: end-to-end route gating, no per-gate isolation. Independent
isolation would require granting one resource but not the other in
SpiceDB, which depends on a reachable spark-authz (out of scope today
per Gap 1 in #276 — pubsec-dev runs raw SpiceDB without spark-authz).
dm36 added a commit that referenced this pull request Jun 4, 2026
Per Greptile review: ``make test-sub-resources`` and ``make test-event``
referenced ``tests/test_event_authz.py``, which only exists on #277.
Running either target on this branch in isolation produced
``ERROR: not found: tests/test_event_authz.py`` and a non-zero exit.

Remove ``EVENT_TESTS``, ``SUB_RESOURCE_TESTS``, ``test-sub-resources``,
``test-event``, plus their references in ``.PHONY``, the ``help`` block,
and the README ``Run`` section. The Makefile's header comment retains
the convention so #277 (and follow-up sub-resource tickets) can re-add
the targets when their test files actually land.
dm36 added a commit that referenced this pull request Jun 4, 2026
Per Greptile review on #277: the test's previous name + docstring claimed
to verify "BOTH query params are gated" by passing zero-UUIDs for both
task_id and agent_id, but FastAPI evaluates the ``task_id`` Depends first
and short-circuits before the agent_id gate runs. The test would still
pass if the agent_id gate were removed entirely — it doesn't isolate
either gate.

Rename and rewrite the docstring to be honest about what it actually
verifies: end-to-end route gating, no per-gate isolation. Independent
isolation would require granting one resource but not the other in
SpiceDB, which depends on a reachable spark-authz (out of scope today
per Gap 1 in #276 — pubsec-dev runs raw SpiceDB without spark-authz).
dm36 added a commit that referenced this pull request Jun 4, 2026
…y exists

#276's Greptile-driven cleanup removed test-event / test-sub-resources
from the Makefile to avoid dangling targets that referenced a file
which doesn't exist on that PR's branch. This PR adds the test file,
so it now re-adds the matching targets + README references.

After this commit (#277 in isolation):
  make test-direct-resources    # api_key tests
  make test-sub-resources       # event tests
  make test-api-key             # all AGX1-325 cases
  make test-event               # all AGX1-331 cases
@dm36 dm36 force-pushed the dhruv/agx1-331-e2e-event-authz branch from f4c0961 to 2782f35 Compare June 4, 2026 22:27
dm36 added a commit that referenced this pull request Jun 4, 2026
Per Greptile review on #277: the test's previous name + docstring claimed
to verify "BOTH query params are gated" by passing zero-UUIDs for both
task_id and agent_id, but FastAPI evaluates the ``task_id`` Depends first
and short-circuits before the agent_id gate runs. The test would still
pass if the agent_id gate were removed entirely — it doesn't isolate
either gate.

Rename and rewrite the docstring to be honest about what it actually
verifies: end-to-end route gating, no per-gate isolation. Independent
isolation would require granting one resource but not the other in
SpiceDB, which depends on a reachable spark-authz (out of scope today
per Gap 1 in #276 — pubsec-dev runs raw SpiceDB without spark-authz).
dm36 added a commit that referenced this pull request Jun 4, 2026
…y exists

#276's Greptile-driven cleanup removed test-event / test-sub-resources
from the Makefile to avoid dangling targets that referenced a file
which doesn't exist on that PR's branch. This PR adds the test file,
so it now re-adds the matching targets + README references.

After this commit (#277 in isolation):
  make test-direct-resources    # api_key tests
  make test-sub-resources       # event tests
  make test-api-key             # all AGX1-325 cases
  make test-event               # all AGX1-331 cases
@dm36 dm36 force-pushed the dhruv/agx1-331-e2e-event-authz branch from 2782f35 to fd3f448 Compare June 4, 2026 22:29
dm36 added a commit that referenced this pull request Jun 5, 2026
Per Greptile review on #277: the test's previous name + docstring claimed
to verify "BOTH query params are gated" by passing zero-UUIDs for both
task_id and agent_id, but FastAPI evaluates the ``task_id`` Depends first
and short-circuits before the agent_id gate runs. The test would still
pass if the agent_id gate were removed entirely — it doesn't isolate
either gate.

Rename and rewrite the docstring to be honest about what it actually
verifies: end-to-end route gating, no per-gate isolation. Independent
isolation would require granting one resource but not the other in
SpiceDB, which depends on a reachable spark-authz (out of scope today
per Gap 1 in #276 — pubsec-dev runs raw SpiceDB without spark-authz).
@dm36 dm36 force-pushed the dhruv/agx1-331-e2e-event-authz branch from fd3f448 to e4fdf0c Compare June 5, 2026 19:16
dm36 added a commit that referenced this pull request Jun 5, 2026
…y exists

#276's Greptile-driven cleanup removed test-event / test-sub-resources
from the Makefile to avoid dangling targets that referenced a file
which doesn't exist on that PR's branch. This PR adds the test file,
so it now re-adds the matching targets + README references.

After this commit (#277 in isolation):
  make test-direct-resources    # api_key tests
  make test-sub-resources       # event tests
  make test-api-key             # all AGX1-325 cases
  make test-event               # all AGX1-331 cases
dm36 added a commit that referenced this pull request Jun 5, 2026
Per Greptile review on #277: the test's previous name + docstring claimed
to verify "BOTH query params are gated" by passing zero-UUIDs for both
task_id and agent_id, but FastAPI evaluates the ``task_id`` Depends first
and short-circuits before the agent_id gate runs. The test would still
pass if the agent_id gate were removed entirely — it doesn't isolate
either gate.

Rename and rewrite the docstring to be honest about what it actually
verifies: end-to-end route gating, no per-gate isolation. Independent
isolation would require granting one resource but not the other in
SpiceDB, which depends on a reachable spark-authz (out of scope today
per Gap 1 in #276 — pubsec-dev runs raw SpiceDB without spark-authz).
dm36 added a commit that referenced this pull request Jun 5, 2026
…y exists

#276's Greptile-driven cleanup removed test-event / test-sub-resources
from the Makefile to avoid dangling targets that referenced a file
which doesn't exist on that PR's branch. This PR adds the test file,
so it now re-adds the matching targets + README references.

After this commit (#277 in isolation):
  make test-direct-resources    # api_key tests
  make test-sub-resources       # event tests
  make test-api-key             # all AGX1-325 cases
  make test-event               # all AGX1-331 cases
@dm36 dm36 force-pushed the dhruv/agx1-331-e2e-event-authz branch from e4fdf0c to c6ab6da Compare June 5, 2026 21:43
Establishes the spark-authz e2e test harness in
``scripts/spark-authz-e2e-tests/`` and ships the first resource:
events (AGX1-331). Subsequent PRs (api_keys / AGX1-325, future
resources) layer test files on top of this infrastructure.

Models the EGP suite shape from scaleapi#142983 — same clients,
conftest pattern, cleanup-tracker, config.json layout.

What's in this PR
-----------------
- clients/agentex_client.py: HTTP client for scale-agentex routes
  (agents + api_keys + events). api_key methods land here ahead of
  AGX1-325 so the client doesn't grow in two places.
- clients/spark_authz_client.py: direct SpiceDB-state client (HTTP-
  transcoded). Repo-agnostic; copied verbatim from the EGP suite.
- conftest.py: config loader, identity credentials, two AgentexClient
  instances (user_a / user_b), SparkAuthzClient with graceful-skip
  semantics (authz_reachable probes /healthz, returns True only on
  2xx), ``parent_agent`` fixture that falls back to a pre-existing
  ``agentex.agent_id`` when the test user lacks ``agent.create``,
  function-scoped cleanup tracker.
- helpers/{cleanup,factories}.py: LIFO teardown + unique-name gens.
- Makefile / README: four-terminal local-run recipe + per-resource
  + logical-group targets.
- tests/test_event_authz.py: AGX1-331 event-route tests.

AGX1-331 scope: event get/list delegated to parent agent view
(read-only); no agent view -> 404 on get and filtered/empty list.

Tests (test_event_authz.py)
---------------------------
- test_get_event_nonexistent_returns_404 — sanity: ItemDoesNotExist
  short-circuits before any authz fires.
- test_list_events_without_agent_view_returns_404 — user_b lacks
  read on user_a's agent -> DAuthorizedQuery collapses to 404.
- test_list_events_denied_on_both_query_params_returns_404 — route
  is gated end-to-end (does not isolate which gate fires; isolation
  needs SpiceDB-side grants).
- test_get_event_with_view_returns_200 — SKIPPED. No public
  POST /events; would need an event-seeding helper that doesn't
  belong in an authz-scoped e2e PR.

Verified against pubsec-dev
---------------------------
3 passed / 1 skipped / 0 failed against
``agentex.sgp-pubsec-dev.scale.com`` on 2026-06-04 with
``spark_authz.host`` port-forwarded to the in-cluster spark-authz
(``kubectl -n spark port-forward svc/spark-authz 8090:8090``).

Linear: AGX1-331. Companion follow-up: AGX1-325 (api_keys) lands its
test files on top of this scaffolding once pubsec-dev's agentex-auth
is redeployed with AUTH_PROVIDER=spark (deployment-config blocker
tracked separately in scaleapi/sgp#3334).
@dm36 dm36 changed the base branch from dhruv/agx1-325-e2e-api-key-authz to main June 8, 2026 17:31
@dm36 dm36 force-pushed the dhruv/agx1-331-e2e-event-authz branch from c6ab6da to cfa002d Compare June 8, 2026 17:32
@rpatel-scale

rpatel-scale commented Jun 8, 2026

Copy link
Copy Markdown
Contributor

Can we just reuse https://github.com/scaleapi/scale-agentex/tree/3a75e52a042e8aa9928fab66867f43ff6c71771b/agentex/tests/integration. I guess the main difference is that these changes hit REST api but integration tests hit isolated_client which is an httpx.AsyncClient wired to the real FastAPI app via ASGITransport

return f"{scheme}://{self.host}"


class SparkAuthzClient:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Spark AuthZ is a downstream microservice, we shouldnt be surfacing the client to be hit directly here. It should be tested in isolation downstream. Here we just want to test agentex functionality (if that in turns calls spark thats fine)

@dm36

dm36 commented Jun 8, 2026

Copy link
Copy Markdown
Contributor Author

Closing — moving this suite to the private scaleapi/agentex repo to avoid publishing internal FGAC surface area (resource types, dual-write semantics, 404-collapse-from-403 rationale) in a public repo.

Replacement: scaleapi/agentex#365 — same scaffolding + AGX1-331 event tests, with the reviewer's spark-authz-boundary fix already applied (no SparkAuthzClient, no direct downstream calls; all assertions go through agentex HTTP).

The AGX1-325 api_keys follow-up is scaleapi/agentex#366, stacked on #365.

@dm36 dm36 closed this Jun 8, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants