Skip to content

feat: add ValkeyMemoryService with vector similarity search (Valkey Search module)#156

Open
daric93 wants to merge 7 commits into
google:mainfrom
daric93:feat/valkey-memory-service
Open

feat: add ValkeyMemoryService with vector similarity search (Valkey Search module)#156
daric93 wants to merge 7 commits into
google:mainfrom
daric93:feat/valkey-memory-service

Conversation

@daric93
Copy link
Copy Markdown

@daric93 daric93 commented Jun 5, 2026

Summary

Closes: #155

Implements ValkeyMemoryService for the ADK community memory module, backed by Valkey using the Valkey Search module for vector similarity search (HNSW). Uses the valkey-glide client library (v2.4.0+). This provides functionality analogous to VertexAiRagMemoryService for developers with Valkey infrastructure.

Changes

  • src/google/adk_community/memory/valkey_memory_service.py — New memory service implementing BaseMemoryService
    • ValkeyMemoryServiceConfig: Pydantic config with similarity_top_k, vector_distance_threshold, embedding_dimensions, key_prefix, index_name, distance_metric, ttl_seconds
    • ValkeyMemoryService: Accepts a configurable async embedding function (users bring their own embedder)
    • Stores memories as Valkey Hash keys with FLOAT32 vector embeddings
    • Uses FT.CREATE with VECTOR field (HNSW algorithm) + TAG fields for app_name/user_id
    • Uses FT.SEARCH with KNN for vector similarity retrieval, pre-filtered by TAG
    • Implements add_session_to_memory and add_events_to_memory
    • Configurable distance threshold, distance metric (COSINE/L2/IP), TTL
    • Batch embedding generation for efficient ingestion
    • Pipelined writes via Batch for single round-trip ingestion
    • asyncio.Lock with double-check locking for index creation
    • Comprehensive TAG character escaping (including ? wildcard)
  • src/google/adk_community/memory/__init__.py — Exports ValkeyMemoryService and ValkeyMemoryServiceConfig
  • src/google/adk_community/memory/README.md — Documentation with usage examples
  • pyproject.toml — Added valkey-glide>=2.4.0 as optional dependency under [valkey] extra
  • tests/unittests/memory/test_valkey_memory_service.py — 46 unit tests (mocked client)
  • tests/integration/test_valkey_memory_service_integration.py — 12 integration tests against live Valkey with Search module

How it works

  1. Ingestion (add_session_to_memory / add_events_to_memory): text is extracted from events, embeddings are generated in batch via the user-provided embedding function, and each event is stored as a Valkey Hash containing the text content, metadata, and the embedding vector. All writes are pipelined via Batch for a single network round-trip.

  2. Search (search_memory): an embedding is generated for the query text, then FT.SEARCH performs a KNN search over the HNSW index with pre-filtering by app_name and user_id TAG fields. Results are ranked by vector distance and optionally filtered by vector_distance_threshold.

Testing plan

Unit tests (46): All public methods tested with mocked valkey-glide client

pytest tests/unittests/memory/test_valkey_memory_service.py -v
46 passed

Integration tests (12): Run against Valkey 9.1 (valkey-bundle image with Search module)

podman run -d --name valkey-test -p 6379:6379 valkey/valkey-bundle:9.1
pytest tests/integration/test_valkey_memory_service_integration.py -v
12 passed

Integration test coverage:

  • Vector similarity search (add + KNN search)
  • Results ranked by similarity
  • User isolation (TAG filter on user_id)
  • App isolation (TAG filter on app_name)
  • Multiple sessions accumulate
  • Empty store search
  • similarity_top_k limit enforcement
  • Function call / empty events filtered
  • Memory entry metadata preserved (author, timestamp)
  • create_index idempotency
  • add_events_to_memory incremental ingestion
  • vector_distance_threshold filtering

Requirements

  • Valkey server with the Search module loaded (e.g., valkey/valkey-bundle image)
  • valkey-glide >= 2.4.0
  • An async embedding function (e.g., OpenAI, Google Gemini, sentence-transformers)

Ref: AEA-497

daric93 added 7 commits June 2, 2026 11:13
…eMemoryService backed by Valkey using the valkey-glide client library. Stores memories as JSON in Valkey lists keyed by app_name and user_id, with simple text-based substring search. - ValkeyMemoryServiceConfig: configurable search_top_k, key_prefix, ttl - ValkeyMemoryService: add_session_to_memory, search_memory, close - Optional dependency: valkey-glide>=2.4.0 under [valkey] extra - 23 unit tests (mocked client) - 5 integration tests (requires running Valkey instance) Ref: AEA-497

Signed-off-by: Daria Korenieva <daric2612@gmail.com>
…s the simple substring-matching implementation with full-text search powered by the Valkey Search module (FT.CREATE / FT.SEARCH). Changes: - Memories stored as Valkey Hash keys (indexed automatically) - FT.CREATE with TEXT field for content, TAG fields for app_name/user_id - FT.SEARCH for full-text search with TAG filtering - Expanded integration tests (11 tests covering isolation, top_k, etc.) - Added memory module README with usage documentation Ref: AEA-497

Signed-off-by: Daria Korenieva <daric2612@gmail.com>
…) Replaces full-text search with vector similarity search powered by the Valkey Search module, matching VertexAiRagMemoryService in functionality. Key changes: - Configurable embedding function (users bring their own embedder) - FT.CREATE with VECTOR field (HNSW, FLOAT32) for KNN search - FT.SEARCH with KNN pre-filtered by app_name/user_id TAG fields - Configurable vector_distance_threshold for filtering low-quality matches - Configurable distance_metric (COSINE, L2, IP) - Batch embedding generation for efficient ingestion - Implements add_events_to_memory for incremental ingestion - 30 unit tests, 12 integration tests (all passing) Ref: AEA-497

Signed-off-by: Daria Korenieva <daric2612@gmail.com>
…ation - Fix incomplete TAG value escaping in _build_knn_query: now escapes all Valkey Search metacharacters (dots, colons, @, spaces, etc.), not just hyphens. This prevents query injection and ensures correct scoping when app_name/user_id contain special characters. - Add _escape_tag_value() static helper with full Valkey Search spec coverage. - Add type annotation for client parameter (Union[GlideClient, GlideClusterClient]). - Add Pydantic field_validator for distance_metric to reject invalid values at config time instead of silently falling back to COSINE. - Fix TTL check to use 'is not None' instead of truthiness for Optional[int]. - Fix create_index docstring (removed incorrect 'timestamp: NUMERIC field'). - Add unit tests: special char escaping (dots, colons, @, spaces), distance_metric validation, _escape_tag_value coverage. Ref: AEA-497

Signed-off-by: Daria Korenieva <daric2612@gmail.com>
…s in docstring and README highlighting that client_name should be set on GlideClientConfiguration for visibility in CLIENT LIST, monitoring dashboards, and CloudWatch metrics.

Signed-off-by: Daria Korenieva <daric2612@gmail.com>
…, DRY refactor - Fix tenant isolation bypass: add '?' to _TAG_SPECIAL_CHARS escape set (single-char wildcard glob in Valkey Search TAG queries) - Use Batch pipelining for hset/expire calls (1 round trip vs 2N) - Add asyncio.Lock with double-check locking in _ensure_index() to prevent redundant FT.CREATE calls under concurrent access - Extract shared _ingest_events() method to DRY up add_session_to_memory and add_events_to_memory - Update unit tests for Batch-based approach; add tests for ? escaping and _ensure_index concurrency safety

Signed-off-by: Daria Korenieva <daric2612@gmail.com>
…s, remove dead code - Move field_validator import to top-level (was polluting class namespace) - Re-raise RuntimeError on batch exec failure instead of swallowing - Remove unreachable ternary fallback (guarded by earlier len check) - Update unit test to expect RuntimeError on batch exec failure

Signed-off-by: Daria Korenieva <daric2612@gmail.com>
@google-cla
Copy link
Copy Markdown

google-cla Bot commented Jun 5, 2026

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

@daric93 daric93 changed the title Feat/valkey memory service feat: add ValkeyMemoryService with vector similarity search (Valkey Search module) Jun 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat: Add ValkeyMemoryService with vector similarity search (Valkey Search module)

1 participant