Skip to content

[codex] Stream IMAP fetch literals#111

Open
mpscholten wants to merge 2 commits into
qnikst:masterfrom
mpscholten:codex/imap-fetch-streaming
Open

[codex] Stream IMAP fetch literals#111
mpscholten wants to merge 2 commits into
qnikst:masterfrom
mpscholten:codex/imap-fetch-streaming

Conversation

@mpscholten

@mpscholten mpscholten commented Jun 5, 2026

Copy link
Copy Markdown
Contributor

Summary

Fixes the high memory usage reported in #39 when fetching full IMAP message bodies.

This changes the IMAP fetch path so literal response bodies are read as ByteStrings directly from the stream instead of being parsed through the existing packrat/String parser and then packed back into ByteString.

Root Cause

fetch returned a ByteString, but internally it went through fetchByString, causing large IMAP literals to be represented as:

  • the full buffered server response
  • packrat parser derivations per byte
  • a Haskell String
  • a re-packed ByteString

That made full-message fetches allocate far more memory than the message size.

Changes

  • Added fetchByByteString and fetchByByteStringR.
  • Rewired fetch, fetchPeek, fetchHeader, fetchR, and related APIs through the ByteString-native path.
  • Kept fetchByString and fetchByStringR as compatibility wrappers and documented the ByteString variants as preferred for large or non-textual values.
  • Stream FETCH responses directly from BSStream and read {n} literals with bsGet n.
  • Preserve atom values such as NIL as raw response bytes instead of converting them to empty values.
  • Added regression and compatibility coverage for raw bytes, large literals, fetchPeek, fetchSize, fetchFlags, header-field sections, range UID mapping, fetchByString scalar/literal/NIL values, and STORE responses.

Validation

  • nix shell github:NixOS/nixpkgs/nixos-25.05#ghc github:NixOS/nixpkgs/nixos-25.05#cabal-install -c cabal test imap-parsers --test-options='--hide-successes'
  • 33 tests passed locally after rebasing onto upstream master.

A compiled synthetic benchmark showed much lower peak RSS after streaming:

  • 2 MB literal: about 15 MB max RSS
  • 10 MB literal: about 23 MB max RSS

@mpscholten mpscholten force-pushed the codex/imap-fetch-streaming branch from fe7f0c0 to ceb90ce Compare June 5, 2026 15:36
@mpscholten mpscholten marked this pull request as ready for review June 10, 2026 13:05
@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, add credits to your account and enable them for code reviews in your settings.

@mpscholten mpscholten force-pushed the codex/imap-fetch-streaming branch from bb73a34 to 6a32ebb Compare June 10, 2026 17:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant