memex MCP server

memex

A production-grade persistent memory service for AI agents. Agents forget everything between sessions by default — memex fixes that. It stores, retrieves, and ranks conversation memory using semantic search with recency decay, so agents surface what's relevant and recent, not just what's semantically closest.

POST /v1/memories          → store a memory, embed it, persist to Postgres
POST /v1/memories/search   → retrieve top-k memories ranked by similarity + recency
DELETE /v1/memories/{id}   → forget a specific memory
GET  /v1/memories/count    → how many memories does this agent/user have
GET  /health               → liveness + DB connectivity check
GET  /metrics              → Prometheus metrics

Architecture

caller (agent / app)
        │
        ▼
  FastAPI (async)
        │
   ┌────┴────┐
   │         │
embeddings  asyncpg pool (min=5, max=20)
(fastembed  │
 ONNX,      ▼
 local)  PostgreSQL 16
           pgvector extension
           ivfflat index (cosine)

Write path: content → fastembed ONNX inference (local, ~12 ms CPU, BAAI/bge-small-en-v1.5) → INSERT with 384-dim vector → return memory ID.

Read path: query → embed → pgvector cosine search (topk × 3 candidates) → re-rank with recency decay in Python → return topk results with scores.

Design decisions

1. Recency decay on top of semantic search

Pure vector similarity returns the most semantically similar memories, not the most useful ones. A fact from 90 days ago that's a 0.95 similarity match is often less useful than a 0.80 match from yesterday.

Score formula:

score = α × cosine_similarity + (1 − α) × exp(−λ × age_days)

Where λ = ln(2) / halflifedays (default: 30 days, so a 30-day-old memory has 50% recency weight).

α is configurable per request (default 0.7). Task-focused agents use higher α (semantic dominates). Conversational agents use lower α (recency matters more).

2. Fetch 3× candidates, re-rank in Python

The pgvector query returns topk × 3 candidates sorted by pure similarity. Python re-ranks with the decay formula and slices to topk. This prevents recency decay from starving high-similarity older memories — they're still in the candidate pool.

At 10× scale (>1M memories per agent): push the scoring into a Postgres function using pg_proc to eliminate the Python re-ranking round-trip.

3. asyncpg + explicit pool sizing over SQLAlchemy async

SQLAlchemy adds ORM overhead on every query. The hot retrieval path — embed, query, re-rank — needs to be tight. asyncpg gives direct control over pool min/max (same instinct as tuning HikariCP in Java). pgvector queries require raw SQL for the <=> operator anyway.

Pool defaults: min=5, max=20. Right-size for a single-instance deployment. Override via DBMAXPOOL_SIZE env var.

4. Rate limiting in Postgres, not Redis

Sliding window counter via upsert. One fewer dependency. Correct under concurrent requests (transactional upsert). At 10× scale with distributed deployments: replace with Redis INCR + EXPIRE — atomic operations, no lock contention.

5. ivfflat index, not HNSW

Shortened. The full README is on GitHub.