3 TB of data, custom-stored with rules
Store, enrich, and recall very large data through the API.
sources ──► POST /custom/store ──► ┌──────────────┐
(json, csv, + rules (map, tag, │ Memoception │
sql, ts, weight, redact) │ records + │
jsonl…) │ content_id │
└──────┬───────┘
/organize ─────────┤──► SQL schema · Neo4j graph
/query ─────────►│ filter · project · traverse
recall ─────────►│ fast fetch-and-rank @ 3 TBThe challenge
A data-intelligence team sits on multi-terabyte stores with thin metadata. They need structure, enrichment, and fast recall at scale — without hand-writing a pipeline per source or losing performance as the corpus grows.
How Memoception solves it
Feed JSON/JSONL/CSV/SQL/TimescaleDB through /custom/store with a rules document — map titles, tag by field, weight by importance, redact PII — so every record gains metadata as it lands.
The same rules vocabulary drives every format pipeline, so onboarding a new source is a config, not code. Each original keeps a content_id linking its representations.
/organize materializes the records as a relational schema or a Neo4j graph; /query filters, projects, and traverses them — SQL over the data, BFS over the relationships.
Recall stays a fast fetch-and-rank; exact lookups are indexed (id + tag), the API runs at 20 req/s per key with batching, and Pro storage holds 1.5 TB per project / 9 TB per account.
custom/store (rules)organize (SQL, Neo4j)query (sql + graph)batchindexed id/tag lookups