Ingestion Plan (Phase 1)
Ingestion Plan (Phase 1)
Goals
- Fetch Kalshi and Polymarket market metadata and top-of-book prices.
- Normalize both venues into a shared schema.
- Persist market metadata and mapping candidates.
- Emit real-time updates to UI through socket.io.
Data Sources
- Polymarket:
https://gamma-api.polymarket.com/events?...
- Kalshi: public market endpoints (read-only).
Loop Design
- Market discovery loop (slow, e.g., every 5-10 minutes):
- Fetch active markets for each venue.
- Normalize into
MarketMetadata.
- Store/upsert into Postgres.
- Price loop (fast, e.g., 1-5 seconds or WS):
- Fetch top-of-book or subscribe to orderbook updates.
- Normalize into
MarketUpdate or OrderBookDepth.
- Publish to RabbitMQ and emit to socket.io.
Normalization Rules
- Convert venue-specific identifiers into
market_id and symbol.
- Standardize YES/NO outcomes and pricing to cents.
- Store venue tokens (if available) for later execution.
Mapping Strategy (Phase 1)
- Create a mapping table that links a Kalshi market to a Polymarket market.
- Start with heuristic matching (title similarity, end time, category).
- Surface mismatches to UI for manual confirmation.
- Persist confirmed mappings for future comparisons.
Polymarket Series Pattern (BTC Hourly)
- Series:
btc-up-or-down-hourly
- Event slug format:
bitcoin-up-or-down-<month>-<day>-<hour><am|pm>-et
- Store this pattern in
app/data/patterns.py to keep it centralized.
Early Opportunity Signals
- Compute simple edge detection using best YES/NO asks.
- Emit a
signal.arb.detected event even before execution exists.