# MarketFeed API

MarketFeed serves live market prices scraped from **Yahoo Finance**, **Google Finance** and **Investing.com**. For each instrument it reports every source's price side by side, a cross-source consensus, a live tick stream, 30 days of 1-minute history that the server records itself, and long-term daily history back to each instrument's first trade (for example Bitcoin since September 2014).

- Base URL: `https://platform.rotan.app`
- Formats: JSON over HTTPS, Server-Sent Events, WebSocket, and MCP (Model Context Protocol).
- This guide is public. Machine-readable copies: [`/docs.md`](https://platform.rotan.app/docs.md) (Markdown), [`/llms.txt`](https://platform.rotan.app/llms.txt) (index), [`/openapi.json`](https://platform.rotan.app/openapi.json) (OpenAPI 3 schema).

Prices carry whatever delay each website applies: US indices and stocks are usually realtime, while futures on Yahoo are often about 10 minutes behind. Nothing here is investment advice.

## Quick start

```bash
# every tracked instrument, consensus + per-source quotes
curl https://platform.rotan.app/api/quotes -H "X-API-Key: $MARKETFEED_KEY"

# a few symbols, forcing a fresh check of every source
curl "https://platform.rotan.app/api/quotes?symbols=JKSE,BBCA,SPX&max_age=0" -H "X-API-Key: $MARKETFEED_KEY"

# start tracking a ticker: give its Yahoo id, the rest is looked up
curl -X POST https://platform.rotan.app/api/symbols -H "X-API-Key: $MARKETFEED_KEY" \
  -H "Content-Type: application/json" -d '{"yahoo":"UNVR.JK"}'

# what the server recorded for JKSE over the last 7 days, as 15-minute candles
curl "https://platform.rotan.app/api/stored/JKSE?interval=15m&days=7" -H "X-API-Key: $MARKETFEED_KEY"

# Bitcoin's last 10 years of daily candles, as CSV
curl "https://platform.rotan.app/api/daily/BTC?range=10y&format=csv" -H "X-API-Key: $MARKETFEED_KEY"

# put Bank Mandiri and the IDX Composite in a group called "JKSE"
curl -X POST https://platform.rotan.app/api/groups/move -H "X-API-Key: $MARKETFEED_KEY" \
  -H "Content-Type: application/json" -d '{"symbols":["BMRI","JKSE"],"group":"JKSE"}'
```

## Authentication

Every `/api/*` route except `/api/health`, plus `/mcp` and `/ws`, needs an API key. There are two kinds:

- **Client keys** (`mfk_…`), created by the owner in the console under **API keys** (or `POST /api/keys`). Each has a name, an access level and an optional rate limit and expiry, and can be disabled or deleted on its own. Ask the owner for one.
- The **master key**, set on the server (`MF_API_KEY`, or the Worker's `API_KEY` secret). It can do everything, including managing client keys and switching sources.

Send the key one of three ways:

| Method | Example |
|---|---|
| Header | `X-API-Key: <key>` |
| Bearer token | `Authorization: Bearer <key>` (use this for MCP clients) |
| Query string | `?api_key=<key>` (for EventSource and WebSocket, which can't set headers) |

| Access | Allows |
|---|---|
| `read` | Every `GET` route, `/api/stream`, `/ws`, and the read-only MCP tools |
| `write` | Also `POST`, `PUT`, `PATCH` and `DELETE` on instruments and groups, `POST /api/refresh`, and the MCP tools `add_symbol`, `update_symbol`, `remove_symbol` and `move_to_group` |
| master key or console | Also `/api/keys` and switching sources (`PATCH /api/sources/{source}`) |

Responses: `401` for a missing, unknown, disabled or expired key (the `detail` says which); `403` when the key's access level doesn't cover the request; `429` with a `Retry-After` header when a key passes its per-minute rate limit. The web console uses a separate PIN that only unlocks browsers. These docs, `/llms.txt`, `/openapi.json` and `/api/health` need no key.

## Concepts

### Instruments

An instrument is something MarketFeed tracks. Each one has a canonical `symbol` used everywhere in this API, plus its id on each website.

| Field | Meaning |
|---|---|
| `symbol` | Canonical symbol: `SPX`, `JKSE`, `BBCA`, `WTI`, `GOLD`, ... |
| `name` | Display name |
| `asset_class` | `index`, `equity`, `commodity`, `fx`, `crypto`, `etf` or `other` |
| `country` | ISO 3166 code (`US`, `ID`, `JP`, ...) or `GL` for global markets (commodities, FX, crypto) |
| `exchange` | Display exchange: `IDX`, `NASDAQ`, `NYMEX`, ... |
| `yahoo` | Yahoo ticker: `^JKSE`, `BBCA.JK`, `CL=F`, `AAPL` |
| `google` | Google Finance id `TICKER:EXCHANGE`: `COMPOSITE:IDX`, `BBCA:IDX`, `.INX:INDEXSP` |
| `investing` | Investing.com pair id (number): `29049` for the IDX Composite |
| `compare` | `true`: poll all three sources. `false`: Yahoo only (live stream + batch), which is the default for most stocks |
| `rank` | Display order inside its group |
| `group` | Custom group name, or `null` for the automatic group of its type (see [Groups](#groups)) |

The default watchlist is the S&P 500, NASDAQ Composite, NASDAQ 100, IDX Composite (JKSE), WTI and Brent crude, Apple, Bitcoin (`BTC`), Ethereum (`ETH`), Solana (`SOL`), and the 45 largest IDX stocks by market cap. All 835 IDX-listed stocks are in a built-in catalog that `/api/search` draws on, and any ticker Yahoo knows can be added.

### Sources and freshness

| Source | How it's read | Cadence |
|---|---|---|
| Yahoo Finance | Live WebSocket stream, plus a batched quote request (20 tickers per call) | Ticks as they trade; full refresh every poll (30 s by default) |
| Google Finance | Quote page, parsed from its embedded data | Every poll, only for instruments with `compare: true` |
| Investing.com | Chart-data endpoint | Every poll, one request at a time, only with `compare: true`. Its bot check sometimes blocks servers; the server then pauses Investing.com for 1 to 10 minutes and keeps the last good price, marked stale |

Yahoo is the master source and is always on. The owner can switch Google Finance or Investing.com off for every instrument (`PATCH /api/sources/{source}`); while a source is off it isn't polled, its quotes disappear from consensus results (`sources_total` shrinks), and requests for that source alone (`/api/quotes/{symbol}/google`, `/api/history/{symbol}?source=google`) return `409`. Check `sources_enabled` in `/api/health`.

While someone watches the stream, instruments whose market is open but whose Yahoo stream has been quiet for 10 s are re-checked every 2 s. Pre-market and after-hours prints from the stream are ignored, so `last` always reflects the regular session.

### Quote

One source's view of one instrument.

| Field | Meaning |
|---|---|
| `source` | `yahoo`, `google` or `investing` |
| `source_id` | The id used on that site |
| `price`, `change`, `change_pct` | Last price, change vs previous close, and that change in % |
| `prev_close`, `open`, `day_high`, `day_low`, `volume` | When the site provides them |
| `currency` | ISO currency code |
| `market_time` | Time of the last trade, per the site (UTC) |
| `fetched_at` | When this server got it (UTC) |
| `via` | `stream` (Yahoo tick), `fast` (2-second re-check), `batch` (Yahoo batch poll), `poll` (single request) |
| `market_open` | Whether the regular session is open now (Yahoo only) |
| `ok`, `error` | `ok: false` means the request failed. An `error` starting with `stale:` means the last good price is being kept after a failure |

### Consensus

The cross-source view returned by `/api/quotes`.

| Field | Meaning |
|---|---|
| `last` | The freshest price: the live Yahoo tick when it is under 15 s old, otherwise the median. Use this for display |
| `last_change_pct` | Change % matching `last` |
| `last_source` | `live` or `median` |
| `price` | Median price across the sources that answered |
| `change_pct` | Median change % |
| `spread_pct` | `(max - min) / median * 100` across sources. Under 0.02 means the sources agree; 1 or more means they disagree (for example different futures contracts) |
| `sources_ok`, `sources_total` | How many polled sources answered |
| `quotes` | Object keyed by source, each a Quote |

Example (`GET /api/quotes/JKSE`, shortened):

```json
{
  "symbol": "JKSE",
  "name": "IDX Composite",
  "last": 5098.057,
  "last_change_pct": -3.3399,
  "last_source": "median",
  "price": 5098.057,
  "change_pct": -3.3399,
  "spread_pct": 0.0,
  "sources_ok": 2,
  "sources_total": 3,
  "quotes": {
    "yahoo": {
      "source": "yahoo", "source_id": "^JKSE", "price": 5098.057, "change_pct": -3.1916,
      "prev_close": 5266.13, "day_high": 5320.895, "day_low": 5098.057, "currency": "IDR",
      "market_time": "2026-10-29T09:00:05Z", "via": "batch", "market_open": false, "ok": true, "error": null
    },
    "google": { "source": "google", "source_id": "COMPOSITE:IDX", "price": 5098.057, "ok": true },
    "investing": { "source": "investing", "source_id": "29049", "ok": false, "error": "HTTP 403" }
  }
}
```

## Quotes

### GET /api/quotes

Consensus for many instruments.

| Query | Default | Meaning |
|---|---|---|
| `symbols` | all tracked | Comma-separated symbols, e.g. `JKSE,BBCA,SPX`. Unknown symbols are looked up on Yahoo by ticker |
| `sources` | all | Comma-separated subset of `yahoo,google,investing` |
| `max_age` | 15 | Accept cached quotes up to this many seconds old. `0` forces a fresh check |

Returns an array of Consensus.

### GET /api/quotes/{symbol}

Consensus for one instrument. Same `sources` and `max_age` parameters. `{symbol}` can be a tracked symbol (`BBCA`) or any Yahoo ticker (`MSFT`, `EURUSD=X`, `BTC-USD`).

### GET /api/quotes/{symbol}/{source}

One source's Quote. `{source}` is `yahoo`, `google` or `investing`.

### GET /api/sparklines

Today's 5-minute closes from Yahoo for many instruments in one call.

| Query | Default | Meaning |
|---|---|---|
| `symbols` | all tracked | Comma-separated symbols |

Returns `{"SPX": [[1793194200, 7721.4], ...], ...}`, pairs of Unix seconds and close.

### POST /api/refresh

Re-checks every tracked instrument now and pushes a `snapshot` to stream clients. Returns the Consensus array once Yahoo and Google have answered; Investing.com results follow in a later `snapshot` event.

## Instruments

### GET /api/symbols

Tracked instruments. Optional filters: `asset_class` (e.g. `equity`) and `country` (e.g. `ID`).

### GET /api/search

Ticker suggestions for adding instruments: tracked matches first, then the IDX catalog (835 stocks), then Yahoo's worldwide search.

| Query | Default | Meaning |
|---|---|---|
| `q` | empty | Text: a code (`TLKM`), Yahoo ticker (`TLKM.JK`), or name (`bank central`). Empty returns popular instruments and the largest IDX stocks not yet tracked |
| `limit` | 12 | 1 to 40 |

Each result: `symbol`, `yahoo`, `name`, `asset_class`, `country`, `exchange`, `tracked` (bool), `origin` (`tracked`, `idx`, `yahoo` or `popular`). Add one by posting its `yahoo` value to `/api/symbols`.

### POST /api/symbols

Start tracking an instrument.

**By Yahoo ticker (recommended).** Name, type, country, exchange, currency and Google id are looked up; the Google id is kept only if Google's page actually parses.

```json
{"yahoo": "TLKM.JK"}
```

Optional fields: `symbol` (override the canonical symbol), `compare` (default `true` for indices and commodities, `false` otherwise), `investing` (Investing.com pair id, which can't be looked up automatically).

**By hand.** Send a full Instrument with `symbol`, `name` and at least one of `yahoo`, `google`, `investing`.

Returns the Instrument: `201` when created, `200` when the ticker was already tracked, `404` when Yahoo has no quote for it.

### PATCH /api/symbols/{symbol}

Change any of `name`, `asset_class`, `country`, `exchange`, `google`, `investing`, `compare`, `note`, `rank`, `group`. Example: `{"compare": true}` starts polling Google and Investing.com for that instrument; `{"investing": 29049}` sets its Investing.com pair id.

### DELETE /api/symbols/{symbol}

Stops tracking the instrument and deletes its stored history. Returns `{"deleted": "BBCA"}`, or `404`.

## Groups

The console lists instruments in groups. Every instrument starts in the **automatic group** for its type: `Indices`, `Stocks`, `Commodities`, `Currencies`, `Crypto`, `ETFs` or `Other`. Move any instrument into a **custom group** (for example `JKSE` for the Jakarta index and its stocks) and it shows there instead. Groups only affect display and filtering; quotes and history don't change.

### GET /api/groups

Groups in display order: `[{"name": "JKSE", "count": 4, "custom": true, "symbols": ["JKSE", "BBCA", "BBRI", "BMRI"]}, ...]`.

### POST /api/groups/move

Body: `{"symbols": ["BMRI", "BBCA"], "group": "JKSE"}`. A new name creates the group (placed right after the group the first instrument came from). `"group": null` (or `""`, or the type's own automatic name) puts the instruments back in their automatic group. Returns the moved instruments. `404` if a symbol isn't tracked.

### PUT /api/groups

Body: `{"order": ["JKSE", "Indices", "Crypto", "Stocks"]}` sets the display order. Groups you leave out keep their default place.

### PATCH /api/groups/{name}

Body: `{"name": "IDX blue chips"}` renames a group. Renaming an automatic group turns its members into a custom group of that name.

### DELETE /api/groups/{name}

Dissolves a custom group; its instruments go back to their automatic groups (nothing is untracked). Returns `{"group", "returned"}`. Automatic groups can't be deleted (`400`).

## History

There are three kinds of history:

- **Remote**: each website's own chart data, available immediately for any instrument (`/api/history`).
- **Stored**: what this server recorded from the stream and polls: one-minute bars per source, kept for 30 days, only for tracked instruments (`/api/stored`).
- **Daily**: long-term daily candles from Yahoo, back to the instrument's first trade (`/api/daily`). For tracked instruments the full history is downloaded once and kept in the database; later requests only fetch the last few days.

### GET /api/daily/{symbol}

| Query | Default | Meaning |
|---|---|---|
| `range` | `1y` | `1mo`, `3mo`, `6mo`, `ytd`, `1y`, `2y`, `5y`, `10y`, `20y` or `max` |
| `interval` | `1d` | `1d`, `1wk` (weeks start Monday) or `1mo` |
| `start`, `end` | | ISO dates; override `range` |
| `refresh` | `false` | Fetch the latest days now instead of waiting for the hourly top-up |
| `format` | `json` | `json` or `csv` (columns `date,open,high,low,close,volume`) |

Returns `{"symbol", "source": "yahoo", "interval", "range", "start", "end", "candles": [{"t", "o", "h", "l", "c", "v"}], "first_available", "stored"}`. `first_available` is the oldest day Yahoo has (BTC: 2014-09-17; S&P 500: 1927). `stored` is `true` when the data came from (and is kept in) this server's database; untracked tickers are fetched live and not stored. Prices are split-adjusted but not dividend-adjusted. Each candle's `t` is the trading day (midnight UTC of that date). `404` when Yahoo has no such ticker.

Examples: `/api/daily/BTC?range=10y`, `/api/daily/SPX?range=max&interval=1mo`, `/api/daily/BBCA?range=5y&interval=1wk&format=csv`.

### GET /api/history/{symbol}

Remote candles from one source.

| Query | Default | Meaning |
|---|---|---|
| `source` | `yahoo` | `yahoo`, `google` (today's intraday line only) or `investing` |
| `interval` | `5m` | `1m`, `5m`, `15m`, `30m`, `1h`, `1d`, `1w` |
| `points` | 78 | 2 to 500, most recent last |

Returns `{"symbol", "source", "interval", "candles": [{"t", "o", "h", "l", "c", "v"}]}`.

### GET /api/stored/{symbol}

Stored bars, aggregated to the interval you ask for. Candles are aligned to UTC.

| Query | Default | Meaning |
|---|---|---|
| `source` | `yahoo` | Which source's bars |
| `interval` | `5m` | `1m`, `5m`, `15m`, `30m`, `1h`, `4h`, `1d` |
| `days` | | Window ending now, 0 to 30. Overrides `start`/`end` |
| `start`, `end` | today (UTC) to now | ISO 8601 timestamps |

Add `format=csv` for a CSV download. Returns `{"symbol", "source", "interval", "start", "end", "candles": [...], "first_stored", "minutes_stored"}`. `first_stored` is the oldest minute kept for that series. `minutes_stored` is the number of 1-minute bars in your window. Gaps mean the market was closed or the server wasn't running. `404` if the instrument isn't tracked.

### GET /api/storage

Database size and contents: `bars`, `bars_by_source`, `oldest`, `newest`, `size_bytes`, `instruments`, `retention_days`, `daily_bars`, `daily_series`.

## Live stream

### GET /api/stream (Server-Sent Events)

Two event types:

- `snapshot`: `{"type": "snapshot", "at": "<iso>", "data": [Consensus, ...]}` for every tracked instrument. Sent on connect (from cache, with `"cached": true`), after every poll (30 s), and whenever instruments change.
- `ticks`: `{"type": "ticks", "at": "<iso>", "items": [{"symbol", "q", "c"}, ...]}`, at most 4 per second while prices move. `q` is the instrument's new Yahoo Quote. `c` holds the updated consensus fields (`last`, `last_change_pct`, `last_source`, `price`, `change_pct`, `spread_pct`, `sources_ok`, `sources_total`). Merge `q` into `quotes.yahoo` and `c` into the instrument's consensus.

A comment line `: keep-alive` is sent every 15 s.

```js
const es = new EventSource("https://platform.rotan.app/api/stream?api_key=<key>");
es.addEventListener("snapshot", e => render(JSON.parse(e.data).data));
es.addEventListener("ticks", e => JSON.parse(e.data).items.forEach(update));
```

### GET /ws (WebSocket)

The same messages as JSON text frames. Authenticate with `?api_key=<key>`.

## MCP (Model Context Protocol)

- Endpoint: `https://platform.rotan.app/mcp` (Streamable HTTP, stateless, JSON responses)
- Auth: `Authorization: Bearer <key>`. A `read` key can call every tool except `add_symbol`, `update_symbol`, `remove_symbol` and `move_to_group` (those return `403`).

| Tool | Arguments | Returns |
|---|---|---|
| `list_symbols` | `asset_class?`, `country?` | Tracked instruments |
| `search_symbols` | `query`, `limit?` | Suggestions, as `/api/search` |
| `get_quote` | `symbol`, `sources?`, `max_age_seconds?` | Consensus |
| `get_quotes` | `symbols?`, `sources?`, `include_source_detail?` | Consensus list |
| `get_history` | `symbol`, `source?`, `interval?`, `points?` | Remote candles |
| `get_stored_history` | `symbol`, `source?`, `interval?`, `days?` (up to 30) | Stored candles |
| `get_daily_history` | `symbol`, `range?` (`1mo` … `10y`, `max`), `interval?` (`1d`, `1wk`, `1mo`) | Long-term daily candles |
| `list_groups` | | Groups with their members |
| `move_to_group` | `symbols`, `group?` (null = automatic) | Moved instruments |
| `add_symbol` | `yahoo`, `compare?`, `investing?` | `{created, instrument}` |
| `update_symbol` | `symbol`, `compare?`, `google?`, `investing?`, `name?`, `group?` | Instrument |
| `remove_symbol` | `symbol` | `{removed}` |
| `source_health` | | Per source: switched on or off, success counts, latency |
| `storage_stats` | | As `/api/storage` |

Resource: `marketfeed://snapshot`, the cached consensus for all tracked instruments.

Claude Code:

```bash
claude mcp add --transport http marketfeed https://platform.rotan.app/mcp --header "Authorization: Bearer <key>"
```

Claude Desktop, Cursor and other clients (`mcp.json`):

```json
{"mcpServers": {"marketfeed": {"url": "https://platform.rotan.app/mcp", "headers": {"Authorization": "Bearer <key>"}}}}
```

## Sources

### GET /api/sources

Each source with its switch, health and a diagnosis: `source`, `enabled`, `master` (true for Yahoo), `instruments` (how many instruments use it), `ok`, `failed`, `avg_latency_ms`, `last_ok`, `last_error`, `last_error_at`, plus:

| Field | Meaning |
|---|---|
| `state` | `ok`, or the kind of the latest failure (table below) |
| `state_label`, `advice` | The state in words, and what to do about it |
| `failing_since` | When the current run of failures began (`null` while working) |
| `recent_ok_pct` | Share of the last 50 requests that got a price |
| `errors_by_kind` | Failures since the server started, by kind, e.g. `{"blocked": 12, "network": 1}` |
| `recent_errors` | The last 20 failures, newest first: `at`, `symbol`, `kind`, `error` |

| Kind | Means | Typical message |
|---|---|---|
| `blocked` | The site refused this server (bot check) | `HTTP 403 (bot challenge)`, `google: blocked (captcha)` |
| `rate_limited` | Too many requests | `HTTP 429 (rate limited)` |
| `paused` | MarketFeed stopped asking for 1 to 10 minutes after failures in a row; the message keeps the last one | `paused for 120 s after 3 failed requests in a row (last: HTTP 403 (bot challenge))` |
| `not_found` | The site doesn't know the ticker or id | `HTTP 404` |
| `site_error` | The site answered with a server error | `HTTP 503` |
| `network` | No answer: timeout, DNS, connection or TLS failure | `Timeout: … timed out` |
| `no_data` | An answer without a price: wrong id, or the site changed its page | `quote for X not found on page` |
| `other` | Anything else | |



### POST /api/sources/{source}/test

Master key or console only. Makes one live request to the source now, ignoring the cache and any pause, for one of its instruments (or `?symbol=BTC`), and returns `ok`, `state`, `state_label`, `advice`, `symbol`, `price`, `error`, `latency_ms` and `tested_at`. A successful test also ends a pause.

### PATCH /api/sources/{source}

Master key or console only. `{"enabled": false}` switches Google (`google`) or Investing.com (`investing`) off for every instrument; `{"enabled": true}` switches it back on and checks it right away. Each instrument's own `compare` setting is kept. Yahoo can't be switched off (`400`). Returns the list as `GET /api/sources`.

## API keys

Master key or console only. The console's **API keys** page does the same with a form.

### GET /api/keys

Client keys, newest first: `id`, `name`, `hint` (e.g. `mfk_Ab3x…9QzT`), `scope` (`read` or `write`), `rate_limit` (requests per minute, 0 = none), `created_at`, `expires_at`, `disabled_at`, `last_used_at`, `last_ip`, `requests`, and `status` (`active`, `disabled` or `expired`).

### POST /api/keys

```bash
curl -X POST https://platform.rotan.app/api/keys -H "X-API-Key: <master key>" -H "Content-Type: application/json" \
  -d '{"name": "Trading bot", "scope": "read", "rate_limit": 120, "expires_days": 90}'
```

Returns `201 {"key": {...}, "secret": "mfk_..."}`. The full key appears only in this response; the server keeps a hash of it.

### PATCH /api/keys/{id}

Any of `name`, `scope`, `rate_limit`, and `enabled` (`false` disables the key at once, `true` re-enables it).

### DELETE /api/keys/{id}

Deletes the key; apps using it get `401` from then on.

## Status

### GET /api/health (public)

`status`, `version`, `poll_interval`, `cache_ttl`, `instruments`, `watching` (open stream clients), `last_poll`, `stream` (Yahoo stream: `connected`, `ticks`, `last_tick_at`, `last_error`), `fast_poll`, `sources_enabled` (`{"yahoo": true, "google": true, "investing": false}`), `retention_days`, and `sources` (per-source health).

### GET /api/bootstrap

Everything a dashboard's first screen needs in one response, served from memory without contacting any website: `instruments`, `groups`, `snapshot` (the cached consensus for every instrument, as `/api/quotes` but without refreshing), `sparklines`, `health`, `session` and `at`. The web console gets the same data embedded in its page, so prices show without any API call.

### GET /api/sources/health

Per source: `ok`, `failed`, `avg_latency_ms`, `last_ok`, `last_error`, `last_error_at`.

## Errors

| Status | Meaning |
|---|---|
| 400 | Bad parameter, e.g. an unknown source name |
| 401 | Missing, unknown, disabled or expired API key |
| 403 | The key's access level doesn't allow this (read-only key writing, or a client key managing keys or sources) |
| 409 | The request needs a source that's switched off |
| 429 | The key passed its rate limit; see `Retry-After` |
| 404 | Instrument not tracked, or unknown ticker on add |
| 422 | Validation error (FastAPI format: `{"detail": [...]}`) |
| 502 | The source website failed for a history request |

Errors are JSON: `{"detail": "..."}`.

## Notes and limits

- Polling: every 30 s (`MF_POLL_INTERVAL`). Cached quotes are reused for 15 s (`MF_CACHE_TTL`) unless you pass `max_age=0`.
- Storage: one SQLite database, 1-minute OHLC bars per instrument and source, deleted after 30 days (`MF_RETENTION_DAYS`).
- Daily history is kept indefinitely; it is small (ten years of one instrument is about 2,500 rows). The first request for an instrument downloads its full history (BTC: ~4,400 days, S&P 500: ~14,000), later ones only the last ten days, at most hourly.
- Crypto trades around the clock, so BTC, ETH and SOL tick and record 24/7. Sources differ by a few tenths of a percent because each quotes a different exchange mix.
- Futures roll on different days per site, so small gaps between sources around expiry are normal. Google lists NYMEX Brent Last Day Financial (`BZW00`), which trades about 3% above the front-month Brent the others show.
- Investing.com ids can't be looked up automatically; find the pair id in the page source of the instrument on investing.com (`data-pair-id`) and set it with `PATCH`.
- Scraping these websites may conflict with their terms of use and can break when they change. For production trading, use a licensed data feed.
