Data Flow
How NEPSE market data travels from the upstream MDP API to the TradingView chart
The TV module is purely read-side. This page shows where the data it serves comes from, what tables hold it, and how each piece gets refreshed.
End-to-end pipeline
Sources
The scheduler polls MDP during NPT trading hours and UPSERTs each tick into nepse_intraday_prices (companies), nepse_intraday_indices (top-level indices), and nepse_intraday_sub_indices (sector sub-indices).
Companies are keyed by (symbol, polled_at); indices and sub-indices by (index_id, polled_at) and (sub_index_id, polled_at) respectively.
| Table | Key | Notes |
|---|---|---|
nepse_intraday_prices | (symbol, polled_at) | 2000+ tickers |
nepse_intraday_indices | (index_id, polled_at) | 4 rows per poll — ^NEPSE, ^SENSITIVE, ^FLOAT, ^SFLOAT |
nepse_intraday_sub_indices | (sub_index_id, polled_at) | 13 rows per poll — sector indices |
Rows older than the retention window (currently 3 trading days) are pruned by the cleanup cron (runs 12:45 AM NPT Mon–Fri).
When /history gets an intraday request, GetIntradayStockCandles / GetIntradayIndexCandles / GetIntradaySubIndexCandles aggregate these raw ticks into buckets of interval_secs seconds in a single SQL pass — no materialized views.
The EOD cron (3:15 PM NPT Mon–Fri, after session close at 15:00) snapshots each active company's final figures and INSERTs one row per symbol into nepse_price_history.
| Column | Type | Notes |
|---|---|---|
company_id | uuid | FK to nepse_companies.id |
trading_date | date | EOD date |
open_price, high_price, low_price | numeric(12,2) | |
ltp | numeric(12,2) | last traded price = EOD close |
volume | int | whole shares |
turnover, avg_price, year_high, year_low, market_cap | various |
This table is permanent. Weekly and monthly /history aggregate directly from here; there's no separate weekly/monthly storage.
nepse_companies is the authoritative symbol list. Seeded once and reconciled on startup (company:bootstrap-sync task) and daily.
| Column | Type | Notes |
|---|---|---|
id | uuid | primary key |
symbol | text | ticker, unique |
name | text | company name |
instrument_type | text | |
sector_id | uuid | FK to nepse_sectors |
status | char | 'A' = active, 'D' = delisted |
image_url, isin_no, nepse_company_id, nepse_security_id | various |
Only status = 'A' rows are returned from /symbols and /search.
nepse_indices (4 rows) and nepse_sub_indices (13 rows) are master tables the poller upserts every tick. For TradingView they contribute a single column each: tv_symbol.
| Column | Type | Notes |
|---|---|---|
id | uuid | primary key |
index_name / name | varchar(100) | human label — "NEPSE Index", "Banking", ... |
tv_symbol | text NOT NULL UNIQUE | short ticker for TV, no ^. The service prepends ^ when serializing to clients. |
| (other) | — | live OHLC, turnover, volume, market cap — used by /api/nepse/market not TV |
Curated mappings (migration 000018_add_tv_symbol backfill, poller tvSymbolForIndex / tvSymbolForSubIndex):
| Index | tv_symbol |
|---|---|
| NEPSE Index | NEPSE |
| Sensitive Index | SENSITIVE |
| Float Index | FLOAT |
| Sensitive Float Index | SFLOAT |
| Banking | BANKING |
| Development Bank | DEVBANK |
| Finance | FINANCE |
| Hotels And Tourism | HOTELS |
| HydroPower | HYDRO |
| Investment | INVEST |
| Life Insurance | LIFEINS |
| Manu.& Pro. | MFG |
| Microfinance | MICROFIN |
| Mutual Fund | MUTUAL |
| Non Life Insurance | NONLIFE |
| Others | OTHERS |
| Trading | TRADING |
Any future index NEPSE adds gets a deterministic slug via slugifyTVSymbol (uppercase, alnum-only, 16-char cap) so the NOT NULL upsert never fails.
Daily EOD rows for indices and sub-indices land in nepse_index_history (~6500 rows, since 2012) and nepse_sub_index_history.
| Column | Type | Notes |
|---|---|---|
index_id / sub_index_id | uuid | FK to master |
trade_date | date | EOD date |
open, high, low, close | numeric(12,2) | |
volume | bigint nullable |
Same storage model as stocks: daily is permanent; weekly/monthly are aggregated on demand with date_trunc('week', ...) / date_trunc('month', ...) in the same bucketing query shape.
Resolution → table routing
Given a TV resolution string, the service picks one of three paths:
Reference: service.go → GetHistory.
The same flow runs for every kind. After resolve() identifies the ticker as kindStock, kindIndex, or kindSubIndex, the resolution switch picks the same-shape helper from the right table family (nepse_price_history vs. nepse_index_history vs. nepse_sub_index_history; nepse_intraday_prices vs. nepse_intraday_indices vs. nepse_intraday_sub_indices). The retention-window check, nextTime hint, and OHLCV marshalling are shared.
Symbol lookup flow
/symbols is the only TV endpoint backed by a persistent Redis cache. The full lookup:
- Cache TTL: 5 minutes (
cache.TTLNepseTVSymbol()) - Key:
nepse:tv:symbol:<UPPER_SYMBOL> - Redis unavailability is non-fatal: a cache read error logs a warning and falls through to the DB; a cache write failure logs a warning but still returns the resolved symbol.
Search flow
/search goes straight to Postgres — no Redis caching because queries are highly variable (every keystroke from the frontend).
The SQL prefix-boost makes NAB return NABBC, NABIL, NABILD2089, NABILD87 (symbol-prefix matches) before name-substring matches like KDBY (which contains "nab" inside Dhanabriddhi Yojana). Typing ^ alone surfaces every index and sub-index; typing ^NEP returns ^NEPSE; typing BANK returns ^BANKING, ^DEVBANK, and any company ticker containing BANK.
-- Postgres forbids expression-based ORDER BY on a bare UNION, so the three
-- branches are wrapped in a subquery. The outer ORDER BY then runs CASE
-- against the merged column set.
SELECT tv_symbol, description, kind
FROM (
(
SELECT symbol AS tv_symbol, name AS description, 'stock'::text AS kind
FROM nepse_companies
WHERE status = 'A'
AND (symbol ILIKE '%' || @search || '%' OR name ILIKE '%' || @search || '%')
) UNION ALL (
SELECT '^' || tv_symbol AS tv_symbol, index_name AS description, 'index'::text AS kind
FROM nepse_indices
WHERE tv_symbol ILIKE '%' || @search || '%'
OR index_name ILIKE '%' || @search || '%'
OR ('^' || tv_symbol) ILIKE '%' || @search || '%'
) UNION ALL (
SELECT '^' || tv_symbol AS tv_symbol, name AS description, 'index'::text AS kind
FROM nepse_sub_indices
WHERE tv_symbol ILIKE '%' || @search || '%'
OR name ILIKE '%' || @search || '%'
OR ('^' || tv_symbol) ILIKE '%' || @search || '%'
)
) AS merged
ORDER BY
CASE WHEN tv_symbol ILIKE @search || '%' THEN 0 ELSE 1 END,
tv_symbol
LIMIT @lim;Where live ticks fit in (future)
/history ends where the Postgres database ends. Live bars currently forming during market hours are only visible after the poller's next flush — typically within a minute.
For v1, the frontend uses TV's default UDFCompatibleDatafeed, which polls /history every ~10s for the current bar. Bars update at poll cadence — perceptually fine for OHLC candles, and nginx caching plus the Go app's rate limiter absorb the traffic easily at a few hundred concurrent chart viewers.
The parked Phase 6 plan (see docs/TRADINGVIEW_UDF_PHASE6.md) adds a WebSocket gateway for tick-level latency. Decision: deferred until measured pain justifies the build — laggy charts, polling bottleneck on /history, or >500 concurrent chart viewers. See Integration for the frontend seam when it's time to wire it up.
Design rule: never push bar aggregation logic into the WebSocket gateway. Send raw ticks to the browser; aggregate in the datafeed's subscribeBars callback per-resolution. Server-side aggregation per-subscriber-per-resolution wastes CPU.
Freshness guarantees
| Endpoint | Freshness |
|---|---|
/config | Changes only on redeploy. In-process cache is populated at first call. |
/time | Wall-clock accurate — returns time.Now().UTC().Unix() on every call. |
/symbols | Cached in Redis for 5 minutes. A new listing appears after TTL expires or when the cleanup cron invalidates. |
/search | Reads directly from nepse_companies — as fresh as the last company-sync job. |
/history intraday | Trails live by at most one poll cycle (seconds, not minutes). |
/history daily | Updated after the EOD cron at 15:15 NPT Mon–Fri. |
/history weekly/monthly | Aggregated on every request from current daily data. |