# Market data API integration and pipelines

URL: https://fincodelabs.com/en/services/market-data-pipelines · Publisher: Fincode (Fincode Yazılım ve Teknoloji A.Ş.), Istanbul, Turkey · Updated: 2026-11-06

Fincode builds the software between a data source and your product: it collects trades, order books and candles from sources you are licensed to use, stores them so they can be queried and replayed, and reports where the data has gaps. The pipeline places no orders, and Fincode does not sell data.

## When a folder of downloads stops being enough

### Teams whose backtests run on files

Nobody can say which file is current, which days are missing or whether delisted symbols were dropped. A test built on that data inherits every hole, as our guide to [backtesting pitfalls](https://fincodelabs.com/en/blog/backtesting-trading-strategy-pitfalls) explains, and last month's result cannot be reproduced.

### Products that show market data

A market data API integration, whether a crypto market data API or a real-time stock data API, needs one internal model for prices, symbols and timestamps, so that changing a provider touches one adapter and not every screen.

### Rule-based trading software

Your software needs live data it can check, plus a record of what it saw when it acted. If you also need order placement, see [exchange API integration](https://fincodelabs.com/en/services/exchange-api-integration).

### Research on crypto tick data and order books

Many sources keep little depth history. To study level 2 data in crypto, you either record it from the day you start or buy it from a vendor that did.

### Teams that need Turkish market data

BIST market data reaches applications through licensed distributors, and Borsa Istanbul's FAQ sends automated use to an agreement signed directly with the exchange. Disclosures on KAP, Turkey's public disclosure platform, come through the data dissemination API of MKK, the central securities depository, or through a distributor. TEFAS, the fund platform, shares no public API, according to Takasbank's FAQ. Fincode reads those documents in Turkish. See [Borsa Istanbul market data](https://fincodelabs.com/en/blog/borsa-istanbul-api-market-data) and [Turkish fund and macro data](https://fincodelabs.com/en/blog/tefas-kap-evds-data-apis).

## What your data license has to cover

Being able to download data does not give you the right to build on it. One large exchange's free bulk archive, for example, comes under dataset terms updated 26 August 2026 that allow non-commercial use only and require a separate written license for anything else. Another's API agreement, updated 28 July 2026, says public access to market data grants no right to redistribute it. Both were read on 5 November 2026. Our guides to [stock market data APIs](https://fincodelabs.com/en/blog/stock-market-data-api) and [crypto price APIs](https://fincodelabs.com/en/blog/crypto-price-api) give the detail.

**General questions, not legal advice. Your contract with the data source decides.**

| Use | What the license has to answer |
| --- | --- |
| Display: people read it on a screen | Who may see it, inside or outside your company, in real time or delayed, and whether fees run per user |
| Non-display: software reads it with nobody watching | Whether machine use is covered at all. Borsa Istanbul's FAQ, for example, sends automated and algorithmic use to an agreement signed directly with the exchange |
| Redistribution: it leaves your company | Whether you may pass it to customers or partners, and in what form |
| Derived data: values you calculate from it | Whether you may publish or sell what you calculate, and whether fees or attribution still apply |
| Storage: you keep history | How long you may keep it, whether you may replay it, and what happens when the contract ends |

## What a market data pipeline has to survive

Most bad data raises no error. Which of these cases apply depends on the source; those that do are simulated on purpose and appear in the test report. Dated examples come from official documents read on 5 November 2026.

### The connection stays open and the data stops

A stale stream looks like a quiet market. The pipeline tracks the age of the last message on each stream, reconnects past a limit you set and records the hole.

### An order book update goes missing

The local book drifts away from the exchange and nothing fails. Sequence numbers are checked on every message; on a gap, the book is dropped and rebuilt from a fresh snapshot. Checksums are used only where the source still supports them: one large exchange fixed its checksum field to 0 in June 2026, while leaving the field in its messages.

### Trades arrive twice or out of order after a reconnect

Volume inflates and nobody notices. Rows are keyed on the source's own trade ID, so a repeat is ignored and a late message lands in its proper place.

### A candle is saved before it closes

The stored bar differs from the one the source later publishes, and a backtest runs on numbers that never existed. Only closed candles count as final, and the rule is written down.

### A timestamp changes unit or meaning

One large exchange's bulk spot files moved to microseconds from 1 January 2025, so a parser that assumes milliseconds writes dates far in the future. Sources also differ on whether a candle's time marks its open or its close. Timestamps are range-checked on arrival and stored in UTC, with the source's event time and the receive time kept apart.

### A published value is corrected later

Overwriting it hides what your software actually saw. Corrections are stored as new rows with their own time, so the first version and the current one both stay available.

### A symbol is renamed or delisted

History splits in two or disappears, and backtests see only the survivors. Symbols live in a table with valid-from and valid-to dates, and delisted instruments stay in the store.

### A split or dividend looks like a crash

Raw prices and adjustment factors are stored side by side, so an adjusted series can be rebuilt and checked instead of trusted.

### A holiday looks like an outage

Without a session calendar, a gap report flags every holiday as missing data and teaches people to ignore it. Where a market has sessions, gaps are measured against its calendar, half days included.

### A backfill hits the rate limit hours in

The job starts again from zero, or the source blocks it. Backfills run as resumable jobs with a checkpoint per instrument and a request budget below the published limit.

### The database falls behind the stream

Messages back up in memory until the process dies or the source drops a slow client. Receiving and writing run separately, with a queue between them and its lag measured.

## Decisions your scope document records

There is no default stack. Each choice is written down with its reason, after a sample of your real feed has been measured.

### What to store

OHLCV candles are small, but they cannot be turned back into trades. Trades can rebuild candles. Depth updates are usually the largest of the three and are worth keeping only if something reads them. The scope says what is stored and what you give up by leaving the rest out.

### Raw and normalized

Raw messages are kept unchanged next to the normalized tables, so a parsing bug is fixed by replaying history instead of losing it. Prices and sizes stay exact decimals from source to storage.

### Files or a database

Parquet files suit research and batch backtests. PostgreSQL suits candles, reference data and smaller symbol sets. A columnar database such as ClickHouse suits large trade tables read by aggregate queries, with one catch: its [documentation](https://clickhouse.com/docs/engines/table-engines/mergetree-family/replacingmergetree) says ReplacingMergeTree removes duplicates only during background merges and does not guarantee their absence, so reads have to allow for that.

### Kafka, or nothing in between

A message broker such as Kafka earns its cost when several consumers read the same stream or you need replay at that layer. One collector feeding one database does not need it, and every extra component is something you have to run.

### Reference data by date

Tick sizes, lot sizes, calendars and symbol mappings change. They are versioned by date, so a past day can be reproduced under that day's rules. Our [BIST backtesting example scenario](https://fincodelabs.com/en/projects/bist-strategy-backtesting-platform) shows why this matters on a market with corporate actions and price limits.

### Where it runs

By default on your own server or cloud account. If your project needs a different setup, the written scope says where it runs.

## What you receive

- A data contract: sources, instruments, data types, timestamps and gap rules, approved by you before coding
- Collectors for live streams, and backfill jobs that can stop and resume
- A raw store and normalized tables, on your own server or cloud account by default
- A documented way for your product or backtests to read the data: tables, files or an internal API
- A gap and quality report per instrument and day, which you can query yourself
- Freshness alerts to Telegram, email or mobile notifications, at limits you set
- A test report covering the failure cases that apply to your sources
- Technical documentation, including how to replay history

[How we work, stage by stage](https://fincodelabs.com/en/how-we-work)

### What Fincode does not do

- Sell, resell or redistribute market data
- Hold your data license or sign with providers for you
- Turn data into signals or trade recommendations
- Operate a hosted data service
- Get around a source's rate limits, terms or access rules: no scraping against its terms, no extra accounts or rotating addresses

## What a data contract looks like

You approve this before coding starts. It decides what the pipeline may do with each source, and it is the document a gap report is checked against.

**Sample data contract, not a real project. Brackets mark what your project fills in.**

| Item | Sample entry |
| --- | --- |
| Source and license | [data provider], under a license held by [your company] that covers [display, machine use or both] |
| Instruments | [your list], each with valid-from and valid-to dates |
| Data types | Trades; best bid and offer; depth updates to [number of levels]; closed candles at [intervals] |
| Timestamps | UTC. Source event time and receive time stored separately |
| Prices and sizes | Exact decimals, as published by the source |
| Gap rule | Reported per instrument and day. Never filled by interpolation |
| Corrections | Stored as new rows with their own time. The first version is kept |
| Backfill | From [date], within [request budget below the source's published limit] |
| Freshness alert | Last message older than [your limit], sent to [your channel] |
| Retention | Raw: [period your license allows]. Normalized: [period your license allows] |

## What is checked first

### What your license allows

You hold the license, so we ask what it says about each use in the table above. A stored copy does not change those terms. If your plan looks outside them, we say so before quoting, and the answer is for you and your lawyer.

### What history actually exists

How far back the source goes, whether delisted instruments stay available, how paging and limits work, and whether order book history exists at all. A period the source no longer offers stays a gap.

### How exact the data must be

A dashboard needs the latest value. A backtest needs what was known at the time. A study of fills needs depth and receive times. The answer changes the design and the scope.

### A sample of your real feed

A short capture from your own instruments gives message rate, message size and compressed size. Storage is chosen from those numbers, not from a guess.

### Your country and the platform's terms

For customers outside Türkiye, which platforms can be used depends on the law of your country and the platform's terms. For customers resident in Türkiye, we work with platforms on the Capital Markets Board's list that offer an API; being listed does not mean a platform is authorized.

## When you do not need us

If you need daily candles for a few dozen symbols, a scheduled script and one database table will do. If you need crypto historical data you never recorded, buy it from a crypto historical data provider: no collector can capture the past. If a vendor's market data API already covers your symbols and its license covers your use, use it and keep your own code small.

A pipeline of your own makes sense when nobody sells the data in the form you need, when you must be able to show what your software saw at a given moment, or when your use outgrows what a vendor's plan allows. Compare it with the vendor's price first: running a collector has a cost too.

### The one real example

Fiva, Fincode's own iPhone app, is in beta and in Turkish only. It brings stocks, funds, gold, FX and crypto data into one place, which means reconciling symbols, trading hours, holiday calendars and corporate actions, and it tells users which data is real-time and which is delayed. It does not execute trades, and it is not client work. Your project gets its own sources and licenses.

[Fiva case study](https://fincodelabs.com/en/projects/fiva-fund-return-estimate-platform)

## Questions about market data pipelines

### Do you supply the market data?

No. Fincode builds the software. The data comes from sources you are licensed to use, and those contracts stay between you and the source.

### Can you sign a Borsa Istanbul market data agreement for us?

No. That agreement is between your company and Borsa Istanbul or a licensed distributor, and Fincode cannot hold it for you. We can read the documents in Turkish and build to what your agreement allows. See [Borsa Istanbul market data](https://fincodelabs.com/en/blog/borsa-istanbul-api-market-data).

### Is the data real time?

It is as current as the source and your license allow, which varies by source and plan, so this page states no latency figure. If latency matters to your project, the test report measures it on your feed and says how it was measured.

### Can you collect crypto tick data and level 2 order book data?

Yes, where the source publishes it and its terms allow your use: trades through streams and REST, and depth through a snapshot plus updates with sequence checks. Depth is usually far larger than trades, so the scope fixes which instruments and how many levels.

### Can you backfill crypto historical data and OHLCV candles?

As far back as the source allows and its terms permit. Backfills run as resumable jobs within published limits. A period the source no longer offers stays marked as a gap and is never filled by interpolation.

### Should market data go in ClickHouse, TimescaleDB, Parquet or Kafka?

It depends on volume and on who reads the data. The scope document names the choice and the reason, based on a sample of your feed. Files are often enough for research, and Kafka earns its cost only with several consumers or a need for replay at that layer.

### Can TradingView be a data source for the pipeline?

No. TradingView's [Terms of Use](https://www.tradingview.com/policies/), read on 5 November 2026, license its charts, alerts, webhooks and other information for display use only and prohibit machine-driven use. Use an exchange or vendor whose license covers what you want to do.

### Does the pipeline need my API keys?

Public market data usually needs no key. Where a source requires one, the pipeline uses read permission only and the key stays on your server. This work never needs trading or withdrawal rights.

### Can the data feed my trading software or backtests?

Yes, if your license covers that use. Your software reads from the store like any other source. The pipeline itself places no orders, and tests show how the software behaves, not future results.

### How is the price set?

You get a written proposal with work items, dates and cost before work starts. Fees depend on scope and effort, not on trading volume or results. Any data fees are between you and the source.

## Read before you scope a pipeline

- [Local order book via WebSocket](https://fincodelabs.com/en/blog/websocket-order-book): Snapshot and update sync, and what to do when an update goes missing.
- [KAP disclosure alert bot: example scenario](https://fincodelabs.com/en/projects/kap-disclosure-alert-bot): Ingest, deduplication, corrections and reconciliation for one Turkish data feed. Not a client project.
- [Trading dashboards](https://fincodelabs.com/en/services/trading-dashboards): Read-only screens and reports on top of the data you collect.

---

Published by Fincode (Fincode Yazılım ve Teknoloji A.Ş.), a software company in Istanbul, Turkey: https://fincodelabs.com/en/services/market-data-pipelines