# Trading bot architecture: events, state and crash recovery

URL: https://fincodelabs.com/en/blog/trading-bot-architecture · Publisher: Fincode (Fincode Yazılım ve Teknoloji A.Ş.), Istanbul, Turkey · Category: Trading bots · Published: 2026-11-06 · Updated: 2026-11-06

Trading bot architecture below the component list: event log, reconcile on start, backpressure, a separate risk process and a crash test you can run.

**Short answer:** **Trading bot architecture** covers how the parts talk and what survives a crash: log each order intent before sending it, rebuild state from that log, reconcile with the exchange on every start, and give the order path its own process. A tested asyncio skeleton built this way, run on 5 November 2026, recovered from a crash mid-order and a partial fill with one order at the fake venue. Architecture decides how failures behave; it does not make any rule profitable, and automated trading can lose money.

The components are listed in [trading bot development](https://fincodelabs.com/en/blog/trading-bot-development); this page covers the boundaries between them and what happens when a process dies between two lines of code.

Everything measured here ran on synthetic data on one Windows 11 machine with Python 3.11.3; the trading rule is a stub. No exchange is recommended; which platforms you may use depends on your jurisdiction and their terms. Article 99/A of Capital Markets Law No. 6362 treats activity by foreign crypto-asset platforms directed at Turkish residents as unauthorized.

## Trading bot structure: one event loop or separate processes?

Start with one event loop if a single strategy trades a single venue, and move the order path into its own process once a stalled strategy could leave orders unmanaged.

**Three layouts, with the failure each one invites**

| Layout | What it buys | What can go wrong |
| --- | --- | --- |
| One process, one event loop | One order of events, simple replay | One blocking call stops everything, limits included |
| Strategy process plus gateway | A strategy bug cannot skip limits or hold up cancels; keys live in one place | The gateway needs a supervisor |
| Services on a message broker | Several strategies and accounts on one view | Delivery order and duplicates at every hop |

Order events must never be dropped or reordered; market data can be thinned.

## How to code a trading bot: start with the event log

Write the event log and a fake exchange first, then the real adapter, and the strategy last, because the failure path cannot be added later.

1. Define the events: intent, acknowledged, fill, absent.
2. Write the log and a pure function that folds events into state.
3. Build a fake exchange that can fail at any line, and test crashes against it.
4. Add the real adapter, on a test environment.

Because the intent is durable before the send, a restart knows what it may have sent.

Durability has a price: on this machine on 5 November 2026, an append with `fsync` took a median of 1.14 to 1.22 ms over three runs of 1,000, against 0.47 to 0.52 ms without. NautilusTrader takes the other side: its built-in Redis and PostgreSQL backends write asynchronously, and its documents warn of a "persistence gap": a venue can receive an order before the local record is durable.

## Trading bot code: a crash test with a fake exchange

This skeleton survives a crash after the venue accepted an order, a partial fill during the downtime and a repeated fill, restarting from the log alone.

```python
import asyncio, json, math, os, tempfile

class Crash(BaseException):
    """The process dying mid-call: except Exception cannot catch it."""

class FakeExchange:
    """Outlives restarts. A repeated client ID creates nothing new."""
    def __init__(self):
        self.orders, self.die_after_accept = {}, False

    async def place(self, cid, qty):
        await asyncio.sleep(0)
        self.orders.setdefault(cid, {"qty": qty, "fills": []})
        if self.die_after_accept:
            self.die_after_accept = False
            raise Crash

    def fill(self, cid, fill_id, qty):
        self.orders[cid]["fills"].append({"id": fill_id, "qty": qty})

class EventLog:
    """Append-only JSON lines. An event exists once fsync returns
    (on Linux, fsync the directory too after creating the file)."""
    def __init__(self, path):
        self.path = path
        if os.path.exists(path):              # dying mid-append leaves a torn last line
            with open(path, "rb+") as f:
                f.truncate(f.read().rfind(b"\n") + 1)

    def append(self, **event):
        with open(self.path, "a", encoding="utf-8") as f:
            f.write(json.dumps(event) + "\n")
            f.flush()
            os.fsync(f.fileno())

    def read(self):
        if not os.path.exists(self.path):
            return []
        with open(self.path, encoding="utf-8") as f:
            return [json.loads(line) for line in f]

def fold(events):
    orders, seen = {}, set()
    for e in events:
        if e["type"] == "intent":
            orders[e["cid"]] = {"qty": e["qty"], "filled": 0.0, "state": "INTENT"}
        elif e["type"] == "acked" and orders[e["cid"]]["state"] == "INTENT":
            orders[e["cid"]]["state"] = "OPEN"
        elif e["type"] == "absent":
            orders[e["cid"]]["state"] = "ABSENT"
        elif e["type"] == "fill" and (e["cid"], e["id"]) not in seen:   # repeats change nothing
            seen.add((e["cid"], e["id"]))
            o = orders[e["cid"]]
            o["filled"] += e["qty"]
            o["state"] = "FILLED" if o["filled"] >= o["qty"] else "PARTIAL"
    return orders, sum(o["filled"] for o in orders.values())

class Bot:
    def __init__(self, log, exchange, name="bot1"):
        self.log, self.ex, self.name, self.halted = log, exchange, name, False

    def next_id(self):                        # repeats only if the log is lost
        used = sum(e["type"] == "intent" for e in self.log.read())
        return f"{self.name}-{used + 1:06d}"

    async def submit(self, qty):
        if self.halted:
            raise RuntimeError("halted")
        if not (math.isfinite(qty) and qty > 0):   # "1e400" parses to inf, "1e-400" to 0.0
            raise ValueError(f"bad quantity {qty}")
        cid = self.next_id()
        self.log.append(type="intent", cid=cid, qty=qty)     # durable before the send
        await self.ex.place(cid, qty)
        self.log.append(type="acked", cid=cid)
        return cid

    def on_fill(self, cid, fill_id, qty):
        if not (fill_id and math.isfinite(qty) and qty > 0) or cid not in fold(self.log.read())[0]:
            self.halted = True
            return
        self.log.append(type="fill", cid=cid, id=fill_id, qty=qty)

    async def reconcile(self):
        """Runs on every start, before the strategy sees one event."""
        local, _ = fold(self.log.read())
        venue = dict(self.ex.orders)          # a real fetch: halt if an ID matches twice
        for cid, order in local.items():
            if cid in venue:
                if order["state"] == "INTENT":
                    self.log.append(type="acked", cid=cid)
                for f in venue[cid]["fills"]:
                    self.on_fill(cid, f["id"], f["qty"])
            elif order["state"] == "INTENT":
                self.log.append(type="absent", cid=cid)
        if set(venue) - set(local):           # an order nobody here wrote
            self.halted = True
        return fold(self.log.read())

async def main(path):
    ex = FakeExchange()
    bot = Bot(EventLog(path), ex)
    ex.die_after_accept = True
    try:
        await bot.submit(1.0)
    except Crash:
        print("crash; log holds:", [e["type"] for e in bot.log.read()])
    ex.fill("bot1-000001", "t1", 0.4)         # partial fill while the process is down

    bot = Bot(EventLog(path), ex)             # restart: only log and venue survive
    orders, position = await bot.reconcile()
    print("after restart:", orders, "position", position, "halted", bot.halted)

    bot.on_fill("bot1-000001", "t1", 0.4)     # the stream repeats a fill
    ex.fill("bot1-000001", "t2", 0.6)
    bot.on_fill("bot1-000001", "t2", 0.6)
    print("after fills:", fold(bot.log.read()), "orders at venue", len(ex.orders))

if __name__ == "__main__":
    with tempfile.TemporaryDirectory() as d:
        asyncio.run(main(os.path.join(d, "events.jsonl")))
```

Output:

```
crash; log holds: ['intent']
after restart: {'bot1-000001': {'qty': 1.0, 'filled': 0.4, 'state': 'PARTIAL'}} position 0.4 halted False
after fills: ({'bot1-000001': {'qty': 1.0, 'filled': 1.0, 'state': 'FILLED'}}, 1.0) orders at venue 1
```

> **What happened.** The log held only the intent, so the restarted bot asked the venue, recorded the partial fill and reported 0.4, not the 1.0 it asked for. Other checks passed on 5 November 2026: a crash before the send, a torn last line, a fill before its acknowledgement, an unknown order at the venue, a fill with no ID or a bad quantity, or for an unknown order (halts; logging it would break restarts), a self-trade's shared trade ID, and inf, NaN, zero or negative quantities (refused).

A real venue may answer a lookup late, and its rule for repeated IDs may be narrower than this fake's. A timed-out order request leaves the same unknown as a crash; retry rules are in [CCXT error handling](https://fincodelabs.com/en/blog/ccxt-rate-limit-error-handling), and the full order state machine in the [crypto OMS guide](https://fincodelabs.com/en/blog/crypto-order-management-system).

## Backpressure and clocks: when a consumer falls behind

A slow strategy must not grow an unbounded queue, because it then acts on old prices while every health check stays green.

**2,000 synthetic ticks, three arriving per tick handled (deterministic)**

| Policy | Deepest queue | Dropped | Worst lag |
| --- | ---: | ---: | --- |
| Unbounded queue | 1,334 | 0 | 1,333 ticks |
| Bounded at 100, producer blocks | 100 | 0 | 99 ticks here; the backlog moves into the socket |
| Bounded at 100, drop oldest | 100 | 1,234 | 99 ticks |
| Keep newest only | 1 | 1,333 | 0 ticks |

Dropping is only safe for data a newer message fully replaces, such as a ticker. Order book deltas cannot be skipped without a resync, as the [WebSocket order book guide](https://fincodelabs.com/en/blog/websocket-order-book) shows. Count drops and alert on them.

- **Timeouts and heartbeats:** a monotonic clock, which Python's documentation says "cannot go backwards".
- **Request timestamps:** a synchronized wall clock; signed requests fail when it drifts.
- **Event order:** store venue time and receive time; never sort fills by arrival alone.
- **Replay:** inject the clock so recorded data drives the same code.

## A risk process that can stop orders when the strategy hangs

Keep the limits and the order path in a process separate from the strategy, because a watchdog inside a blocked event loop cannot run.

```python
import asyncio, json

async def gateway(exchange, strategy_cmd, silence_limit=0.5, startup_limit=10.0):
    """Holds the keys and the order path. The strategy only writes lines to a pipe."""
    child = await asyncio.create_subprocess_exec(*strategy_cmd, stdout=asyncio.subprocess.PIPE)
    limit = startup_limit
    try:
        while line := await asyncio.wait_for(child.stdout.readline(), limit):
            limit = silence_limit
            message = json.loads(line)
            if message.get("type") == "intent":   # add limits here; json.loads accepts NaN
                await exchange.submit(message)
        reason = "strategy exited"
    except asyncio.TimeoutError:
        reason = "strategy silent"            # alive or not, it no longer gets to trade
    except Exception as e:                    # a stray line or failed send still cancels
        reason = "gateway error: " + repr(e)
    finally:
        if child.returncode is None:
            child.kill()
    await exchange.cancel_all(reason)
    await child.wait()
    return reason
```

**Time to react to a stalled strategy, limit 0.5 s, three runs, 5 November 2026**

| Watchdog | The stall | Reacted after |
| --- | --- | --- |
| Task in the same event loop | `time.sleep(2)` in a callback | 2.0 s in all three runs, only once the block ended |
| Gateway, separate process | Child alive but silent | 0.515, 0.5 and 0.515 s after its last message |

In these runs the earlier intent was handled before `cancel_all`, and an abrupt exit, a non-JSON line or a failed send also ended in `cancel_all`. What the cancel should cover and who may reset it are in [trading bot risk management](https://fincodelabs.com/en/blog/trading-bot-risk-management).

## Crypto trading bot architecture in three open-source projects

Three published designs answer these questions differently; this describes their documents and source, read 5 November 2026, and ranks nothing.

**Three designs (alphabetical)**

| Question | Freqtrade | Hummingbot | NautilusTrader |
| --- | --- | --- | --- |
| Control flow | One bot loop; `process_throttle_secs` defaults to 5 s | A clock (`tick_size` 1.0) ticks child iterators; strategies read state each tick | A single-threaded core; network and persistence run elsewhere and feed it through channels |
| How parts talk | Direct calls and strategy callbacks | Connectors standardize each venue's API for strategies | A message bus: publish/subscribe, request/response, commands and events |
| Durable state | SQL database, SQLite by default | A recorder saves and restores connector tracking states | Cache with optional backing; durability depends on that store |
| On start | Re-fetches orders its database lists as open, except in dry-run | Restores saved tracking states | Reconciles against venue reports before strategies start |
| On a fault | Temporary errors retry after 30 s; operational errors stop the bot | Tick errors are logged and the loop continues; an order not found over 3 times is marked lost | Release builds abort on panic; an external supervisor restarts |
| License, latest release (GitHub, 5 November 2026) | GPL-3.0; 2026.10, 30 October | Apache-2.0; v2.17.0, 22 September | LGPL-3.0; 1.231.0, 2 August; 2.0.0 candidates since |

NautilusTrader's architecture page says: "Venue commands are not generally safe to retry blindly." Repository checks are in the [open source bot guide](https://fincodelabs.com/en/blog/open-source-crypto-trading-bot).

## Trading bot programming language: what each layer needs

No language suits every layer; match each layer's needs and keep boundaries simple enough to change one side later.

**What each layer asks of a language**

| Layer | What it needs |
| --- | --- |
| Exchange adapter | Async I/O, signing, a maintained library for your venue |
| Strategy and research | Data tools, fast iteration, the same code in replay and live |
| Event log and state | Exact decimals (the sketch uses floats for brevity), simple serialization |
| Limits and gateway | Small code, few dependencies, easy for someone else to review |
| Latency-critical path | Predictable pauses; measure before blaming the language |

The projects above show the range: GitHub showed Freqtrade at 98.5% Python, Hummingbot at 97.2% Python and NautilusTrader at 93.7% Rust on 5 November 2026. The [algorithmic trading](https://fincodelabs.com/en/services/algorithmic-trading) page describes commissioning such a system; rules and decisions stay with you.

## Frequently asked questions

### What is the best architecture for a trading bot?

No single one. One event loop suits one strategy on one venue; a gateway process makes sense once a stalled strategy could leave orders unmanaged.

### Which programming language is best for a trading bot?

None for every layer; the limits and order path mainly need small, testable code.

### Why log an order before sending it?

So a restart after a crash between send and reply knows what it may have sent, and asks the exchange instead of sending a duplicate.

### Does a good architecture make a bot profitable?

No. It controls how the software behaves under failure. Trades can still lose money.

## Sources

- [NautilusTrader documentation: Architecture (threading model, crash-only design)](https://github.com/nautechsystems/nautilus_trader/blob/develop/docs/concepts/architecture.md)
- [NautilusTrader documentation: Execution policies (command outcomes, persistence before transport)](https://github.com/nautechsystems/nautilus_trader/blob/develop/docs/concepts/execution/policies.md)
- [NautilusTrader documentation: Execution reconciliation](https://github.com/nautechsystems/nautilus_trader/blob/develop/docs/concepts/execution/reconciliation.md)
- [Freqtrade documentation: Freqtrade basics (bot execution logic)](https://www.freqtrade.io/en/stable/bot-basics/)
- [Freqtrade documentation: Configuration (process_throttle_secs, db_url)](https://www.freqtrade.io/en/stable/configuration/)
- [Freqtrade source: worker.py (error handling)](https://github.com/freqtrade/freqtrade/blob/develop/freqtrade/worker.py)
- [Freqtrade source: freqtradebot.py (startup_update_open_orders)](https://github.com/freqtrade/freqtrade/blob/develop/freqtrade/freqtradebot.py)
- [Hummingbot documentation: Strategies (clock tick)](https://hummingbot.org/strategies/)
- [Hummingbot documentation: Connectors](https://hummingbot.org/connectors/)
- [Hummingbot source: core/clock.pyx](https://github.com/hummingbot/hummingbot/blob/master/hummingbot/core/clock.pyx)
- [Hummingbot source: connector/client_order_tracker.py (lost orders)](https://github.com/hummingbot/hummingbot/blob/master/hummingbot/connector/client_order_tracker.py)
- [Hummingbot source: connector/markets_recorder.py (tracking states)](https://github.com/hummingbot/hummingbot/blob/master/hummingbot/connector/markets_recorder.py)
- [Python documentation: time.monotonic](https://docs.python.org/3/library/time.html#time.monotonic)
- [Linux manual page: fsync(2), directory entries need their own fsync](https://man7.org/linux/man-pages/man2/fsync.2.html)
- [Capital Markets Law No. 6362, consolidated text, Article 99/A (Turkish)](https://www.mevzuat.gov.tr/mevzuatmetin/1.5.6362.pdf)

> This guide is general information about software and markets, not investment, legal or tax advice. Automated trading can lose money, past or backtested performance does not guarantee future results, and the rules that apply depend on your jurisdiction and your exchange's terms. Sources, checks and corrections: see the [editorial policy](https://fincodelabs.com/en/editorial-policy).

---

Published by Fincode (Fincode Yazılım ve Teknoloji A.Ş.), a software company in Istanbul, Turkey: https://fincodelabs.com/en/blog/trading-bot-architecture

Cite as: Fincode, "Trading bot architecture: events, state and crash recovery", updated November 6, 2026, https://fincodelabs.com/en/blog/trading-bot-architecture