# Machine learning trading bot development: tested failures

URL: https://fincodelabs.com/en/blog/machine-learning-trading-bot-development · Publisher: Fincode (Fincode Yazılım ve Teknoloji A.Ş.), Istanbul, Turkey · Category: Trading bots · Published: 2026-11-06 · Updated: 2026-11-06

Machine learning trading bot development: a leakage test on random data, feature parity, rollback, NaN at inference and why RL results can mirror the simulator.

**Short answer:** **Machine learning trading bot development** adds a training pipeline, a model file and a second copy of every feature to an ordinary bot, and much of what breaks lives in those additions. On a synthetic random walk, a look-ahead feature with a shuffled split scored 70.8% accuracy, while past-only features under TimeSeriesSplit scored 48.5% (seed 7, run 5 November 2026). A high score proves nothing until leakage and feature parity are tested, and never forecasts returns.

This guide uses no real asset data, and each script exists to show a failure, so no feature, model or hyperparameter here is a setting to use. Scripts ran on 5 and 6 November 2026 on Python 3.11.3, scikit-learn 1.8.0, numpy 2.0.0 and pandas 2.2.3.

Fincode works from rules the customer wrote and supplies no models or strategies, so this page does not sell ML trading, and nothing here suggests a model can predict markets. Automated trading can lose money. For readers in Turkey: Article 99/A of Capital Markets Law No. 6362 treats activity by foreign crypto-asset platforms directed at Turkish residents as unauthorized.

## What changes when a model sits inside a trading bot

The order, risk and exchange layers described in [trading bot development](https://fincodelabs.com/en/blog/trading-bot-development) stay the same; you add artifacts that can fail with no code change.

**Rule-based bot and bot with a model**

| Item | Rule-based | With a model |
| --- | --- | --- |
| Decision logic | Code you can read | A file produced by training |
| Proof it is correct | Expected output for a given input | A score on unseen data, easy to inflate |
| Inputs | Market data | A feature pipeline that exists twice, in research and live |
| Typical failure | An error or a wrong rule | A confident output from bad input |

## Machine learning trading bot in Python: a leakage test on random data

This script produces strong validation accuracy on returns that are random by construction.

The label is the sign of the sum of the next 10 returns. Three features use past returns only; a fourth is a centered average that reaches 5 bars ahead.

```python
import numpy as np, pandas as pd
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import KFold, TimeSeriesSplit, cross_val_score

SEED, N, H = 7, 4000, 10                   # H: bars ahead in the label
rng = np.random.default_rng(SEED)
ret = pd.Series(rng.normal(0, 0.01, N))    # synthetic random walk

fwd = ret.rolling(H).sum().shift(-H)       # sum of the next H returns
y = (fwd > 0).astype(int)
past = pd.concat({f"mean_{w}": ret.rolling(w).mean() for w in (5, 20, 60)}, axis=1)
leak = past.assign(centered=ret.rolling(11, center=True).mean())
keep = leak.notna().all(axis=1) & fwd.notna()

shuffled = KFold(5, shuffle=True, random_state=SEED)
tests = [("look-ahead feature, shuffled KFold", leak, shuffled),
         ("look-ahead feature, TimeSeriesSplit", leak, TimeSeriesSplit(5)),
         ("past-only features, shuffled KFold", past, shuffled),
         ("past-only features, TimeSeriesSplit", past, TimeSeriesSplit(5)),
         ("past-only features, TimeSeriesSplit gap=10", past, TimeSeriesSplit(5, gap=H))]
print("seed", SEED, "rows", keep.sum(), "share of class 1", round(y[keep].mean(), 3))
for name, X, cv in tests:
    score = cross_val_score(RandomForestClassifier(random_state=SEED), X[keep], y[keep], cv=cv)
    print(f"{name:43s}{score.mean():.3f}")
```

Output:

```
seed 7 rows 3931 share of class 1 0.485
look-ahead feature, shuffled KFold         0.708
look-ahead feature, TimeSeriesSplit        0.664
past-only features, shuffled KFold         0.550
past-only features, TimeSeriesSplit        0.485
past-only features, TimeSeriesSplit gap=10 0.494
```

- **0.708:** the look-ahead feature sees half of the label's window.
- **0.664:** walk-forward validation does not cure a leaky feature. `TimeSeriesSplit` only orders the rows.
- **0.550:** no look-ahead feature here. Labels overlap, so shuffled neighbors share most of their future, and the split itself leaks.
- **0.485 and 0.494:** back to chance. `gap` drops rows just before each test fold; its default is 0 and it knows nothing about your label, so set it to at least the label's horizon.

Over seeds 0 to 19, the last row averaged 0.502 and ranged from 0.475 to 0.522, while the past-only shuffled row never fell below 0.536. So one run at 52% is ordinary noise. Run your own pipeline on synthetic returns: a score clearly above chance means it leaks. scikit-learn defines leakage as information unavailable at prediction time; general causes are in [backtesting pitfalls](https://fincodelabs.com/en/blog/backtesting-trading-strategy-pitfalls), and engines in [backtesting Python libraries](https://fincodelabs.com/en/blog/backtesting-python-libraries).

## Feature parity between research and live code

The same feature name can hold different numbers in the notebook and the bot, so compare both code paths on the same bars.

```python
from collections import deque
import numpy as np, pandas as pd

ret = pd.Series(np.random.default_rng(7).normal(0, 0.01, 500))   # synthetic bars
research = ret.rolling(20).std()                                 # pandas default: ddof=1
for ddof in (0, 1):                                              # numpy default: ddof=0
    window, live = deque(maxlen=20), []
    for x in ret:                                                # live code: one bar at a time
        window.append(x)
        live.append(np.std(window, ddof=ddof) if len(window) == 20 else np.nan)
    assert (research.isna() == pd.isna(live)).all(), "NaN on different bars"
    gap = (research - pd.Series(live)).abs().max()
    print(f"live ddof={ddof}: largest gap {gap:.8f}")
```

Output:

```
live ddof=0: largest gap 0.00032525
live ddof=1: largest gap 0.00000000
```

The defaults differ, as the pandas reference notes, so two correct-looking lines differ by a steady 2.5% and no error ever appears. A second case: `ewm(span=10)` with pandas' default `adjust=True` against a recursive live average differed by 0.00154 at bar 5 and by under 1e-17 at bar 400, and `adjust=False` removed the gap, so compare the first bars after a restart as well as the last. Run this in continuous integration, or better, import one feature module in both paths.

## Model versioning, retraining and rollback

Treat a trained model as a release: versioned, reproducible, tied to its data and replaceable by the previous one.

**Items to settle; scikit-learn and FreqAI points read 5 November 2026**

| Item | What to require |
| --- | --- |
| Environment | Load in the same package versions. scikit-learn says pickle, joblib, cloudpickle and skops do not support loading a model trained with a different version |
| File safety | Pickle and joblib can execute code when loaded, so load only trusted, verified files |
| Metadata | Data snapshot, source code, library versions and cross-validation score (scikit-learn's list), plus the label definition and `gap` |
| Retraining | A retrain is a deploy: same tests, then a dry run |
| Rollback | Keep the previous model with its features. FreqAI's `purge_old_models` removes older models, so set it with rollback in mind |
| Expiry | Refuse new entries from an old model, as FreqAI's `expiration_hours` does |

Check the data license before training. The Binance Vision Dataset Terms (updated 26 August 2026, read 5 November 2026) grant only a non-commercial license (section 3), extend to models built from the files (2.4) and bar live proprietary trading and commercial trading bot platforms (4.2, 4.4). Read any source's terms with a lawyer before a commercial build.

## Drift monitoring and fallback when inference fails

Decide what the bot does when the model is down, stale or fed bad input, because some failures raise no error.

```python
import numpy as np, sklearn
from sklearn.ensemble import RandomForestClassifier
from sklearn.linear_model import LogisticRegression

rng = np.random.default_rng(7)
X, y = rng.normal(size=(2000, 3)), rng.integers(0, 2, 2000)   # synthetic, no NaN in training
nan_row = np.array([[np.nan, 0.1, 0.2]])
for m in (LogisticRegression(), RandomForestClassifier(random_state=7)):
    m.fit(X, y)
    try:
        print(type(m).__name__, "default:", m.predict(nan_row))
    except ValueError:
        print(type(m).__name__, "default: ValueError")
    with sklearn.config_context(assume_finite=True):
        print(type(m).__name__, "assume_finite:", m.predict(nan_row))
```

Output:

```
LogisticRegression default: ValueError
LogisticRegression assume_finite: [0]
RandomForestClassifier default: [1]
RandomForestClassifier assume_finite: [1]
```

A random forest trained on clean data returned a class for a NaN row with no error. scikit-learn's tree page documents the rule for single trees: such values go to the child with the most samples. Logistic regression raises by default, but under `assume_finite=True`, which the performance page offers to cut validation time, it also returned a class. Check `np.isfinite(row).all()` yourself before predicting.

**Behavior to write into the specification (a suggestion, not a setting)**

| Event | Bot behavior |
| --- | --- |
| NaN feature or stale bar | No new entries, alert |
| Inference error or timeout | No new entries; open positions follow deterministic exit rules |
| Live features or outputs drift from the training window | Alert and human review, no automatic retrain |

Drift shows the data changed, not that the model is wrong, so log feature and output distributions rather than judging by recent results. Limits outside the model are in [trading bot risk management](https://fincodelabs.com/en/blog/trading-bot-risk-management). Measure latency too: on one busy Windows machine (three runs, 5 November 2026), a single-row `predict_proba` took a median of about 0.1 ms for logistic regression, 9 to 12 ms for a default forest and 32 to 35 ms with `n_jobs=-1`; not a benchmark, but threads slowed a single row.

## Reinforcement learning trading bot: why results often reflect the simulator

An RL agent optimizes the simulator's reward, so any free fill or missing cost in the environment becomes something to exploit.

- **Seeds.** Stable-Baselines3's guide says results vary when only the seed changes and asks for several runs and a separate test environment. Henderson et al. found that two groups of five runs of TRPO on HalfCheetah, differing only in seed, gave statistically different distributions.
- **Sample size.** The same guide says these algorithms can need millions of interactions, and that most RL successes came in games or simulation.
- **Frictions.** The 2020 FinRL paper treats transaction cost and bid-ask spread as parameters, and its text does not discuss market impact or slippage. What you enter is the market the agent learns.
- **Reward.** FreqAI's RL page says its training environment is much simpler than a backtest, that its default reward is not for production and that poorly trained agents may find "cheats" that raise reward without winning trades.

> **Worked example.** One million steps is 3,968 years of daily bars at 252 a year, or 1.9 years of one-minute bars in a 24-hour market, so an agent on daily data replays the same history many times.

A null test in Python catches the worst of this: on pure noise, every policy must earn zero less costs.

```python
import numpy as np

r = np.random.default_rng(7).normal(0, 1, 200_000)   # synthetic returns, no signal exists
pos = np.sign(r)                                     # any policy works; this one reacts to r
same_bar = pos * r                                   # bug: paid the return it just saw
next_bar = pos[:-1] * r[1:]                          # correct: position now, paid next bar
turnover = np.abs(np.diff(pos[:-1], prepend=0))
with_cost = next_bar - 0.05 * turnover               # hypothetical cost per unit traded
print(f"same-bar fill {same_bar.mean():+.3f}")
print(f"next-bar fill {next_bar.mean():+.3f}")
print(f"next-bar fill with cost {with_cost.mean():+.3f}")
```

Output, in units of one standard deviation per bar:

```
same-bar fill +0.797
next-bar fill -0.001
next-bar fill with cost -0.051
```

The first line is a one-line environment bug paying a policy on noise. Stable-Baselines3's `check_env` checks the interface, not this. These figures describe a simulator on synthetic data and say nothing about any market.

## Machine learning trading bot frameworks: what each one is

Each tool leaves the problems above to you.

**From each project's documents and license file, 5 November 2026 (alphabetical, not a ranking)**

| Tool | License | What it is | Watch for |
| --- | --- | --- | --- |
| FinRL | [MIT](https://github.com/AI4Finance-Foundation/FinRL/blob/master/LICENSE) | Deep RL library with market environments and backtesting; its README says education, benchmarking and research prototyping | Costs and spread are yours to set |
| FreqAI (Freqtrade) | [GPL-3.0](https://github.com/freqtrade/freqtrade/blob/develop/LICENSE) | Trains and retrains models beside a bot | Example strategy is not for production; in backtests, features are built once over the training range and must not look ahead; no dynamic volume pairlists; GPL terms apply if you distribute it |
| scikit-learn | [BSD-3-Clause](https://github.com/scikit-learn/scikit-learn/blob/main/COPYING) | General ML library with splitters and pipelines | It knows nothing about bars or label horizons |
| Stable-Baselines3 | [MIT](https://github.com/DLR-RM/stable-baselines3/blob/master/LICENSE) | RL algorithm implementations | You supply the environment and its economics |

## What to ask when reviewing a machine learning trading bot

Ask for evidence of behavior, since no review can say what a model will earn.

1. Does a synthetic random walk score near chance through the full pipeline?
2. Is `gap` at least the label horizon, and can the label be computed only from data after the decision?
3. Do research and live features match bar for bar in continuous integration?
4. Which data snapshot and commit built the running model, and what restores the previous one?
5. What happens on NaN, on a timeout and on an expired model?
6. What does the training data's license allow?

Fincode's [trading bot code review](https://fincodelabs.com/en/services/trading-bot-code-review) can put these to an existing codebase; it checks software behavior and offers no view on a model's results.

## Frequently asked questions

### Can machine learning predict markets for a trading bot?

This guide makes no such claim. Its test uses random data to show how validation mistakes manufacture accuracy.

### Why is my ML trading backtest so accurate?

Suspect leakage first. Feed the same pipeline a synthetic random walk: a score clearly above chance means something leaks.

### Is reinforcement learning better than supervised learning for a trading bot?

This guide ranks neither. RL adds the simulator as one more source of false results.

### Is there AI trading bot Python code I can copy from here?

This page gives test code, not a bot. Run these tests on any code you copy.

## Sources

- [scikit-learn: TimeSeriesSplit (documentation 1.9.1)](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.TimeSeriesSplit.html)
- [scikit-learn: Common pitfalls and recommended practices (data leakage)](https://scikit-learn.org/stable/common_pitfalls.html)
- [scikit-learn: Model persistence (security, version compatibility, metadata)](https://scikit-learn.org/stable/model_persistence.html)
- [scikit-learn: Computational performance (prediction latency, assume_finite)](https://scikit-learn.org/stable/computing/computational_performance.html)
- [scikit-learn: Decision trees (missing values support)](https://scikit-learn.org/stable/modules/tree.html)
- [pandas: Rolling.std reference (ddof default and numpy note)](https://pandas.pydata.org/docs/reference/api/pandas.api.typing.Rolling.std.html)
- [Stable-Baselines3: Reinforcement Learning Tips and Tricks](https://stable-baselines3.readthedocs.io/en/master/guide/rl_tips.html)
- [Henderson et al.: Deep Reinforcement Learning that Matters (arXiv 1709.06560)](https://arxiv.org/abs/1709.06560)
- [Liu et al.: FinRL: A Deep Reinforcement Learning Library for Automated Stock Trading in Quantitative Finance (arXiv 2011.09607)](https://arxiv.org/abs/2011.09607)
- [AI4Finance Foundation: FinRL repository README (scope statement)](https://github.com/AI4Finance-Foundation/FinRL)
- [Freqtrade: FreqAI introduction](https://www.freqtrade.io/en/stable/freqai/)
- [Freqtrade: Running FreqAI (retraining, expiration_hours, purge_old_models)](https://www.freqtrade.io/en/stable/freqai-running/)
- [Freqtrade: FreqAI reinforcement learning](https://www.freqtrade.io/en/stable/freqai-reinforcement-learning/)
- [Binance Vision Dataset Terms, last updated 26 August 2026](https://github.com/binance/binance-public-data/blob/master/TERMS_AND_CONDITIONS.md)
- [Capital Markets Law No. 6362, consolidated text, Article 99/A (Turkish)](https://www.mevzuat.gov.tr/mevzuatmetin/1.5.6362.pdf)

> This guide is general information about software and markets, not investment, legal or tax advice. Automated trading can lose money, past or backtested performance does not guarantee future results, and the rules that apply depend on your jurisdiction and your exchange's terms. Sources, checks and corrections: see the [editorial policy](https://fincodelabs.com/en/editorial-policy).

---

Published by Fincode (Fincode Yazılım ve Teknoloji A.Ş.), a software company in Istanbul, Turkey: https://fincodelabs.com/en/blog/machine-learning-trading-bot-development

Cite as: Fincode, "Machine learning trading bot development: tested failures", updated November 6, 2026, https://fincodelabs.com/en/blog/machine-learning-trading-bot-development