What I Wanted To Know
A technical verdict is easy to make look smart. Price above its moving averages, RSI in a "bullish zone", MACD above its signal line: every chart tells a story after the fact. The question I cared about was narrower and harder: when the engine says bullish on a given day, is the stock more likely to be up N days later than if I had ignored the engine and guessed? If the answer was no, I did not want a green badge on the site implying otherwise.
The Engine Under Test
The verdict comes from one TypeScript function, analyzeStock, in the website's analysis library. It computes around nineteen readings (moving averages, RSI, MACD, Bollinger Bands, Stochastic, ATR, ADX, OBV, MFI, Williams %R, CCI, pivots, Fibonacci levels and fractals), and all of them are shown to the user. But only six of them move the score, and I think that is the first thing worth admitting:
- Price vs the 20-day average: +1 above, -1 below
- Price vs the 50-day average: +1 / -1
- Price vs the 200-day average: +2 / -2
- RSI: +2 oversold, -2 overbought, +1 or -1 in the zones between
- MACD above or below its signal line: +1 / -1
- Money Flow Index: +1.5 in the accumulation zone, -1.5 in the distribution zone
The sum maps to a label: 6 or more is "Strongly Bullish", 3 or more "Bullish", 1 or more "Slightly Bullish", and the same mirrored below zero. The maximum possible score is 8.5. For the backtest, any bullish label counts as an "up" call and any bearish label as a "down" call, so a score of just +1 is already a directional prediction. That detail matters for the result.
How The Replay Works
The test is a plain Node.js script of about 90 lines. There is no machine learning and no tuning. The site's own engine is compiled from the website's TypeScript to JavaScript by a small build step, so the backtest runs the exact code the site runs, not a re-implementation that could quietly drift.
- Data: free Yahoo Finance daily bars from 2012 for 40 liquid US stocks and ETFs. Closes are not adjusted for dividends, and there are no fees, slippage or position sizing.
- No look-ahead: at each test date the engine receives only the last 260 bars up to that day, the same order of size the site loads. It cannot see the future by construction.
- Sampling: one call every 5 trading days per stock, to reduce overlap between neighbouring samples.
- Scoring: the verdict is compared with what happened 1, 5, 10 and 20 trading days later. A move counts as "up" or "down" only if it clears a flat band (0.3% for a day, 0.8%, 1.2%, and 2% for a month), otherwise it was "flat". I set those bands by hand so that each class has a reasonable number of samples.
- Baseline: the best single class you could always guess. It was "up" at every horizon. I chose it after seeing the data, which makes it a deliberately tough bar.
- Confidence: a Wilson 95% interval. A result only counts as an "edge" if the whole interval clears the baseline by more than 2 points.
The Result
The verdict was right less often than always guessing "up", at every horizon, and the intervals do not overlap the baseline.
- Next day: verdict right 37.5%, "always up" 43.3% (27,709 calls)
- 1 week: 37.6% vs 43.9% (27,669 calls)
- 2 weeks: 37.2% vs 44.9% (27,629 calls)
- 1 month: 37% vs 45% (27,549 calls)
Looking closer explains part of it. Over half of all calls (54.6%) were bullish, about a third (32.4%) bearish, and only 13.1% neutral, so the engine rarely chooses the "flat" outcome that happens roughly a fifth of the time at one day. When it did call a direction and the stock actually moved, the direction was right about 51% of the time: a coin flip. And at the one-month horizon the average move after a Bearish verdict was +2.0%, higher than the +1.3% after a Bullish one. The signal, if there is one, points the wrong way.
Six Rules I Tried To Rescue It With
The obvious reaction is to fiddle with the rules until something works, which is exactly how people fool themselves. So I wrote a second script with the rule set fixed in advance and a strict gate: split the history into 2012 to 2019 and 2020 to now, and keep a rule only if it clears the baseline on both periods at the same horizon. The six challengers were: the engine inverted, price versus the 200-day average only, RSI mean reversion at 30 and 70, strong-verdicts-only, all indicators agreeing, and oversold-in-an-uptrend.
None passed. The verdict stays as it is. One lead is worth noting without overselling it: buying RSI-oversold and selling RSI-overbought showed a spread of roughly 1 to 1.5 percentage points in average 20-day return across both periods. That is an observation, not a trading rule, and I ran no correction for having tried several variants. A side note from the output: two of the variants ("strong only" and "all indicators agree") produced identical numbers in every cell, which tells me they are effectively the same rule on this data.
The Live Forward Test
A backtest is still a test I designed on data I had already seen, so I added a forward test that cannot be tuned. Every night the site's rebuild job recompiles the engine, then a script records tonight's verdict for all 40 symbols using only completed sessions (a bar dated today could still be forming, so it is dropped). Each entry stores the close, the verdict, the score, a timestamp and a short hash of the engine code. Later runs fill in the outcome after 1, 5, 10 and 20 days using only bars after the call. A verdict is not judged until a horizon has at least 300 scored calls; until then the page says "early".
Today that is 117 calls over three trading days. The next-day calls are 66.2% right on 77 scored, against a baseline of 76.6% in a market that happened to fall most days. It says nothing yet, which is the point of publishing the 300-call rule.
What I Would Change Next:
- The backtest tests the website engine. The mobile app has a separate, larger engine, which I have not backtested and make no claim about.
- My own copy used to say "15+ indicators", which overstated what the verdict scores. The engine shows many readings but scores six, and I have corrected the wording.
- Samples overlap and the same stocks repeat, so the confidence intervals are tighter than they should be.
- The 40 symbols are names that are liquid today, so there is survivorship bias, which usually flatters a strategy rather than hurting it.
- The forward log is meant to be append-only, but that is a convention in the script, not something the file system enforces. Committing it nightly would make the history auditable.
What It Means
The verdict describes the chart; it does not forecast it. I would rather publish that than let a score look like a prediction. Use SUTOK to read a chart faster, not to decide what happens next. SUTOK is research and education software, not financial advice, and past results do not guarantee future ones. For the product overview, read the SUTOK project page.
🚀 Try SUTOK at sutok.org →