Commit tune maps and research so tunes are reachable from the phone

Removes the *.hex/maps_cache gitignore rule (explicit user call, reversing
the earlier no-redistribution stance) so the official TuneECU catalogue
maps, derived SAI/O2-delete composites, and the checksum/composition
tooling are actually available to pull up on a phone browser when using
the real TuneECU app. Also folds in tonight's KWP2000 fixes (TesterPresent
keep-alive, connect-failure cleanup, slow-init StartCommunication fix) and
the accumulated research docs.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FP2GaxS9HkUdL5sLBnjKje

tunie/research/TUNING_IMPL_PLAN.md

@@ -0,0 +1,436 @@
+# Implementation plan: ECU-streamed, onboard-logged road tuning
+
+Concrete engineering plan for the approach settled on in
+[`TUNING_GUIDE.md`](TUNING_GUIDE.md#better-idea-aug-2026-stream-live-data-straight-off-the-ecu-with-tunie):
+stream RPM/TPS/coolant live off the ECU via `tunie`'s existing read-only
+KWP2000 transport, sample a wideband AFR sensor on the same onboard unit,
+write one synced log, post-process off-bike into F Trim corrections. This
+document is the task list; `TUNING_GUIDE.md` and `COMMUNITY_TUNING.md` are
+the research backing each decision here.
+
+**Status: not started.** Nothing in this plan has bike contact yet — Phase
+0 is pure software and can begin immediately; everything after Phase 1
+is blocked on the cable/connector work already tracked in `../STATUS.md`.
+
+## The near-term validation game plan
+
+Six concrete steps, checked against everything in this document and
+`TUNING_GUIDE.md`, with the gaps filled in:
+
+**0. Fix the cable.** Not one of the six steps below, but it's the actual
+current blocker sitting in front of all of them — the Triumph diagnostic
+connector adapter, tracked in `../STATUS.md` since before this tuning plan
+existed. Nothing past this point has bike contact until it's solved.
+
+**1. Prove basic ECU comms with `tunie` — `tunie info`.** Already the
+correct first step per `../STATUS.md`; this plan doesn't change it. Gives
+ECU identity, current map ID/DTCs, and — as a side effect — confirms the
+cable/protocol work before adding live-poll complexity on top.
+
+**2. Yank the current tune off the bike — real gap, needs a scope
+decision, not just a step.** A ROM/map dump uses KWP2000 service `0x35`
+RequestUpload, which `tunie` **deliberately does not implement** —
+`safety.py` refuses it by construction, on purpose, as this whole
+project's core safety guarantee (`README.md`: "Cannot write to the ECU by
+construction"). Building this into `tunie` means consciously opening a
+door that's been kept shut for a reason, not just adding a feature.
+**Simpler alternative, use before building anything new:** TuneECU's own
+Android app already has a working "Read Map" function
+(`android.html`: "Read Map: Read the map from the ECU, only possible via
+cable connection"). Use that for this one step — it's already built,
+already proven, and its output (a `.hex` file) is exactly the input format
+`decode_map.py`/`reconstruct_rom.py`/`table_map.py` already work with. No
+reason to reimplement upload capability in `tunie` just to get a file
+these tools already know how to read. Once dumped, cross-check its
+signature/ID against `maps.json` to confirm exactly which stock
+calibration is actually flashed — resolves an assumption this whole
+research effort has been making (which map family applies) rather than
+just producing a file.
+
+**If "Read Map" ever fails partway through, don't panic-troubleshoot —
+here's what's actually happening.** Traced TuneECU's Recovery mode
+(`research/RECOVERY_MODE.md`) specifically so this is known *before* it's
+needed rather than discovered live. Short version: Recovery is the same
+reprogram function, relabeled — TuneECU checks whether the currently-open
+map file decodes to a valid signature (same magic-header + directory
+lookup `decode_map.py`/`table_map.py` already implement) and offers
+Recovery instead of Reprogram if so. It is **not** detecting the ECU's
+live fault state — it's checking you have a complete, valid base map ready.
+Practical implication: **keep a matching OEM `.hex` (from `maps.json`,
+already downloaded — `20187`/`20191` etc.) on hand before attempting the
+dump**, so if the read fails partway, the documented community recovery
+procedure is available: cycle ignition off/on, wait ~5s, open that
+matching OEM map, reconnect, retry. Full trace, including what's still
+unconfirmed (whether a live ECU-side signal is also involved), in
+`RECOVERY_MODE.md`.
+
+**3. Get live data, see what it actually gives us.** Two sub-parts, not
+one:
+   - *What's there:* poll the candidate record list (Phase 0/1 below) and
+     see what responds at all.
+   - *Labeling, as its own deliberate task, not passive observation:*
+     ignition on/engine off, move the throttle by hand, watch which record
+     jumps → that's TPS. Start the engine, compare a record's value
+     against the tach → RPM. Let it idle and warm up, watch what climbs
+     slowly → coolant. **Contingency:** the candidate record list
+     (`0x0100`, `0x0001`, `0x5103`, `0x0139`, `0x0007`) is one plausible
+     branch recovered from the decompile, not a confirmed-correct one for
+     this specific bike (see Phase 0/1 above) — if none of them behave
+     sensibly, the fallback is a wider sweep of nearby record values and/or
+     going back to trace the `Oe()` switch case selection more rigorously
+     rather than assuming the guess was right.
+   - **Basic bench-test safety:** ignition-on/engine-off first, always;
+     if it gets to engine-running tests, that needs a stand and
+     ventilation considerations like any other bench run — not a research
+     question, just don't skip it.
+
+**4. Design and build the onboard live-capture system.** Pi Zero (or
+similar) running `tunie`'s poll loop + an ADC reading the wideband
+controller's analog output, writing one synced log. CSV or SQLite both
+fine for this — SQLite's nicer once you're joining/querying against table
+geometry later, CSV is simpler to just eyeball. This is Phase 3 in this
+document; **the wideband sensor/controller/mounting setup you already
+flagged as probably-missing is real and belongs here** — see "Phase 2"
+above (hardware, bung reuse, sensor count decision) for what that actually
+involves; it's not just "buy a sensor," it's a genuine sub-project of its
+own (mounting decision, wiring, warm-up handling, thread confirmation).
+
+**5. Verify the captured data locally, build logs from it.** Sanity-check
+against known references before trusting it: idle RPM should match a
+handheld tach, TPS should track smoothly with a slow throttle sweep,
+coolant should rise monotonically from cold start, AFR should read near
+14.7 at a stable idle before any tune changes. This step is really "does
+the whole capture chain produce believable numbers," which is a
+prerequisite before trusting it for step 6.
+
+**6. Compare live ride samples against the current map.** This is Phase 4
+(post-processing) — map each logged point to a fuel-table cell using the
+already-known RPM/throttle axes (`table_map.py`), see how far off the
+stock table is from what's actually happening on the modified bike.
+
+**What this game plan does *not* yet cover, worth naming explicitly:**
+it's a validation/prototyping pass — proving the whole chain works and
+seeing what the data looks like — not yet the actual iterative tuning
+loop (F Trim corrections → Commit trims → reflash → repeat →
+final AF1/AF2 targets, Phase 5 above). That's the next phase after this
+one succeeds, not part of it.
+
+---
+
+## Phase 0 — software groundwork (no bike required, start now)
+
+**Goal:** `tunie` can poll 2-byte extended local identifiers in a loop,
+not just the one-shot single-byte sweep it does today.
+
+- [ ] Add 2-byte record support to the local-identifier path. Today
+  `src/tunie/identify.py` sends `0x21 <1 byte>` (`LOCAL_ID_RECORDS`, single
+  byte). TuneECU's own dashboard sends `0x21 <hi> <lo>` for a wider record
+  space — confirmed in the decompile (`m.java`'s `se()`, records like
+  `0x0100`, `0x5103`, `0x0139`). Extend `Transport.request()` callers to
+  accept a 2-byte record variant; `safety.py`'s allowlist already permits
+  `0x21` generally (per `triumph.py`'s `OBSERVED_READ_SERVICES`), just
+  confirm it doesn't assume single-byte payload length anywhere.
+- [ ] Add a **polling-loop mode**, distinct from `identify`'s one-shot
+  sweep: given a list of records, poll each in round-robin at the fastest
+  safe rate, timestamp every response, write to a log file (CSV or
+  similar). This is the core of what's needed — everything else in this
+  plan depends on it existing.
+- [ ] Candidate record list to poll, from the recovered `ch` array
+  (Keihin-family default case, `m.java:204`): `0x0100`, `0x0001`,
+  `0x5103`, `0x0139`, `0x0007`. Unlabeled — Phase 1 resolves which is
+  which. Poll all of them; drop the unused ones once labeled.
+- [ ] Write the labeling capture script: connects, polls all candidate
+  records continuously, dumps a live table to the terminal (record → raw
+  value, updating) so a person can watch and correlate against what
+  they're doing to the bike (revving, moving throttle by hand, waiting for
+  it to warm up).
+- [ ] Decide the log format now, since it constrains the post-processing
+  script (Phase 4): one row per poll cycle, columns `timestamp_ms, rpm,
+  tps_raw, coolant_raw, <spare records>`. Keep raw values through this
+  layer — scaling/unit conversion happens once records are labeled and
+  their encoding is known (may not be linear; TPS especially could be a
+  raw ADC count needing calibration against the known throttle axis
+  breakpoints in `reference-maps/TABLES.md`).
+
+**Acceptance:** a script exists that, given a connected bike, would print
+a live-updating table of all candidate records with timestamps, with no
+bike required to review/test the code path structurally (mock transport
+for a dry run is enough to validate the polling loop and log-writing
+logic before real hardware is available).
+
+---
+
+## Phase 1 — first ECU contact + record labeling (blocked on cable)
+
+**Blocker:** the Triumph diagnostic connector adapter, per `../STATUS.md`
+— unchanged blocker, not something this plan resolves.
+
+**Goal:** know which of the candidate records is RPM, which is TPS, which
+is coolant temp (and what any leftover ones are).
+
+- [ ] Run `tunie info` for the first time — the original Phase-1 goal from
+  `../STATUS.md`, still the correct first step regardless of this plan.
+  Confirms cable/protocol work before adding the live-poll complexity.
+- [ ] Ignition on, engine off (safe, no combustion risk): run the labeling
+  script from Phase 0. Move the throttle by hand — whichever record jumps
+  in response is TPS. Note idle values for the others.
+- [ ] Engine running, idle: whichever record now shows a plausible RPM
+  value (four-figures, matches tach) is RPM. Blip the throttle to confirm
+  it tracks.
+- [ ] Let the engine sit and warm up: whichever record climbs slowly over
+  minutes is coolant temp.
+- [ ] Determine each record's raw→real-units scaling by comparing against
+  a known reference (tach for RPM, a multimeter on the TPS sensor wire or
+  the throttle's marked positions for TPS, a rough ambient/operating-temp
+  sanity check for coolant). Write the results into `triumph.py` or a new
+  `live_records.py` module, with the same "recovered from X, confidence Y"
+  documentation style the rest of this project uses.
+
+**Acceptance:** a small table, committed to the repo, mapping record ID →
+name → raw-to-real-units formula → confidence level, for RPM/TPS/coolant
+at minimum.
+
+---
+
+## Phase 2 — wideband hardware (parallel-able with Phase 1, needs purchase)
+
+Per `TUNING_GUIDE.md`'s shopping list — Innovate LM-2/MTX-L, AEM X-Series
+UEGO, or **Zeitronix ZT-3** (a third real option, confirmed via an
+external research pass Aug 2026 — Woolich Racing sells it as a
+tuning-package wideband kit), all with a 0-5V analog output. This project
+can't execute this phase (physical purchase, welding, wiring) — tracked
+here so the software side knows what interface to design against.
+
+- [ ] **Don't just thread the wideband into whatever bung is available —
+  masking risk, confirmed.** Long/angled stock-style bungs and
+  thread-reducing adapters (e.g. 18mm→12mm) can prevent the sensor tip
+  from reaching the core exhaust gas stream, giving delayed/inaccurate
+  readings — directly confirmed against a real installer's page
+  (RB Racing): *"Many manufacturers use long 18mm O2 slanted bungs. These
+  'mask' the 18mm O2 sensor signal"*; thread reducers are *"junk... the
+  threaded section is too long masking the signal."* Check bung depth/
+  angle against the sensor's own spec before assuming any available bung
+  works, whether stock or new.
+- [ ] Check whether the Arrow 2-in-1's headers retain provisions at the
+  stock O2 bung locations before assuming a weld is needed — M18x1.5 is
+  the universal O2 sensor thread (narrowband and wideband both use it;
+  confirmed for this Triumph twin family specifically by the German
+  tuning primer), so if a bung survives, the wideband sensor threads
+  straight in, no fabrication required.
+- [ ] Decide one sensor or two: the stock design has **two** O2 bungs
+  (one per cylinder, pre-collector — matches the ECU's per-cylinder
+  `F1`/`F2` fuel tables). One wideband sensor placed post-collector on the
+  2-in-1 gives a blended average AFR and loses per-cylinder correction
+  ability; two sensors (one per header, if those bungs survive on the
+  Arrow pipe) preserves it, at roughly double the sensor/controller cost.
+  Pick based on whether per-cylinder tuning resolution is worth it.
+- [ ] Sensor + controller acquired, bung fitted, wired to switched 12V.
+- [ ] Confirm the controller's analog-output voltage-to-AFR/lambda mapping
+  (from its manual) — needed for Phase 3's ADC scaling.
+
+**O2 delete is not required by this method — that's specific to the
+original German guide's setup, not ours.** The 2003 primer had the wideband
+probe *physically replace* the stock narrowband sensor in its own port,
+which is why it explicitly required disabling closed loop ("Ab jetzt muss
+ein Tune ohne Lambdaregelung gefahren werden!" — "from now on a tune
+without lambda regulation must be run") — the ECU's own narrowband signal
+was gone, so closed loop couldn't function regardless. **Our plan doesn't
+have that constraint**, because the wideband sensor sits in an independent
+bung, feeding only our own logger — the stock O2 sensor(s) can stay fully
+wired, active, and in closed loop the whole time if desired.
+
+This matters because closed loop and the F Trim/road-tuning method operate
+on *different parts of the map*: closed-loop O2 correction only runs at
+idle and steady low-mid throttle; everything the road-tuning method is
+actually correcting (WOT, hard acceleration, deceleration) is **already
+open-loop on the stock ECU regardless of O2 presence**. So keeping O2
+active doesn't get in the way of the tuning — if anything it's
+complementary: O2 keeps handling live idle/cruise correction (one less
+thing to get exactly right by hand), while the wideband-driven F Trim
+process corrects the open-loop main fuel table, which O2 feedback never
+touches, stock or modified.
+
+**Practical options, now that this is a real choice rather than forced:**
+1. **Keep both O2 sensors, wideband in a separate/reused bung.** Full
+   closed-loop behavior retained; wideband only used for tuning
+   measurement passes. Needs a bung location that doesn't conflict with
+   either stock sensor.
+2. **Delete O2 (as originally planned for the SAI/O2 project), reuse the
+   freed bung(s) for the wideband.** Loses closed-loop idle/cruise
+   correction (richer, more consistent idle/cruise per the original
+   SAI/O2-delete motivation in `COMMUNITY_TUNING.md`, but no more live
+   auto-correction there either — needs the road-tuning method to get that
+   region right by hand instead of relying on O2 feedback).
+3. **Hybrid** — delete/replace on one cylinder's bung for the wideband,
+   leave the other cylinder's O2 sensor active. Asymmetric, more complex,
+   probably not worth it unless bung availability forces it.
+
+Decision isn't urgent — can be made once the Arrow pipe's actual bung
+layout is known (Phase 2's first task above).
+
+**For this build specifically, refined: option 2 (O2 delete), but via
+temporary sensor swap rather than permanent bung reuse.** Instead of
+pulling the stock narrowband sensors permanently, thread wideband sensors
+into the *same stock bungs* for tuning sessions only, then put the stock
+narrowband sensors back afterward — zero new holes, zero permanent
+modification to the exhaust. The O2-delete *behavior* (richer/cooler
+idle-cruise, the original motivation from `COMMUNITY_TUNING.md`) then
+becomes a pure software choice — the ECU's device flag, independent of
+whatever sensor happens to be physically sitting in the bung — rather than
+something tied to permanently removing hardware. Full reasoning, including
+the closed-loop-behavior tension this surfaces (WOT tuning and
+idle/cruise-mixture behavior are separate axes, don't assume fixing one
+fixes the other):
+[`TUNING_GUIDE.md`](TUNING_GUIDE.md#best-version-yet-reuse-the-stock-o2-bungs-make-the-o2-delete-decision-reversible-too).
+Session-scoped exception: closed loop still has to be bypassed *during*
+the tuning rides themselves, since a wideband occupying the narrowband's
+spot can't feed the ECU a valid narrowband signal at the same time — that's
+temporary, not a permanent-mod question.
+
+**For open-sourcing this to other riders who want to keep O2 active** —
+worth supporting as first-class options, not assuming everyone deletes O2:
+
+- **Considered and rejected: reusing the SAI injection point as a wideband
+  mounting location.** Doesn't work mechanically — SAI air enters through
+  openings cast into the cylinder head at the exhaust port junction (reed
+  valves live in the cam cover), not through a bolt-on fitting on a pipe.
+  There's no simple threaded port there to repurpose; doing this would mean
+  machining into the head casting. Not a real shortcut.
+- **What does work: no-weld, clamp-on O2 bung adapters** — a standard
+  product (PLM, GlowShift, ProFlow, others), drill one hole, bolt on a
+  stainless clamp with a gasket, no welding. For O2-keeping riders on a
+  2-in-1: **one** clamp-on bung post-collector (single blended-AFR
+  reading, no per-cylinder resolution, but one new hole instead of two).
+- **Zero-modification option, with a real cost:** clamp-on tailpipe-end
+  "sniffer" probes exist too — clamp around the exhaust tip, no drilling
+  at all. Trade-off: less accurate (ambient dilution, slower response,
+  less representative of true exhaust gas) — a real cost for zero
+  permanent modification, not a free option.
+- **Design implication for the eventual tooling/writeup:** document this
+  as a menu (delete + reuse stock bungs / keep O2 + single clamp-on bung
+  post-collector / keep O2 + zero-modification tailpipe clamp / weld two
+  dedicated pre-collector bungs for full per-cylinder resolution) rather
+  than assuming one hardware path — the logging/post-processing pipeline
+  (Phase 3/4) doesn't care which one a given rider picks, it just needs an
+  AFR channel wired to the ADC either way.
+
+**Don't wire the wideband's output back into the ECU's O2 input.** A
+wideband sensor can't be wired directly into a narrowband-expecting ECU
+pin at all (different sensor physics — pumped/Nernst cell needing its own
+controller, vs. a simple heated-zirconia voltage source); some controllers
+offer a "narrowband emulation" output to fake the ECU into thinking
+nothing changed, but that mode discards the precision this whole project
+needs, and the `0x21` diagnostic stream still wouldn't carry real AFR
+either way. Since O2 delete is already part of the plan, the correct setup
+is a fully standalone wideband — own bung, own controller, output only to
+the Phase 3 logger's ADC, ECU's O2 circuit left disconnected/flagged off.
+
+---
+
+## Phase 3 — onboard unit integration
+
+**Goal:** one small computer, riding along, producing one synced log per
+session.
+
+- [ ] Pick the board. Constraints: needs a USB port (or equivalent) for
+  the K-line/FTDI cable `tunie` already uses, an ADC input (built-in or via
+  a small ADC breakout) for the wideband's analog output, enough compute
+  to run `tunie`'s Python transport comfortably, and to survive vibration/
+  heat/vibration under the seat. A Raspberry Pi Zero 2 W (or similar) with
+  a cheap I2C ADC (e.g. ADS1115) is a reasonable default; an ESP32 is an
+  alternative if the KWP2000 polling loop gets ported to something more
+  embedded, but doing that port is extra work `tunie`'s existing Python
+  code doesn't need if a Pi-class board is used instead.
+- [ ] Wire ADC channel to the wideband controller's analog output; confirm
+  voltage range matches the ADC's input range (may need a divider).
+- [ ] Combine the two data sources into one process: KWP2000 poll loop
+  (Phase 0/1) and ADC sample loop, both timestamped against the same
+  clock, written to one log file per run (CSV: `timestamp_ms, rpm, tps,
+  coolant, afr`).
+- [ ] Power: needs to run off the bike's switched 12V (same source as the
+  wideband controller) with a regulator appropriate to the board, or a
+  battery pack for bench testing before wiring it in permanently.
+- [ ] Basic operational robustness: start logging automatically on power-up
+  (no need to interact with it mid-ride — this was a stated goal, since
+  operating a device while riding and marking throttle positions is a
+  safety concern per `TUNING_GUIDE.md`'s honest complexity assessment),
+  and fail safe (stop logging / flag clearly, don't crash silently) if the
+  K-line connection drops.
+
+**Acceptance:** power the unit on, ride/rev through a test range, power
+off, retrieve one CSV with all four channels populated and sensibly
+timestamped.
+
+---
+
+## Phase 4 — post-processing pipeline (software, buildable now against synthetic data)
+
+**Goal:** turn a captured log into F Trim table corrections, reusing the
+table geometry already reverse-engineered (`reference-maps/table_map.py`'s
+RPM axis, throttle axis, and table offsets).
+
+- [ ] For each logged row, find the nearest fuel-table cell: RPM axis
+  lookup is direct (`table_map.py`'s `rpm_axis`, already extracted per-map);
+  TPS needs converting from whatever raw units Phase 1 lands on into the
+  table's throttle-axis units (`throttle_axis`, 0-1000 = 0-100.0%).
+- [ ] For each cell with enough samples, compute the AFR error (measured
+  vs. the flat baseline target used during capture, per the road-tuning
+  method in `TUNING_GUIDE.md`) and produce a correction value.
+- [ ] Output format: a per-cell correction table matching the F Trim
+  table's shape, ready to be entered into TuneECU's Map Edit F Trim screen
+  by hand (or, if worth the extra work later, generate a diff to apply
+  directly via the same decode/repack/encode pipeline
+  `toggle_devices.py`/`compose_arrow_delete.py` already use — flagged as
+  optional, manual entry into the app is the simpler and lower-risk MVP).
+- [ ] This script can be written and unit-tested **now**, against
+  synthetic/fake log data, without waiting on any other phase — it only
+  needs the table geometry, which is already known.
+
+**Acceptance:** given a CSV in the Phase 3 format, produces a per-cell
+correction table.
+
+---
+
+## Phase 5 — the actual tuning loop (needs everything above, plus riding)
+
+Execute the method from `TUNING_GUIDE.md`: flat rich baseline tune → ride
+with the logger → Phase 4 processing → enter corrections into F Trim →
+Commit trims → reflash → repeat 3-4 passes → set real AF1/AF2 targets →
+done. Not further decomposed here; it's the method already documented, now
+using the improved single-source logging instead of manual correlation.
+
+### After tuning: what happens to the wideband hardware
+
+The wideband sensors are a measurement tool, not a runtime dependency —
+once the F Trim corrections are validated and committed into the permanent
+FuelMap, the ECU doesn't need live AFR feedback to run correctly.
+
+- [ ] Remove the wideband sensor(s).
+- [ ] **Plug the freed bungs with standard M18x1.5 threaded plugs, not a
+  weld.** Keeps them reusable for a future re-tuning pass (e.g. after the
+  eventual airbox removal) without welding again.
+- [ ] Optional, not required: leave **one** wideband sensor permanently
+  installed as an ongoing AFR gauge. Worth considering specifically
+  because this build ends up with no SAI and no O2 feedback at all —
+  zero live safety-net watching for a lean condition afterward (bad
+  injector, air leak, etc.). Common practice for fully O2-deleted setups;
+  doesn't need both sensors, just a dashboard readout.
+
+---
+
+## What can start today vs. what's blocked
+
+| Phase | Blocked on | Can start now? |
+|---|---|---|
+| 0 — polling loop software | nothing | **yes** |
+| 1 — record labeling | cable/connector (`../STATUS.md`) | no |
+| 2 — wideband hardware | purchase + fabrication (not this project's job) | independently, whenever |
+| 3 — onboard unit | Phase 1 (needs labeled records) + Phase 2 hardware | partially — board/ADC selection and wiring plan, yes; full integration, no |
+| 4 — post-processing | table geometry only (already known) | **yes, against synthetic data** |
+| 5 — tuning loop | everything | no |
+
+**Recommended immediate next action:** Phase 0 (polling-loop code) and
+Phase 4 (post-processing script skeleton) are both pure software, need no
+hardware, and are the actual bottleneck-breakers — everything else is
+either already blocked on the known cable issue or is a purchase/fabrication
+task outside this project's scope.