# V6–V8 methods: online image/pose/goal decisions

The environment is our 3D replica of Dust2. This document specifies the runtime and training method; selected checkpoints, executed step counts and final outcomes are recorded in the accompanying evidence JSON. Version numbers describe experiments, not a guaranteed ranking.

## What “online” means

At each 100 ms action step, the renderer produces a new 64 × 36 depth image from the current pose, with a 90° horizontal field of view and a sensor height of 1.6 m. The controller also receives exact current XYZ, body heading and target XYZ. Localization is supplied, not inferred from the image. The policy receives no NAV identifiers, portal geometry, route planner, path history or cached policy decisions. The navigation mesh is used by the simulated body to constrain physical movement.

A trained sensory adapter proposes a direction cue for FC2 cells. Current body heading stimulates EPG; the rendered depth image also stimulates the same retinal inputs as earlier versions. The entire retained graph advances for ten 10 ms neural steps. Only PFL3 activity reaches the motor interface. The body moves, a new image is rendered, and the loop repeats. Weights are fixed during evaluation.

A terminal frame records the final pose and depth view, retains the last computed neural state, and sets applied movement commands to zero because the episode has ended. It does not add another policy call. The terminal metadata identifies that boundary.

The static showcase plays prepared outputs of this loop. It does not run the connectome in JavaScript. The separate local GPU lab computes new, user-specified goals through the same controllers. A prepared replay is not a policy cache inside the experiment.

## Common representation

Depth is clipped to 0–60 m, transformed with log1p(depth)/log(61), and averaged over 4 × 4 pixel blocks to 144 values. Twelve pose/goal values contain current and target XYZ divided by 50, cosine/sine of heading and relative goal bearing, log1p(horizontal goal distance)/5, and vertical difference/10. Another 56 values encode sine/cosine Fourier features of current XY and goal XY at seven frequencies 1,2,4,8,16,32,64 after scaling by π/50. Total: 212 float32 input features.

The coordinates and Fourier features help represent a known map. Their presence means these models must not be described as navigation from vision alone. Image-removal controls assess the frozen system's actual visual dependence.

## V6: supervised sensory adapter

Selected architecture: 212 → 1,024 SiLU → 1,024 SiLU → 512 SiLU → 2. The 1,793,538 weights/biases predict cosine/sine of a relative heading. Absolute cue = body heading + atan2(output_y, output_x).

Training uses 1,000,000 rendered examples (100,000 initial + 900,000 broader samples), split into 950,000 fitting examples and 50,000 diagnostic validation examples. A map-based teacher labels randomly sampled continuous positions, headings and goals. Samples include off-centre positions and entrances. The teacher uses offline shortest-path choices and a local visibility test to propose a nearby heading. Its implementation was numerically compared with the previous geometric teacher on 64 training situations with zero angular difference. It is never imported into runtime policies.

AdamW fits squared error in the 2-D direction target, with batch 2,048, gradient norm cap 2, weight decay 0.0001, initial learning rate 0.001 and seed 130941. Training alternates uniform examples and the highest-error 20%, refreshed every 1,000 updates. A random 95/5 image split is a fitting diagnostic, not an independent navigation test. Navigation selection uses the separately frozen development set.

Development compared hidden widths 512 (503,554 parameters) and 1,024 (1,793,538), at 8,000, 16,000 and 24,000 updates each. Selection maximized goals reached on 32 development situations, then minimized mean duration including 120 s failures. V6 selected width 1,024 at 8,000 updates: 19/32 development goals, mean capped duration 67.521875 s. This is selection evidence, not a final test result.

V3 internal fly weights and the V2 motor adapter are inherited and frozen. The old V2 visual network is loaded by the inherited class but is not called; its unused weights are excluded from inference counts.

## V7: DAgger refinement

V7 starts from a selected V6 sensory adapter. It walks training situations under its own decisions. Every third action step, the teacher labels the current visited state, including locations where the policy gets stuck. Those examples are aggregated with the supervised dataset and the sensory adapter is fitted again. The teacher is restricted to the `rl-training` split; collection rejects development and held-out splits.

Three DAgger rounds each collect from 128 different entries of the frozen 512-case training split (offsets 0, 128 and 256). Every round fits for 8,000 updates, at learning rates 0.0003, 0.00015 and 0.0001. Half of each fitting minibatch is sampled from accumulated DAgger examples; the remainder follows the uniform/hard-example schedule. Checkpoints at 4,000 and 8,000 updates are assessed on the same 32 development cases; the better checkpoint seeds the next collection. The final V7 is selected among all six trained candidates by development success and then capped duration. These steps are optimization and selection choices, not independent validation.

Repeated states from blocked trajectories are retained. Consequently, the diagnostic random image split can contain very similar observations on both sides; it is not an independent navigation benchmark. V7 selected the second-round 8,000-update checkpoint (after 8,000 V6 + 4,000 first-round + 8,000 second-round optimizer updates along its lineage), with 22/32 development goals and mean capped duration 61.509375 s. Its final fitting dataset contains 1,074,000 rows: 1,020,300 fitting and 53,700 diagnostic validation examples. The third-round candidates reached 20/32 and were retained as development evidence.

The final architecture and inference inputs are identical to V6. The internal fly graph and V2 motor adapter stay frozen. This is supervised dataset aggregation, not reinforcement learning. Executed iterations, source datasets, selected checkpoints and development results are listed in the evidence artifact.

Reference: [Ross et al., A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning](https://arxiv.org/abs/1011.0686).

## V8: joint hierarchical PPO

Both learned interfaces start without pretrained V2–V7 adapter weights. V8 uses original fixed V1 anatomical weights and its calibrated constants. “From scratch” refers to learning the sensory and motor adapters from rewards; it does not mean random anatomy, no engineered motor mapping, or training all anatomical synapses.

Sensory actor: 212 → 256 tanh → 256 tanh → 1 (120,577 parameters). A sampled Gaussian latent is squashed through tanh and multiplied by π, then added to the direct goal bearing. The mean is used at evaluation. Its resulting heading cue advances the fly graph.

Motor actor: 24 PFL3 activities multiplied by 10 → 64 tanh → 64 tanh → 2 (5,890 parameters). Gaussian samples during training, means during evaluation. It supplies residuals around the fixed calibrated V1 turn/speed mapping, clipped to shared physical limits. The residual turn range is ±1 rad/s and speed range ±2 m/s. Zero total PFL3 activity forces a stop, as in the inherited motor interface. The actor has no direct access to the sensory image, pose, goal or map.

The joint log probability is:

```
log p(sensory latent | current image, pose, goal)
+ log p(motor latent | PFL3 after the sensory-driven brain transition)
```

Both actors learn from the resulting advantage using policy gradients. The intermediate fly transition is part of the control loop. Anatomical weights are not differentiated or trained in V8.

The critic independently receives the same 212 sensory features and has architecture 212 → 256 tanh → 256 tanh → 1 (120,577 parameters). Three learned log standard deviations control exploration. Critic and exploration parameters are training-only and excluded from deterministic inference counts.

PPO settings: seed 130942; 32 parallel environments; horizon 128; Adam learning rate 0.0003, epsilon 1e-5; clipping 0.2; gamma 0.995; GAE lambda 0.95; four optimization epochs, minibatches 1,024; normalized advantages; entropy coefficient 0.004; gradient norm cap 0.5; early stopping of optimization epochs if approximate KL exceeds 0.03. The value term is 0.25 × squared return error.

Reward = change in a graph-distance progress proxy − 0.03 per action − 0.3 for movement blockage + 20 for success − 10 for timeout failure. Development compares two scalar reward proxies:

- `center`: graph distance from current area to target area + horizontal distance from the body to the current area centre.
- `exit`: the same graph distance + distance from the body to the next shortest-path area's centre − distance between current and next area centres. This supplies a local slope toward an exit instead of attracting the body to the current centre.

Inside the goal area both use direct horizontal distance to the actual target. The selected variant is recorded in the evidence JSON. The next-area data in `exit` is used solely to compute a scalar training reward: neither actor nor critic receives the route, next area, expert heading or action labels. These proxies can be discontinuous at area boundaries; we do not claim exact geodesic optimality or potential-based policy invariance. Reward geometry is unavailable at inference.

Curriculum: updates 1–25 sample nearby goals within 12 m and allow 25 s; 26–50 within 30 m/45 s; 51–75 within 80 m/75 s; later updates draw 75% from the frozen training goals and 25% nearby goals within 80 m, with 120 s. The stage's time limit also applies to ongoing episodes at a transition. This is an explicitly scheduled training curriculum, not a change to final evaluation conditions.

Reference: [Schulman et al., Proximal Policy Optimization Algorithms](https://arxiv.org/abs/1707.06347).

Two independent 600-update PPO runs executed 4,915,200 environment steps in total (2,457,600 each). Development evaluated checkpoints at updates 100, 300 and 600 from both reward variants. Selection chose the `center` run at update 100: 409,600 environment steps seen by the selected checkpoint, 4/32 development goals, mean capped duration 106.059375 s. Later checkpoints and the full logs are retained. More training did not consistently improve navigation in this setup. No held-out outcomes were used to select this checkpoint.

## Network size and interfaces

All versions compute the retained MaleCNS graph: 166,700 neurons, 25,582,938 directed aggregated edges and 124,177,617 anatomical contacts. Edges can aggregate several contacts; “25.6 million” is neither a neuron count nor the count of all individual synapses.

Inputs: 3,335 mapped R1–R6 cells (of 3,377 annotated), 46 EPG heading cells, 92 FC2 goal-cue cells (18 A + 27 B + 47 C), and 7,114 lamina cells receiving tonic drive. Output: all 24 PFL3 cells; left/right assignment uses connectivity to two DNa02 references. Twelve PFL2 cells are monitored; the original braking coefficient is zero. See `neuron-interfaces.csv` for exact IDs and the original methods for calibration and anatomical assumptions.

The viewer shows 176 selected cells = 46 EPG + 92 FC2 + 24 PFL3 + 12 PFL2 + 2 DNa02. It displays 240 strong anatomical edges among that subset. Recorded colour represents continuous rate activity, not individual spikes. The whole graph is still simulated. The flybody mesh is an illustrative body; heading/commands follow the recording, while individual leg animation is not a learned biomechanical controller.

## Inference parameter accounting

| Version | Neural weights/biases used in forward computation | Additional active learned components |
|---|---:|---|
| V1 | 25,582,938 | No MLP; calibrated simulation/motor constants are separate |
| V2 | 25,654,878 | Visual 66,050 + motor 5,890 |
| V3 | 25,654,878 | Same adapters; internal changes merged into existing graph weights |
| V4 | 25,588,828 | Motor 5,890; saved map choices replace running the 2,429,958-parameter map MLP |
| V5 | 25,594,718 | Motor 5,890 + PPO motor actor 5,890; same saved map choices |
| V6 | 27,382,366 | Visual 1,793,538 + motor 5,890 |
| V7 | 27,382,366 | Same inference architecture as V6 |
| V8 | 25,709,405 | Sensory actor 120,577 + motor actor 5,890 |

These counts exclude normalization buffers, graph structure indices, activity state, calibration constants, sensory preprocessing, geometry, lookup tables and training critics. V4/V5 additionally use 5,031,049 int8 cached choices. They are explicitly historical map-assisted versions; V6–V8 do not use those choices at inference. A weight count is not a count of currently firing or nonzero-activity cells.

## Evaluation and causal interpretation

Independent splits were written before final testing: development 32 situations (seed130937), held-out100 (seed130938), training512 (seed130939). Half of starts are near T spawn, half map-wide. Positions and goals are continuous points in reachable regions of the known map. Teachers never use final held-out cases.

Select checkpoints by development results, freeze weights and runtime hashes, then run the same100 final cases for all three versions. Success requires horizontal distance ≤2m and vertical difference <0.6m within120s. Speed ≤4.5m/s; turn magnitude ≤2.8rad/s. Report successes, mean successful time, and capped mean counting failures as120s. Retain all failures.

The controls compare frozen policies with the image replaced by an open-depth field and with internal recurrent weights zeroed. A changed-target probe measures sensitivity to a different goal. These are interventions on a trained system, not matched retrained architecture baselines. PFL3-zero stopping is an explicit motor rule, so graph-silencing failure alone is especially insufficient to establish biological topology superiority.

V1–V3 historical B tests used a different input boundary,60s limit and original NAV height approximation. V4/V5 used cached map assistance and a different final split. Those tests remain separate; their percentages cannot be treated as an eight-way controlled ranking.

## Reproduce in the prepared workspace

```
python scripts/prepare_online.py --count 100000 --tag initial --seed 130940
python scripts/prepare_online.py --count 900000 --tag broad --seed 130943
python scripts/train_online.py --data outputs/online/imitation-initial.npz outputs/online/imitation-broad.npz --width 1024 --steps 8000 --tag repeat-v6
python scripts/run_online.py --version v6 --checkpoint outputs/online/checkpoint/repeat-v6.pt --split development --tag repeat
python scripts/run_online.py --version v7 --checkpoint outputs/online/checkpoint/repeat-v6.pt --split rl-training --count 128 --offset 0 --tag repeat-dagger --collect
python scripts/train_online.py --data outputs/online/imitation-initial.npz outputs/online/imitation-broad.npz outputs/online/dagger-repeat-dagger.npz --width 1024 --resume outputs/online/checkpoint/repeat-v6.pt --steps 8000 --lr .0003 --dagger-fraction .5 --tag repeat-v7
python scripts/train_online_rl.py --updates 600 --batch 32 --reward-mode center --tag repeat-rl-center
python scripts/train_online_rl.py --updates 600 --batch 32 --reward-mode exit --tag repeat-rl-exit
```

The full V6 capacity study and three-round V7 schedule are implemented in `scripts/develop_v6.py` and `scripts/develop_v7.py`; run them in a fresh project copy to preserve the recorded development artifacts. Existing split files are preserved. Use fresh tags to preserve selected weights. Exact learned bytes may differ across software, GPU and CUDA versions. Commands above illustrate reproduction; the evidence JSON lists the actual selected experiment's full datasets and iterations. Run `start-lab.ps1` for new user-specified goals after selected checkpoints exist.

## Česky stručně

V6 se z příkladů učí navrhnout další směr podle nového obrazu, polohy a cíle. V7 dostává navíc opravy v situacích, do kterých se dostane vlastní chůzí. V8 učí vstupní i motorové rozhraní společně pomocí odměn; používá původní pevné mušší váhy. Ve všech třech verzích se rozhoduje znovu při každém kroku a pohyb prochází přes mušší síť. Web přehrává záznamy, lokální laboratoř počítá nové cíle. Výsledky dokládají chování tohoto systému na známé mapě, nikoli věrnou simulaci živého zvířete nebo nadřazenost biologického zapojení.


## Graphics runtime on the training machine

Recorded GPU runs use CUDA on an NVIDIA GeForce RTX 5090. On this Windows machine the default Python launcher created its OpenGL depth context on the integrated AMD GPU. We created a project-local `python-gpu.exe` launcher and set its Windows graphics preference to the RTX. A fresh PPO run replaced the slower development pilot without transferring learned weights. This changes computation speed, not the policy information boundary.

The controllers and training environment use the original single-view `DepthRenderer`. An optional batched-renderer benchmark is retained as a separate performance experiment; it is not used to generate the selected training runs or final recordings.