Senses and motor decoder

The engine of the previous chapter takes a rate per input neuron and returns a spike per step. This chapter is the boundary around that engine: how a camera frame and a gyroscope reading become those rates, how a tonic bias keeps the inhibitory retina from silencing the lamina, how a drifting grating decides whether the frozen wiring computes motion at all, and how the 83 wing and haltere motor neurons become four command channels. Wiring stays frozen. The learnable set here is six photoreceptor-encoder parameters, four haltere-encoder parameters, three lamina biases, and about 340 decoder weights.

The first real-imago probe, recorded in the notebook entry Optic-flow probe, first run, failed in every condition. The chapter cites that outcome rather than a hypothetical pass.

Eye model

Photoreceptors in the imago carry an eye and a pair of hex coordinates (hex1, hex2) on the Reiser lab’s medulla column grid, inferred at build time from the cartridge each cell innervates. The eye model assigns every unique (eye, hex1, hex2) a viewing direction.

Hex coordinates are shifted to the conventional eye centre (18, 19) and laid on the plane with the second axis at 120 degrees from the first:

\[ x = p - q/2, \qquad y = q \cdot \sqrt{3}/2. \]

The sign of the shear is a measurement, not a convention. Under \(x = p + q/2\) the placed columns of the v1.0 imago span 48 units of \(x\) (240 degrees at \(\Delta\phi = 5\)) as a sheared parallelogram. Under \(x = p - q/2\) they form a round disk spanning 143 by 165 degrees, which is the shape of an eye. That layout is therefore the one the code uses.

Directions then follow a three-parameter model. Defaults, from EyeMapParams: \(\Delta\phi = 5\) degrees (the interommatidial angle), \(\mathrm{az}_0 = 60\) degrees, \(\mathrm{el}_0 = 0\) degrees.

\[ \mathrm{azimuth} = \mathrm{az}_0 + \Delta\phi \cdot x \quad\text{(negated on the left eye)}, \qquad \mathrm{elevation} = \mathrm{el}_0 + \Delta\phi \cdot y. \]

Left-eye mirroring reflects the right-eye map through the midline, so body azimuth is positive toward the animal’s right and the eye centre looks \(\mathrm{az}_0\) off the front. The probe stimulus, which should look the same to both eyes relative to each eye’s own front, uses the absolute value local_azimuth_deg instead. The orientation of \(x\) along the horizon and the sign of elevation remain unchecked conventions: a consistent inversion of every direction-selective type on one axis is reported as a suspected axis flip, not as a probe failure.

Literature extents (Zhao et al. 2025: azimuth about \(-10\) to \(+155\) degrees per eye, elevation about \(-70\) to \(+90\)) are a sanity check, not an assertion. The first v1.0 map reports 877 columns on the left and 891 on the right, local azimuth \([0.0, 135.0]\) degrees on both eyes, elevation \([-77.94, 86.60]\) degrees, and a warning that the elevation floor sits below the literature bound. Those numbers live on the eye map data page.

A measured table with header eye,hex1,hex2,azimuth_deg,elevation_deg (eye L or R) can replace the parametric model. Every column of the imago must be present; missing rows raise. The Reiser lab’s eyemap_T4 data is the intended source. The first probe used the parametric model.

Camera model

The AI-deck camera is a 324-pixel-wide monochrome sensor with an 87 degree field. The stock streamer sends a 324 by 244 window of that sensor. Focal length is computed from the sensor, not the window:

\[ f = \frac{w_{\mathrm{sensor}}/2}{\tan(\mathrm{FOV}/2)}, \]

so cropping the height does not pretend the optics changed. CameraIntrinsics defaults are width 324, height 244, sensor_width_px 324, fov_deg 87. The principal point is \(((W-1)/2, (H-1)/2)\). An explicit focal_px overrides the formula.

Body frame: \(x\) forward along the optical axis, \(y\) left, \(z\) up. Image \(u\) grows to the right, \(v\) down. A direction \((\mathrm{az}, \mathrm{el})\) projects as

\[ \begin{aligned} x &= \cos \mathrm{el}\,\cos \mathrm{az}, & y &= \cos \mathrm{el}\,\sin \mathrm{az}, & z &= \sin \mathrm{el}, \\ u &= c_x - f\, y/x, & v &= c_y - f\, z/x. \end{aligned} \]

and is in the field only if \(x > 0\) and \((u, v)\) lands inside the window.

Each column inside the field is sampled with a Gaussian acceptance cone of 5 degrees full width at half maximum, the default VisionParams.acceptance_fwhm_deg. Angular sigma is FWHM over \(2\sqrt{2\ln 2}\); pixel sigma is \(f \tan(\sigma_{\mathrm{ang}})\). The kernel covers two pixel-sigma and is renormalized per column. Columns outside the field, or whose kernel has no valid pixel, read constant mid-gray luminance 0.5 and therefore adapt to silence.

Photoreceptor encoder

Per column, luminance \(\ell \in [0, 1]\) is log-compressed, high-pass filtered, mixed with a DC fraction of the deviation from mid-gray, and read out as a rate:

\[ L = \log(\ell + \varepsilon),\qquad \dot{L}_{lp} = \frac{L - L_{lp}}{\tau_{\mathrm{adapt}}},\qquad c = (L - L_{lp}) + d\,(L - L_{\mathrm{mid}}),\qquad r = \mathrm{clip}(r_0 + g\,c,\,0,\,r_{\max}). \]

The code is forward Euler on the low-pass at every brain step, with the frame value held between pushes (zero-order hold). Mid-gray is sent through the same ops as a real frame so a mid-gray column reads \(r_0\) bit-exactly rather than \(r_0\) plus a float32 ulp.

VisionParams defaults, in as_vector order:

\(\varepsilon = 0.01\), \(L_{\mathrm{mid}} = \log(0.5 + \varepsilon)\), \(r_{\max} = 300\) Hz. R1–R6 of a placed column receive \(r\); R7 and R8 receive \(r\) times r78_gain. Unplaced photoreceptors (hex column \(-1\)) receive \(r_0\) and so sit at the dark baseline.

The adaptation time constant is longer than a frame interval on the stock streamer. A 10 Hz stream held for 100 ms through a 200 ms high-pass produces a contrast transient at every frame boundary, which is flicker, not optic flow. That is why the probe records both a 1 ms ideal stream and a 100 ms hold.

Input layout

InputLayout is the map from column rates and haltere rates onto the brain’s input-rate vector of length \(K\) (the count of imago.is_input). Photoreceptors, virtual photoreceptors, and haltere afferents occupy disjoint slots in that vector; a member that is not an input neuron is an error at construction.

assemble(column_rates, haltere_rates, params) writes, for batch \(B\), a tensor \([B, K]\):

• placed photoreceptors: the matching column rate times 1 or r78_gain;
• unplaced photoreceptors: r_base_hz;
• virtual photoreceptors: the column rate with no extra gain (they stand in for missing R1–R6);
• haltere afferents: the encoder’s rates, or left at zero if none are supplied.

The probe drives vision only and passes haltere_rates=None.

Haltere encoder

Halteres are the fly’s gyroscopes. Version one does not attempt the measured directional fields of campaniform sensilla. It groups the 439 afferents by cell type: the distinct type labels present (24 in v1.0, including the untyped label <unknown>) are sorted by name and assigned round-robin to six groups, positive and negative for roll, pitch, and yaw, in that order. Every cell of a type, on either side, shares a group, so the type’s downstream wiring stays coherent.

The grouping is written to the imago sidecar as haltere_groups at build time. An encoder reading an older imago, whose sidecar predates that entry, recomputes the same function from the type table. Per-cell round-robin, counted separately per side, exists only as an ablation.

Given body-frame angular rates \(\omega = (\omega_{\mathrm{roll}}, \omega_{\mathrm{pitch}}, \omega_{\mathrm{yaw}})\) in rad/s, a cell in group \(a\) with sign \(\pm\) fires at

\[ r = \mathrm{clip}(r_0 + k_a \max(0, \pm\omega_a),\, 0,\, r_{\max}). \]

HaltereParams defaults: k_roll, k_pitch, k_yaw each 50 Hz per rad/s, r_base_hz 10. The clip ceiling is the same 300 Hz as the photoreceptor encoder. The four values are the learnable haltere vector.

Lamina bias

Photoreceptors are histaminergic and therefore inhibitory. In the fly, L1, L2, and L3 are graded neurons held at an operating point by tonic transmitter release, so more light hyperpolarizes them and less light depolarizes them. In a spiking network with no tonic drive, inhibition onto a cell at rest does nothing, the lamina stays silent, and every motion circuit behind it stays silent with it.

The policy is a per-type tonic bias \(b\) (the per-neuron term of the LIF equation) on L1, L2, and L3, initialized so that under the encoder’s dark baseline the lamina fires at a mid-range target and photoreceptor rate changes modulate it in both directions. Transmitter scales stay non-negative, so the optimizer cannot get the same effect by flipping the histamine sign.

The initialization is mean-field. A photoreceptor spike adds \(w\) mV to the synaptic variable \(g\), which decays with \(\tau_s\), so under a Poisson barrage of rate \(r_0\) the mean of \(g\) is \(r_0 \tau_s W_{\mathrm{in}}\) with \(\tau_s\) in seconds (the code writes r_base_hz * (tau_s_ms / 1000) * W_in). A noiseless LIF with steady input \(v_\infty\) fires at rate \(f\) when

\[ v_\infty(f) = \frac{E\, v_{th} - v_{\mathrm{reset}}}{E - 1},\quad E = \exp\!\left(\frac{1/f - t_{\mathrm{ref}}}{\tau_m}\right), \]

which the code evaluates in milliseconds as \(E = \exp((1000/f - t_{\mathrm{ref}})/\tau_m)\). The per-type bias is

\[ b = v_\infty(f^*) - v_{\mathrm{rest}} - r_0\, \tau_s\, \tilde W_{\mathrm{in}}, \]

where \(f^*\) defaults to 50 Hz and \(\tilde W_{\mathrm{in}}\) is the median incoming photoreceptor weight over the type’s cells that have any such input. Histamine weights are negative, so the last term raises the bias above \(v_\infty - v_{\mathrm{rest}}\).

Only lamina cells that receive photoreceptor input are biased. A cell nothing modulates would add tonic noise, so the 720 column-less lamina cells of v1.0 stay silent. bias_undriven=True applies the type’s bias to every cell of the type, which is the literal reading of the original spec sentence, kept as an option.

On the v1.0 imago at \(r_0 = 10\) Hz and target 50 Hz the first probe recorded L1 15.6276 mV, L2 15.3426 mV, L3 12.8401 mV. Those values are identical on the shuffled control, because they are computed from the imago’s weights, which the shuffle preserves, not from the rewired targets.

Optic-flow probe

Before any training and before any flight, a synthetic drifting grating is shown to the encoder in the four cardinal directions (progressive, regressive, up, down) and the direction-selective sets are read out. The grating lives in eye-local coordinates so both eyes see the same motion relative to their own front:

\[ \ell = \tfrac12 + \tfrac12\, C \sin\bigl(2\pi(\pm \theta / \lambda - f_t t)\bigr), \]

with default period \(\lambda = 30\) degrees, temporal frequency \(f_t = 1\) Hz, contrast \(C = 0.8\). Horizontal directions use local azimuth; vertical directions use elevation. Each direction runs 0.5 s of warm-up plus 2 s of counted spikes. The frame interval is a parameter: 1 ms is the ideal stream, 100 ms mimics the stock streamer, and both are recorded.

Sets are taken from the type table. T4a–d and T5a–d match those names exactly, so T4_unclear and T5a_unclear are excluded. HS matches any type beginning HS (eight cells in v1.0: HSN, HSE, HSS, HST), VS any type beginning VS (34 cells). Empty sets are omitted.

The verdict is on selectivity, not on literature direction. Each subtype is compared across the two directions of its axis (a and b horizontal, c and d vertical; HS horizontal, VS vertical). Selectivity is the larger mean rate over the smaller, each floored at 0.5 Hz so that silence does not pass:

\[ s = \frac{\max(r_a, r_b) + 0.5}{\min(r_a, r_b) + 0.5}. \]

The probe passes when every present subtype of T4, T5, HS, and VS reaches the configured ratio (default 1.5). evaluate_probe scopes that check to the required groups; the CLI default keeps all four. Which direction wins is reported against the literature (T4a/T5a front-to-back, b back-to-front, c up, d down, HS front-to-back, VS down). If every passing T4/T5 subtype on one axis prefers the opposite of the literature, that axis is listed in axis_flip_suspected. A flip is a calibration fact about the eye map, not evidence against the wiring, and it does not fail the probe.

The first run on male-cns-v1.0.imago failed in all three conditions. On the real imago, both the 1 ms stream and the 100 ms hold produced exactly zero spikes in every probed set in every direction for the full two-second window: selectivity 1.0 is the floor formula at 0/0, not a signal. The shuffled control fired (T4/T5 about 0.43–0.90 Hz, HS about 55.6–57.4 Hz, VS about 30.0–30.9 Hz) but was flat (selectivity 1.00–1.01). No axis flip was suspected, because none of the sets passed.

The report also carries a diagnostic stages block that is not part of the verdict: mean rates for the photoreceptor input pool (the photoreceptors and virtual_photoreceptors named sets), L1–L5, C2, C3, and the medulla types Mi1, Tm3, Mi4, Mi9, Tm1, Tm2, Tm4. A follow-up run found the lamina firing (L1 ~38 Hz, L2/L3 ~21 Hz) and the medulla silent on the real wiring, while the shuffled control carried signal through the same medulla types. The break is lamina → medulla, not a silent lamina; see the stage-localization notebook.

flatline stage-drive accounts for that break without simulating: per target type it sums the signed effective weight from each source type, the cells reached, and the mean-field input at measured source rates against the threshold gap. On v1.0 it finds the ON pathway held below rest by glutamatergic (inhibitory) L1 and the OFF pathway under threshold because L2 fires at 20 Hz where 40–53 Hz is needed; see the lamina drive notebook. Raising --r-base wakes Tm1, Tm2 and Tm4, and T5 stays silent on the real imago; see the operating-point sweep.

Motor decoder

The 67 wing motor neurons followed by the 16 haltere motor neurons form an 83-element vector. Spikes pass through an exponential rate filter with learnable time constant \(\tau_{\mathrm{dec}}\) (default 30 ms) and then a linear map onto four channels, in order right, forward, yaw_right, up:

\[ r \leftarrow r\, e^{-\Delta t / \tau_{\mathrm{dec}}} + s\, \tfrac{1000}{\tau_{\mathrm{dec}}}, \qquad u = \tanh(W r + b). \]

\(W\) is 4 by 83, \(b\) is 4, 336 parameters, “about 340” in the spec. Outputs are in \((-1, 1)\). Type names of the form b1 MN are stripped to a muscle label by dropping the MN suffix.

Structured initialization gives the optimizer a sensible start. Each channel’s nonzero weights are then divided by (the count of nonzeros times 100 Hz) so that a cell firing at 100 Hz contributes on the order of one unit before the tanh.

Haltere motor neurons and unmatched types start at zero weight. Shadow mode — running the decoder on a live flight without acting on it — is the flight sub-project; nothing here touches the radio.