Device Connect D2D: discovery returns [] / invoke 'no responders' on multi-homed / multicast-hostile networks

[armwaheed]

Issue: In an insecure zero-infrastructure D2D (device-to-device) deployment on https://github.com/Arm/device-connect, an agent host cannot query a device on the same LAN. discover_devices() returns [] and invoke_device() fails with no responders after a 30 s timeout. Both hosts are on the same subnet and both correctly auto-select the Zenoh D2D peer backend.

Environment: two multi-homed hosts on one Wi-Fi LAN — a robot (private internal motor-control NIC on eth0, Wi-Fi on wlan0, plus docker0) and a laptop (Wi-Fi plus awdl0/llw0/utun*/bridge0). Terminal-only; D2D has no browser dashboard (discovery rides Zenoh presence, not a server registry). The "sensor" is the repo quick-start demo (device_id=sensor-001) — any two hosts reproduce it; no physical sensor required.

Task:

• Reproduce on a multi-homed robot + laptop on the same Wi-Fi network.
• Find and fix the root cause in code.
• Document the problem, the reproduction, and the fix for field/customer sharing and for the team.

Resolution: see the comment below and #48. Robot sensor/effector access notes: https://github.com/armwaheed/robotics-connect

[armwaheed]

Resolution — root cause, reproduction, and fix

Fixed in #48. This comment is written to be shared with the field/customer and to brief the team on why the fix is necessary.

TL;DR

D2D (zero-infrastructure, device-to-device) discovery rides Zenoh's UDP multicast scouting, which is fragile on real robot/Wi-Fi networks. The first read was "network layer, not a code bug." Half true — but the symptom reproduces on a LAN where multicast provably works, which isolates two real code defects. Both are now fixed and backward compatible.

Symptom

Two hosts, same subnet, insecure D2D mode. Agent side:

discover_devices(device_type="sensor")  ->  []
invoke_device("sensor-001", "get_reading")  ->  "no responders" (30 s timeout)

Root cause

1. The reliable unicast escape hatch was booby-trapped. When multicast is unreliable, the documented workaround is a direct unicast link via ZENOH_CONNECT=tcp/<device>:7447. But the agent's connect() treated any explicit endpoint as "infrastructure mode" and routed discovery to a registry service a zero-infra device doesn't run — so it timed out and returned [] unless the caller also set DEVICE_CONNECT_DISCOVERY_MODE=d2d. The MCP bridge already had the correct logic; the connect() path diverged from it. That inconsistency is the bug.
2. Multicast scouting bound to the wrong NIC. The Zenoh adapter never set a scouting interface, so Zenoh's default interface:"auto" picks one of the host's many NICs. Robots and laptops are heavily multi-homed (private motor-control NIC, docker0, VPN utun*, Apple awdl0/llw0, …), so two peers on the same LAN intermittently never form a session.

How to reproduce

Two hosts on one LAN, both insecure D2D (device = repo quick-start sensor demo, device_id=sensor-001; agent = agent-tools client). No physical sensor needed.

# Device:
DEVICE_CONNECT_ALLOW_INSECURE=true python sensor.py
# Agent:
DEVICE_CONNECT_ALLOW_INSECURE=true python client.py    # -> [] then "no responders"

Decisive test — pin a direct unicast link (bypasses multicast):

# Device:
ZENOH_LISTEN=tcp/0.0.0.0:7447 DEVICE_CONNECT_ALLOW_INSECURE=true python sensor.py
# Agent — BEFORE the fix you also needed DEVICE_CONNECT_DISCOVERY_MODE=d2d, or it returned []:
ZENOH_CONNECT=tcp/<device-ip>:7447 DEVICE_CONNECT_ALLOW_INSECURE=true python client.py

Differential diagnosis — speculative causes eliminated

#	Hypothesis	How it was tested	Observation	Verdict
1	Misconfigured credentials / auth	Both ends run DEVICE_CONNECT_ALLOW_INSECURE=true (auth bypassed)	No auth in the path	❌ Eliminated
2	Wrong backend / not really in D2D	Compared device startup logs	Both auto-select Zenoh D2D peer mode; healthy, identical logs	❌ Eliminated
3	Multicast physically blocked on the LAN	Raw Zenoh pub/sub, device → agent	16/16 messages delivered	❌ Eliminated (multicast works here)
4	Zenoh session never forms (real adapter)	Ran the real adapter on both hosts; dumped connected peer ZIDs + presence	Session forms; presence both ways	❌ Eliminated as a blanket failure
5	Agent leaves D2D when given an explicit endpoint	Ran agent with ZENOH_CONNECT only	Registry timeout → DISCOVERED: []	✅ Root cause #1
6	Multicast scout binds the wrong NIC	interface=auto vs pinned, multicast-only	auto intermittently fails; pinned is deterministic	✅ Root cause #2

The fix (#48)

Area	Change
agent-tools/connection.py	auto (default) = D2D whenever the backend is Zenoh, including with ZENOH_CONNECT (keeps the unicast endpoint, still discovers via presence). Opt into the registry with DEVICE_CONNECT_DISCOVERY_MODE=infra.
edge/zenoh_adapter.py	New ZENOH_MULTICAST_INTERFACE env (and multicast_interface kwarg) to pin the multicast scout to a NIC.

Fix validation (on hardware)

Scenario	Before	After
Agent with ZENOH_CONNECT=tcp/<device>:7447 only (no DISCOVERY_MODE)	discovery timed out (no responders), DISCOVERED: []	DISCOVERED: [sensor-001], invoke → {temperature: 22.5, humidity: 45}
Multicast-only with ZENOH_MULTICAST_INTERFACE pinned to the LAN NIC	flaky / no session	session + presence both directions

Suite	Result
device-connect-edge unit (excl. fuzz)	529 passed, 1 skipped
test_zenoh_adapter.py	72 passed (incl. 2 new)
test_connection_unit.py	26 passed (incl. 4 new)

Field guidance (share with the customer)

When D2D discovery returns [] on a robot/Wi-Fi network:

• Multi-homed host? Pin the LAN NIC: ZENOH_MULTICAST_INTERFACE=wlan0.
• Multicast blocked (AP/client isolation, managed switch)? Use a direct unicast link — device: ZENOH_LISTEN=tcp/0.0.0.0:7447; agent: ZENOH_CONNECT=tcp/<device-ip>:7447. After the fix the agent needs no second env var.