← Madiyar Zhunussov

MeshSync

Calibrate face gestures once at one small board, and every board nearby recognizes you.

Two ESP32 development boards on a wooden desk, one connected by USB.

How it came together

1. Calibration that adds up

Problem: Each board learns from the people who calibrate at it. When boards share what they learned, the result has to match training on everyone’s data at once, whatever order the boards merge in.

Choice: Boards store calibration as integer totals (counts, sums and sums of squares) instead of models or raw samples. Merging two boards is then just addition. Each class is capped at 1,000,000 samples so the 64-bit math can’t overflow.

Result: After merging, accuracy was 90.7%, the same as central training, compared with about 48% for one board alone. In 600 random merge orders the totals were exact every time, and the C++ matched the Python prototype bit for bit on 450 test drawings.

2. Deleting data from boards that weren’t listening

Problem: A deletion marker can only remove data a board already has. A board that was offline during a deletion could keep recording the deleted user and bring them back later.

Choice: Each user gets a deletion counter that only ever goes up. Any board that sees a higher counter drops that user’s data, including data it learned about after the deletion.

Result: With this rule turned off, deleted data came back in 3 of 20 simulated runs.

Fig. 1. A board that was offline during a deletion rejoins, sees the higher deletion counter, and drops the user. Illustrative.

3. Syncing state that doesn’t fit in a packet

Problem: The full state, up to 64 slots, 256 deletion markers and 64 users, is far larger than one 250-byte radio packet.

Choice: Every second, each board broadcasts a short fingerprint of its state. When fingerprints differ, the boards page through each other’s contents and send only the missing pieces, in chunks of up to 232 bytes with a checksum. Boards in the middle relay. I tested it in a simulator that runs the same C++ with 6 boards and a bad radio: 20% of packets dropped, 10% duplicated and 1% corrupted.

Result: 500 of 500 simulated runs reached the same state, in 8.7 seconds on average and 22.2 seconds at worst. Two real boards sync in 1.3 seconds.

4. Keeping recognition fast while the radio works

Problem: Recognizing a gesture has to stay quick even while a board is busy syncing.

Choice: The browser runs face tracking locally and sends the board 12 numbers per frame, 55 to 70 bytes instead of a 921,600-byte camera frame. On the board, the radio runs on its own core, and recognition never waits for the sync code.

Result: From the page to the board and back took 18 ms typically, and under 26 ms for 95% of 300 predictions. Video never leaves the laptop.

What broke

In the playground, a deletion counter jumped although nobody had deleted anything. The inventory pages had no checksum, so one corrupted byte could turn a normal entry into a deletion marker and erase live data. I added a CRC to every page, a test that flips one bit and expects the page to be rejected, and a simulator check that nothing is deleted by mistake. Simulated runs went from 490/500 passing to 500/500.

Next

Eye gaze in the browser, and a head-controlled cursor. Known limits: deletions aren’t authenticated, deletion markers are never cleaned up, and only two physical boards have been tested.