Myriad

AI for everyone

Description

Gemma 4 31B needs about 62 GB of GPU memory, but the GPUs people own have 10–24 GB. Myriad lets a swarm of ordinary GPUs, owned by different people, run it together, like BitTorrent for AI. The model is split by layers across peers. Your own machine keeps the first and last layers, so peers only ever see numbers in the middle, never your prompt or the answer.

Two ideas make it practical:

• Speculative decoding across the swarm. Distance makes a swarm slow, since every word travels through every peer. So your PC guesses the next few words with Gemma 4’s official drafter, which runs locally on state your PC already holds. The swarm then checks all the guesses in one trip. The output is identical to the full model’s. With Gemma 4 31B split between Montreal and Sweden, it went from 2.05 to 3.60 tokens/s (1.76x).
• Tit-for-tat credits, without a token. Peers sign a receipt for every piece of work, and each node keeps its own ledger. Busy peers serve contributors first, and keep a slot open for newcomers. In our test, a contributor’s wait halved, and freeloaders still made progress.

A live dashboard shows each request crossing the swarm, the guesses accepted, and each peer’s credits. You can ask questions from any browser.

How we built it

• Stack: Python, PyTorch and Hugging Face Transformers. The peers speak WebSockets and msgpack, and a small tracker handles discovery.
• Correctness: checked against Transformers. The split model is bit-identical to running it in one piece, on E2B and E4B, and the same holds across real peer processes. 92 automated tests.
• Downloads: each peer fetches only its own layers’ weights, by HTTP range requests.
• Deployment: three RunPod GPUs in three regions (Montreal, Sweden, Missouri).

Gemma 4 and open source

• Why Gemma 4: one shared vocabulary across sizes lets a small model guess for a big one, and its official MTP drafter runs entirely on the client.
• License: the code is Apache-2.0, like Gemma 4.
• Repo: https://github.com/Delta025/myriad

What’s next

Faster GPU code (CUDA graphs), hiding who sent a request, and catching peers that cheat.