Why Your Yellowstone gRPC Stream Lags or Misses Events

A Yellowstone gRPC stream can fail in two ways that look the same from your bot. It can be late: every event arrives, but slots after it happened. Or it can be incomplete: it looks fast, but some events never arrive. Both get reported as "my stream is slow", and the fixes are different. This post shows how to tell which one you have, the usual causes of each in the order we would check them, and the code for the fixes. It applies to any Yellowstone provider and to a node you run yourself.

First, measure which problem you have

Two checks, run from the server your bot runs on, tell lag and gaps apart.

How far behind is the stream? Compare the newest slot your code has read from the stream with the slot the node is on right now. Yellowstone endpoints answer a unary GetSlot call, so you can ask the same node every second. A gap of a slot or two is normal. A gap that is large from the start points at distance or at the node. A gap that keeps growing while the stream runs means updates are queueing between the node and your code. Slots are about 200 ms now (we measured 210 to 216 ms on mainnet on 28 October 2026), so 5 slots behind is about a second.

Is the stream missing anything? Lag numbers can't show an event that never arrived. For that, record every signature the stream delivers for one account over a fixed time, then ask an RPC node for that account's signatures over the same slots and compare. A signature the RPC node has and the stream doesn't is a missed event. Two things to get right:

• Compare like with like. If your filter sets failed: false, leave failed transactions (err not null) out of the RPC list too.
• A missing slot number is not a gap. Leaders skip slots. Check a suspicious range with getBlocks before you count it.

This script does both. Point it at an account with tens of transactions a minute, not a whole busy program, or the RPC check will take many pages.

// stream-check.mjs: how far behind is the stream, and is it missing anything?
// npm install @grpc/grpc-js @grpc/proto-loader @solana/web3.js bs58
// geyser.proto and solana-storage.proto from rpcpool/yellowstone-grpc go in this folder.
import grpc from "@grpc/grpc-js";
import protoLoader from "@grpc/proto-loader";
import { Connection, PublicKey } from "@solana/web3.js";
import bs58 from "bs58";

const ENDPOINT = process.env.GRPC_ENDPOINT ?? "88.216.36.108:10001"; // AllenHark Frankfurt
const ACCOUNT = process.env.WATCH_ACCOUNT; // a wallet or pool with steady activity
const MINUTES = Number(process.env.MINUTES ?? 10);
const rpc = new Connection(process.env.RPC_URL ?? "https://api.mainnet-beta.solana.com", "finalized");

const def = protoLoader.loadSync("geyser.proto", {
  keepCase: true, longs: String, enums: String, defaults: true, oneofs: true,
});
const { geyser } = grpc.loadPackageDefinition(def);
const client = new geyser.Geyser(ENDPOINT, grpc.credentials.createInsecure(), {
  "grpc.max_receive_message_length": 64 * 1024 * 1024, // the 4 MB default is too small for big updates
});
const meta = new grpc.Metadata(); // empty from an allowlisted IP
if (process.env.ALLENHARK_KEY) meta.set("x-token", process.env.ALLENHARK_KEY);

let streamSlot = 0;     // newest slot read from the stream
let firstTxSlot = 0;
let lastTxSlot = 0;
const seen = new Set(); // signatures the stream delivered
let finished = false;

const stream = client.subscribe(meta);
stream.on("data", (u) => {
  if (u.update_oneof === "ping") {
    stream.write({ ping: { id: 1 } }); // answer the server's ping
  } else if (u.update_oneof === "slot") {
    streamSlot = Math.max(streamSlot, Number(u.slot.slot));
  } else if (u.update_oneof === "transaction") {
    const slot = Number(u.transaction.slot);
    firstTxSlot ||= slot;
    lastTxSlot = Math.max(lastTxSlot, slot);
    seen.add(bs58.encode(u.transaction.transaction.signature));
  }
});
stream.on("error", (err) => {
  if (finished) return; // our own cancel() at the end
  console.error(`stream error ${err.code}: ${err.details}`);
  process.exit(1);
});
stream.write({
  accounts: {},
  slots: { tip: { filter_by_commitment: true } },
  transactions: {
    watch: { vote: false, failed: false, account_include: [ACCOUNT], account_exclude: [], account_required: [] },
  },
  transactions_status: {}, blocks: {}, blocks_meta: {}, entry: {}, accounts_data_slice: [],
  commitment: "PROCESSED",
});

// Lag: the node's slot now against the newest slot this process has read
const lagTimer = setInterval(() => {
  client.getSlot({ commitment: "PROCESSED" }, meta, (err, res) => {
    if (err) return console.error("GetSlot:", err.details);
    const behind = Number(res.slot) - streamSlot;
    console.log(`node ${res.slot}  stream ${streamSlot}  behind ${behind} slots (~${behind * 200} ms)`);
  });
}, 1_000);

// Completeness: after MINUTES, compare with RPC over the same slots
setTimeout(async () => {
  finished = true;
  clearInterval(lagTimer);
  stream.cancel();
  await new Promise((r) => setTimeout(r, 5_000)); // let the newest slots finalize
  const from = firstTxSlot + 1; // the first slot may have been joined halfway
  const to = lastTxSlot - 5;
  let streamed = 0;
  const missing = [];
  let before;
  scan: for (;;) {
    const page = await rpc.getSignaturesForAddress(new PublicKey(ACCOUNT), { before, limit: 1000 });
    if (page.length === 0) break;
    for (const s of page) {
      if (s.slot < from) break scan;
      if (s.slot > to || s.err !== null) continue; // outside the window, or failed
      if (seen.has(s.signature)) streamed++;
      else missing.push(s);
    }
    before = page[page.length - 1].signature;
  }
  console.log(`slots ${from}-${to}: ${streamed} delivered, ${missing.length} missing`);
  for (const s of missing.slice(0, 20)) console.log("missing", s.slot, s.signature);
  process.exit(0);
}, MINUTES * 60_000);

If the lag line climbs, read the next section. If the last line reports missing signatures, skip to "Why streams miss events" further down.

Why streams lag

1. Your read loop does too much

This is the first thing we would check. A gRPC stream is flow-controlled: the server only sends as fast as your client reads. If the loop that reads the stream also parses, logs, writes to a database or waits on an RPC call, reading stops while that work runs. Updates queue up on the server side, and every queued update is a late update.

When the queue fills, the open-source Yellowstone server stops waiting. It ends the stream with INTERNAL: lagged to send an update, or with lagged to receive geyser messages if the client fell behind the server's internal broadcast. If you see "lagged" in an error, the stream was not read fast enough: usually because of the client, sometimes because the connection is too thin for the data you asked for. The updates still queued for you when the stream ended are lost too, so lag turns into missing events.

The fix is to make the read loop do nothing but read. Hand each update to a queue and do the work somewhere else. In Rust:

// Cargo.toml: yellowstone-grpc-client = "16.0.1", yellowstone-grpc-proto = "14.0.1",
// tokio = { version = "1", features = ["full"] }, futures = "0.3", anyhow = "1"
use std::{collections::HashMap, time::Duration};

use futures::{SinkExt, StreamExt};
use tokio::sync::mpsc;
use yellowstone_grpc_client::GeyserGrpcClient;
use yellowstone_grpc_proto::geyser::{
    subscribe_update::UpdateOneof, CommitmentLevel, SubscribeRequest, SubscribeRequestFilterSlots,
    SubscribeRequestFilterTransactions, SubscribeRequestPing, SubscribeUpdateTransaction,
};

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    let mut client = GeyserGrpcClient::build_from_shared("http://88.216.36.108:10001")?
        .x_token(std::env::var("ALLENHARK_KEY").ok())?
        // A dead connection fails instead of hanging
        .http2_keep_alive_interval(Duration::from_secs(15))
        .keep_alive_timeout(Duration::from_secs(5))
        .keep_alive_while_idle(true)
        // The 4 MB default is too small for large blocks and accounts
        .max_decoding_message_size(64 * 1024 * 1024)
        .connect()
        .await?;

    let mut transactions = HashMap::new();
    transactions.insert(
        "pump".to_string(),
        SubscribeRequestFilterTransactions {
            vote: Some(false),
            failed: Some(false),
            account_include: vec!["6EF8rrecthR5Dkzon8Nwu78hRvfCKubJ14M5uBEwF6P".to_string()],
            ..Default::default()
        },
    );
    // Slot updates arrive every slot, so they double as a heartbeat
    let mut slots = HashMap::new();
    slots.insert(
        "tip".to_string(),
        SubscribeRequestFilterSlots { filter_by_commitment: Some(true), ..Default::default() },
    );
    let request = SubscribeRequest {
        transactions,
        slots,
        commitment: Some(CommitmentLevel::Processed as i32),
        ..Default::default()
    };
    let (mut sink, mut stream) = client.subscribe_with_request(Some(request)).await?;

    // The slow part (parsing, strategy, storage) runs here, off the read loop
    let (queue, mut work) = mpsc::channel::<SubscribeUpdateTransaction>(100_000);
    tokio::spawn(async move {
        while let Some(tx) = work.recv().await {
            handle(tx);
        }
    });

    // The read loop only reads, answers pings and queues
    loop {
        let update = match tokio::time::timeout(Duration::from_secs(5), stream.next()).await {
            Ok(Some(update)) => update?,
            Ok(None) => anyhow::bail!("stream ended: reconnect"),
            Err(_) => anyhow::bail!("no update for 5 s, not even a slot: reconnect"),
        };
        match update.update_oneof {
            Some(UpdateOneof::Ping(_)) => {
                let pong = SubscribeRequest { ping: Some(SubscribeRequestPing { id: 1 }), ..Default::default() };
                sink.send(pong).await?;
            }
            Some(UpdateOneof::Transaction(tx)) => {
                if queue.capacity() < 1_000 {
                    eprintln!("worker queue nearly full: the worker is too slow");
                }
                queue.send(tx).await?;
            }
            _ => {}
        }
    }
}

fn handle(tx: SubscribeUpdateTransaction) {
    // Your code: decode the instructions, decide, store
    let _ = (tx.slot, tx.transaction);
}

If the queue stays near full, the worker is the bottleneck: add workers or do less per update. When the queue is full, send waits, which pushes the backlog back to the server. A bot that only cares about the newest data can use try_send instead and drop updates on purpose, which keeps the stream current at the cost of losing data.

In Node.js the same rule applies, with one more catch: @grpc/grpc-js decodes every message on the main thread. Don't await inside the data handler, don't console.log whole updates, and if the stream is heavy, run the gRPC client in its own worker thread or process and pass small results to the rest of your bot.

2. You ask for more data than you use

Every update has to cross the network and be decoded before the next one can be read. A transaction stream for a whole busy program, or an account stream for a big owner such as the Token program, can carry far more data than your bot needs. Filters run on the server, so a tighter filter cuts the bytes sent and your decode time.

• Transactions: set vote: false, and failed: false unless you need failed transactions. Use account_required to demand every account of the pattern you want, not just one.
• Accounts: add memcmp and datasize filters to an owner subscription, so you only receive the account type you parse.
• Account data: accounts_data_slice sends only the byte ranges you read, for example a discriminator and one field, instead of the whole account.

3. Distance to the endpoint

Every update crosses the network once, so put your bot in the same region as the endpoint. To see the round trip, time a unary Ping or GetSlot call from your server. For AllenHark that means Frankfurt or Amsterdam. If your servers are in the US or Asia, an endpoint near them will beat any endpoint in Europe.

4. The commitment level you chose

confirmed and finalized updates are sent when a slot reaches that level, so they arrive after processed by design. Under Alpenglow the gap is small: the Solana Foundation puts finality at roughly 150 ms. That is not lag. If you need the earliest signal, subscribe at processed and handle bank_id, as our Alpenglow guide explains.

5. The node itself is behind

If GetSlot on the endpoint trails another source at processed (a different RPC node, sampled at the same moment), the node is behind the cluster. Nothing in your client fixes that. Tell your provider, or try another region.

6. You need data earlier than any gRPC stream has it

A gRPC stream only sees a transaction after the node has received its shreds and executed it. When that is still too late, read shreds directly: they carry the block's data as the leader sends it out. Our comparison of shreds, gRPC, RPC and WebSockets covers the trade-offs.

Why streams miss events

1. The stream died quietly

A TCP connection can die without either side noticing: a NAT or load balancer drops an idle mapping, or a host disappears. Your client sees no error and no data. It just waits. Three defences, all cheap:

• Answer the server's pings. Yellowstone servers send a ping update on every open stream (every 10 seconds in the current open-source code; the README says 15). Reply on the same stream with { ping: { id: 1 } }. The server answers with a pong and keeps your filters, because a ping-only request does not replace them. The reply also keeps proxies that close streams with no client traffic from closing yours.
• Turn on HTTP/2 keepalive so a dead connection fails instead of hanging. In Rust, set http2_keep_alive_interval, keep_alive_timeout and keep_alive_while_idle on the builder, as above. In Node.js, set grpc.keepalive_time_ms, grpc.keepalive_timeout_ms and grpc.keepalive_permit_without_calls in the channel options. Don't set the interval very low: many gRPC servers answer pings that come too often with a GOAWAY (too_many_pings).
• Run a watchdog. Add a slots filter even if you don't need slots. Slot updates arrive about every 200 ms, so if nothing arrives for a few seconds, the stream is stuck: cancel it and reconnect.

2. You reconnected at the tip

A reconnect that starts from the current slot loses everything that happened during the gap. To get it back, resubscribe with from_slot set to the last slot you finished, send your full request again (the server keeps no state between connections), and drop the duplicates the replay sends. How far back you can go depends on the provider; AllenHark keeps about 48 hours. Our guide to reconnecting without missing transactions has the full code.

3. The server cut you off for lagging

See the first lag cause. A stream ended with lagged to send an update loses the updates queued for it. Fix the read loop first, then reconnect with from_slot.

4. One message was too big for your client

Both tonic (Rust) and @grpc/grpc-js (Node.js) refuse incoming messages over 4 MB by default. A large block, or a big account, then kills the stream. In Rust the error reads decoded message length too large; in Node.js it is RESOURCE_EXHAUSTED with "Received message larger than max". Raise the limit: max_decoding_message_size on the Rust builder, or grpc.max_receive_message_length in the Node.js channel options.

5. Your filter doesn't mean what you think

• AND inside a filter, OR between filters. Inside one named filter, every field must match: a transaction must pass vote and failed, contain at least one account from account_include, none from account_exclude and all from account_required. Separate named filters in one request are alternatives. Each update lists the names it matched in filters.
• failed: false drops failed transactions. That is often what you want, but a reconciliation against RPC has to drop them too.
• Account filters send writes, not current state. An account that nobody writes to sends nothing. Read its current state once with getMultipleAccounts when you subscribe, then apply the stream's updates on top.

6. Forks and skipped slots look like gaps

At processed, a slot can have more than one candidate block, and Yellowstone labels each with a bank_id. If you key data by slot alone, two candidates merge into one, or one overwrites the other. Key by slot and bank_id, and keep the bank that reaches confirmed. The Alpenglow guide shows the buffering. Skipped slots are normal: a slot number with no data is not a missed event.

Symptom, cause, fix

AllenHark Yellowstone gRPC

AllenHark Yellowstone gRPC runs in Frankfurt (88.216.36.108:10001) and Amsterdam (84.32.104.38:10001), plaintext HTTP/2, so connect with http:// or createInsecure().

• Authentication: connect from an IP address on your allowlist, or, on new subscriptions, send your API key in the x-token header.
• Replay: from_slot replays about 48 hours, so a reconnect, or a long outage, can be filled in from the stream itself. SubscribeReplayInfo returns the oldest slot you can ask for.
• Alpenglow: streams carry bank_id today.
• Limits: on gRPC Standard, each connection you buy is one Subscribe stream with 500 accounts per subscription. Dedicated gRPC has no rate limits and no account limit. Plans are on the pricing page.

The gRPC docs cover setup, and the Subscribe reference covers every filter field.

FAQ

Why is my Yellowstone gRPC stream several slots behind?

Usually because your code reads it too slowly: work done inside the read loop holds back every update after it. Check that first, then the filter size, the distance to the endpoint and the commitment level. If GetSlot on the endpoint itself trails the cluster, the node is behind.

What does "lagged to send an update" mean?

The open-source Yellowstone server sends it when it ends a stream because the client was not reading fast enough and its queue filled. Move the work off the read loop, then reconnect with from_slot to recover what was lost.

Why does my gRPC stream stop without an error?

The connection died without a clean close, often at a NAT or a load balancer. Answer the server's pings, turn on HTTP/2 keepalive, and reconnect when no slot update has arrived for a few seconds.

How do I know if my stream is missing transactions?

Record the signatures the stream delivers for one account, then compare them with getSignaturesForAddress over the same slots, applying the same rules as your filter. Any signature in the RPC list but not in yours was missed.

Do I need to answer pings?

Answering costs nothing, and some proxies close streams on which the client never sends anything. Reply to each ping update with a ping request on the same stream.

Related reading

• Reconnect Yellowstone gRPC without missing transactions
• Alpenglow: what Solana RPC and gRPC users need to change
• Best Yellowstone gRPC providers in 2026, with a test you can run on any provider
• Shreds vs gRPC vs RPC vs WebSockets

Sources

Checked October 29, 2026.

• Yellowstone gRPC README (filters, pings), server code (the "lagged" errors, the 10-second ping) and geyser.proto
• yellowstone-grpc-client 16.0.1 and tonic 0.14 source, for the builder options and the 4 MB default
• Solana Foundation, Alpenglow, for finality and bank_id
• Slot times: our measurement on mainnet RPC, 28 October 2026