Nuclei

Chapter 15 of 17

Backends

io_uring, epoll, kqueue and I/O completion ports: how one is picked, and what each does.

How the backend is picked

The first of these that says something decides, when a runtime's first shard starts:

  1. NUCLEI_BACKEND, when set and not empty: auto, io_uring, or the platform's poller by name (epoll, kqueue, iocp).
  2. The runtime's own Builder::backend.
  3. NucleiConfig::backend, the process's choice for every runtime that names none.
  4. auto: io_uring where the kernel lets the process have the rings its shards use, the platform's poller otherwise.

A backend named in any of these ways that the process cannot have fails with the reason; it never falls back. Proactor::backend() says which one the calling thread's shard runs.

src/main.rs
use nuclei::{Builder, IoBackend, Proactor};

fn main() -> std::io::Result<()> {
    // What the process picked: NUCLEI_BACKEND if set, else io_uring where it may, else the poller.
    println!("default: {:?}", Proactor::backend());

    // One runtime on the platform's poller, whatever the default is.
    let poller = if cfg!(target_os = "linux") {
        IoBackend::Epoll
    } else if cfg!(windows) {
        IoBackend::Iocp
    } else {
        IoBackend::Kqueue
    };
    let runtime = Builder::new().worker_threads(1).backend(poller).build()?;
    let used = runtime.block_on(runtime.spawn(async { Proactor::backend() }));
    // It runs the poller, unless NUCLEI_BACKEND names another: the variable wins.
    if std::env::var_os("NUCLEI_BACKEND").is_none_or(|v| v.is_empty()) {
        assert_eq!(used, poller);
    }
    Ok(())
}

io_uring

  • Linux and Android 6.1 or later, with the iouring feature (on by default). The ring is Nuclei's own, on libc alone.
  • The probe is safe even under a seccomp filter that kills a process for calling io_uring_setup; where io_uring is refused (an older kernel, a container's policy), auto runs epoll.
  • Each shard owns one ring. A worker's ring takes submissions from whichever thread runs its shard, one at a time, since a handoff may move the shard; the shard of a thread in block_on, and a worker with handoff_after(None), use a single-issuer ring with deferred task work. A kernel submission thread (sqpoll_wake_interval) replaces both.
  • A stream keeps one multishot receive armed, and accept_multi one multishot accept, filled from the shard's receive group (recv_buffers, recv_buffer_size). The group fits in half the completion queue, and an armed receive or accept that waits on an idle connection takes no room from other operations.
  • Regular files are read, written, synced and allocated with ring operations, at the handle's cursor or at an offset (Files). A write_zero_copy of at least zero_copy_threshold bytes is sent without a copy.

epoll

  • Linux, Android and illumos. On Linux it is what auto picks where io_uring cannot be had.
  • A readiness poller: the shard's driver makes each call when the descriptor may be ready, into the shard's buffers. Your code sees the same completion API.
  • Regular files run as jobs on the blocking pool: pread and pwrite (preadv and pwritev for several buffers), syncs and the space calls. A child's exit comes through its pidfd.

kqueue

  • macOS, iOS, FreeBSD, NetBSD, OpenBSD and DragonFly.
  • A readiness poller, served like epoll; a child's exit comes through EVFILT_PROC, wakes through EVFILT_USER.
  • Regular files run on the blocking pool.

I/O completion ports

  • Windows. One completion port per shard; operations are overlapped calls the kernel completes.
  • A handle takes std's TCP and UDP sockets, files, and a child's pipes. There are no Unix sockets on Windows.
  • A pipe or console opened for synchronous IO cannot complete on a port, so its calls run as jobs of the blocking pool and come back to the shard like the kernel's completions.
  • A TCP write, once issued, is always finished: a cancelled send there may have handed bytes to the network stack while saying it took none.
  • Timed waits use a high-resolution waitable timer where the system offers one, so timers are not rounded to the 15.6 ms system tick, and the handoff's watch acts at its threshold.

What is tested where

CI runs the suite on io_uring and on epoll on Linux 6.1, 6.6 and 6.12 and on the runner's own kernel, on kqueue on macOS, and on completion ports on Windows; it runs it under AddressSanitizer on both Linux backends. The examples on these pages run on Linux, on io_uring and on epoll.