Chapter 14 of 17

Production

Every runtime setting, the hooks, the counters, a clean shutdown, and tests.

Every runtime setting

Every setting is optional, and an unset one takes a default that needs no configuration to work well. Set them on a Builder, then build a runtime of your own or build_global the default one (Runtime). build checks them all before it starts a thread: a value outside what the runtime can work with fails with InvalidInput naming the setting, its value and the range it must be in (Errors).

src/main.rs
use std::time::Duration;

use nuclei::runtime::{RngSeed, UnhandledPanic};
use nuclei::{Builder, IoBackend};

fn main() -> std::io::Result<()> {
    let backend = if cfg!(target_os = "linux") {
        IoBackend::Epoll
    } else {
        nuclei::Proactor::backend()
    };
    let runtime = Builder::new()
        .worker_threads(4) // default: one per core
        .max_worker_threads(8) // default: four times worker_threads
        .pin_workers(true) // Linux; default true
        .thread_name("edge") // default nuclei-worker, nuclei-blocking
        .thread_stack_size(2 * 1024 * 1024) // default 8 MiB
        .max_nesting(64) // default 256
        .max_blocking_threads(128) // default 512
        .thread_keep_alive(Duration::from_secs(5)) // default 10 seconds
        .local_spawn_margin(16) // default 32
        .handoff_after(Some(Duration::from_millis(20))) // default 10 ms
        .io_buffer_cap(32 * 1024 * 1024) // default 64 MiB per worker
        .write_behind_linger(Duration::from_secs(5)) // default one minute
        .event_interval(31) // default 61
        .global_queue_interval(15) // default 31
        .max_io_events_per_tick(256) // default: all that are ready
        .disable_lifo_slot()
        .rng_seed(RngSeed::from_bytes(b"replay"))
        .unhandled_panic(UnhandledPanic::ShutdownRuntime)
        .backend(backend) // default: the process's choice
        .build()?;

    let used = runtime.block_on(runtime.spawn(async { nuclei::Proactor::backend() }));
    println!("workers run {used:?}");
    runtime.shutdown_timeout(Duration::from_secs(1));
    Ok(())
}
SettingWhat it sets (its default)
worker_threadsWorker threads, each running one shard. The default is one per core of worker_cores when it is set, else one per core the process may run on, held to max_worker_threads when that is set. Zero runs every task on the thread that spawns it, while that thread is inside block_on.
max_worker_threadsThe most worker threads the runtime may start over its life, the ones it starts with included (default four times the workers it starts with): what spawn_more_threads may add up to. A stopped worker’s place is not taken again. Below an explicit worker_threads, build fails with InvalidInput; with worker_threads unset, the runtime starts with at most this many.
pin_workersWhether each worker thread is pinned to its own core (Linux and Windows; default true). A pinned shard keeps its cache warm; unpinned, the scheduler may move it. With worker_cores set, each worker is pinned to its core of that list, or, unpinned, runs on any core of the list.
worker_coresThe cores the workers run on, in the order the workers take them: worker k runs on cores[k % cores.len()], pinned to it, or, with pin_workers(false), on any core of the list. The worker count defaults to the list’s length. Every core must be one the process may run on and the list must not be empty; build fails with InvalidInput otherwise. Pinning takes effect on Linux and Windows; elsewhere the list sets the worker count and is checked against the cores the machine has.
blocking_coresThe cores the blocking pool’s threads run on, any of them (default: the cores of the thread that builds the runtime). Checked as worker_cores are, and in effect on Linux and Windows.
thread_nameThe name of every thread the runtime starts (default nuclei-worker and nuclei-blocking).
thread_name_fnA function naming each thread the runtime starts.
thread_stack_sizeThe stack size of every thread the runtime starts, in bytes (default 8 MiB). At least 128 KiB; build fails with InvalidInput below.
max_nestingHow deep block_on may nest on one thread (default 256). A call one level deeper panics instead of running the thread out of stack. At least 1; build fails with InvalidInput below.
max_blocking_threadsThe most threads the blocking pool runs at once (default 512). At least 1; build fails with InvalidInput below.
thread_keep_aliveHow long an idle blocking thread waits for work before it exits (default 10 seconds).
local_spawn_marginHow many more live tasks than another shard a spawning thread’s shard may hold and still take a task it spawns (default 32). A task spawned on a worker stays on it (so a task that waits for what it spawned keeps it on its own thread), and a task spawned on any other thread goes to that thread’s home shard (where its spawns went last), unless that shard holds that many more tasks than a shard picked at random; then it goes to the other, which becomes the home. Zero balances strictly; a larger margin keeps more work together.
handoff_afterHow long one poll of a task may run on a worker before the worker’s other tasks go on without it (default 10 ms; None: never).
io_buffer_capThe most bytes each worker’s IO buffers hold (default 64 MiB). Reads that had to wait land in these buffers, and writes that had to wait are copied into them; at the cap, IO waits for buffers to come back, so a worker’s IO memory never grows past it. On io_uring the worker’s receive buffers count against the cap too (recv_buffers of recv_buffer_size bytes, 4 MiB by default, with their descriptors), and the rest is left for the others. A cap that leaves less than 512 bytes beside the receive buffers is refused with InvalidInput when the worker starts; otherwise any value is accepted, and a request that cannot fit under a cap too small for it fails with an error that names this setting.
write_behind_lingerHow long the bytes a dropped handle wrote behind may take to reach the kernel (default one minute): a task delivers them after the drop, as the kernel delivers what a closed socket holds. Past it the rest is given up, and a TCP stream’s connection is reset, so its peer reads an error rather than the end of a stream cut short. get_mut and into_inner wait for the bytes as long at most, and the runtime’s drop waits for the tasks. More than zero; build fails with InvalidInput at zero.
zero_copy_thresholdThe least a write_zero_copy must have left to send for its next send to go zero-copy, in bytes (default 1 MiB): below it, the kernel copies the bytes, as it does for every write. Copying costs less than pinning the pages for small writes; where the threshold pays depends on the machine and its network. Only io_uring sends zero-copy, on the runtime’s workers: every value is accepted, and 0 sends every write zero-copy, usize::MAX none.
send_file_chunkThe most bytes one call of a send_file sends (default 1 MiB). Each call runs on the blocking pool and may stall on a cold page there, so this bounds how long a pool thread is held, and how much a cancelled send_file still sends after its future went; where the system has no call for it, each chunk is read into a buffer of this size. From 1 byte to 0x7ffff000 (what one sendfile(2) sends); build fails with InvalidInput outside.
event_intervalTask runs between a worker’s looks at its timers and IO when it is busy (default 61): lower answers IO sooner, higher runs tasks with fewer interruptions. At least 1; build fails with InvalidInput at 0.
global_queue_intervalTask runs between the times a worker takes from its inbox (tasks woken by other threads) before its own queue (default 31). At least 1; build fails with InvalidInput at 0.
max_io_events_per_tickThe most IO events a worker takes from its driver in one look (default: all that are ready). At least 1; build fails with InvalidInput at 0.
disable_lifo_slotTurns off each worker’s LIFO slot, where a task woken by the task running on the worker goes to run next. Without it the woken task goes to the back of the worker’s queue: fairer when the queue is long, slower for a task that hands work to another and waits for it.
rng_seedSeeds the runtime’s random choices, so that they repeat run after run; see RngSeed.
unhandled_panicWhat the runtime does when one of its tasks panics (default UnhandledPanic::Ignore: the panic goes to the task’s handle).
start_pausedWhether the runtime’s clock starts paused (default false): see time::pause. Its tasks then see time move only as they advance it, or as it jumps to the next timer whenever every shard of the runtime is idle.
backendThe kernel interface the workers run their IO on. Unset, it is the process’s: the backend of NucleiConfig, or io_uring where the kernel lets this process have it and the platform’s poller otherwise. A backend this process cannot have makes build fail with the reason. A non-empty NUCLEI_BACKEND overrides it.
io_uringThe settings the workers’ io_uring rings are made with, for this runtime alone (default: the process’s, from Proactor::with_config). Checked in build, which fails with InvalidInput naming the setting a ring cannot work with, on every platform; only the workers of a runtime whose IO runs on io_uring make rings. Whether io_uring is chosen when no backend is named is decided with these settings. The shard of a thread inside block_on keeps the process’s settings: it serves every runtime the thread enters.
record_latenciesCounts how long each task poll runs, and (with the io-stats feature) how long each IO operation waits for the kernel, by kind, in log2 histograms each worker’s ShardStats carries (default false: no clock is read for it). The poll times cost two clock reads and a call around every poll, of every runtime of the process while one records them; the IO latencies two clock reads per operation of this runtime’s workers.

A busy worker looks at its timers and IO every event_interval task runs, and also, checked every 16 runs, whenever its last look is more than a millisecond old, so a run of slow tasks cannot keep IO waiting for long.

Hooks

A runtime calls hooks of yours on its threads and around its tasks: for metrics, tracing or thread setup. Each task hook is given the task's Id.

src/main.rs
use std::sync::Arc;
use std::sync::atomic::{AtomicUsize, Ordering};

use nuclei::Builder;

fn main() -> std::io::Result<()> {
    let spawned = Arc::new(AtomicUsize::new(0));
    let polls = Arc::new(AtomicUsize::new(0));
    let runtime = Builder::new()
        .worker_threads(2)
        .on_thread_start(|| println!("worker up"))
        .on_task_spawn({
            let spawned = Arc::clone(&spawned);
            move |_task| {
                spawned.fetch_add(1, Ordering::Relaxed);
            }
        })
        .on_before_task_poll({
            let polls = Arc::clone(&polls);
            move |_task| {
                polls.fetch_add(1, Ordering::Relaxed);
            }
        })
        .build()?;

    runtime.block_on(async {
        for n in 0..10 {
            nuclei::spawn(async move { n }).await;
        }
    });
    assert_eq!(spawned.load(Ordering::Relaxed), 10);
    assert!(polls.load(Ordering::Relaxed) >= 10);
    Ok(())
}
HookWhen it is called
on_thread_startCalled on each thread the runtime starts (workers and blocking-pool threads), on that thread, before it does anything else.
on_thread_stopCalled on each thread the runtime starts, on that thread, as it ends (after a panic too).
on_thread_parkCalled on a worker thread just before it sleeps with nothing to run.
on_thread_unparkCalled on a worker thread just after it wakes from a sleep.
on_task_spawnCalled with each task this runtime spawns (not the blocking pool’s jobs), on the spawning thread, once the task exists.
on_before_task_pollCalled just before each poll of a task of this runtime, on the thread that polls it.
on_after_task_pollCalled just after each poll of a task of this runtime, on the thread that polled it.
on_task_terminateCalled when a task of this runtime is done with: its future completed or was dropped by a cancel, on the thread that dropped it.
on_long_pollCalled with each poll of a task of this runtime that its worker’s watch sees run past handoff_after, on the watch’s own thread: the task’s id and name say which one holds its worker up. The task is the innermost one whose poll was running when the watch looked; the poll may have ended by the time the hook runs. A poll is told once, unless a block_on inside it ran a long poll of its own and the shard stayed on that thread: the outer poll may then be told again as it goes on. The shard is handed on first, so a slow hook delays no rescue, but every other worker waits for its next look while the hook runs: keep it short. A panic in it is caught (after the panic hook reported it) and the watch goes on. Only workers are watched; with handoff_after(None) build fails with InvalidInput.

A hook runs in the middle of the runtime's work: keep it short, and never panic in one. Task hooks get a TaskMeta: the task's id(), and its name() when task::Builder gave one.

on_long_poll finds the task that holds a worker up: it is called, on the watch's own thread, for each poll that runs past handoff_after (Tasks).

src/main.rs
use std::sync::mpsc;
use std::time::Duration;

use nuclei::{Builder, task};

fn main() -> std::io::Result<()> {
    let (told, heard) = mpsc::channel();
    let runtime = Builder::new()
        .worker_threads(1)
        .handoff_after(Some(Duration::from_millis(10)))
        // Called on the watch's thread for a poll that runs past the handoff.
        .on_long_poll(move |task| {
            let _ = told.send(task.name().map(str::to_owned));
        })
        .build()?;

    let name = runtime.block_on(async move {
        let blocker = task::Builder::new()
            .name("report")
            .worker(0)
            .spawn(async move {
                // A blocking call inside a poll, on purpose: it waits for the report of itself.
                heard.recv_timeout(Duration::from_secs(10))
            })?;
        Ok::<_, std::io::Error>(blocker.await)
    })?;
    assert_eq!(name, Ok(Some("report".to_owned())));
    Ok(())
}

The io_uring configuration

The rings are configured once per process, before any IO, with Proactor::with_config: every runtime starts from these settings. A second call, or one after IO has started (which configures the defaults), panics. Builder::io_uring gives one runtime's workers ring settings of their own instead, and build answers a setting a ring cannot work with as an error rather than a panic.

src/main.rs
use nuclei::Proactor;
use nuclei::config::{IoUringConfiguration, NucleiConfig};

fn main() {
    // Once, before any IO: the io_uring settings of every shard.
    Proactor::with_config(NucleiConfig {
        iouring: IoUringConfiguration {
            queue_len: 4096,
            recv_buffers: 512,
            recv_buffer_size: 16 * 1024,
            ..IoUringConfiguration::default()
        },
        backend: None, // io_uring where this process may have it, else the poller
    });
    nuclei::block_on(async {
        println!("IO runs on {:?}", Proactor::backend());
    });
}
FieldWhat it sets (its default)
queue_lenEntries in each ring’s submission queue (default 2048); the completion queue holds twice as many.
recv_buffersBuffers in each shard’s group for multishot receives (default 256, or queue_len rounded up to a power of two when that is fewer): the kernel fills one as bytes arrive, so a stream’s receive stays armed for every arrival and a receive that waits holds no buffer of its own.
recv_buffer_sizeThe size in bytes of each of those buffers (default 16384); for UDP, one that holds the largest datagram expected.
sqpoll_wake_intervalWith Some(ms), a kernel thread polls the submission queue and sleeps after that many idle milliseconds: fewer system calls for more CPU (default None: no polling thread).
iopoll_enabledBusy-waits for file completions instead of taking interrupts (default false): lower latency for more CPU, and only for files opened with O_DIRECT on file systems that support polling.
per_numa_bounded_worker_countThe most io_uring worker threads per NUMA node for IO bound in time, on regular files and block devices (default Some(256)).
per_numa_unbounded_worker_countThe most io_uring worker threads per NUMA node for IO that may never complete, such as on sockets (default Some(512)).
aggressive_pollHas no effect, and is kept so code written against earlier versions builds (default false). Every shard already looks at its ring’s completions while it spins before it sleeps, and a kernel thread that polls the submission queue is sqpoll_wake_interval; no third kind of polling is left for this flag to turn on.

Four presets fill the struct from a queue length: interrupt_driven (the one to pick when unsure), low_latency_driven and kernel_poll_only (a kernel submission thread), and io_poll. NucleiConfig::backend is the backend of every runtime that names none.

The sizes are checked where they take effect: queue_len at least 64, recv_buffers 0 or a power of two below twice queue_len, and recv_buffer_size from 512 bytes to 1 GiB while there are receive buffers. A value outside fails the first runtime built with it, with InvalidInput naming the setting.

Counters

nuclei::stats() takes a snapshot of every shard alive in the process (each runtime's workers, then the shard of each thread in block_on) and of the blocking pools; Runtime::stats takes one of a single runtime's workers and pool. It reads the shards without stopping them; most counts are published by a shard as it is about to wait for work, and by a busy one every few hundred task runs.

src/main.rs
#[nuclei::main]
async fn main() {
    nuclei::spawn(async {}).await;

    let stats = nuclei::stats();
    for shard in &stats.shards {
        println!(
            "{:?} on {:?}: {} tasks, {} polls, {} IO in flight, {} of {} buffer bytes",
            shard.worker,
            shard.backend,
            shard.tasks,
            shard.polls,
            shard.ops_in_flight,
            shard.buffer_bytes,
            shard.buffer_cap,
        );
        for (kind, count) in shard.errors.iter() {
            println!("  {count} errors of kind {kind:?}");
        }
    }
    println!("{} blocking threads", stats.blocking_threads);
}
FieldWhat it counts
workerThe shard’s place among its runtime’s workers; None for the shard of a thread that runs block_on.
runtimeThe runtime the shard is a worker of; None for the shard of a thread that runs block_on, which serves every runtime the thread enters.
backendThe kernel interface the shard’s IO runs on.
tasksTasks placed on the shard and not done yet.
pollsTask polls the shard has run (published).
queue_depthRunnables waiting in the shard’s own queue, its LIFO slot included (published).
remote_wakesRunnables of the shard’s tasks it took from other threads: a task woken there, or spawned onto it from there (published).
parksTimes the shard slept with nothing to run (published).
parkedTime the shard spent asleep, in all (published).
busyTime since the shard was made that it did not spend asleep: running tasks, driving its IO and timers, spinning before a park (from the published time asleep).
ops_in_flightIO operations the shard handed to the kernel that have not ended: on io_uring the operations submitted, on IOCP the ones issued. A readiness poller (epoll, kqueue) hands the kernel none, as its driver makes each call once the descriptor is ready: 0 there (published).
cq_overflowsTimes the shard found completions the kernel was holding for want of room in the ring’s completion queue (none is lost: the kernel keeps them until there is room); 0 on other backends (published).
buffer_bytesBytes the shard’s IO buffers hold: its pool’s, in use and free, and on io_uring its receive buffers’ (published).
buffer_capThe most bytes the shard’s IO buffers may hold, its receive buffers included (Builder::io_buffer_cap).
buffer_exhaustedTimes a buffer was wanted while every byte under the pool’s cap was in use; the IO waits for one to come back (published).
timersTimers armed on the shard (published).
handoffsTimes the shard went to another thread because the thread running it stayed in one poll past Builder::handoff_after.
longest_pollThe longest poll the runtime saw on the shard, at the resolution of its watch (a quarter of the handoff threshold); zero when no poll lasted that long, or with the handoff off.
budget_yieldsTimes a task’s poll on the shard spent its budget of operations and yielded to the shard’s other tasks (see task::coop).
poll_timesHow long each task poll on the shard ran, while its runtime records it (Builder::record_latencies); None when it does not, which is not a shard with no polls.
errorsErrors of IO run on the shard.

Stats also has blocking_threads and blocking_queue_depth, off_shard_errors for IO run on threads that run no shard, and receive_buffers_missing for shards that started without the io_uring receive buffers they asked for.

IO counts and latencies

A runtime built with record_latencies(true) keeps a log2 Histogram of how long each task poll ran, in ShardStats::poll_times. With the io-stats feature (Setup) it can also count and time each worker's IO:

  • Builder::count_io(true) fills ShardStats::io, an IoCounts: reads and writes with the bytes of each, and accepts.
  • record_latencies(true) also fills ShardStats::io_latencies, an IoLatencies: the time from submission to completion of reads, writes, accepts, connects and file calls. On epoll and kqueue it is None: the driver makes each call once the kernel says it is ready, so there is no wait in the kernel to time.
src/main.rs
use std::time::Duration;

use nuclei::Builder;
use nuclei::io::{AsyncReadExt, AsyncWriteExt};
use nuclei::net::{TcpListener, TcpStream};

fn main() -> std::io::Result<()> {
    let runtime = Builder::new()
        .worker_threads(1)
        .count_io(true) // the io-stats feature
        .record_latencies(true) // poll times; with io-stats, IO latencies too
        .build()?;
    runtime.block_on(runtime.spawn(async {
        let listener = TcpListener::bind("127.0.0.1:0").await?;
        let mut client = TcpStream::connect(listener.local_addr()?).await?;
        let (mut server, _) = listener.accept().await?;
        client.write_all(b"ping").await?;
        client.flush().await?;
        let mut buf = [0u8; 4];
        server.read_exact(&mut buf).await?;
        Ok::<_, std::io::Error>(())
    }))?;
    // A worker publishes its counts each time it looks at its IO and timers: a sleep makes it look.
    runtime.block_on(runtime.spawn(nuclei::time::sleep(Duration::from_millis(1))));

    let stats = runtime.stats();
    let worker = &stats.shards[0];
    let io = worker.io.unwrap();
    assert_eq!((io.accepts, io.bytes_read, io.bytes_written), (1, 4, 4));
    let polls = worker.poll_times.as_ref().unwrap();
    for (from, until, samples) in polls.buckets() {
        println!("{samples} polls of {from:?} to {until:?}");
    }
    // Readiness backends (epoll, kqueue) have no wait in the kernel to time.
    if let Some(io) = &worker.io_latencies {
        let timed = [&io.reads, &io.writes, &io.accepts, &io.connects, &io.files];
        println!(
            "{} IO operations timed",
            timed.iter().map(|h| h.count()).sum::<u64>()
        );
    }
    Ok(())
}

Without the feature, no IO path carries counting or timing code, and these items do not exist. With it, every IO call costs a few instructions, counted or not, so leave it off where nobody reads the counts. The API reference here is built with io-stats on, so it lists them; they exist only with io-stats on.

A clean shutdown

A server stops taking connections when asked, lets the open ones finish, and gives up after a deadline. A CancellationToken carries the request to every task, Ctrl-C raises it, a JoinSet holds the connections, and timeout bounds the drain.

src/main.rs
use std::time::Duration;

use nuclei::net::{TcpListener, TcpStream};
use nuclei::sync::CancellationToken;
use nuclei::task::JoinSet;
use nuclei::time::timeout;

async fn serve(listener: TcpListener, shutdown: CancellationToken) -> std::io::Result<()> {
    let mut connections = JoinSet::new();
    // Take connections until the token is cancelled.
    while let Some(accepted) = shutdown.run_until_cancelled(listener.accept()).await {
        let (stream, _) = accepted?;
        let stop = shutdown.child_token();
        connections.spawn(async move {
            let mut buf = [0u8; 1024];
            while let Some(Ok(n @ 1..)) = stop.run_until_cancelled(stream.recv(&mut buf)).await {
                let _ = stream.send(&buf[..n]).await;
            }
        });
    }
    // Let the open connections finish, for five seconds at most.
    let drained = timeout(Duration::from_secs(5), connections.join_all()).await;
    if drained.is_err() {
        eprintln!("some connections did not finish in time");
    }
    Ok(())
}

#[nuclei::main]
async fn main() -> std::io::Result<()> {
    let listener = TcpListener::bind("127.0.0.1:0").await?;
    let addr = listener.local_addr()?;
    let shutdown = CancellationToken::new();

    // Ctrl-C cancels the token.
    let on_ctrl_c = shutdown.clone();
    nuclei::spawn(async move {
        if nuclei::signal::ctrl_c().await.is_ok() {
            on_ctrl_c.cancel();
        }
    })
    .detach();

    let server = nuclei::spawn(serve(listener, shutdown.clone()));

    // Here a client comes and goes, then the program asks for the shutdown itself.
    let client = TcpStream::connect(addr).await?;
    client.send(b"hi").await?;
    let mut buf = [0u8; 2];
    client.recv(&mut buf).await?;
    shutdown.cancel();
    server.await
}

Tests

#[nuclei::test] makes an async test: its body runs in block_on on the default runtime, or on a runtime of its own with threads = N or start_paused = true (paused time). A test may return a Result, like any test.

tests/echo.rs
use std::time::Duration;

use nuclei::net::{TcpListener, TcpStream};
use nuclei::time::{self, Instant};

#[nuclei::test]
async fn echoes_a_message() -> std::io::Result<()> {
    let listener = TcpListener::bind("127.0.0.1:0").await?;
    let addr = listener.local_addr()?;
    nuclei::spawn(async move {
        let (stream, _) = listener.accept().await?;
        let mut buf = [0u8; 64];
        let n = stream.recv(&mut buf).await?;
        stream.send(&buf[..n]).await?;
        std::io::Result::Ok(())
    })
    .detach();

    let client = TcpStream::connect(addr).await?;
    client.send(b"ping").await?;
    let mut buf = [0u8; 4];
    client.recv(&mut buf).await?;
    assert_eq!(&buf, b"ping");
    Ok(())
}

// A runtime of its own with one worker, for a test that needs a known shape.
#[nuclei::test(threads = 1)]
async fn times_out() {
    let never = std::future::pending::<()>();
    assert!(
        time::timeout(Duration::from_millis(10), never)
            .await
            .is_err()
    );
}

// A paused clock: the day-long timeout passes at once.
#[nuclei::test(start_paused = true)]
async fn a_day_passes_at_once() {
    let start = Instant::now();
    let never = std::future::pending::<()>();
    assert!(
        time::timeout(Duration::from_secs(86_400), never)
            .await
            .is_err()
    );
    assert_eq!(start.elapsed(), Duration::from_secs(86_400));
}

#[nuclei::bench] does the same for a benchmark on the nightly test harness.

The earlier entry points

Programs written against earlier versions set up the default runtime with init or init_with_config; both still work, on the shards. GlobalExecutorConfig sets the workers to start (with_min_threads, or from an environment variable with with_env_var), the most to run (with_max_threads) and their names. spawn_more_threads starts more workers, up to the maximum, and stop_thread drains one: it takes no new task and exits once its tasks are done.

src/main.rs
use nuclei::GlobalExecutorConfig;

fn main() {
    // The default runtime, set up the way earlier versions of nuclei did it.
    nuclei::init_with_config(
        GlobalExecutorConfig::default()
            .with_env_var("APP_WORKERS") // read the worker count from here first
            .with_min_threads(2)
            .with_max_threads(4),
    );
    nuclei::block_on(async {
        let started = nuclei::spawn_more_threads(2).await.expect("workers start");
        println!("started {started} more workers");
        let stopped = nuclei::stop_thread().await;
        println!("stopped one: {stopped}");
    });
}

Code that drove the proactor by hand still can: Proactor::get().wait(max_events, timeout) drives the calling thread's shard once, and wake ends every wait in progress.