Skip to content

fix: return an error when the tty is gone - #1067

Open
wtn wants to merge 2 commits into
crossterm-rs:masterfrom
wtn:fix-dead-tty-busy-loop
Open

fix: return an error when the tty is gone#1067
wtn wants to merge 2 commits into
crossterm-rs:masterfrom
wtn:fix-dead-tty-busy-loop

Conversation

@wtn

@wtn wtn commented Jul 10, 2026

Copy link
Copy Markdown

I left atuin search -i (a ratatui app) open in a terminal tab, and after the session went away the process pegged a CPU core. A spindump put all 1001 samples inside cursor::position(), 975 of them in back-to-back select(2) and read(2) calls with no sleeping.

A dead tty stays permanently readable while read returns 0 or fails, and three loops respond by retrying immediately:

  • read_position_raw swallows poll errors (Err(_) => {}) and retries with no delay.
  • The use-dev-tty event source conflates EOF with WouldBlock, so try_read re-polls a permanently readable descriptor until its timeout expires, or forever when it has no timeout.
  • The mio event source's read loop never exits on Ok(0) and swallows read errors other than WouldBlock/Interrupted.

I propose to report the dead tty as an error: read_position_raw propagates poll errors, and both unix event sources return UnexpectedEof at EOF (the mio source also propagates other read errors). This changes behavior for event::poll/event::read on a vanished terminal: instead of timing out, hanging, or spinning, they fail with an error the caller can act on.

This also affects piped or redirected input that reaches EOF, not just a vanished terminal: event::poll/event::read now return UnexpectedEof there too instead of timing out. It's the same permanent-readable-at-EOF condition, so the same fix applies.

@wtn
wtn force-pushed the fix-dead-tty-busy-loop branch 2 times, most recently from f8ee30c to 8971b97 Compare July 10, 2026 05:11
@wtn
wtn marked this pull request as ready for review July 10, 2026 05:57
@wtn
wtn requested a review from TimonPost as a code owner July 10, 2026 05:57
@abbyssoul

Copy link
Copy Markdown

Independent corroboration of this, on Linux — the mio path specifically. Your description of that loop matches what I landed on exactly, so mostly I want to add a second platform and one detail about the impact that I think is sharper than "high CPU".

Repro: crossterm 0.29.0, ratatui TUI in a tmux pane on Linux 6.17, killed with tmux kill-session. The pty hangs up under the app and it goes to 100% CPU, permanently, in event::poll.

The reason my app was still alive to hit this is its own fault, not crossterm's: it installs a SIGHUP handler for config reload, which replaces SIGHUP's default action of terminating. So it survived losing its terminal. That's a downstream bug and I've fixed it on my side. But it's also exactly the shape of program that reaches this loop — anything with a SIGHUP handler, anything under sudo (as in #793), anything double-forked. The app doesn't have to do anything unusual with crossterm to get here, it just has to outlive its tty.

The detail I'd add: poll's timeout is not a safety net here, and I think that's worth calling out. The timeout check is in the outer loop (mio.rs:149), but the read loop at mio.rs:95 never breaks out to reach it — on Ok(0) the read_count > 0 guard skips the body and re-reads; on a non-WouldBlock/Interrupted error the Err arm falls through and re-reads. So event::poll(Duration::from_millis(120)) doesn't return late or spuriously, it never returns. Control never comes back to the caller.

That's what made this hard to find from the outside, and it's why a caller can't defend against it: I assumed my terminal.draw()? would eventually fail on the dead tty and propagate, but it never got another turn to run. The busy loop isn't just wasted cycles — it's a permanently stuck thread inside a call that has a timeout on it.

Still reproduces on master as of today; the Err arm has no break for anything but WouldBlock/Interrupted, and Ok(0) still falls through the guard. Returning UnexpectedEof is the right call as far as I'm concerned — it gives the caller back the control it needs to shut down. Would be good to see this land.

Probably also fixes #793, which has been open since 2023 with the same stack trace.

@roazanas

Copy link
Copy Markdown

Thanks for this — I hit the same loop in a downstream consumer (Codex CLI pins a crossterm fork) and tested your branch. The try_read fix works, but on macOS the async EventStream path still never surfaces the error, so event-stream consumers trade a busy loop for a silent stall.

What happens after this patch

The background wake thread runs:

if let Ok(true) = internal::poll(None, &EventFilter) {
    break;
}

It observes the new error exactly once and discards it — Err matches no arm, so it loops straight back into poll(None).

Foreground polls cannot recover it, for two independent reasons:

  1. internal::poll(None, …) takes the global EVENT_READER lock via lock_event_reader() and holds it for the whole blocking kevent(). poll_next uses internal::poll(Some(Duration::ZERO), …), which goes through try_lock_event_reader_for and hits return Ok(false) when the lock is unavailable — it never looks at the reader at all.
  2. mio registers the descriptor with EV_CLEAR (sys/unix/selector/kqueue.rs), so the readiness notification is reset once it has been retrieved by kevent(). Even with the lock free, the next kevent() has no new transition to report.

Measurements

macOS 26.5.2 (Darwin 25.5.0, arm64), tokio current-thread consumer, pty torn down by closing the last master fd, SIGHUP ignored in the child so the process survives teardown (test isolation, so signal delivery is not a variable):

build CPU before → after teardown process state Err delivered to consumer
crossterm 0.29.0 (control) 0% → 101% Rs+ running no
this branch (8971b97) 0% → 0% Ss+ sleeping no
branch + Err(_) => break in the wake loop 0% → 0% Ss+ sleeping no
branch + store the error, return it from poll_next 0% → 0% exited yes (UnexpectedEof)

So the patch does remove the CPU burn — that part works. But the stream stays pending indefinitely unless some other event or shutdown path intervenes.

To check whether an active executor recovers on its own, I instrumented poll_next on your branch, kept an independent 500 ms interval running, and wrote the harness's teardown moment into the same log so pre- and post-teardown polls can be separated exactly. In the 5 s window after the pty hangup: 14 poll_next calls, all 14 returning Ok(false), none returning Err. A longer 16 s run stayed in the same state.

Breaking the wake loop on Err is not enough

That is row 3 above. By the time the woken foreground poll runs, the error has been dropped and the readiness notification has already been cleared, so it gets Ok(false), re-arms the wake task, and returns Pending again.

What did work (row 4) was preserving the terminal condition across the two threads: store it in shared state, wake the executor, and have poll_next return it ahead of its own poll. Storing the io::Error itself is only one option — a Result channel, or a terminal-state flag that regenerates the error, would do as well; the necessary part is that the fact of the error survives the hand-off. My version was a throwaway prototype to test the hypothesis, not production-shaped: I did not consider multiple concurrent EventStream instances, or a single non-permanent error.

Happy to open a follow-up PR if that is useful.

Worth noting the current tests cannot catch this: they exercise UnixInternalEventSource::try_read directly via UnixStream::pair(), and this failure only appears through the async EventStream wrapper. An event-stream regression test that tears down a pty and asserts the stream yields Err would cover it.

I have not tested Linux. mio uses EPOLLET there too, but whether a hung-up pty re-reports EPOLLHUP is a separate question.

@roazanas

roazanas commented Jul 30, 2026

Copy link
Copy Markdown

Following up on my analysis above — I went ahead and implemented the EventStream half and validated it end to end, so here it is in case you'd like to fold it into this PR rather than have a second one competing with it.

be6cfec, branched directly off 8971b97: branch eventstream-dead-tty-handoff.

What it does. The background wake task no longer discards the error. It stores it in a per-EventStream state machine (Active → Error(io::Error) → Terminated), so poll_next yields the error once and then ends the stream. That closes the gap I described: without it, the error is observed once by the wake thread and thrown away, and the foreground zero-timeout poll can never recover it — the EVENT_READER lock is held during the blocking kevent(), and EV_CLEAR has already consumed the readiness notification.

Also included: tests/event_stream_dead_tty.rs (fails by timeout on 8971b97, passes in ~100 ms with the change) and a CHANGELOG line.

On the test needing a real pty. A socketpair dup2'd onto fd 0 is not enough — tty_fd() only uses stdin when isatty(0), so the source never picks it up. It uses rustix-openpty as a dev-dependency; rustix itself is already a dependency, so only that one is new. Happy to switch it to libc::openpty if you would rather not add a dev-dependency, since libc is already there.

On keeping the state per-instance. Deliberate. I also built and measured a process-global terminal condition, and it is worse for consumers that recreate the stream on error: every freshly constructed stream errors immediately, and since EventStream::default() spawns its wake thread unconditionally, that turns into a thread-churn loop. In a real consumer it went from a 1-core spin to ~4 cores. Reopen policy belongs to the consumer, not the library.

Verified in a real consumer. Codex CLI (which pins a crossterm fork) goes from burning a full core after a pty hangup to exiting cleanly — measured with pty.fork(), SIGHUP ignored in the child so only the application's own EOF path can end it, and teardown by closing the last master fd:

build CPU after teardown outcome
crossterm 0.29.0 as released ~101% of a core busy loop in try_read
this PR alone ~0% no spin, but the consumer never learns
this PR + be6cfec ~0% error delivered; its main screen exits, but its own re-arm logic still stalls the onboarding screen
this PR + be6cfec + the consumer-side fix ~0% exits in ~0.1 s, +0.00 s CPU, on both screens

It needed a change on its own side too — its event broker classified EOF as "restartable" and re-armed a fresh stream — which is exactly why I think the library should report the condition and stop there.

Cherry-pick it, take the diff, or ignore it entirely — whichever is least friction for you. I have not opened a competing PR and don't intend to.

wtn and others added 2 commits July 30, 2026 23:25
The previous commit makes the unix event sources report a dead tty as an
error, but the async `EventStream` still drops it. The background wake
task runs `if let Ok(true) = internal::poll(None, ..) { break }`, so an
`Err` matches no arm and it loops straight back into `poll(None)`.

The foreground poll cannot recover it either: `poll_next` uses a
zero-timeout `internal::poll`, which returns `Ok(false)` whenever the
global `EVENT_READER` lock is held by the blocking wake task, and mio
registers the descriptor with `EV_CLEAR`, so the readiness notification
has already been consumed. `event-stream` consumers therefore traded a
busy loop for a silent stall.

Carry the condition across the hand-off instead: the wake task stores the
error in a per-`EventStream` state machine (Active -> Error -> Terminated),
and `poll_next` yields it once before ending the stream.

The state is deliberately per-instance rather than process-global. A
global terminal condition makes every freshly constructed stream error
immediately, and since `EventStream::default()` spawns its wake thread
unconditionally, that turns a one-core spin into thread churn for
consumers that recreate the stream on error. Reopen policy belongs to the
consumer.

The test needs a real pty: `tty_fd()` only uses stdin when `isatty(0)`,
so a socketpair dup2'd onto fd 0 is never picked up. It uses
`rustix-openpty`, matching the non-optional `rustix` dependency, so the
test runs in the default configuration rather than only when the optional
`libc` feature is enabled.

Co-authored-by: Nikita <me@rznz.ru>
@wtn
wtn force-pushed the fix-dead-tty-busy-loop branch from 8971b97 to c006ee6 Compare July 31, 2026 04:40
@wtn

wtn commented Jul 31, 2026

Copy link
Copy Markdown
Author

@roazanas Rebased and merged yours in.

@oborchers

Copy link
Copy Markdown

Proqi maintainer here. This change would materially help our terminal lifecycle architecture.

Proqi currently uses an outer supervisor and nested Crossterm reader because event::poll(timeout) can remain trapped after terminal loss. Since the underlying reader may not return, we cannot truthfully join and destroy the input worker within our bounded shutdown deadline.

Returning EOF and nonretryable terminal errors would let us classify terminal revocation and perform owned cleanup instead of detaching a potentially live reader. We track the downstream work in oborchers/proqi#52 and have deterministic PTY coverage for macOS and Linux ready to qualify a published release.

For accuracy, this PR addresses a major part of our requirement, but we will also verify bounded return during continuous input, unfinished escape or bracketed-paste sequences, and resize handling before removing the supervisor.

Thank you for working on this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants