Skip to content

Expose the spans that widths are additive over #94

Open

Description

@Aetf

Expose the spans that widths are additive over

Hi, and thanks for maintaining this crate — unicode-truncate has been built on it from the start, and the contextual width rules in 0.2 were a clear correctness win.

They also surfaced a gap I don't think a downstream crate can close on its own, so I'd like to propose an addition. Feel free to push back on any of it.

unicode-truncate cuts strings to a display width budget. Anything that cuts text needs grapheme clusters as the unit, and therefore sums cluster.width(). That sum is not str::width():

let lam_alef = "\u{0644}\u{0627}";                                  // لا
assert_eq!(lam_alef.width(), 1);
assert_eq!(lam_alef.graphemes(true).map(|g| g.width()).sum::<usize>(), 2);

The contextual rules documented in the README are exactly the cases where the two differ, and each spans a cluster boundary, so no per-cluster width assignment can close the gap.

This is not a rounding error downstream (Aetf/unicode-truncate#40): unicode-truncate 3.0.0 mixed the two models and could return a slice wider than the requested budget ("لا".repeat(8) centered to 1 column returned a 5-column slice), and its padding came up short of the target width. 3.1.0 fixes that by scoring candidates consistently in the per-cluster sum model and measuring the returned slice, which is the best a caller can do today: never over budget, honestly reported, but conservative — on strings containing these ligatures it keeps less than would fit, and centering drifts.

Request

An iterator over the spans widths are additive over:

/// Contiguous byte ranges covering the string, each boundary a grapheme cluster boundary.
fn width_spans(&self) -> impl Iterator<Item = (Range<usize>, usize)>;

with the guarantee that for every span boundary i, s[..i].width() + s[i..].width() == s.width().

That is the whole ask: the positions where cutting does not change what the pieces measure. Truncation, padding, wrapping and cursor placement all become exact on top of it. In shaping terms these are the safe-to-break positions (the complement of HarfBuzz's HB_GLYPH_FLAG_UNSAFE_TO_BREAK), a correctness constraint, unlike UAX #14's break opportunities.

This is deliberately not the truncation API of #84: no position on what to keep, no claim that cutting text is meaningful in general — only surfacing what the crate already computes, since the spans are where WidthInfo carries state across a cluster boundary. I'd rather not reimplement that detection downstream: it would mean copying the tables here and inevitably drifting out of sync with them.

On direction: str_width folds right to left and the Arabic rule allows unbounded transparent characters between lam and alef, so a forward iterator would need unbounded lookahead. An iterator that yields spans back to front is fine for my use.

No attachment to the exact shape — a different signature achieving the same guarantee works just as well for me. Happy to implement it and send a PR if this direction seems reasonable to you.

Activity

  1. Manishearth commented on Aug 24, 2026

    @Manishearth
    Member

    Have you gone through the later proposal in #84 to expose the width_in_str numbers? That is not exactly what you're asking for but it might be helpful.

    I think what you are asking would be somewhat complicated to maintain and I am wary of maintaining it.

    (Also, I would prefer to not converse with an LLM on an issue: feel free to use LLM output in <details> blocks, but I would like to converse with humans)

  2. Aetf commented on Sep 5, 2026

    @Aetf
    Author

    Hi @Manishearth, sorry for the late reply. First re LLM, yes I do use one to help me phrase things, since I'm not a native English speaker, but the content is mine and not unsupervised output :)

    width_in_str is almost exactly what I need, provided there is also some way to tell when the carried state is back to WidthInfo::DEFAULT during the str_width rfold. That is the boundary I'm after: if the state after consuming s[i..] is DEFAULT, then continuing over s[..i] is the same computation as s[..i].width() from scratch, so s[..i].width() + s[i..].width() == s.width() holds at i.

    What I ultimately need is the set of all such positions, intersected with grapheme boundaries. The iterator in my original post is basically that; a backwards iterator of (width, is_default) per char, or an opaque public WidthInfo with an is_default(), would let me build it downstream without touching any tables.

    The widths alone aren't enough: a lam that contributed 0 because of a following alef and a genuinely zero-width mark look identical in the delta stream, but the position right after the lam (between it and the alef) is exactly where I must not cut, while cutting after the mark is fine.

  3. Manishearth commented on Sep 8, 2026

    @Manishearth
    Member

    Yeah, I'm not sure if exposing that would be easy to maintain, unfortunately. I'm still open to width_in_str itself. Truncation is a complicated concept when it comes to Unicode.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions