Using std::io::{Read, Write, Cursor} in a nostd environment

I'm surprised there's no ticket for this, so here we go.

It isn't possible in nostd to use std::io::Cursor. That seems to be mostly because it requires std::io::Read, Write, and Seek. However, I'd argue that those traits should also be part of libcore, and only the specific implementations be in libstd. After all, the Read/Write/Seek traits and the Cursor struct don't seem to rely on anything that's part of the stdlib: they don't need allocation, etc...

My use-case is, I have to write data into an array, and treating that array in a file would lead to much more idiomatic code. Furthermore it'd allow using crates like byteorder in nostd environment.

The problem with this, which has been covered in previous issues, is that Read and Write have std::io::Error hardcoded in their method signatures which requires quite a bit of std machinery including the ability to allocate.

Do you have a link to previous issues ? I tried looking for it, but nothing came up :(.

I... can't find the previous issues. Maybe it was just scattered comments over the place and IRC discussions...

These traits being absent is the reason rust-core_io exists. Relibc would thank you if core::io became a thing, and I could then continue on my newly started quest to make more crates support no_std.

Anything here requires the std::error::Error trait, which requires std::backtrace::Backtrace, but maybe all that only requires core+alloc. Assuming so..

We should explore replacing std::io::Error with some more flexible error type, vaguely inspired by the SmallError type suggested in rust-lang/rfcs#2820 (comment) that encodes the io::error::Repr enum with some vtable pointer.

In essence, it should act like Box<dyn Error+Send+Sync+'static> except small error types get passed by value.

We could maybe preserve mem::size_of::<SmallError>() being only twice mem::size_of::<usize>() too.

Also, I'm unsure if std::error::Error could feature gate the std::backtrace::Backtrace methods on alloc, but maybe. We must encode Drop inside the errorr::Error trait though, which I suspect either requires magic similar to Box, or else makes Drop a supertrait for errorr::Error. We'd wind up with an Error trait like

pub trait Error: Drop + Debug + Display {
    fn description(&self) -> &str { "description() is deprecated; use Display" }
    fn cause(&self) -> Option<&dyn Error> { self.source() }
    fn source(&self) -> Option<&(dyn Error + 'static)> { None }
    fn type_id(&self, _: private::Internal) -> TypeId where Self: 'static { TypeId::of::<Self>() }
    #[cfg(feature = "alloc")]
    fn backtrace(&self) -> Option<&Backtrace> { None }
}

We must then use wrapper types that handle drop correctly.

#[derive(Clone)]
pub struct BoxError {
    inner: Box<dyn Error+Send+Sync+'static>,
}
// delegate Deref+DerefMut+Drop to inner

impl Drop for BoxError {
    fn drop(&mut self) {
        self.deref_mut().drop()
    }
}

use core::mem;
use core::raw::TraitObject;

const SMALL_ERROR_SIZE : usize = mem::size_of::<usize>();

#[derive(Clone)]
pub struct SmallError {
    vtable: *mut (),
    data: [u8; SMALL_ERROR_SIZE],
}

impl Drop for SmallError {
    fn drop(&mut self) {
        self.deref_mut().drop()
    }
}
impl Send for SmallError {}
impl Sync for SmallError {}

impl SmallError {
    pub fn new<E: Error+Clone+Send+Sync>(err: E) -> SmallError 
    // where mem::size_of::<E>() <= SMALL_ERROR_SIZE  // sigh
    {
        assert!(mem::size_of::<E>() <= SMALL_ERROR_SIZE);
        let data = [0u8; SMALL_ERROR_SIZE],
        core::intrinsics::copy(&err as *const E, &mut data as *mut [u8; SMALL_ERROR_SIZE] as *mut E , 1);
        let vtable = unsafe { mem::transmute::<&mut dyn Error,TraitObject>(&mut dyn err).vtable };
        // mem::forget(err);
        SmallError { vtable, data }
    }
}

impl Deref for SmallError {
    type Target = dyn Error+Send+Sync;  // Should we add +Clone when mutli-trait objects exist
    fn deref(&self) -> &Self::Target {
        let SmallError { vtable, ref data } = self;
        mem::transmute::<TraitObject,&mut dyn Error>(TraitObject { vtable, data })
    }
}

impl DerefMut for SmallError {
    fn deref(&mut self) -> &mut Self::Target {
        let SmallError { vtable, ref data } = self;
        mem::transmute::<TraitObject,&mut dyn Error>(TraitObject { vtable, data })
    }
}

We then need some outer type that calls Drop for BoxError and SmallError, but forwards Error methods, like

pub struct DerefError(&dyn Deref<Target = dyn Error+Send+Sync+'static>+DerefMut+Drop)
impl Drop for DerefError {
    fn drop(&mut self) { self.drop() }
}
impl Error for DerefError {
    // delegate all methods to self.deref()
}

Backtrace cannot really exist outside std. It is highly dependent on OS features. For instance, to get function names (symbolication step of printing a backtrace), it requires opening the executable as a file and parsing it to access the debug information table, e.g. it requires doing open(argv[0]). Or it might require dlresolve or other kind of libc stuff.

In an ideal world, binaries compiled in no_std could just have an "empty" implementation of Backtraces. I don't think that's possible however, since libstd simply reexports everything in libcore verbatim. It's not possible for libstd to use a different implementation of things than libcore AFAICT.

Unfortunately, moving backtraces to be an exposed interface of std::error::Error seems to permanently kill any possibility of std::error and std::io being available in libcore.

Could we replace &Backtrace with some opaque pub struct BacktraceRef(usize); type that exists libcore? We'd then provide inherent methods for that type only in std, assuming that flies with whatever magic.

Std cannot (currently) add inherent impls for structs it does not itself define. It plays by the same rule as the rest of the ecosystem when it comes to coherence, there's no magic going on here (modulo some escape hatches for fundamental types that don't apply here). Libstd can implement libstd traits on libcore structs, or implement libcore traits for libstd structs. But it can't add its own inherent impls for (libcore) structs, only the crate defining a type can do that.

The best I can figure out is changing the backtrace function to return a &'a dyn BacktraceTrait and do away entirely with the Backtrace structs (they'd be an implementation detail). This would prevent adding new functionality to backtraces (since they're now a trait that anyone can impl) but it means libcore's Error could return an empty backtrace impl.

Except that's also undesirable because now all the errors living in libcore won't get to use backtraces - only errors in libstd would use the "right" backtrace implementation.

Maybe that'd be fixable using the same mechanism global allocators use? Have some kind of backtrace factory, have libstd give out a default implementation, and ask libcore users to provide their own (probably stubbed) implementation.

I suppose std cannot use features for dependencies like many crates do if we want std to ever actually be the standard library on some future operating system. In other words, if features are additive then std should always have all its features enabled.

I think &dyn BacktraceTrait sounds fine because std::backtrace::Backtrace indicates it'd resemble

trait BacktraceTrait : Debug + Display {
    fn status(&self) -> BacktraceStatus;
}

I still think BacktraceRef could be defined in core, but you define BacktraceTrait in std. Also we've several proposals for extensions like (a) doing inherent methods across distinguished crate boundaries, or (b) allowing similar opaque extern types like you gets from C header files, but your &dyn BacktraceTrait works well enough here I think.

As for implementation, there are several options here:

We could treat Error special with the trait object tricks I wrote above.
We could implement those same tricks via some proc macro, and some SmallBoxDyn type, so that other trait objects could benefit form a small Copy types being passed, and do not require even alloc, while larger types get boxed, requiring alloc.
We could do some SmallBox type that handled the forwarding via compiler magic for the forwarding vtables, but also worked without always passing through trait objects.

I think anyhow::Error already provides no_std wrapper for std::io::Error. At present, it requires alloc but it looks like a reasonable starting point if we want to make this dependency optional.

We'd need to remove the Box in https://docs.rs/anyhow/1.0.26/src/anyhow/lib.rs.html#322-324 which increases the size from one to two usizes.

It appears anyhow::Error already manually implements its trait object internally, which I considered overkill but if they needed it then maybe my suggestions here also do.

We should reimplement the std::error::Error trait so that its new method does not depend upon std.

I think anyhow::Error addresses the Backtrace machinery by depending upon the backtrace crate, so maybe that already works no_std but requires alloc. We need a BacktraceTrait that avoids even needing alloc, as discussed above.

We should also remove the backtrace that anyhow::Error attaches https://docs.rs/anyhow/1.0.25/src/anyhow/error.rs.html#672 because it requires alloc. We could however permit layering one internally when using alloc via another internal wrapper type.

After all this we could then reimplement all traits from std::io and their async cousins. Appears doable although not sure until you try.

Export `std::io` instead of always using thin reimplementation

I haven't seen ciborium mentioned yet, which has its own I/O abstraction layer for no_std/no_alloc occasions. enarx/ciborium@main/ciborium-io

Relevant xkcd

We do not want to use an alternative. We want the standard implementation to be actually standard.

I haven't seen ciborium mentioned yet, which has its own I/O abstraction layer for no_std/no_alloc occasions. https://github.com/enarx/ciborium/tree/main/ciborium-io

I would very much like to drop this code in ciborium.

If we're providing evidence that core::io would be used..

ark-std exists entirely because early SNARK crates use std::io for serialization, but cryptography should ultimately work under alloc, without std.

ark-std has 234 dependents published in crates.io. Anecdotally SNARK crates are rarely published on crates.io, so searching github for ark-std language:TOML finds ark-std in 23,000 .toml files.

embedded-io has 148 dependents published in crates.io and searching github for embedded-io language:TOML finds almost 12k .toml files. I suppose embedded code rarely gets published either.

i too need this

in my case i have an embedded system, a GNU TAR. file that is encrypted not compressed :-)
i want to read it from a spi flash decrypt into a byte buffer or strip one of the files out of the tar file

or encrypt something and send it to a spi flash.. i also really would like to be able to create a vector of physical addresses for use with a dma controller

would be cool if the basic part could act like a caching buffer ie read a 20meg tar file chunk by chunk (block read/write?) pumping it through a hardware dma scheme that does encrypt/decrypt/sha hash using a dma engine

but all of that needs to be no_std

ie think a sw update image and you need to loop over all files in the tar file and extract file foobar.bin
and write it to a specific micro controller over a bitbang interface

I just stumbled into the issue that Write will be useful in no-std. I need to write data into the buffer like this:

let mut buf = [0u8; 65535];
let mut w = buf.as_mut_slice();

w.write(b"abc").unwrap();
w.write(b"def").unwrap();

This code is not possible in no-std since Write is not available. I can workaround by keep track next index to write the data but it is cumbersome.

Apologies to be yet another person chiming in. I see quite a few people discussing core::io when the next step would simply be alloc::io. Are there known blockers for that specifically, or what would be the next step forward?

The last time I looked into it, one of the main blockers for alloc::io was io::Error::last_os_error. That method needs to access an OS interface (it is in the name...) and I'm not sure if there's yet a way that can be worked around

@fintelia

@fintelia
as shown in my previous demo PR for moving io::Error to core, last_os_error can be defined in std by using #[rustc_allow_incoherent_impl]

I'm having trouble using the Write trait from no_std_io for my alloc::Vec. I'm getting unsatisfied trait bound... = help: the trait no_std_io::io::Write is not implemented for no_std_io::io::Cursor<Vec<u8>>.

It works if I enable the std feature in no_std_io, but the whole point is to use it in a no_std environment. (The alloc feature seem to make no difference for this.)

Over at https://crates.io/crates/ciborium-io, I see "If the alloc feature is enabled, we provide implementations for Vec" That looks promising, I guess I need to ditch no_std_io and go for ciborium-io.

Ah, looking into ciborium-io, I see Vec writing is for appending to the vector only. No Seek trait, no Cursor.

Ideally I'd like the Seek and Write traits over (something that wraps) alloc::Vec, so I can do random-access writes to the Vec. If std is enabled, my type should wrap any T: Seek + Write type from the standard library. Without std it would be nice to have someone else's Seek + Write to use. Looks like I have to write my own.

Try some rope type instead of Vec: https://crates.io/search?q=ropec

Inserts into a Vec would be quadratic time, and cause allocations, so it's problematic to make inserts user friendly like that.

Around this, there is a peformance footgun in that Vec::insert takes element: T instead of elements: &[T], but the latter would require T: Clone or Box<[T]> or similar.

Try some rope type instead of Vec: https://crates.io/search?q=ropec

Inserts into a Vec would be quadratic time, and cause allocations, so it's problematic to make inserts user friendly like that.

std::io::Cursor<Vec<u8>> will overwrite existing data until it reaches the end of the Vec at which point it will extend the Vec. So that never calls Vec::insert or similar so is not horribly slow.

For what it's worth, I have sent a PR (#152918) to try to move std::io::Error and std::io::IoSlice[Mut] to alloc. It it gets merged, the rest (including the main traits) should follow easily.

Navigation Menu

Repository navigation

Using std::io::{Read, Write, Cursor} in a nostd environment #48331

24 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Using std::io::{Read, Write, Cursor} in a nostd environment #48331

Description

Activity

retep998 commented on Feb 18, 2018

roblabla commented on Feb 18, 2018

retep998 commented on Feb 18, 2018

jD91mZM2 commented on Apr 21, 2019

burdges commented on Jan 3, 2020

roblabla commented on Jan 3, 2020

burdges commented on Jan 3, 2020

roblabla commented on Jan 3, 2020

burdges commented on Jan 4, 2020

burdges commented on Jan 21, 2020

24 remaining items

Altair-Bueno commented on Mar 15, 2025

npmccallum commented on Aug 6, 2025

burdges commented on Aug 6, 2025

duaneellissd commented on Aug 8, 2025

ultimaweapon commented on Aug 9, 2025

kayabaNerve commented on Sep 28, 2025

fintelia commented on Sep 29, 2025

programmerjake commented on Sep 29, 2025

alexkingnz commented on Dec 15, 2025

alexkingnz commented on Dec 15, 2025

burdges commented on Dec 15, 2025

programmerjake commented on Dec 15, 2025

a1phyr commented on Feb 20, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions