A little story about the `yes` command in Unix

How do you know the simplest Unix command? There is an echo that prints a string to stdout, and there is a true that does nothing but just ends with zero code.

Among the many simple Unix commands, the command yes hidden. If you run it without arguments, you will get an infinite stream of characters "y", each from a new line:

 y y y y (...   ) 

Although at first glance the team seems pointless, but sometimes it is useful:
')

 yes | sh boring_installation.sh 

Ever installed a program that requires you to enter "y" and press Enter to install? The yes command comes to the rescue! She will carry out this task carefully, so you can stay in touch with the Pootie Tang .

Write yes

Here is the basic version on ... hm ... BASIC.

 10 PRINT "y" 20 GOTO 10 

And here is the same thing in Python:

 while True: print("y") 

Seems simple? Wait a minute

As it turns out, such a program is rather slow.

 python yes.py | pv -r > /dev/null [4.17MiB/s] 

Compare with the built-in version on my Mac:

 yes | pv -r > /dev/null [34.2MiB/s] 

So I tried to write a faster version on Rust. Here is my first attempt:

 use std::env; fn main() { let expletive = env::args().nth(1).unwrap_or("y".into()); loop { println!("{}", expletive); } } 

Some explanations:

• The line we are typing in a loop is the first command line parameter called expletive . This word I learned from the manual yes .
• I use unwrap_or to get expletive from parameters. If no parameters are set, the default is "y".
• The default parameter is converted from a string fragment ( &str ) to owned() on a heap ( String ) using into() .

We are testing.

 cargo run --release | pv -r > /dev/null Compiling yes v0.1.0 Finished release [optimized] target(s) in 1.0 secs Running `target/release/yes` [2.35MiB/s] 

Oops, nothing really improved. It is even slower than the Python version! This interested me, so I searched for the source code for implementation on C.

Here is the very first version of the program , which was released as part of Version 7 Unix for the honorary authorship of Ken Thompson January 10, 1979:

 main(argc, argv) char **argv; { for (;;) printf("%s\n", argc>1? argv[1]: "y"); } 

No magic

Compare with the 128-line version of the GNU coreutils kit, a mirror of which is on Github . After 25 years, the program is still in active development! The last code change happened about a year ago. She's pretty quick:

 # brew install coreutils gyes | pv -r > /dev/null [854MiB/s] 

The important part is at the end:

 /* Repeatedly output the buffer until there is a write error; then fail. */ while (full_write (STDOUT_FILENO, buf, bufused) == bufused) continue; 

Aha So here a buffer is simply used to speed up write operations. The buffer size is set to a constant BUFSIZ , which is selected for each system in order to maximally optimize I / O operations (see here ). On my system, it was installed as 1024 bytes. In reality, the best performance was at 8192 bytes.

I expanded my Rust program:

 use std::io::{self, Write}; const BUFSIZE: usize = 8192; fn main() { let expletive = env::args().nth(1).unwrap_or("y".into()); let mut writer = BufWriter::with_capacity(BUFSIZE, io::stdout()); loop { writeln!(writer, "{}", expletive).unwrap(); } } 

Here it is important that the buffer size is divided by four, this ensures alignment in memory .

This program gives 51.3 MiB / s. Faster than the version installed on my system, but much slower than the version from the author of the post I found on Reddit . He says he achieved a speed of 10.2 GiB / s.

Addition

As usual, the Rust community did not disappoint. As soon as this article got into the sub-section about Rust , user nwydo pointed to the previous discussion on this topic. Here is their optimized code that breaks 3 GB / s on my machine:

 use std::env; use std::io::{self, Write}; use std::process; use std::borrow::Cow; use std::ffi::OsString; pub const BUFFER_CAPACITY: usize = 64 * 1024; pub fn to_bytes(os_str: OsString) -> Vec<u8> { use std::os::unix::ffi::OsStringExt; os_str.into_vec() } fn fill_up_buffer<'a>(buffer: &'a mut [u8], output: &'a [u8]) -> &'a [u8] { if output.len() > buffer.len() / 2 { return output; } let mut buffer_size = output.len(); buffer[..buffer_size].clone_from_slice(output); while buffer_size < buffer.len() / 2 { let (left, right) = buffer.split_at_mut(buffer_size); right[..buffer_size].clone_from_slice(left); buffer_size *= 2; } &buffer[..buffer_size] } fn write(output: &[u8]) { let stdout = io::stdout(); let mut locked = stdout.lock(); let mut buffer = [0u8; BUFFER_CAPACITY]; let filled = fill_up_buffer(&mut buffer, output); while locked.write_all(filled).is_ok() {} } fn main() { write(&env::args_os().nth(1).map(to_bytes).map_or( Cow::Borrowed( &b"y\n"[..], ), |mut arg| { arg.push(b'\n'); Cow::Owned(arg) }, )); process::exit(1); } 

So it's a completely different thing!

• We have prepared a filled string buffer, which will be reused in each loop.
• The standard output stream (stdout) is protected by blocking . So instead of continuous capture and release, we keep it all the time.
• We use the platform native std::ffi::OsString and std::borrow::Cow to avoid unnecessary memory locations.

The only thing I can add is mut .

Lessons learned

The trivial yes program was actually not so simple. It uses output buffering and memory alignment to improve performance.

Recycling standard Unix tools is a fascinating experience, and it makes you appreciate the elegant tricks that make our computers fast.