Rust: swing the tape and parsing JSON

I'll show you how to write a small program that downloads a feed ( feed ) in JSON format, parses and prints a list of notes to the console in a formatted format.

We have all resulted in a very concise code. How? Look under the cut.

Download Rust

The usual way to download Rust is to download it to your computer using rustup . Check to rustup already available in the repository of your distribution.

rustup manages toolchains of development utilities. It allows you to change the version of Rust used, manage additional development tools, such as RLS, and download development utilities for different platforms.

When you have Rust installed, type the following command:

 rustup install stable 

For this example, we need to use at least Rust version 1.20, because it requires some dependencies.

Let's see what we have.

This command has installed:

• rustc compiler
• cargo - package manager and project builder
• documentation
• some other tools.

To view the documentation in your browser, type rustup doc .

Project Setup: cargo

cargo manages Rust projects. We want to build a small executable file, so we tell cargo that we need to build a program, not a library:

 cargo init --bin readrust-cli 

This command will create a readrust-cli .

Let's see what is in this directory:

 . |-- Cargo.toml |-- src |-- main.rs 

You will notice that the project has a simple structure: it contains only the code of our program ( src/main.rs ) and ( Cargo.toml ). Let's see what is contained in Cargo.toml :

 [package] name = "readrust-cli" version = "0.1.0" authors = ["Florian Gilcher <florian.gilcher@asquera.de>"] [dependencies] 

Currently the configuration file contains some descriptive information about the project. Notice that the dependencies section is main.rs empty, and main.rs contains a small "hello world" by default.

Run:

 $ cargo run Finished dev [unoptimized + debuginfo] target(s) in 0.0 secs Running `target/debug/readrust-cli` Hello, world! 

Fine. Everything is working. cargo run itself launched the rustc compiler, compiled the program and then launched it. cargo can also detect any changes we have made to the code and recompile them.

Now let's get started!

We plan the course of our actions in advance. We want to write a utility with which you can interact via the command line interface.

On the other hand, we want to solve our problem and not do extra work.

What do we need:

 *     * HTTP-    *  JSON    *       . 

Regarding the set of features, I want to add a little flexibility, enough to get started:

• the program will accept --count , which tells the program to display the number of notes in the tape
• the program will accept --number [NUMBER] , which specifies the number of notes that the program should output.

CLAP

- stands for command line argument parser .
It was easy, wasn't it? CLAP has extensive documentation, and we will only use a little functionality.

First, we must add the clap package as a dependency.

To do this, we must specify the name and version in Cargo.toml :

 [dependencies] clap = "2.29" 

If you run cargo build , clap will be compiled with our program. In order to use CLAP, we have to specify Rust that we use the library ( crate in the terminology of Rust). We must also add the types we use to the namespace. CLAP has a very user-friendly API, which gives us the ability to configure as deeply as we need it.

 extern crate clap; use clap::App; fn main() { let app = App::new("readrust") .version("0.1") .author("Florian G. <florian.gilcher@asquera.de>") .about("Reads readrust.net") .args_from_usage("-n, --number=[NUMBER] 'Only print the NUMBER most recent posts' -c, --count 'Show the count of posts'"); let matches = app.get_matches(); } 

The next step is to compile and run the program, specifying the option --help , in order to get the instruction message:

 readrust 0.1 Florian G. <florian.gilcher@asquera.de> Reads readrust.net USAGE: readrust [FLAGS] [OPTIONS] FLAGS: -c, --count Show the count of posts -h, --help Prints help information -V, --version Prints version information OPTIONS: -n, --number <NUMBER> Only print the NUMBER most recent posts 

Good! A couple of simple lines, and here we already have a full instruction on the use of our program.

Getting the necessary information

In order to test our program, we need the necessary material.

Wrap this into a function with the following signature:

 fn get_feed() -> String { // implementation } 

One of the good HTTP clients is reqwest . There is also a hyper from the same author. Hyper is a more "low-level" library, while reqwest allows reqwest to solve "let's do it quickly" tasks.

 [dependencies] reqwest = "0.8" 

This function is implemented quite simply:

 extern crate reqwest; pub static URL: &str = "http://readrust.net/rust2018/feed.json"; fn get_feed() -> String { let client = reqwest::Client::new(); let mut request = client.get(URL); let mut resp = request.send().unwrap(); assert!(resp.status().is_success()); resp.text().unwrap() } 

This is very similar to how you would do it in other programming languages: create a client that allows us to make requests.

Calling send() , we make a request and get an answer.
Calling text() on the response, we get it as a string.

Pay attention to the word mut . In Rust, mutable variables are declared this way. By sending a request and receiving a response, we change the internal state
of the request object, so it must be mutable.

Finally, unwrap and assert .
Sending a request or reading a response is an operation that can fail, for example, the connection is broken.

Therefore, send (sends the request) and text ("reads" the response) returns a Result object.

Rust expects us to analyze the contents of the returned object and
take the necessary action. unwrap leads to a panic ( panic ) - the program is terminated, but before that it “cleans up” the used memory behind it.

If there was no error, we get the required value. The request may be successful in the sense that the server responded, but the HTTP return code is not 200 SUCCESS (Internal Server Error?).

assert prevents us from reading the contents of the response from a request that failed.

In many scripting languages ​​in this place, we would get an unhandled exception ( exception ), which leads to a similar effect.

There are no exceptions in Rust - instead we use ADT (like Maybe in Haskell).

Do not be afraid to use unwrap often during your training.

You will learn how to use other tools later.

JSON parsing: we connect serde

Now we need to parse the JSON tape.
To do this, Rust has a serde library for (se / des) implementation.
serde supports not only JSON, but also other formats.
serde also provides convenient ways to
job serializable types, the so-called derive 's.

For this reason, we need to add the following 3 dependencies:
serde , serde_derive , serde_json .

 [dependencies] serde = "1.0" serde_derive = "1.0" serde_json = "1.0" 

serde_json gives you the opportunity to handle JSON in two ways: to parse a string in a JSON tree or to inform the Serde structure of the expected information.
The second method is much faster and more convenient.
Looking at the definition of the feed , we see that there are 3 main types:

• the authors
• items
• tape

Ribbon and elements have the author.

Make changes to the code:

 extern crate serde; #[macro_use] extern crate serde_derive; extern crate serde_json; #[derive(Debug, Deserialize, Serialize)] struct Author { name: String, url: String, } 

In order to make the example more visual, we presented the URL as a regular string, but in the future we can change this. We also define a simple data structure with two fields. The names of these fields match the corresponding fields in JSON. The most interesting is hidden in the line with derive .

It tells the compiler to generate additional code based on this structure.

• Debug generates a debug representation in the form of lines, which is often useful.
• Serialize / Deserialize generates code to serialize our data structure. This is still not tied to the JSON format that we plan to use with you.

Create a structure to represent the element in the ribbon:

 #[derive(Debug, Deserialize, Serialize)] struct FeedItem { id: String, title: String, content_text: String, url: String, date_published: String, author: Author, } 

This is similar to the structures that we have already set. You can see that we used composition to include the field type Author .

We called our type FeedItem , because it more eloquently indicates why this type is needed.

Let's see how our tape will look like:

 #[derive(Debug, Deserialize, Serialize)] struct Feed { version: String, title: String, home_page_url: String, feed_url: String, description: String, author: Author, items: Vec<FeedItem>, } 

There is nothing new here, except that we have included the items field, which is a vector that includes ribbon elements.

Vec is a standard type in Rust to represent a list of something. It can contain any set of objects of the same type.

For those who are used to generic `s in other languages, this designation is already familiar.

Let get_feed return a Feed instead of a String :

 fn get_feed() -> Feed { let client = reqwest::Client::new(); let mut request = client.get(URL); let mut resp = request.send().unwrap(); assert!(resp.status().is_success()); let json = resp.text().unwrap(); serde_json::from_str(&json).unwrap() } 

There are only two things left to add: we assign the returned text to the json variable and call the function to parse the variable.

Since the parsing may fail, the program may return a Result containing an erroneous value. If the function succeeds, in order to retrieve the value you need to call unwrap .

By the way we changed the return type in the get_feed function, Rust found out that we want to convert JSON text into a variable of type Feed .

If json does not contain the correct ( valid ) JSON, then the program will end with an error, so if readrust.net changes the tape coding format, we will immediately notice it.

Count

We are close to completion, but have not yet written the code for displaying the result to the user.
Correct - let's teach our program to show the user the number of elements in the tape.

 fn print_count(feed: &Feed) { println!("Number of posts: {}", feed.items.len()); } 

Look at & : Rust is a system programming language that provides two ways to pass arguments:

• transfer of ownership
  ( pass ownership , owned )
• Link transfer - borrowing ( borrow , borrowed ).

Ownership - this means that the calling code will lose access to the transmitted object (will transfer ownership of the object). With the values ​​you own, you can do everything: destroy them, ignore them, use them.

Borrowing - this means that you can only "look" at the object, after which you will have to return the object to its owner. The owner can give or not give you permission to change the object. Since now we don’t need to change the object, we borrow it via an immutable link.

Here is how it looks in main :

 let matches = app.get_matches(); let feed = get_feed(); if matches.is_present("count") { print_count(&feed); } 

gen_matches determined which arguments were passed to the program.

The is_present call allows us to know if the user passed the argument --count . Notice that we must use & and here to tell the compiler that we want to pass an object by reference.

Run:

 [ skade readrust-cli ] cargo run -- --count Compiling readrust v0.1.0 (file:///Users/skade/Code/rust/readrust-cli) Finished dev [unoptimized + debuginfo] target(s) in 2.46 secs Running `target/debug/readrust --count` Number of posts: 82 

Formatted output

Now we have to teach the program to display the results on the screen. I decided to print the table using the prettytable library prettytable

 [dependencies] prettytable-rs = "0.6" 

Let's look at one of the examples of using the library and adapt it to our case:

 #[macro_use] extern crate prettytable; fn print_feed_table<'feed, I: Iterator<Item = &'feed FeedItem>>(items: I) { let mut table = prettytable::Table::new(); table.add_row(row!["Title", "Author", "Link"]); for item in items { let title = if item.title.len() >= 50 { &item.title[0..49] } else { &item.title }; table.add_row(row![title, item.author.name, item.url]); } table.printstd(); } 

Here are a few points that are worth paying attention to:

• the loop is used to "run through" the elements and
  bring them out
• some authors like to take notes with headlines,
  containing more than 50 characters, we shorten them

Let's look at if , which in Rust is an expression that returns a value .
This means that we can assign the result of the if calculation to the title variable. If you look at the two possible execution branches, you will again see the & characters — this is called a “slice”. If the title is too long, then we take the link to the first 50 characters, so we don’t have to copy it. The similarity with & to denote borrowing is not accidental: we borrow a slice.

This leads to the fact that the signature looks like this:
fn print_feed_table<'feed, I: Iterator<Item = &'feed FeedItem>>(items: I)

Functions can work with generics and I decided to add them to the print_feed_table implementation. This function accepts an object that implements Iterator and provides us with borrowed elements.

Entities that Iterator gives us are called Item - a type parameter, in this case FeedItem . Finally, there is a 'feed .

Rust checks that all links point to something: what they point to should exist.

This semantics is expressed in function signatures. In order to return references to elements, we must make sure that these elements are in memory. Roughly speaking, <'feed, I: Iterator<Item = &'feed FeedItem>> means that there is some entity outside the function that exists for a time of 'feed

This entity provides us with the items we are borrowing. We get an iterator I, which "runs" through the elements, giving us elements to borrow. The lifetime ( lifetime ) expresses these ratios.

It looks like this:

  if matches.is_present("count") { print_count(&feed); } else { let iter = feed.items.iter(); if let Some(string) = matches.value_of("number") { let number = string.parse().unwrap(); print_feed_table(iter.take(number)) } else { print_feed_table(iter) } } 

Here we see the reason why I chose this particular implementation. In order to enable support for the --number option, I decided to use Iterator.

If the user provides a number, I convert it to a string (this, of course, can fail if a random string is passed).

After I convert the set of remaining elements into a Take iterator. Take returns a certain number of elements from the source iterator and then completes its execution.

All is ready! Source code you can find here.

What to do next?

We have written a program that you can expand.

For example, you can try the following:

• calculate the statistics of notes in the tape
• select only those notes whose headings match the regular expression
• improve error handling with more verbose messages
• present dates with variables of the appropriate type, not strings ( String )
• follow the links and get the contents of the notes.

Total

We received a program in which we check the data for errors, completing
program after processing them.

JSON data is parsed safely, detecting for errors.

• The program copes well with unexpected situations. For example, using iteration instead of indexed access to an array, we are protected from accessing an element outside the array => no segfault.

  
  

• The program is written using well-documented libraries, with which you can quickly move on to writing your own code. The situation with the library in Rust is now much better than it was, say, 2 years ago, we already have many mature libraries.

  
  
• The program is portable ( cross-platform ) => you can share the program by simply copying the executable file.

Here is the complete source code:

 extern crate clap; #[macro_use] extern crate prettytable; extern crate reqwest; extern crate serde; #[macro_use] extern crate serde_derive; extern crate serde_json; use clap::App; pub static URL: &str = "http://readrust.net/rust2018/feed.json"; #[derive(Debug, Deserialize, Serialize)] struct Author { name: String, url: String, } #[derive(Debug, Deserialize, Serialize)] struct FeedItem { id: String, title: String, content_text: String, url: String, date_published: String, author: Author, } #[derive(Debug, Deserialize, Serialize)] struct Feed { version: String, title: String, home_page_url: String, feed_url: String, description: String, author: Author, items: Vec<FeedItem>, } fn print_count(feed: &Feed) { println!("Number of posts: {}", feed.items.len()); } fn print_feed_table<'feed, I: Iterator<Item=&'feed FeedItem>>(items: I) { let mut table = prettytable::Table::new(); table.add_row(row!["Title", "Author", "Link"]); for item in items { let title = if item.title.len() >= 50 { &item.title[0..49] } else { &item.title }; table.add_row(row![title, item.author.name, item.url]); } table.printstd(); } fn get_feed() -> Feed { let client = reqwest::Client::new(); let mut request = client.get(URL); let mut resp = request.send().unwrap(); assert!(resp.status().is_success()); let json = resp.text().unwrap(); serde_json::from_str(&json).unwrap() } fn main() { let app = App::new("readrust") .version("0.1") .author("Florian G. <florian.gilcher@asquera.de>") .about("Reads readrust.net") .args_from_usage( "-n, --number=[NUMBER] 'Only print the NUMBER most recent posts' -c, --count 'Show the count of posts'", ); let matches = app.get_matches(); let feed = get_feed(); if matches.is_present("count") { print_count(&feed); } else { let iter = feed.items.iter(); if let Some(string) = matches.value_of("number") { let number = string.parse().unwrap(); print_feed_table(iter.take(number)) } else { print_feed_table(iter) } } } 

Many thanks to everyone from the Rustycrate community who participated in the translation, proofreading and editing of this article. Namely: born2lose, vitvakatu.

UPDATE : added full source code.