Part 2. Generators

In the introductory article of the series, I hope you have already managed to get acquainted with the generators. In this tutorial we will consolidate the knowledge gained, learn to write our own (including recursive) generators. Although he is dedicated to generators, we will not forget about the properties either. Moreover, we will actively use them, demonstrating the full power of the generator mechanism. Consider the mechanism of preconditions. Perhaps it would be more logical to devote the second article of the series to the properties, and perhaps this would be the right decision. However, according to my personal observations, it is the generators that cause the greatest difficulties. Properties we consider in the next article.

Cycle structure

• Introduction
• Generators
• Properties
• Minimization and state testing
• Integration and Settings

Generators

The forAll method from the Prop object uses data generated in an arbitrary way. It can be both simple data types and container ( compound ). In ScalaCheck, any generator is Gen’s successor:

 sealed abstract class Gen[+T] { def apply(prms: Gen.Params): Option[T] ... } 

The class is parametric by the type of value generated. Accordingly, for strings we will have Gen[String] , and for notorious Person - Gen[Person] . You can also use the arbitrary method from an Arbitrary object, provided that the necessary vises exist in the scope.

Add implicit support to our generators

Since we are talking about implications, let's figure out how to make an arbitrary
for your type. First, we need the type itself:

 sealed trait LED case object Red extends LED case object Green extends LED case object Blue extends LED 

Now let's create a generator for our diodes:

 val ledGenerator: Gen[LED] = Gen.oneOf(Red, Green, Blue) 

And now let's create an implicit Arbitrary instance from the generator we have:

 implicit val arbitraryLed: Arbitrary[LED] = Arbitrary(ledGenerator) Arbitrary.arbitrary[LED].sample // Some(Green) 

Now, having LEDs in our scope, we can check some properties:

 val ledProp = forAll { diode: LED => //    } 

For a number of primitive types, as well as containers from the standard library, implicit conversions are already predefined inside the Arbitrary object:

 import org.scalacheck.Arbitrary.arbitrary val arbitraryString = arbitrary[String] val arbitraryInteger = arbitrary[Int] 

Gen.resultOf can greatly help you in this matter. This function accepts a case-class, for the parameters of the constructor of which arbitrary values ​​are already defined:

 case class Coord(x: Double, y: Double) //     resultOf. val genCoord = Gen.resultOf(Coord) //    Arbitrary. implicit val arbitraryCoord = Arbitrary(genCoord) //         . val prop = Prop.forAll { coord: Coord => //... } 

sample method

Unfortunately, the toString method is not defined for instances of the Gen class, but there is a method that is better and more convenient. You even have to use it very often when writing generators. This is the Gen.sample method. It returns an example of the generated value inside Option :

 // ScalaCheck     ,   //  10 . Arbitrary.arbitrary[String].sample map (_ take 10) //     . // Some(Ｑ冋执龭轙⮻拍箘㖕) Arbitrary.arbitrary[Double].sample // Some(-5.180668081211655E245) 

Transformations

As already mentioned in the previous section, the map method can also be applied to generators. We illustrate this with the following example:

 val octDigitStr = choose(0, 7) map (_.toString) octDigitStr.sample // Some(3) octDigitStr.sample // Some(5) 

Filtration

For generators, there is also a filter method, although, in fact, this is just an alias for calling suchThat method:

 val evenOct = octDigit.filter(_ % 2 == 0) evenOct.sample // None //  ,   : evenOct.sample // None //  ,    ? evenOct.sample // Some(2) 

The example above demonstrates why you should not use filter or suchThat . The new generator simply rejects values ​​that do not pass the filter. This greatly slows the testing process.

Preconditions act like filters.

Preconditions

The preconditions represented in logic by the operation of implication in ScalaCheck are the methods used inside properties. They allow you to limit the input data. Recorded by the operator ==> . In order to take advantage of the preconditions, you need to add Prop.propBoolean . So let's check the statement that for any even number, the last digit will also be an even number:

 import org.scalacheck.Prop.{forAll, propBoolean} def isEven(number: Int) = number % 2 == 0 //    . def lastChar(number: Int) = number.toString.last.toInt // . val p2 = forAll(arbitrary[Int] suchThat isEven) { number => isEven(lastChar(number)) } //  . val p1 = forAll(arbitrary[Int]) { number => isEven(number) ==> isEven(lastChar(number)) } 

To run a property, we can use the check function:

 //     p1.check // + OK, passed 100 tests. //   p2.check // + OK, passed 100 tests. 

The harder the filter, the more values ​​will be ignored. This can be critical for a small set of values ​​or for values ​​that are very far apart. For example, prime numbers on the Int range. From a similar venture ScalaCheck can easily get tired and surrender to you:

 scala> p1.check ! Gave up after only 1 passed tests. 100 tests were discarded 

You will not notice this in the example with even numbers, and in the example with simple numbers - it is easy. You should also know that:

The filter for even numbers can be implemented in different ways. For example, you can take
any integer and just multiply it by 2. In the case of prime numbers
it will be much easier to calculate them in advance, using, for example,
sieve Eratosthenes and then push the list
computed values ​​inside Gen.oneOf , which we will look at later in this
article.

If you combine generators using suchThat , you may also have problems. In such cases, you should use the retryUtil method. Unlike suchThat , it makes ScalaCheck work harder, which can end in an endless cycle. Called retrtyUtil similar to fiter or suchThat :

 arbitrary[Int] retryUntil isEven 

You may also notice that the collect and flatten , as well as fromOption , are declared Deprecated . This is done so that you have less temptation to produce empty generators.

Create your own generator

As noted earlier, you can create your own generators by combining the ones that already exist with for comprehension:

 val coordGen = for { i <- arbitrary[Int]; j <- arbitrary[Int] } yield (i, j) coordGen.sample // Option[(Int, Int)] = Some((45,60)) coordGen.sample // Option[(Int, Int)] = Some((29, 37)) 

You can declare generators for any data structure. A simple case of using ADT:

 sealed trait LightColor case object Red extends LightColor case object Orange extends LightColor case object Green extends LightColor case object FlashingGreen extends LightColor case object Extinct extends LightColor val colorGen = Gen.oneOf(Red, Orange, Green, Extinct) 

If you complicate your life, so thoroughly:

 //       ,     . type Color = LightColor object PedestrianTrafficLight { sealed class State(_1: Color, _2: Color) case object Stop extends State(Red, Extinct) case object Move extends State(Extinct, Green) case object FinishMove extends State(Extinct, FlashingGreen) def states: Seq[State] = Seq(Move, FinishMove, Stop) } object VehicularTrafficLight { sealed class State(_1: Color, _2: Color, _3: Color) case object Stop extends State(Red, Extinct, Extinct) case object Ready extends State(Red, Orange, Extinct) case object Move extends State(Extinct, Extinct, Green) case object Warn extends State(Red, Orange, Extinct) case object FinishMove extends State(Extinct, Extinct, FlashingGreen) def states: Seq[State] = Seq(Stop, Ready, Move, Warn, FinishMove) } sealed trait TrafficLight case class PedestrianTrafficLight(state: PedestrianTrafficLight.State) extends TrafficLight case class VehicularTrafficLight(state: VehicularTrafficLight.State) extends TrafficLight 

Let's announce our generators:

 import Gen._ //      val pedestrianTrafficLightStateGen = Gen.oneOf(PedestrianTrafficLight.states) //      val vehicularTrafficLightStateGen = Gen.oneOf(VehicularTrafficLight.states) 

Now let's generate a tuple of pedestrian and traffic lights. There is only one thing: the state of traffic lights should not be mutually exclusive. To begin with, we will define the condition under which a pedestrian can perform a maneuver. Since we create a generator by composition, we use retryUntil instead of suchThat :

 val pedestrianCanGo = pedestrianTrafficLightStateGen.retryUntil { state => state != PedestrianTrafficLight.Stop } 

Now we will generate the state of the car traffic light and, starting from it, we will determine the permissible condition for the pedestrian, and then we will combine them into the list. The generator we received will be of the following type: Gen[(VehicularTrafficLight.State, PedestrianTrafficLight.State)] . Gen[(VehicularTrafficLight.State, PedestrianTrafficLight.State)] .

 val states = vehicularTrafficLightStateGen flatMap { case vehState @ VehicularTrafficLight.Stop => pedestrianCanGo map (pedState => (vehState, pedState)) case pedestrianCanNotGo => (pedestrianCanNotGo, PedestrianTrafficLight.Stop) } 

Now we will display the states of the traffic lights in the objects themselves:

 val trafficLightPair = states map { case (v, p) => (VehicularTrafficLight(v), PedestrianTrafficLight(p)) } 

Check the results? I got:

 (VehicularTrafficLight(FinishMove),PedestrianTrafficLight(Stop)) 

and

 (VehicularTrafficLight(Move),PedestrianTrafficLight(Stop)) 

which is quite like the truth. Maybe somewhere I was wrong. The main thing was to demonstrate the mechanism itself.

ScalaCheck also allows you to generate recursive data structures,
However, it is difficult to call this process easy and trivial. Further in this
This article will cover recursive generators and problems that you
may encounter when using them.

Generators in the properties

In most of the previously used examples, ScalaCheck independently selects the appropriate generator instance and uses it to calculate the property. However, we did not write the generators ourselves, so that ScalaCheck could take away from the field of visibility, which was horrible. Remember our "compatible" traffic lights? Let's tackle them somewhere:

 import Prop.{forAll, propBoolean} def isRed(trafficLight: VehicularTrafficLight) = trafficLight.state == VehicularTrafficLight.Stop def notRed(trafficLight: PedestrianTrafficLight) = trafficLight.state != PedestrianTrafficLight.Stop //        Prop.forAll val canNotBothBeRed = forAll(trafficLightPair) { case (vehicular, pedestrian) => isRed(vehicular) ==> notRed(pedestrian) } 

Check?

 canNotBothBeRed.check 

And make sure all is well:

 + OK, passed 100 tests. 

Often, one generator is not enough, then you can use several:

 val p = forAll(Gen.posNum[Int], Gen.negNum[Int]) { (pos, neg) => neg < pos } 

Properties, like generators, can be nested in each other, so the example above can be rewritten as:

 val p = forAll(Gen.posNum[Int]) { pos => forAll(Gen.negNum[Int]) { neg => neg < pos } } 

Instead of nested forAll calls forAll you can also assemble a tuple of generators you want to use. And do not call forAll twice.

Generator Tags

Using tags is good practice. This allows improved error reports and improved readability of code using generators. To add a label, you can use the operators |: and :| . A label can be either a string or a symbol. Several tags may be added to the generator. To illustrate, take the previous example, just change the sign, making the property absurd :

 //  |:  :|   . val p = forAll('positive |: Gen.posNum[Int]) { pos => forAll(Gen.negNum[Int] :| "negative numbers") { neg => neg > pos } } 

Agree, it became better?

 ! Falsified after 0 passed tests. > positive: 0 > positive_ORIGINAL: 1 > negative numbers: 0 

Primitive Generators

Constants

It's hard to come up with something simpler than Gen.const . It takes any value and carefully packs it into the generator. Any object in the right place at the right time will be implicitly cast to Gen Therefore, most likely, you will not have to call this method explicitly.

Umirashka

Inside ScalaCheck, the Gen.fail method is Gen.fail to return a non-viable generator. When you try to call the sample method, you always get None . Most likely, he will not meet on your way, and if he does, consider that you are familiar with him:

 Gen.fail.sample // Option[Nothing] = None 

Numbers

You can use Gen.posNum and Gen.negNum to generate positive and negative numbers, respectively:

 import org.scalacheck.Gen import org.scalacheck.Prop.forAll val negative = Gen.negNum[Int] val positive = Gen.posNum[Int] val propCube = forAll (negative) { x => x * x * x < 0 } 

Gen.chooseNum has an interesting functionality: this generator generates numbers in a given range (inclusive), with more weight for zero (if it is within the specified range), minimum and maximum values, as well as for the presented list of values:

 //  specials  vararg. Gen.chooseNum(minT = 2, maxT = 10, specials = 9, 5) 

Gen.choose can be used to generate numeric values ​​within a specified range. Moreover, this is the rare type of generator that never fails. It can also be used with symbols. Speaking of symbolic generators ...

Characters

ScalaCheck has many character generators. You are invited to choose from the following:

• Gen.alphaUpperChar
• Gen.alphaLowerChar
• Gen.alphaChar
• Gen.numChar
• Gen.alphaNumChar

Now let's try to generate coordinates for the game “Sea Battle”:

 val coord = for { letter: Char <- Gen.alphaUpperChar number: Char <- Gen.numChar } yield s"$letter$number" coord.sample // Some(L1) 

Thanks to for comprehension, you can generate any aggregates: lists, tuples, strings. Speaking of strings ...

Strings

You can also easily generate them. Gen.alphaStr generates strings only from alphabetic characters. Gen.numStr generates numeric strings. It is possible to separately generate strings from different case characters: Gen.alphaLowerStr and Gen.alphaUpperStr are designed for this purpose. Gen.idenifier gives you a non-empty string that always starts with an alphabetic lower case character followed by alphanumeric characters. For some grammars, this is quite an identifier.

 import org.scalacheck.Gen val stringsGen = for { key <- Gen.identifier value <- Gen.numStr } yield Bucket(key take 8, value take 2) stringsGen.sample // Some(Bucket(kinioQqg,60)) 

Functions

You can also generate functions in Scalacheck (function0 instances: () => T ). To do this, pass the generator responsible for the return value to Gen.function0 :

 val f = Gen.function0(Arbitrary.arbitrary[Int]).sample //  tostring  ,      - : // some(org.scalacheck.gen$$$lambda$40/1418621776@4f7d0008) 

Container generators

You can generate instances of standard collections using arbitrary :

 import Arbitrary.arbitrary val listgen = arbitrary[List[Int]] val optgen = arbitrary[Option[Int]] 

But this is often not enough, so ScalaCheck provides advanced tools for working with collections, as well as their generation. Next, we will omit the Option s, which the sample method returns, and will show you the values ​​immediately, so as not to be led into a misleading.

Generate Option

If you use Gen.some , then the "income" is guaranteed:

 //     : Gen.some("") // Some() 

Worse, when money has additional degrees of freedom:

 // . Gen.option("") // None 

We generate lists and dictionaries

Gen.listOf produces lists of arbitrary lengths, and Gen.listOfN generates lists of a given length.

 //  3     Gen[Int]. Gen.listOf(3) map (_ take 5) // List(3, 3, 3, 3, 3) //    . Gen.listOfN(5, Gen.posNum[Double]) map (_ take 5) 

Gen.listOf generator can return an empty list. If you are guaranteed a non-empty list, you can use Gen.nonEmptyListOf . To generate dictionaries or maps ( Map ), there are similar methods: Gen.mapOf , Gen.mapOfN , as well as Gen.nonEmptyMapOf . Unlike methods for a list, generators for dictionaries require that a two-element tuple generator be passed to them:

 import Arbitrary._ val tupleGen = for { i <- arbitrary[Short] j <- arbitrary[Short] } yield (i, j) Gen.mapOfN(3, tupleGen) map (_ take 3) // Map(10410 -> -7991, -19269 -> -18509, 0 -> -18730) 

We generate a list of elements of a given set

Gen.someOf will return you an ArrayBuffer from some set of elements included in the set you specify, for example:

 import org.scalacheck.Gen.someOf val numbers = someOf(List(1, 2, 3, 4, 5)).sample // Some(ArrayBuffer(5, 4, 3)) val leds = someOf(List(Red, Green, Blue)).sample // Some(ArrayBuffer(Blue, Green)) 

Gen.pick works similarly to Gen.someOf . The only difference is that Gen.pick allows you to specify the required number of elements:

 import org.scalacheck.Gen.pick val lettersGen = Gen.pick(2, List('a', 'e', 'i', 'o', 't', 'n', 'm')).sample // Some(ArrayBuffer(m, n)) 

Spawn sequence

Gen.sequence creates an ArrayList based on the received sequence of values. If at least one of the provided generators falls, the whole sequence will fall.

 import Gen.{const, choose} val ArrayListGen = Gen.sequence(List(const('a'), const('b'), choose('c', 'd'))) // [a, b, d] 

We generate an endless stream

ScalaCheck supports the ability to generate endless streams:

 val stream = Gen.infiniteStream(Gen.choose(0,1)) //   8  stream.sample map (stream => stream.take(8).toList) // Some(List(1, 0, 0, 0, 1, 1, 1, 0)) 

Container generator

The methods listed above for dictionaries and lists are, in fact, special cases of using Gen.containerOf and its associates: containerOfN and nonEmptyContainerOf`. This is the most generalized form that allows you to generate most of the collections known to us. Let's consider the following examples:

 import Arbitrary._ //   «»     . val prettyShortSeq = Gen.containerOfN[Seq, Short](3, arbitrary[Short]) // Vector(0, -24114, 32767) //  :  . val genNonEmptyList = Gen.nonEmptyContainerOf[Map[K, V], (K, V)] (oneOf("foo", "bar")) 

Generators created using containerOf abstracted over types, but not genders (kinds). Let's look at the signature of the containerOf method:

 def containerOf[C[_],T](g: Gen[T])(implicit evb: Buildable[T,C[T]], evt: C[T] => Traversable[T] ): Gen[C[T]] = 

and its implementation:

 buildableOf[C[T],T](g) 

So, if you need to generate a list or something that can be represented as * ⟶ * , then containerOf will be useful to you. And if you need to generate something like * ⟶ * ⟶ * (for example, Map ), you will have to use an even more abstract method, which we can spy on in the mapOf implementation:

 def mapOf[T,U](g: => Gen[(T,U)]) = buildableOf[Map[T,U],(T,U)](g) 

Similar to the methods of container , there are methods buildableOf , buildableOfN and nonEmptyBuildableOf . Scalacheck usually decides how to build the collection passed to it as a type. For this, it uses an implicit instance of org.scalacheck.util.Buildable . Such instances are declared in ScalaCheck for all standard collection types. It's easy enough to implement your collection and get support for containerOf , as well as instances of Arbitrary .

Higher order generators

I think you, dear reader, know what are the highest order functions . Similar behavior is peculiar to generators: the generator can accept other generators as arguments, generating these other generators. The container generators considered earlier by us, with rare exceptions, are also generators of a higher order.

One of many

Gen.oneOf is capable of receiving from two to N generators, thereby generating a new generator.

 import Gen.{oneOf, const, choose} //      Gen[T] val dogies = oneOf("Lucy", "Max", "Daisy", "Barney") //        val windows = oneOf(choose(1, 8), const(10)) 

In oneOf you can pass any successor to scala.collection.Seq , for example, a list.

Frequency generator

Gen.frequency accepts a list of tuples as input. Each of which consists of a generator and its probabilistic weight. The output is a new generator that will use the integer values ​​passed to it as probabilistic weights:

 val russianLettersInText = Gen.frequency ( (9, ''), (9, ''), (8, ''), (7, ''), (6, '') //..   ) 

Lazy generator

Using Gen.lzy will allow you to postpone the generation until it is really needed. It is very useful in composite generators, as well as in the generation of recursive data structures, such as AST. In the case of Gen.lzy , calculations are performed once. Gen.delay also Gen.delay calculations, however, they will be executed every time.

Generator Size

Gen.size takes as its only parameter a lambda, which takes an integer as its parameter. This number is the size that ScalaCheck indicates during generator execution. Gen.resize allows Gen.resize to set the size for the generator where it is needed. We illustrate this with the following example:

 def genNonEmptySeq[T](genElem: Gen[T]): Gen[Seq[T]] = Gen.sized { size => for { listSize <- Gen.choose(1, size) list <- Gen.containerOfN[Seq, T](listSize, genElem) } yield list } val intVector = genNonEmptySeq(Arbitrary.arbitrary[Int]) val p = Prop.forAll(Gen.resize(5, intVector)) { list => list.length <= 5 } 

Check:

 p.check // + OK, passed 100 tests. 

Gen.resize creates a new generator based on an existing one, but with a resized size. This is useful when creating recursive generators, as well as creating complex generators from simple ones using composition.

Recursive generators

I often have to work with recursive data structures, namely AST. . , , , . :

 trait IntTree case class Leaf (value: Int) extends IntTree case class Node (children: Seq[IntTree]) extends IntTree 

:

 import org.scalacheck.Gen._ def treeGen: Gen[IntTree] = oneOf(leafGen, nodeGen) def leafGen: Gen[Leaf] = arbitrary[Int] map (value => Leaf(value)) def nodeGen: Gen[Node] = listOf(treeGen) map (children => Node(children)) 

:

 treeGen.sample Exception in thread "main" java.lang.StackOverflowError at org.scalacheck.ArbitraryLowPriority$$anon$1.arbitrary(Arbitrary.scala:70) at org.scalacheck.ArbitraryLowPriority.arbitrary(Arbitrary.scala:74) at org.scalacheck.ArbitraryLowPriority.arbitrary$(Arbitrary.scala:74) at org.scalacheck.Arbitrary$.arbitrary(Arbitrary.scala:61) 

treeGen . . . treeGen :

 def treeGen: Gen[IntTree] = lzy(oneOf(leafGen, nodeGen)) 

Run:

 treeGen.sample // Some(Leaf(857998833)) // Some(Leaf(2147483647)) // Some(Leaf(489549235)) 

 Exception in thread "main" java.lang.StackOverflowError at org.scalacheck.rng.Seed.next(Seed.scala:17) at org.scalacheck.rng.Seed.long(Seed.scala:39) at org.scalacheck.Gen$Choose$.org$scalacheck$Gen$Choose$$chLng(Gen.scala:309) at org.scalacheck.Gen$Choose$$anon$5.$anonfun$choose$1(Gen.scala:358) at org.scalacheck.Gen$$anon$3.doApply(Gen.scala:254) at org.scalacheck.Gen.$anonfun$map$1(Gen.scala:75) at org.scalacheck.Gen$$anon$3.doApply(Gen.scala:254) at org.scalacheck.Gen.$anonfun$flatMap$2(Gen.scala:80) at org.scalacheck.Gen$R.flatMap(Gen.scala:242) at org.scalacheck.Gen$R.flatMap$(Gen.scala:239) 

, , . . Gen.sized Gen.resize :

 def nodeGen: Gen[Node] = sized { size => choose(0, size) flatMap { currSize => val nGen = resize(size / (currSize + 1), treeGen) listOfN(currSize, nGen) map (child => Node(child)) } } 

:

 val nGen = resize(size / (currSize + 1), treeGen) 

we are creating a smaller generator: the size will be reduced in proportion to the level of nesting. For a specific N, an empty generator of size 0 will be created, and we will not get a stack overflow. The size of the generator should be reduced each time, since the generator does not know what level of nesting it is at.

So we figured out the recursive generators. An example of generating AST for JSON can be found on the Eric Torreborre blog (the creator of specs2).

This concludes our acquaintance with the generators and go to the properties. More about them will be discussed in the next article in the series. See you!