Allure-Framework. Work with code

Continuing the series of publications about the possibilities of the Allure-framework , today we will talk about working with the code. Under the cut, we analyze what a test step is, how to display information in a report when performing steps, and what categories of defects there are. In addition, tell about the annotations Allure. Further more interesting!

Step test. Definition and annotation @ Step

To visualize the processes occurring in your tests, you must use steps. A step must represent some kind of atomic action or check. Actually the sequence of steps and represents your test. For clarity, the step should describe the input parameters, as well as the expected and actual results, if the step is a check.

In Allure, to denote the test step above the method, you must annotate the @ Step. Then the necessary steps-methods are included in the body of the test method: tests with the @ Test annotation.

Consider an example
')
Create a method with the annotation @ Step, which checks the consistency of the sum of the two terms to the expected result:

@Step public static void checkSumStep(int num1, int num2, int expectedSum) { Assert.assertTrue("     ", num1 + num2 == expectedSum); } 

Now create a test method using this step:

 @Test public void simpleTest2() { checkSumStep(3, 2, 5); checkSumStep(5, 4, 9); } 

When we run this autotest, we will receive the following report:



Annotation @ Step can accept the parameter “value”, which allows you to specify the name of the step in Russian. In addition, the step name can be parameterized by passing the values ​​of the arguments passed to the step to it.

Let's demonstrate by example

 @Step("   {num1}   {num2}") public static void checkSubtractionStep(int num1, int num2, int expectedResult) { Assert.assertTrue("     ", num1 - num2 == expectedResult); } 

When using this step in the test, we get the report:

It should be noted that in the case of using an object as an argument of a method, it is possible to display the value of one of the fields of this object in the step name. To do this, specify the field name in value in the form {object.fieldName}

Information output in the report when performing test steps. Attachments

When performing autotest steps, it can be very useful to display various kinds of information in a report. For these purposes in Allure serve attachments.

An attachment is information of a different type in the form of a file that can be attached to a test step or test.

Attachments can have 4 characteristics:

• value / name - the name of the attachment;
• type - type of information in accordance with the MIME standard;
• content - the content of the attachment;
• fileExtension — An optional attachment file extension starting with a dot.

Attachments to the report can be attached in two ways: using the @ Attachment annotation and using the Allure static addAttachment method.

Attaching attachments using the @Attachment annotation

This method creates a method that returns an array of bytes, above it is placed the @ Attachment annotation. When using this method, the read information will be added to the report as a file with the appropriate extension. The name of the attachment will be the same as the name of the method being called.

Let's demonstrate by example

Create a method that will read the attachment:

 @Attachment public static byte[] getBytes(String resourceName) throws IOException { return Files.readAllBytes(Paths.get("src/main/resources", resourceName)); } 

Let's create a test step that will use this method:

 @Step("   {str1}  {str2}") public static void checkStringEqualsStep(String str1, String str2) throws IOException { Assert.assertTrue("  ", str1.equals(str2)); getBytes("picture.jpg"); getBytes("text.txt"); } 

Finally, we will write a test in which our step will be used:

 @Test public void simpleTest4() throws IOException { String darkSouls = "Dark souls 3"; checkStringEqualsStep(darkSouls, darkSouls); } 

As a result of this test run, we get the report:



In the @ Attachment annotation, you can set additional refinement parameters:

 @Attachment(value = "", type = "application/json", fileExtension = ".txt") public static byte[] getBytesAnnotationWithArgs(String resourceName) throws IOException { return Files.readAllBytes(Paths.get("src/main/resources", resourceName)); } 

When using this method in its steps, the attachment will receive the name “Attachment”, the contents of the attachment will be highlighted in accordance with the “json” template, and the attachment itself will be saved in the “.txt” format:



If you change the type to “plain / text”, then the key-value highlighting will no longer be characteristic of the json-style:

You can also experiment with fileExtension, for example, specify ".doc". In this case, the attachment will be saved in a format typical of MS Word '97.

Attaching Attachments with the Allure static addAttachment Method

In this method, you can attach an attachment to the test / test step using the overloaded static addAttachment method from the Allure class. Up to 4 arguments can be passed to this method - 2 of them are required (the name of the attachment and the attached content itself), 2 optional (file extension and MIME type).

 @Step("   ") public static void addLinkSber() { String link = "http://sberbank.ru"; Allure.addAttachment("", "text/plain", link); } 

Despite the fact that the MIME types need to be specified quite often, they have to be written “manually”. Allure does not include a class with constants of admissible types.

Other Allure annotations

In addition to the most famous annotations @ Step and @ Attachment, Allure includes a number of other annotations:

Annotation @ Description

@ Description - annotation placed above the test or step. Allows you to attach a description to the test or test step. This annotation can take “value” parameters - description text and “useJavaDoc”. If the “true” value is set to the last parameter, the Javadok located above the method will be taken as the description.

Let us give an example of using this annotation.

 @Test @Description(value = "    ") public void simpleTest7_1() { Assert.assertTrue(1 == 1); } 

Functional annotations

@ Epic - abstract, placed above the test. Allows you to group tests by epic. This annotation takes the parameter “value” - the name of the epic.
@ Epics is the same as @ Epic, but it takes an array of Epic as the “value” parameter.
@ Feature - annotation placed above the test. Allows you to group tests by checked functionality. This annotation takes the parameter “value” - the name of the functional.
@ Features is the same as @ Feature, but it takes an array of Features as the “value” parameter.
@ Story - annotation placed above the test. Allows you to group tests for User story. This annotation takes the parameter “value” - the name of the User story.
@ Stories is the same as @ Story, but it takes a Story array as the “value” parameter.

Annotations @ Epic / Epics, @ Feature / Features, @ Story / Stories can be attributed to the annotation of functionality. These annotations group tests by functionality in the Behaviors section of the Allure report.

Let us give an example of using these annotations.

 @Epic(value = "") @Feature(value = "  ") @Story(value = "") @Test public void sumTest() { Steps.checkSummationStep(5, 4, 9); } @Epic(value = "") @Feature(value = "  ") @Story(value = "") @Test public void subTest() { checkSubtractionStep(8, 2, 6); } @Epics(value = {@Epic(value = ""), @Epic(value = "")}) @Features(value = {@Feature(value = ""), @Feature(value = "  ")}) @Stories(value = {@Story(value = ""), @Story(value = "")}) @Test public void checkSinTest() { checkSinStep(0, 0); } 

When executing the tests and then generating the report, in the Behaviors section we will see that the tests are grouped into a multi-level list ( @ Epic → @ Feature → @ Story):

Abstract @ Flaky

@ Flaky - annotation placed above the test. Allows you to mark autotest as unstable. If the autotest falls in at least one restart (for example, the “target” folder between the runs of the same test is not cleared), you will see the sign of the bomb in the report for this test. In addition, this annotation can mark classes with unstable tests.

Let us give an example of using this annotation.

Please note that you should not do branchings in tests: in this case, the if - else block is used only to make the test unstable!

 @Test @Flaky public void testDemoFlaky() { int randomNum = ThreadLocalRandom.current().nextInt(0, 2); if (randomNum == 0) { Assert.assertTrue(true); } else { Assert.assertTrue(false); } } 

For multiple test runs in restart mode, the report will look as follows:

Annotations of references to test cases and defects

@ Issue - abstract placed above the test. Allows you to link AutoTests with Issue. This annotation accepts the “value” parameter, in which the ID of the defect in the bug-tracking system is indicated.

@ Issues is the same as @ Issue, but the Issue array is taken as the “value” parameter.

@ TmsLink - annotation placed above the test. Allows linking autotests with test cases, the steps of which are described in test management systems. This annotation accepts the “value” parameter, in which the test ID is specified in the test management system.

@ TmsLinks is the same as @ TmsLink, but it takes an array of TmsLinks as the “value” parameter.

Annotations of this group add references to the defect / test case in the Allure report.

In order to get a full link from the defect ID / test case specified in the “value” parameter, it is necessary in the allure.properties (which should be placed in the classpath, for example, in src / test / resources) to describe the reference template before the corresponding defect / test -case

 allure.link.issue.pattern=https://example.org/issue/{} allure.link.tms.pattern=https://example.org/tms/{} 

When generating a report, Allure replaces the {} characters with the values ​​that were specified in the value parameter of your annotation, and you will receive full links.

Let us give an example of using these annotations.

 @Test @Issue(value = "FGY-4627") @TmsLinks({@TmsLink(value = "TL-135"), @TmsLink(value = "TL-158")}) public void simpleTest15() { Assert.assertTrue(1 == 1); } @Test @TmsLink(value = "TL-678") public void simpleTest18() { Assert.assertTrue(1 == 1); } 

Other link annotations

@ Link - abstract placed above the test. Allows you to attach to the autotest links to some external resources. This annotation can take the parameters “name” - the name of the link (by default, url), “value” - a synonym for “name”, “url” - the address to which the transition should be made and “type” - the parameter used to create an icon for links.
@ Links is the same as @ Link, but as an argument the "value" takes an array of links.

Let us give an example of using these annotations.

 @Test @Link(name = "", url = "http://yandex.ru") public void checkSubtractionWithLinkTest() { checkSubtractionStep(15, 5, 10); } @Test @Links(value = {@Link(name = "1", url = "http://sberbank.ru"), @Link(name = "2", url = "http://yandex.ru")}) public void checkSubtractionWithLinksTest() { checkSubtractionStep(14, 5, 9); } 

Annotation @ Owner

@ Owner - annotation placed above the test. Allows you to specify the person in charge of the test. This annotation accepts the “value” parameter, in which information about the author of the autotest is indicated.

Let us give an example of using this annotation.

 @Test @Owner(value = "  ") public void testDemoOwner() { checkSumStep(1, 2, 3); } 

Abstract @ Severity

@Severity - abstract placed above the test. Allows you to specify the level of criticality of the functional checked by the autotest. This annotation accepts the “value” parameter, which can be one of the enum SeverityLevel elements: BLOCKER, CRITICAL, NORMAL, MINOR or TRIVIAL.

Let us give an example of using this annotation.

 @Test @Severity(value = SeverityLevel.BLOCKER) public void testDemoSeverity() { checkSubtractionStep(6, 1, 5); } 

Parameterized Autotests in Allure

Allure can work with parameterized autotests. Consider the example of JUnit 4.12.
First, create a test class with parameterized tests of the following form:

 @RunWith(Parameterized.class) public class ParamsTests { @Parameter public int operand1; @Parameter(1) public int operand2; @Parameter(2) public int expectedResult; @Parameters(name = "operand1 = {0} | operand2 = {1} | expectedResult = {2}") public static Collection<Integer[]> dataProvider() { return Arrays.asList(new Integer[][]{ {1, 2, 3}, {2, 4, 6}, {5, 6, 11}, {7, 5, 12}, {2, 4, 5}, {4, 1, 5}, {8, 2, 11} }); } @Test public void checkSum() { Assert.assertTrue("     ", operand1 + operand2 == expectedResult); } } 

Now let's run our test and collect the report:



Allure presents our parameterized test as a set of tests, if we fall into any test, we will receive detailed information about the run of this particular case.

Categories of defects. Defect distribution by category

By default, on the Allure-report tab of the Allure report, all of the run-away test methods are divided into 2 categories: product defects (failed tests) and test defects (broken tests). In order to make a custom classification, you need to attach the categories.json file to the target / allure-results directory.

If the allure-maven plugin is connected to you in pom.xml, then categories.json can be placed in the resources subdirectory of the test directory

Let us give an example of a custom classification of defects. Create a categories.json file:

 [ { "name": " ", "matchedStatuses": ["broken", "failed"], "messageRegex": ".*-   .*" }, { "name": " ", "matchedStatuses": ["broken"], "traceRegex": ".*NullPointerException.*" }, { "name": "Product defects", "matchedStatuses": ["failed"] }, { "name": "Test defects", "matchedStatuses": ["broken"] } ] 

Attribute Description :

• name - the name of the category. It will be displayed on the Categories tab at the highest level of the classification. Required attribute.

• matchedStatuses - a list of statuses, one of which must complete the test in order to fall into this category. Out of the box, the states “failed”, “broken”, “passed”, “skipped” and “unknown” are available. Optional attribute.

• messageRegex - a regular expression that an Exception message must match to fall into this category. Optional attribute.

• traceRegex is a regular expression that must meet Exception-StackTrace to fall into this category. Optional attribute.

Now let's run the tests that detect such defects, and see what the report will look like on the Categories tab:

Test environment. ENVIRONMENT block on the main report page

The Allure report allows you to display on your main page in a special section information about the test environment on which the tests were run. It looks like this:



The information displayed in this block gets there from a special environment.properties file.

Here is an example of this file:

 OS.version=Windows 10 Pro JDK.version=jdk1.8.0_162 MAVEN.version=Apache Maven 3.5.2 allure-junit4.version=2.6.0 allure-report.version=2.6.0 

This file must be attached to the “target / allure-results” directory before building the html report. You can do this manually, or you can use the maven-resources-plugin .

Let us give an example of its use in this situation, provided that environment.properties are located in the resources subdirectory of the test directory. To do this, modify pom.xml:

 <build> <plugins> ... <plugin> <artifactId>maven-resources-plugin</artifactId> <version>3.0.2</version> <executions> <execution> <id>copy-resources</id> <phase>validate</phase> <goals> <goal>copy-resources</goal> </goals> <configuration> <outputDirectory>${allure.results.directory}</outputDirectory> <resources> <resource> <directory>src/test/resources</directory> <includes> <include>environment.properties</include> </includes> </resource> </resources> </configuration> </execution> </executions> </plugin> ... </plugins> </build> 

Now when building the project, environment.properties will fall into the “target / allure-results” and participate in the construction of the html report.

When running tests on Jenkins, categories.json will not copy by itself into the “target / allure-results”. It is also very convenient to add it to the includes maven-resources-plugin section.

Conclusion

In the second part of Longrid, we tried to tell in detail about the annotations in Allure, gave examples of their use. We touched the categories of defects, the test environment and how to display information in the report. We invite you to discuss in the comments.

Subscribe to the ESF blog, follow new publications. Soon we will again tell something new and useful about Allure.