BDD Python Framework Comparison: Pros and Cons

Hello. Already this month, the course "Python QA Engineer" starts at OTUS. On the eve of the start of this course, we are sharing with you the translation of interesting material.

Almost all major programming languages ​​have frameworks for BDD testing , and Python is no exception. And in fact, he has several of them! So how do you compare them and which one is the best? Let's try to figure it out.

behave

behave is one of the most popular BDD python frameworks. Despite the fact that it is not officially part of the Cucumber project, its functionality is quite similar to the Cucumber frameworks.

Sources:

• Python Testing 101: behave
• Behavior-driven Python
• Behave project on github
• The behave-parallel project on GitHub
• Behave documentation

Logo

Pros:

• Fully supports the Gherkin language;
• The functional environment and fixture (fixture) make it easy to install and remove;
• There is integration with Django and Flask;
• Popular with BDD Pitonists;
• Good documentation and guidance;
• There is support for PyCharm Professional Edition .

Minuses:

• There is no support for parallel execution, this requires a separate behave-parallel framework;
• behave is a standalone framework;
• Sharing steps between feature files can be a lot of trouble.

pytest-bdd

pytest-bdd is a pytest plugin that allows users to write tests as Gherkin feature files instead of test functions. Because it is integrated into pytest, it can work with any other pytest plugins, such as pytest-html for creating beautiful reports and pytest-xdist for parallel testing. It also uses pytest fixtures for dependency injection.

Sources:

• Python Testing 101: pytest-bdd
• Python Testing 101: pytest
• The pytest-bdd project on GitHub
• Documentation for pytest-bdd
• pytest.org

Logo

Pros:

• Fully compatible with pytest and its main plugins;
• You can enjoy all the benefits of the pytest community;
• Fixtures are a great way to manage context between actions;
• Tests can be filtered and performed along with other pytest tests;
• Definitions of actions and hooks can be easily shared using conftest.py;
• Tabular data can be better processed for data-driven tests ;
• Excellent online documentation and tutorials;
• There is support for PyCharm Professional Edition .

Minuses:

• Modules for determining actions should have explicit declarations for feature files (via “@scenario” or the “scenarios” function);
• The behavior plan should be handled differently.

radish

radish is a BDD framework with a twist: it adds a new syntax to the Gherkin language. Language features such as script cycles, script prerequisites, and constants make Gherkin in radish more programmatic for test cases.

Sources:

• Project radish on github ;
• radish-bdd.io
• Documentation radish

Logo

Pros:

• Extensions to the Gherkin language allow testers to write tests better;
• Website, documentation and logo - everything is in place;
• Feature files and behavior definitions are very clear.

Minuses:

• It is a standalone framework with a limited number of extensions;
• BDD purists may not like adding Gherkin syntax.

lettuce

lettuce is another “vegetable” BDD Python framework, which has been known for several years. However, his site and code have not been updated for a long time.

Sources:

• Project lettuce on github;
• lettuce.it .

Logo:

Pros:

• Simple code;
• Proven and good.

Minuses:

• Not as rich in functionality as in other frameworks;
• It seems that he does not have active ongoing support.

freshen

freshen is one of the very first BDD testing frameworks in Python. He was a plugin for nose . However, freshen and nose are no longer supported, and on the page for their documentation it is clearly said to use other frameworks.

Recommendations

None of these frameworks are perfect, but many of them have significant advantages. In general, I would recommend using pytest-bdd, since it has all the advantages of pytest. I believe that pytest is one of the best frameworks for testing in any language because of its brevity, fixtures, assertions and plug-ins. 2018 Python Developers Survey showed that pytest is the most popular framework for testing and in Python. Although pytest-bdd does not look as polished as behave, I believe that some TLC from the open source community have the power to fix it.

And some more recommendations:

• Use behave if you need a reliable, clean development experience and a large community.
• Use pytest-bdd if you need integration with other plugins that already contain a bunch of pytest tests, or if you need parallel execution of tests.
• Use radish if you want more programmatic testing control at the Gherkin level.
• Do not use lettuce and freshen.

What is your favorite BDD framework? Please share in the comments!