Automated API Testing with Test Frameworks

Sebastian Sanchez

November 3, 2020

Companies are increasingly adopting innovative practices for their software projects, to improve the quality of project development and delivery.

This implies leaving behind traditional methodologies and adding agile development as a new way of carrying out projects. 

This innovative, agile ecosystem, where there is continuous integration and deployment, and which involves frequent iterations, requires more deliverables than the traditional one, since the product grows in functionality in each iteration and is built integrating the customer.

Image: Continuous Integration and Deployment

When talking about agility, it is common that the first word we think about is Scrum, since it is one of the most used methodologies in agile development. However, there are many others less used or uncommon, such as XP, Kanban, Lean

These methodologies allow for better results, and to respond to changes more quickly and flexibly, integrating clients into the entire development process. The philosophy of these methodologies is supported by the Agile Manifesto.

About the application of agile development methodologies, you should consider building the product under these fundamental principles:

  • The software should be kept as simple as possible.
  • You should have a quick feedback.
  • structure should allow for incremental change.
  • Make sure the software is a quality job.

Under these principles and thinking about the nature of software development, where changes are integrated in an incremental and continuous way, it is key to maintain the quality and functionality.

How to maintain the quality and functionality of the software? 

You can achieve this through automated testing.

Automated testing helps in this continuous process of changes and deliveries, because once you automate the test set, you can visualize early on that a change caused the loss of some functionality in the software or modified part.

Frameworks for Automated Testing

Karate and Cucumber are two frameworks for automated testing and we want to show you which one delivers better results in terms of usability and compatibility.

Cucumber and Karate are based on the BDD (Behavior Driven Development) methodology, which is an improvement of TDD (Test Driven Development), an evidence-based development methodology. BDD, unlike TDD, includes the business and allows both the technical and business teams to relate through a common language called Gherkin, used to describe the behavior of the software. It uses five general statements:

  • Feature: indicates the feature of the software.
  • Scenario: it tells us in which scenario the test will take place.
  • Given: indicates that "given that" a scenario exists, the test will be performed.
  • When: indicates that "when a condition exists" the test will be performed.
  • Then: indicates that "then" an expected behavior will occur in the test.
Example Gherkin language - © Kranio.io

Now that you've seen how the gherkin language works, we can review the frameworks we'll use to do the automated testing.

Cucumber

For this exercise we will use Cucumber with JVM. If you are using a language other than Java, there are other integrations. We will use Maven for the dependencies. In the pom.xml we add the following: 

Cucumber Spring integration

To add cucumber-jvm:

-- CODE language-xml -- <dependency>     <groupId>io.cucumber</groupId>     <artifactId>cucumber-java</artifactId>     <version>6.8.0</version>     <scope>test</scope> </dependency>

‍If you can better understand these conversations, rank (word cloud), prioritize and engage users, you can improve a lot and build a true cult following.

Then we add Junit and cucumber's dependence:

-- CODE language-xml -- <dependency>     <groupId>io.cucumber</groupId>     <artifactId>cucumber-junit</artifactId>     <version>6.8.0</version>     <scope>test</scope> </dependency> <dependency>     <groupId>io.cucumber</groupId>     <artifactId>cucumber-junit</artifactId>     <version>6.8.0</version>     <scope>test</scope> </dependency>


Finally, we add, Spring's dependence:

-- CODE language-xml -- <dependency>     <groupId>io.cucumber</groupId>     <artifactId>cucumber-spring</artifactId>     <version>6.8.0</version>     <scope>test</scope> </dependency>

‍If you can better understand these conversations, rank (word cloud), prioritize and engage users, you can improve a lot and build a true cult following.


You must then configure the REST service. For this example a REST api will be used with a list of available games.

Cucumber Steps Definitions

To run the cucumber tests with Junit, we need to define the steps, create an empty class with the @RunWith(Cucumber.class) notation and @CucumberOptions to define the options we will use.

Then, in the path "src/test/resources/features/" you must write the files containing the characteristics of the tests written in Gherkin language.

This scenario makes a GET call to the service that delivers the game list and validates that the information is displayed correctly.

After having this scenario created in gherkin, you must code the steps of each feature in java, for this create a class that contains the same steps so that the steps are recognized.

NOTE: we must use Rest-Assured to validate the Api responses.

Now you have everything you need to run the test. Run the test with Junit in the RunTest class we've created. If everything is correct, the test was executed correctly.

To see the report you can use cucumber-reporting.

Image: Report generated by cucumber report.

Now we move on to Karate.

Karate BDD

Karate is a simple testing framework just like Cucumber and has tools included that make it very effective. This is how its creators describe it: 

"Karate is the only open source tool that combines automation of API testing, simulations, performance testing and even user interface automation into a single, unified framework. The BDD syntax popularized by Cucumber is a neutral and easy language even for non-programmers. It incorporates powerful JSON and XML statements, and you can run tests in parallel for greater speed "

For this example we will not use all the tools, but several to perform the same exercise as with Cucumber and make the comparison.

First add the following rooms to the pom:

-- CODE language-xml -- <!-- https://mvnrepository.com/artifact/org.mockito/mockito-core -->         <dependency>             <groupId>org.mockito</groupId>             <artifactId>mockito-core</artifactId>             <version>${mockito.version}</version>             <scope>test</scope>         </dependency> <!-- https://mvnrepository.com/artifact/com.intuit.karate/karate-apache -->         <dependency>             <groupId>com.intuit.karate</groupId>             <artifactId>karate-apache</artifactId>             <version>${karate.version}</version>             <scope>test</scope>         </dependency> <!-- https://mvnrepository.com/artifact/com.intuit.karate/karate-junit5 -->         <dependency>             <groupId>com.intuit.karate</groupId>             <artifactId>karate-junit5</artifactId>             <version>${karate.version}</version>             <scope>test</scope>         </dependency>            <dependency> <groupId>com.intuit.karate</groupId> <artifactId>karate-junit4</artifactId> <version>0.9.5</version> <scope>test</scope> </dependency>          <build>         <testResources>             <testResource>                 <directory>src/test/java</directory>                 <excludes>                     <exclude>**/*.java</exclude>                 </excludes>             </testResource>         </testResources>          <plugins>             <plugin>                 <groupId>org.springframework.boot</groupId>                 <artifactId>spring-boot-maven-plugin</artifactId>             </plugin>              <plugin>                 <groupId>org.apache.maven.plugins</groupId>                 <artifactId>maven-surefire-plugin</artifactId>                 <version>2.22.1</version>                 <configuration>                     <excludes>                         <exclude>karate/**/*.java</exclude>                     </excludes>                     <includes>                         <include>com/**/*.java</include>                     </includes>                 </configuration>             </plugin>         </plugins> </build>

‍If you can better understand these conversations, rank (word cloud), prioritize and engage users, you can improve a lot and build a true cult following.

Then create a runner class to add the Karate libraries and then run the tests from there. In the class, the method that runs the tests must have the @Karate.Testnotation:


Then create the feature file that will contain the scenarios and steps to be executed:

-- CODE language-gherkin -- @Tag games Feature: Get list of games Background: * def baseUrl = 'https://api-gamelist.herokuapp.com' * def gameBase = '/games' Scenario: Get the game list Given baseUrl+gameBase And header Accept = 'application/json' When method GET Then status 200

‍If you can better understand these conversations, rank (word cloud), prioritize and engage users, you can improve a lot and build a true cult following.

Then run the tests. This will automatically generate an html report:


Image: Results with Karate

Image: Results with Karate

Now that we've seen both frameworks, we can draw conclusions! 

While both frameworks work similarly, Karate is simpler to use than Cucumber. Let's see the advantages of both frameworks:

Advantages of Karate:

  • With Karate you don't need to be an expert in programming to be able to implement it.
  • Unlike Cucumber, you can run tests simultaneously.
  • In Karate we make the validations of the answers directly in the feature file and it even allows us to compare answers, since it works natively with Json files.
  • By including certain tools, it automatically generates reports to visualize the result of the tests, while Cucumber needs to be configured and a dependency must be added to visualize the results with a graphic interface.
  • Fewer settings are needed to achieve the same results.
  • We do not need to include other additional libraries to do the same job (Rest-assured, Serenity BDD, cucumber-report).

Advantages of Cucumber:

  • Cucumber can be run from Maven or from Junit using the same configuration.
  • Cucumber allows to use different libraries or frameworks that interpret the results to be displayed in pie charts with a much more friendly and better understanding interface.
  • You can use the reports generated by cucumber in project management tools such as Jira via the integration of Xray.
  • You can use Cucumber with Selenium, if your tests go beyond the backend and you also need to test the frontend.

Karate or Cucumber: which is better?

Both tools are powerful and the one you choose will depend on each case and the way you need to visualize the results. If what you need is to deliver information to a management level, with a better graphic interface, and better interpretation of the results, Cucumber is the best alternative if you complement it with Serenity BDD or another component to visualize results. If what you need is to graph the results for a technical area, Karate is the most agile alternative, because you will get the same results with less steps and complexity.

Sebastian Sanchez

November 16, 2020

Previous entries