4

4.2 Functional tests
4.2.1 Test framework

For creating and running functional tests three the most popular test frameworks such as Mocha, Jasmine and Cucumber had been considered. This section describes each of these frameworks briefly, compares them in terms of following requirements and elaborates on why it was a fairly easy decision for choosing Cucumber and what makes it a suitable test framework for functional tests of application’s frontend.

We Will Write a Custom Essay Specifically
For You For Only $13.90/page!


order now

There were two main the most desired features while looking for a test framework for automated UI functional tests:
Support for Behaviour Driven Development (BDD).
Ease of reading and understanding tests by non programmers. The reason for that is to initiate collaboration between business facing people – Penneo client support team and developers.
Support for browser automation utilities is also a requirement as the testing framework is supposed to work alongside with it. (Ref link to: 2.2 Requirement Specification)

Mocha
Starting from arguably the most popular (http://www.npmtrends.com/mocha-vs-cucumber-vs-jasmine) library, Mocha(https://mochajs.org/). It is a JavaScript testing framework which runs tests on Node.js. It got the attention while deciding upon what testing framework should be used for UI tests mostly because of the fact that it supports BDD and because of its popularity and huge developer community. Mocha has been used to test everything under the sun, including functional UI testing so the decision to take a closer look at this framework from the perspective of functional testing was made. Mocha as a biggest testing framework is compatible with most (if not all of them) browser automation tools like Selenium, Protractor, WebdriverIO, etc.

By default mocha provides BDD style interface, which specifies a way to structure the code into test suites and test cases modules for execution and for the test report.
Mocha’s tests suites are defined by the `describe` clause and the test case is denoted by the `it` clause. Both clauses accept callback functions and can nest inside each other, which means that one test suite can have many inner test suites which can either contain another test suites or the actual test cases. The API provides developers with hooks like: `before()`, `after()`, `beforeEach()`, and `afterEach()` which are used for setup and teardown of the test suites and test cases. Mocha allows also to use any library for assertions such as ChaiJS or Should.js which are great BDD assertion libraries. (https://mochajs.org/#assertions)
This is very useful for unit tests or for functional tests that are going to be seen only by developers, but may not be preferred format for business-facing users. Test specification mixes up with the actual implementation of tests and it is not an especially friendly approach for non programmers and could actually complicate communication between both sides.

Jasmine
From this point of view Jasmine(https://jasmine.github.io/) is very similar to Mocha, it is also perfect for unit testing. It is compatible with many browser automation tools. Similarly to Mocha its tests are also supporting BDD style written tests. (https://jasmine.github.io/tutorials/your_first_suite) This is what both of these frameworks have in common.

The `describe` clause defines a test suite and the `it` implements the BDD method that is describing the behaviour that a function should exhibit.
Test case denoted by the `it` clause is build in the same way as in case of Mocha. Its first parameter is a description of the behaviour under test written in plain-English and the second parameter is the implemented function which calls assertions that either pass or fail, depending on whether the described behaviour was, or was not confirmed.

Both of them are fine choices if separation of the textual specification that everyone can read and write, from the technical implementation is not required. Due to not meeting this requirement Jasmine same as Mocha has been dismissed as inadequate.

Cucumber
Cucumber(https://cucumber.io/) is more user story based testing framework which allows writing tests in BDD style.
It targets higher levels of abstraction. Instead of writing tests purely in code like it is done in Mocha or Jasmine, Cucumber separates tests into human-readable user stories and the code that runs them. Cucumber uses Gherkin language to write its tests. Gherkin is a Business Readable, Domain Specific Language (https://martinfowler.com/bliki/BusinessReadableDSL.html) that makes it possible to describe software’s behaviour without going in to details of how that behaviour is implemented. Since the tests are written in a plain language and anyone on the team can read them that improves communication and collaboration of the whole team.

Cucumber tests are constructed in a completely different way than a typical Mocha tests, and that construction makes them so unique. In Mocha there are wrapping clauses that nest inside of each other, mixing up test scenario description with their code implementation. Cucumber tests seem to be more ordered, they consist of feature files written in Gherkin and step definitions written in JavaScript.

Feature files have a single `Feature` description at the top and one or more `Scenarios`. `Feature` is a brief description of what is being tested and presents the justification for having the feature. It describes also the role of the user/s being served by the feature and what is expected to be done from the feature. `Scenarios` are like user stories, they do not go into details of what the software would do, they rather focus on the perspective of the person the feature is intended to serve. Each `Scenario` starts with a line that describes it and is followed by some combination of `steps` which compose the whole description of a user story.
Every `step` has its code implementation, those implementations are called step definitions.
Unlike the feature files, step definition files are not available for business people, as they are implemented in JavaScript, and can be understood only by developers. They are more like traditional test functions that one would write in Mocha or Jasmine, except they are used to match the scenarios. They handle the whole logical operations behind the `steps`, they execute them.

The division that Cucumber introduces brings a great value for the business. The Features are written in a natural language therefore business people can read them easily, give developers early feedback or even write some themselves. Another great benefit of having functional tests made with cucumber is that those tests provide a living documentation, describing how the UI interface is supposed to work for each specific feature.

4.2.2 Browser automation

In order to implement functional tests, just like in a case of the unit tests an extensive research have been made. Different browser automation options have been looked into and compared to find the best solution for testing some of the crucial parts of Penneo web application.

Since the application to be tested is developed and maintained by JavaScript developers a decision that the implementation of the functional tests is also gonna be written in the same programming language has been made. The main benefit of that was significantly faster expected development of these tests. The simplicity, reliability, ease of setting up and readable syntax were another requirements for desired framework.

Another highly desired feature was a broad support for different browsers. Support for the real browsers was pretty obvious in the business of browser testing frameworks.
Important was that the browsers for tests would be cloud based to ensure all the UI components would render as expected in many different browsers in their various versions, even some of the pretty old ones. Installing and working with all of them on the virtual machines would get annoying and inconvenient very quickly.
Important was also a support for the headless browsers like PhantomJS or Headless Chrome, as execution of tests using these browsers makes the development of tests much more convenient as developer does not need to watch multiple browser windows reload, change and so on. Fortunately it turned out that all of the most popular browser testing frameworks already support these features.

A key factor was also a support from the developer community to make sure that the framework that would be chosen will stay popular for another few years and there will not be a need for changing it. Support of a wide community also brings benefits in terms of new useful plugins or extensions and makes it easier to find answers on websites like StackOverflow for inevitable problems and errors. Another thing is that working with tools that are not very popular does not seem very appealing.

The browser automation tools has been researched together with test frameworks. However very quickly a decision about using the Cucumber has been made, so the browser automation utility received another requirement, that is to be compatible with Cucumber.

WebdriverIO
(https://www.npmjs.com/package/webdriverio)
WebdriverIO (WDIO) is a JavaScript automation library which is a custom Node.js implementation for W3C WebDriver API to perform commands and assertions on DOM elements.(https://www.w3.org/TR/webdriver/) In other words WDIO is a testing utility enabling writing automated selenium tests by providing a variety of tools for controlling a browser or a mobile application. WebdriverIO provides entirely different and smarter bindings for the WebDriver API in comparison to Selenium’s which is the most popular browser automation library on the market and its Node.js implementation is called WebDriverJS. Naming around these technologies can get a little bit tricky and confusing since they all use a similar names.

Figure ?: WebdriverIO, Selenium server and Chrome test communication flow

WebdriverIO, WebDriverJS and Nightwatch communicate over a restful HTTP API with a WebDriver server (usually the Selenium server). The restful API is defined for all of them by the W3C WebDriver API. (source)

Recently WebdriverIO had a major version change with a rewrite of the whole code base into ES2016. As the project community is growing dynamically the project by itself and its plugins are also being developed rapidly. One of the biggest changes that has been made was splitting the entire project into multiple packages which has a positive impact on new plugins that are being developed to write better tests in a simpler way.

WebdriverIO framework has an integrated test runner which is a great advantage that helps to create a reliable test suit that is easy to read and maintain. It supports three main test frameworks which are Mocha, Cucumber and Jasmine. After the newest major version release the WDIO test runner handles async (https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Statements/async_function) which is a very useful change because now asynchronous commands can be written in a synchronous way. It is especially useful for inexperienced with a concept of promises developers, supporting them in writing tests without worrying about how to handle a Promise to avoid racing conditions in tests and also simplifying the code syntax a lot.

WebdriverIO has been chosen after an insightful analysis of other key players on the stage of automated system testing tools powered by Node.js. The most popular one is without doubts WebDriverJS, however there are also Protractor and NightWatch, each of them has been analysed to identify advantages and disadvantages of each to know which would suit the given problem best.

WebDriverJS is an official JavaScript library for Selenium. It consists of a set of bindings for
Protractor
Protractor(http://www.protractortest.org/#/api) has been dismissed as inadequate as a first one, because even though it has support for the same test runners as WebdriverIO and pretty good reporting mechanism its main focus is to test Angular applications as it has build in support for AngularJS element identification strategies.(https://www.protractortest.org/#/api?view=ElementFinder) Also unlike all of the other compared frameworks it is only implemented as a wrapper for WebDriverJS which creates one more layer in between Selenium and the Protractor system tests (https://www.protractortest.org/#/). It means that its highly dependent on WebDriverJS project, so whenever there is an issue with a WebDriverJS, Protractor needs to wait for the WebDriverJS team to fix that.
NightWatch
NightWatch(http://nightwatchjs.org/) is very similar to WebdriverIO since it is also a custom implementation of W3C WebDriver API – works with the Selenium standalone server. An interesting feature in this tool is that it has its own testing framework, the only external testing framework that is also supported is Mocha. One may see this as an advantage as it solves a headache of choosing a testing framework especially that it also implements its own assertions mechanism. However in our particular case the lack of support for Cucumber framework was a major disadvantage and main reason for not choosing NightWatch, the reasoning for that will be explained further in this paper. In comparison with WebdriverIO, Protractor and WebDriverJS it has slightly less support which was also an important factor when comparing these technologies.

An important thing to point out is that all of the mentioned system automation tools support cloud browser testing providers, such as SauceLabs and BrowserStack which is a very desired feature. Those services allow developers to run their automated tests in cloud based browsers to ensure websites run flawlessly on every browser, operating system and device. Both of them offer a wide range of operating systems and browser combinations across desktop and mobile platforms in almost all ever released versions.
This solution gets rid of a cumbersome issue of having many actual devices, or virtual machines with specific operating systems and browsers installed.

WebdriverIO after being split up into multiple packages has to have different necessary packages installed separately which makes it the most flexible and customizable one among all of them. Important thing to point out is that even though it has to have its separate packages installed it is still the easiest one to configure and set up due to its unique command line interface that makes test configuration as easy and simple as possible.

Last but not least great feature that WebdriverIO has acquired to its project is that it is possible to set framework for test runner. WebdriverIO supports such frameworks as Cucumber, Jasmine and Mocha. One of the requirements for the browser automation utility was to support Cucumber, and thus this framework has been chosen.

WebdriverIO
WebDriverJS
NightWatch
Protractor
Architecture
? Custom implementation for selenium’s W3C webdriver API
? Official JavaScript implementation of Selenium
? Custom implementation for selenium’s W3C webdriver API
? It’s a wrapper around WebDriverJS;
Focused on Angular
Syntax
? Easy
? Difficult
? Easy, similar to WebdriverIO
? Medium
Configuration
? Easy
? Difficult, requires advanced skills
? Medium
? Easy
Supported testing frameworks
? Cucumber, Mocha, Jasmine
? Cucumber, Mocha, Jasmine
? Mocha, inbuilt framework
? Cucumber, Mocha, Jasmine
Supported browsers
? Chrome, Firefox, Safari, Opera, PhantomJs
? Chrome, Firefox, Safari, Opera, PhantomJs
? Chrome, Firefox, Safari, Opera, PhantomJs
? Chrome, Firefox, Safari, Opera, PhantomJs
Inbuilt test runner
? Yes
? No
? Yes
? Yes
Cloud execution
– Browserstack
– Sauce Labs
? Supported
? Supported
? Supported
? Supported

? = advantage
? = disadvantage
? = compromise

Figure ?: Browser Automation comparative chart

After combining technology stack which consists of Cucumber and its Gherkin tests with WebdriverIO that can write Selenium tests, a useful and powerful functional UI testing framework has been built.