4. describes each of these frameworks briefly, compares
2 Functional tests4.2.1 Test frameworkFor creating and running functional tests three the most popular test frameworks such as Mocha, Jasmine and Cucumber had been considered.
This section describes each of these frameworks briefly, compares them in terms of following requirements and elaborates on why it was a fairly easy decision for choosing Cucumber and what makes it a suitable test framework for functional tests of application’s frontend.There were two main the most desired features while looking for a test framework for automated UI functional tests:Support for Behaviour Driven Development (BDD).Ease of reading and understanding tests by non programmers. The reason for that is to initiate collaboration between business facing people – Penneo client support team and developers.Support for browser automation utilities is also a requirement as the testing framework is supposed to work alongside with it. (Ref link to: 2.2 Requirement Specification)MochaStarting from arguably the most popular (http://www.
It got the attention while deciding upon what testing framework should be used for UI tests mostly because of the fact that it supports BDD and because of its popularity and huge developer community. Mocha has been used to test everything under the sun, including functional UI testing so the decision to take a closer look at this framework from the perspective of functional testing was made. Mocha as a biggest testing framework is compatible with most (if not all of them) browser automation tools like Selenium, Protractor, WebdriverIO, etc.By default mocha provides BDD style interface, which specifies a way to structure the code into test suites and test cases modules for execution and for the test report.
Mocha’s tests suites are defined by the `describe` clause and the test case is denoted by the `it` clause. Both clauses accept callback functions and can nest inside each other, which means that one test suite can have many inner test suites which can either contain another test suites or the actual test cases. The API provides developers with hooks like: `before()`, `after()`, `beforeEach()`, and `afterEach()` which are used for setup and teardown of the test suites and test cases. Mocha allows also to use any library for assertions such as ChaiJS or Should.js which are great BDD assertion libraries. (https://mochajs.org/#assertions)This is very useful for unit tests or for functional tests that are going to be seen only by developers, but may not be preferred format for business-facing users.
Test specification mixes up with the actual implementation of tests and it is not an especially friendly approach for non programmers and could actually complicate communication between both sides.JasmineFrom this point of view Jasmine(https://jasmine.github.io/) is very similar to Mocha, it is also perfect for unit testing. It is compatible with many browser automation tools. Similarly to Mocha its tests are also supporting BDD style written tests.
(https://jasmine.github.io/tutorials/your_first_suite) This is what both of these frameworks have in common.The `describe` clause defines a test suite and the `it` implements the BDD method that is describing the behaviour that a function should exhibit.Test case denoted by the `it` clause is build in the same way as in case of Mocha. Its first parameter is a description of the behaviour under test written in plain-English and the second parameter is the implemented function which calls assertions that either pass or fail, depending on whether the described behaviour was, or was not confirmed.Both of them are fine choices if separation of the textual specification that everyone can read and write, from the technical implementation is not required.
Due to not meeting this requirement Jasmine same as Mocha has been dismissed as inadequate.CucumberCucumber(https://cucumber.io/) is more user story based testing framework which allows writing tests in BDD style.It targets higher levels of abstraction. Instead of writing tests purely in code like it is done in Mocha or Jasmine, Cucumber separates tests into human-readable user stories and the code that runs them. Cucumber uses Gherkin language to write its tests.
Gherkin is a Business Readable, Domain Specific Language (https://martinfowler.com/bliki/BusinessReadableDSL.html) that makes it possible to describe software’s behaviour without going in to details of how that behaviour is implemented. Since the tests are written in a plain language and anyone on the team can read them that improves communication and collaboration of the whole team.Cucumber tests are constructed in a completely different way than a typical Mocha tests, and that construction makes them so unique. In Mocha there are wrapping clauses that nest inside of each other, mixing up test scenario description with their code implementation.
They are more like traditional test functions that one would write in Mocha or Jasmine, except they are used to match the scenarios. They handle the whole logical operations behind the `steps`, they execute them.The division that Cucumber introduces brings a great value for the business. The Features are written in a natural language therefore business people can read them easily, give developers early feedback or even write some themselves. Another great benefit of having functional tests made with cucumber is that those tests provide a living documentation, describing how the UI interface is supposed to work for each specific feature.4.
The simplicity, reliability, ease of setting up and readable syntax were another requirements for desired framework.Another highly desired feature was a broad support for different browsers. Support for the real browsers was pretty obvious in the business of browser testing frameworks.Important was that the browsers for tests would be cloud based to ensure all the UI components would render as expected in many different browsers in their various versions, even some of the pretty old ones.
Installing and working with all of them on the virtual machines would get annoying and inconvenient very quickly.Important was also a support for the headless browsers like PhantomJS or Headless Chrome, as execution of tests using these browsers makes the development of tests much more convenient as developer does not need to watch multiple browser windows reload, change and so on. Fortunately it turned out that all of the most popular browser testing frameworks already support these features.A key factor was also a support from the developer community to make sure that the framework that would be chosen will stay popular for another few years and there will not be a need for changing it. Support of a wide community also brings benefits in terms of new useful plugins or extensions and makes it easier to find answers on websites like StackOverflow for inevitable problems and errors. Another thing is that working with tools that are not very popular does not seem very appealing.
js implementation for W3C WebDriver API to perform commands and assertions on DOM elements.(https://www.w3.org/TR/webdriver/) In other words WDIO is a testing utility enabling writing automated selenium tests by providing a variety of tools for controlling a browser or a mobile application.
WebdriverIO provides entirely different and smarter bindings for the WebDriver API in comparison to Selenium’s which is the most popular browser automation library on the market and its Node.js implementation is called WebDriverJS. Naming around these technologies can get a little bit tricky and confusing since they all use a similar names.Figure ?: WebdriverIO, Selenium server and Chrome test communication flowWebdriverIO, WebDriverJS and Nightwatch communicate over a restful HTTP API with a WebDriver server (usually the Selenium server).
The restful API is defined for all of them by the W3C WebDriver API. (source)Recently WebdriverIO had a major version change with a rewrite of the whole code base into ES2016. As the project community is growing dynamically the project by itself and its plugins are also being developed rapidly.
It consists of a set of bindings for ProtractorProtractor(http://www.protractortest.org/#/api) has been dismissed as inadequate as a first one, because even though it has support for the same test runners as WebdriverIO and pretty good reporting mechanism its main focus is to test Angular applications as it has build in support for AngularJS element identification strategies.(https://www.protractortest.org/#/api?view=ElementFinder) Also unlike all of the other compared frameworks it is only implemented as a wrapper for WebDriverJS which creates one more layer in between Selenium and the Protractor system tests (https://www.protractortest.
org/#/). It means that its highly dependent on WebDriverJS project, so whenever there is an issue with a WebDriverJS, Protractor needs to wait for the WebDriverJS team to fix that.NightWatchNightWatch(http://nightwatchjs.org/) is very similar to WebdriverIO since it is also a custom implementation of W3C WebDriver API – works with the Selenium standalone server. An interesting feature in this tool is that it has its own testing framework, the only external testing framework that is also supported is Mocha. One may see this as an advantage as it solves a headache of choosing a testing framework especially that it also implements its own assertions mechanism. However in our particular case the lack of support for Cucumber framework was a major disadvantage and main reason for not choosing NightWatch, the reasoning for that will be explained further in this paper.
In comparison with WebdriverIO, Protractor and WebDriverJS it has slightly less support which was also an important factor when comparing these technologies. An important thing to point out is that all of the mentioned system automation tools support cloud browser testing providers, such as SauceLabs and BrowserStack which is a very desired feature. Those services allow developers to run their automated tests in cloud based browsers to ensure websites run flawlessly on every browser, operating system and device.
Both of them offer a wide range of operating systems and browser combinations across desktop and mobile platforms in almost all ever released versions.This solution gets rid of a cumbersome issue of having many actual devices, or virtual machines with specific operating systems and browsers installed.WebdriverIO after being split up into multiple packages has to have different necessary packages installed separately which makes it the most flexible and customizable one among all of them. Important thing to point out is that even though it has to have its separate packages installed it is still the easiest one to configure and set up due to its unique command line interface that makes test configuration as easy and simple as possible.Last but not least great feature that WebdriverIO has acquired to its project is that it is possible to set framework for test runner.