Adopting Katalon for UI Tests

Over the last few years my .NET dev shop has experimented with several different software testing automation frameworks.  First we adoptedProtractor, then WebDriver.io and Cypress and now Katalon. You can see that all of these frameworks utilize Selenium framework, Each platform has it’s only strengths (and weaknesses).  I won’t go into the weeds on the strengths and weaknesses of each product, but I do want to tell you how and why of adopting Katalon for UI tests.

Why we chose katalon

• Katalon allows individuals without coding skills to write tests.
• it allows developers to also write tests utilizing Groovy language (a Java enhancer).
• It’s compatible with Sauce Labs
• Open source and free

First off, let’s look at how you can get started with Katalon.

Getting Started with katalon

To get started adopting Katalon for UI tests you can go to Katalon.com and download the free version of Katalon Studio.  There are various tutorials on their website to get you started.  I was able to pick it up easily after a short demo shared by one of my colleagues during a weekly meeting.  Katalong Studio has the ability to pull source down within the desktop app, but I’ve never used that – I’m accustomed to using GitKraken after I got fed up with a few issues with SourceTree.

First, I just played around with the app, asked questions, then our team had a session where we each sat down and added some tests to a new application that was nearly production ready.  After this we created new Katalon repositories for our many other applications – we would pull that repo down, write tests, branch and make a Pull Request utilizing Team Foundation Server (TFS) before it could be merged into master.

I know what you may be thinking – you guys have Pull Requests mandatory – yes, across almost all of our applications.  Each PR has a set of required reviewers (depending on the app), with QA as the final approval before it can be merged into develop or master – depending on the project.  As a matter of doing business we have made quality important, it’s an embedded part of our processes (and culture).  More on the this later, as reviewing a code change is substantially easier than reviewing a Katalon PR.

The separate repositories worked well for the testers who were writing and using the tests – and it worked well when we ran the tests in devlab on the build server.  However, there was a missing element.  Developers couldn’t easily run the tests against their local branches, hence they couldn’t fix any Katalon tests they broke while making changes.  To solve this problem, we’re actively migrating the katalon solution to a folder within the source of each application.

Our new work flow looks like this:

• pull down the repository of the primary application we wish to edit/add tests too
• make changes
• create a branch
• create a pull request

This new process has made been seamless for developers and testers alike.

Test and File Organization

while completely customizable, we have three different primary levels in the test file system:

Test Steps

Allow you to create repeatable testing steps that can then be called into the Test Scenarios.  Examples of test steps:

• • “go to the login page” – provide a url and verify it takes you to the expected place.
  • “login to the user portal” by providing username and password and selecting the Login button.  A few more examples:
  • “logout of the user portal” by selecting the logout button
  • “page navigation” are steps that will allow you build more complex test scenarios.  Navigation are key test steps, some specific examples may be “select user settings” then “two factor authentication” or “change my password” – which would navigate to the appropriate page/s in the application to do those actions. Later when you add more tests, you can use these navigation steps.
  • “verify specific elements exist on each page” – a user with XYZ products may expect to see access to those products.  One way to test this is verify “element is present” for those products.

TEST SCENARIOS

These are test cases that invoke multiple test steps.  An example using the steps below could be:

• • • Change Password
      1. go to login page
      2. login to user portal
      3. navigate to Change Password page
      4. change password
      5. logout
    • Enable Two-Factor Authentication
      1. go to login
      2. login to user portal
      3. navigate to Two-Factor Authentication page
      4. Enable Two-Factor Authentication
      5. logout

Test Suites

Are used to run 2 or more test scenarios together.  One Test Suite may be “regression test” or “full navigation suite”.  A specific example, as used from above:

• • change password
  • enable two-factor authentication

Object Repository

This is the area where page elements are stored.  I recommend having the file system of the Object Repo mirror the Test Files.  For example, you have have Test Steps (as noted above) as login, logout, change password – in the Object repo you could have the same subset of files.  This will help you easily keep track of elements because many elements have similar names (although you can easily rename things).  I also utilized this since we have micro-apps that have the look and feel of one large app.  However, there may be several elements with the word “username” or “password” or “two-factor” – this will help you keep track of which element is on each page (regardless of app name).  So you could have a File for “Change Password Page”, “Two-Factor Authentication Page”, and “Login Page”.

Profiles structure

Global variables live in the Profiles area.  You can add Global Variables (such as URLs, usernames, API calls) that can be linked to one profile.  If you create multiple profiles, you have all those things linked to different environments.  For us, we have profiles for Local, Develop, Staging and Production where Develop is the default profile.  This allows us to write one test for all environments, then invoke Global Variables in the test cases as needed.

Creating Test Cases in katalon

I’ve long manually tested our many applications, so I have a strong working knowledge of the apps I’m writing tests for.  However, manually testing something is different than writing tests that need to be reliable and reuseable.  When adopting Katalon for UI tests, I’ve developed the following organization strategy:

Test Case Organization Strategy

review manual test cases

first, I review any and all manual test cases I have on the feature or app.  This helps me recall the use cases and some finite details that can be overlooked.

create organization to begin writing tests

I ask myself “which of these tests are repeatable or can be reused?” then I write those down.  Then I formulate a methodology of organization for test steps>>scenarios.

write test steps

I will typically use the recorder feature, which records all your clicks then saves all the elements you clicked on or typed in.

organize your elements

In a folder structure similar to your steps and scenarios.  This allows me to quickly find elements regardless of page or application, since we have micro-apps that, to the user, looks like a single app.  note: one very good thing about these end-to-end UI tests is that it allows you to test integration between apps, as well as gives you the opportunity to solely focus on a single app, depending on your needs.

write test scenarios

Pull in the repeatable/reuseable test steps that you wrote previously to create robust test scenarios.

refactor

I typically reach a point of “I should’ve wrote a test case for this” or “I need to break this test case into smaller test cases” so that they can more easily be consumed in the scenarios.

make sure they all work (after any refactoring)

I didn’t mention it before, but I try to write test steps that are going to work for all Profiles – each test step should pass in development, staging and prod.

write test suites

Pull your test scenarios into larger test suites, depending on need. You may have a test case called “regression” or you may organize them by feature so that you can test subsequent changes to your application more easily by running these smaller suites without running the full regression.

demo

At this point I like to demo the tests to my dev team and my testing peers.  They’ll provide feedback that often results in me making some changes to my tests.  And, it gives them an understanding of the tests (because they will submit code changes with all tests passing).

Last Words

My goal was to teach you the why and how of Adopting Katalon for UI Tests.  Katalon can do about anything you want it to do – and it has a great community that offers tips, tricks and suggestions.  There’s also robust documentation available on the website and tons of online videos and tutorials.  I’m still learning the ropes, but I’ve enjoyed few things better than writing automated UI tests in Katalon.  You can read more about my journey to our Dallas office to spread the gospel of Katalon.  I’m looking forward to a long friendship and lots of learning.