Cucumber and TestRail salad with Ruby dressing

You have finally completed your Automation Test Framework that was designed to save tons of manual labor for your company by automating repetitive and time-consuming test cases. Your fellow teammates are definitely happy that it is finally available for their disposal but it turns out that it is never being used as it was meant to. The problem lies in presenting the test results to your target audience. Everyone has plenty of their own problems and tasks to take on and no time to try to dig deeper to understand test results that are hard-to-read or unclear.
Now, let's talk how to make stuff you wrote useful for everyone!

There are plenty of tools to manage, track and organize software testing efforts. Here at Vevo we picked TestRail from http://www.gurock.com. Its flexibility allows you to run as many tests on as many devices as you want. TestRail is well structured and if used correctly becomes a really powerful tool for QA folks, developers and management. TestRail has all you need to be integrated into your test framework. Public APIs and bindings are available here: http://docs.gurock.com/testrail-api2/start. I ended up writing my own TestRail wrapper to integrate its functionality to the framework. If you are using Ruby, I would highly recommend rewriting the TestRail API client using “Typhoeus" rather than using good old "net/http” gem. "Typhoeus" outperforms "net/http(s)" library by a good 30-40 percent, which definitely helps to keep overall test execution time down.

Here is my implementation of the client using Typhoeus gem:


require 'typhoeus'
require 'uri'
require 'base64'

module TestRail
    class TyphoeusAPIClient

        @url      = ''
        @user     = ''
        @password = ''

        attr_accessor :user
        attr_accessor :password

        def initialize(base_url)
            @url = base_url + '/' + 'index.php?/api/v2/'
        end

        def send_get(uri)
            _send_request method: 'GET',
                                        uri:     uri
        end

        def send_post(uri, data)
            _send_request method: 'POST',
                                        uri:     uri,
                                        data:    data
        end

        def _send_request(opts = {})
            api_method = opts.fetch :method
            uri        = opts.fetch :uri
            data       = opts.fetch(:data, nil)
            url        = File.join(@url, uri)

            request  = Typhoeus::Request.new(
                    url,
                    method:  api_method.downcase.to_sym,
                    body:    JSON.dump(data),
                    headers: { 'Content-Type'  => 'application/json',
                                         'Authorization' => "Basic #{Base64.strict_encode64("#{@user}:#{@password}")}" }
            )
            response = request.run

            if response.code == 504
                Log.error "504 Gateway Time-out. The server didn't respond in time."
                nil
            else
                begin
                    JSON.parse(response.body)
                rescue JSON::ParserError
                    response.body
                end
            end
        end

    end
end

Here is structure of the TestRail wrapper:

Each file in the lib directory contains methods to handle each part of the TestRail.
For example here is content of my plan.rb file:


module TestRail
  class Client

    module Plan

      #
      # test_plan_exist?               Method checks if Given Test Plan already exists
      #
      # Arguments:
      # plan_name                      String representation of Test Plan Name (e.g. 'TEST_Plan')
      # project_id                     ID of given Project
      #
      # Return:                        true/false
      #
      def plan_exist?(plan_name, project_id)
        plans = @client.send_get("get_plans/#{project_id}")
        plans.each { |plan|
          return true if plan.fetch('name') == plan_name
        }
        false
      end


      #
      # add_test_plan                  Method adds new test plan to already existing Project
      #
      # Arguments:
      # project_id                     ID of given project
      # data                           The data to submit as part of the request (as
      #                                Ruby hash, strings must be UTF-8 encoded)
      #                                Data must contain following fields:
      #                                                                   name         (string) The name of the test plan (required)
      #                                                                   description  (string) The description of the test plan
      #                                                                   milestone_id (int)    The ID of the milestone to link to the test plan
      #                                                                   entries      (array)  An array of objects describing the test runs of the plan
      #
      def add_plan(project_id, data)
        @client.send_post("add_plan/#{project_id}", data)
      end


      def build_plan_data(opts = {})
        {
            name:        opts.fetch(:_name),
            description: opts.fetch(:_description)
        }
      end


      #
      # get_test_plan_id_by_name       Method obtains Test Plan ID by its name
      #
      # Arguments:
      # plan_name                      String representation of Test Plan Name (e.g. 'TEST_Plan')
      # project_id                     ID of given Project
      #
      # Return:                        plan_id if plan exists or nil if it doesn't
      #
      def get_plan_id_by_name(plan_name, project_id)
        plans = @client.send_get("get_plans/#{project_id}")
        plan  = plans.select { |plan| plan if plan.fetch('name') == plan_name }.first
        if plan.nil?
          Log.debug "Plan with name '#{plan_name}' does not exist"
          nil
        else
          plan.fetch 'id'
        end
      end




      #
      # delete_test_plan                Method obtains Test Plan ID by its name
      #
      # Arguments:
      # plan_id                        ID of given Test Plan
      # data                           TestRail#send_post() requires data argument to be passed in.
      #                                Default value is set to "empty string"
      #
      def delete_plan(plan_id, data = '')
        if plan_id.nil?
          Log.warn 'There is no Plan to be deleted..'
        else
          @client.send_post("add_plan/#{plan_id}", data)
        end
      end


      #
      # plan_entry_exist?              Method checks if Plan Entry already exists by searching for a certain Entry Name among all the entries that belong to given Plan
      #
      # Arguments:
      # plan_id                        ID of given Test Plan
      # enrty_name                     Name of the Plan entry
      # Return: [boolean]              true/false
      #
      def plan_entry_exist?(plan_id, enrty_name)
        if plan_id.nil?
          Log.warn 'Plan does not exist'
        else
          plan_entries = @client.send_get("get_plan/#{plan_id}")
          entries      = plan_entries.fetch('entries')
          entries.each { |entry|
            runs = entry.fetch('runs')
            runs.each { |run|
              return true if run.fetch('name') == enrty_name
            }
          }
          false
        end
      end


      def add_plan_entry(plan_id, data)
        if plan_id.nil?
          Log.warn 'Entry could not be added, Plan does not exist'
          nil
        else
          @client.send_post("add_plan_entry/#{plan_id}", data)
        end
      end


      def build_entry_data(opts = {})
        if opts.fetch(:_suite_id).nil?
          Log.warn 'Test Suite does not exist and therefore, Entry Data cannot be built'
          nil
        else
          {
              suite_id:    opts.fetch(:_suite_id),
              name:        opts.fetch(:_name, "#{ENV.fetch('DEVICE_NAME')}_#{ENV.fetch('ENVIRONMENT')}"),
              include_all: opts.fetch(:_include_all, true)
          }
        end
      end


      def get_entry_properties(plan_id)
        if plan_id.nil?
          Log.warn 'Plan does not exist'
        else
          entry_properties = {}
          plan_entries     = @client.send_get("get_plan/#{plan_id}")
          entries          = plan_entries.fetch('entries')
          entries.each { |entry|
            runs = entry.fetch('runs')
            runs.each { |run|
              run_name = run.delete('name')
              entry_properties[run_name.to_sym] = run
            }
          }
          entry_properties
        end
      end


    end
  end
end

The rest of the classes/modules in the wrapper have similar functionality for corresponding parts of the TestRail.

When and where to push all the test results to TestRail?

There are could be quite a few option when and where in your code you can place TestRail integration logic. In Cucumber based frameworks if it is set to generate JUnit report XML files, those could be parsed after the test runs then extracted and pushed to TestRail. In this case you will be dealing with parsing quite a large XML file that contains only the name of the test case followed by Pass/Fail result. I chose to use Cucumber After Hook to insert all the TestRail logic. Cucumber provides a number of hooks which allow the user to run blocks of code at various points in the Cucumber test cycle (https://github.com/cucumber/cucumber/wiki/Hooks). The hook is executed right after last step of each test case; it will also be triggered if test fails. This approach definitely has several advantages over parsing the XML JUnit report:
In After Hook you can get a reference to Cucumber Scenario Object that contains test result information such as: passed or failed, scenario steps, exception backtrace if failed.

If the scenario failed and you have a rerun logic implemented in your Framework this scenario will be ran again. If it passes during the rerun sequence After Hook code will be triggered again and new results will be pushed to TestRail.

How to attach files to TestRail?

Another issue that came up during the integration process was a lack of the public API for TestRail that could be used for attaching files (screenshots and log files) to the test result. It could be done manually, but it is not the solution what I was looking for. The solution was found using a Google Drive to store actual screenshots or log files. There is a CLI gdrive (https://github.com/prasmussen/gdrive) project that is available as open source which allows you to upload/download files to the Google Drive cloud from the command line. By sharing those files to be available for everyone, this link could be passed to the test result as a String parameter. Then by the time the test is over, if it failed, a screenshot would be automatically attached to the current TestRail result.
alt

The results

As a result right after Jenkins Test job is over early in the morning all the test results are available on TestRail (what passed, what failed, what was untested and, could not be automated and etc.). Folks who are performing tests manually only need to "cherry pick” what failed or was untested rather than perform top-to-bottom test on the entire test suite run with multiple devices. The main purpose of investing the time and money in Automation is to minimize manual testing time and make the entire testing process as easy as possible on manual QA folks. I think that collaboration of automation test frameworks with integrated TestRail bindings in it definitely helps to archive the desired goal.
alt

This article was written by Max Groshev and Sergei Kostin.