7. Settings

Here is a summary of 2 entities of cross-browser-tests-runrer created towards the aim of providing a uniform/standard interface across various cross-browser testing platforms:

  1. As one can see in Platform Configuration, since different cross-browser testing platform use different names for browsers and OSes, cross-browser-tests-runner helps your write your tests using uniform browser and platform name aliases, and would convert them to platform-specific names internally.
  2. To improve efficiency in writing your test configuration, you write browsers involved in your tests in a Browsers YAML in a smart and compact format, using the uniform aliases from above.

The ‘Settings File’ is the third link of the above chain. It is a JSON format file that is generated based on the Browsers YAML file (using cbtr-init). It eventually contains other details of your tests e.g. unit testing framework, the test file paths etc. It serves the following purposes:

  1. Provides test settings for the in-built JavaScript unit tests and Selenium tests runner
  2. Serves as the common input file to generate multiple third party unit tests runner settings file e.g. testem.json, using executable binaries provided by cross-browser-tests-runner. Since each third-party unit tests runner has their own configuration syntax, generating each of them from a common settings file helps reducing efforts.

7.1. Format

7.1.1. Examples

7.1.1.1. JavaScript Unit Testing

{
  "framework": "jasmine",
  "test_file": "tests/functional/code/tests/jasmine/html/tests.html",
  "retries": 1,
  "limit": "4mb",
  "browsers": {
    "BrowserStack": {
      "JS": [
        {
          "os": "Windows",
          "osVersion": "7",
          "browser": "Chrome",
          "browserVersion": "32.0"
        },
        {
          "os": "iOS",
          "osVersion": "6.0",
          "browser": "Mobile Safari",
          "browserVersion": null,
          "device": "iPad 3rd (6.0)"
        }      ]
    },
    "SauceLabs": {
      "JS": [
        {
          "os": "Windows",
          "osVersion": "7",
          "browser": "Chrome",
          "browserVersion": "32.0"
        },
        {
          "os": "OS X",
          "osVersion": "Sierra",
          "browser": "Mobile Safari",
          "browserVersion": "10.3"
        }
      ]
    },
    "CrossBrowserTesting": {
      "JS": [
        {
          "os": "Android",
          "osVersion": "7.0",
          "browser": "Dolphin Mobile",
          "browserVersion": "12.0"
        },
        {
          "os": "OS X",
          "osVersion": "Sierra",
          "browser": "Safari",
          "browserVersion": "10.0"
        }
      ]
    }
  },
  "capabilities": {
    "BrowserStack": {
      "local": true,
      "localIdentifier": "native-functional-tests",
      "build": "native-runner-build",
      "test": "native-runner-functional-test",
      "project": "cross-browser-tests-runner/cross-browser-tests-runner",
      "screenshots": true
    },
    "SauceLabs": {
      "local": true,
      "localIdentifier": "native-functional-tests",
      "build": "native-runner-build",
      "test": "native-runner-functional-test",
      "timeout": 90,
      "screenshots": true
    },
    "CrossBrowserTesting": {
      "local": true,
      "localIdentifier": "native-functional-tests",
      "build": "native-runner-build",
      "test": "native-runner-functional-test",
      "project": "cross-browser-tests-runner/cross-browser-tests-runner",
      "screenshots": true,
      "timeout": 90
    }
  },
  "server": {
    "port": 8000,
    "host": "127.0.0.1"
  },
  "parallel": {
    "BrowserStack": 2,
    "SauceLabs": 5,
    "CrossBrowserTesting": 5
  }
}

7.1.1.2. Selenium Testing

{
  "test_file": "tests/functional/code/tests/selenium/html/tests.html",
  "test_script": "tests/functional/code/scripts/selenium/script-1.js",
  "browsers": {
    "BrowserStack": {
      "Selenium": [
        {
          "os": "OS X",
          "osVersion": "Mavericks",
          "browser": "Firefox",
          "browserVersion": "39.0"
        },
        {
          "os": "Android",
          "osVersion": "4.0",
          "browser": "Android Browser",
          "browserVersion": null,
          "device": "Motorola Razr"
        }
      ]
    },
    "SauceLabs": {
      "Selenium": [
        {
          "os": "OS X",
          "osVersion": "Mavericks",
          "browser": "Chrome",
          "browserVersion": "33.0"
        },
        {
          "os": "Android",
          "osVersion": "5.0",
          "browser": "Android Browser",
          "browserVersion": null,
          "device": "Android Emulator"
        },
        {
          "os": "iOS",
          "osVersion": "8.3",
          "browser": "Safari",
          "browserVersion": null,
          "device": "iPhone 6 Plus Simulator"
        }
      ]
    },
    "CrossBrowserTesting": {
      "Selenium": [
        {
          "os": "OS X",
          "osVersion": "Yosemite",
          "browser": "Firefox",
          "browserVersion": "39.0"
        },
        {
          "os": "Windows",
          "osVersion": "8.1",
          "browser": "Internet Explorer",
          "browserVersion": "11.0"
        }
      ]
    }
  },
  "capabilities": {
    "BrowserStack": {
      "local": true,
      "localIdentifier": "native-functional-tests",
      "build": "native-runner-build",
      "test": "native-runner-functional-test",
      "project": "cross-browser-tests-runner/cross-browser-tests-runner",
      "screenshots": true,
      "timeout": 120
    },
    "SauceLabs": {
      "local": true,
      "localIdentifier": "native-functional-tests",
      "build": "native-runner-build",
      "test": "native-runner-functional-test",
      "project": "cross-browser-tests-runner/cross-browser-tests-runner",
      "timeout": 120
    },
    "CrossBrowserTesting": {
      "local": true,
      "localIdentifier": "native-functional-tests",
      "build": "native-runner-build",
      "test": "native-runner-functional-test",
      "project": "cross-browser-tests-runner/cross-browser-tests-runner",
      "screenshots": true,
      "timeout": 120
    }
  },
  "server": {
    "port": 7883,
    "host": "127.0.0.1"
  },
  "parallel": {
    "BrowserStack": 2,
    "SauceLabs": 5,
    "CrossBrowserTesting": 5
  }
}

7.1.2. Parameters

Parameter Applies To Description Possible Values Default
framework JavaScript unit tests using the in-built native runner It is the name of the JavaScript unit tests framework used in your tests jasmine jasmine
retries JavaScript unit tests using the in-built native runner Number of retries to try a test once it fails (See Test Retries) >=0 0
limit JavaScript unit tests using the in-built native runner Size limit of data (test reports, and code coverage data) sent by a browser to accept See Request Size Limit "4mb"
test_file JavaScript unit tests and Selenium tests using the in-built native runner Path of test HTML file(s), must be relative to root directory of your project (See Test HTML for details) A string or an array of strings - one per test html file  
test_script Selenium tests Path of Selenium test script(s), must be relative to root directory of your project (See Test Script for details) A string or an array of strings - one per test script file  
browsers All testing The set of browsers to use for your tests - generated by cbtr-init based on browsers specified in your Browsers YAML file    
capabilities All testing Testing “capabilities” other than browser details - generated with defaults by cbtr-init See capabilities See capabilities
server All testing Server host and port - generated with defaults by cbtr-init See server See server
parallel All testing Number of sessions to run in parallel on a cross-browser testing platform - generated with defaults by cbtr-init See parallel See parallel

7.1.2.1. capabilities

Parameter Platforms Description Possible Values Default
local All Enforces testing of local pages if set to true true, false true
localIdentifier All Identifier for tunnel used for local testing string type  
screenshots All Enables taking screenshots if set to true, disables if set to false true, false true for BrowserStack
video All Enables capturing a video of test if set to true, disables if set to false true, false  
timeout All Timeout for a browser/test session in seconds (note: BrowserStack has a minimum 60s timeout requirement) >0 120
project BrowserStack username/repo slug of your project, automatically set if running on Travis, Circle or Appveyor string type "anonymous/anonymous"
build All build id for your tests, automatically set if running on Travis, Circle of Appveyor to commit SHA1 string type Output of git rev-parse HEAD
test All Name for your test session (applies to all browser sessions in your tests), automatically set to a unique identifier if running on Travis, Circle or Appveyor string type A uuid
tags SauceLabs An array of arbitrary tags for a test array type  
customData SauceLabs An object with arbitrary key values object type  
priority SauceLabs To assign higher/lower priority to a test as compared to others number type  
parentTunnel SauceLabs While using sub-accounts, use this to use parent user’s tunnel string type  
noServerFailureScreenshots SauceLabs Do not take screenshots of selenium script failure points true, false true
videoUploadOnPass SauceLabs Upload video even if a test passes true, false true
seleniumVersion BrowserStack SauceLabs Selenium version to use string type  
appiumVersion BrowserStack SauceLabs Appium version to use string type  
timezone BrowserStack SauceLabs Time zone to use for a test string type  
captureConsole BrowserStack Capture console logs of a test string type  
captureNetwork BrowserStack CrossBrowserTesting Capture network packets of a test true, false false
captureLogs SauceLabs Record logs of a test true, false true
captureHtml SauceLabs Capture HTML output of a test true, false false
ieNoFlash BrowserStack Do not use Flash in Internet Explorer true, false false
ieDriver BrowserStack SauceLabs Version of Internet Explorer webdriver string type  
ieCompat BrowserStack Compatibility level of Internet Explorer number type  
iePopups BrowserStack Enable pop-ups in Internet Explorer true, false false
edgePopups BrowserStack Enable pop-ups in Edge true, false false
safariPopups BrowserStack Enable pop-ups in Safari true, false false
safariAllCookies BrowserStack Allow all cookies in Safari true, false true
safariDriver BrowserStack Version of Safari webdriver string type  
geckoDriver BrowserStack Version of gecko (Firefox) driver string type  
chromeDriver SauceLabs Version of Chrome webdriver string type  
automationEngine SauceLabs Automation engine to use on devices string type Appium
autoAcceptAlerts SauceLabs Automatically accept JavaScript created alerts true, false false
prerun SauceLabs An object that describes an executable to run before the test object type  

7.1.2.2. server

Parameter Description Possible Values Default
host The IP address on which the test server listens ip address, host name "127.0.0.1"
port The port on which the test server listens number type 7982

No changes in this section are needed unless:

  • You need to connect to the server from a separate machine (probably for your local testing) and not using a tunnel. In such case you may want to change it to "0.0.0.0".
  • The port 7982 is in use by some other process

7.1.2.3. parallel

Parameter Description Possible Values Default
BrowserStack Number of sessions that can be run in parallel on BrowserStack >0 2
SauceLabs Number of sessions that can be run in parallel on SauceLabs >0 5
CrossBrowserTesting Number of sessions that can be run in parallel on CrossBrowserTesting >0 5

7.2. Multiple Copies

You can have more than one test settings files, if you need to break your work down into multiple tests.

7.3. Samples

./node_modules/cross-browser-tests-runner/samples/cbtr/*/*.json