7. Settings¶
Here is a summary of 2 entities of cross-browser-tests-runrer created towards the aim of providing a uniform/standard interface across various cross-browser testing platforms:
- As one can see in Platform Configuration, since different cross-browser testing platform use different names for browsers and OSes, cross-browser-tests-runner helps your write your tests using uniform browser and platform name aliases, and would convert them to platform-specific names internally.
- To improve efficiency in writing your test configuration, you write browsers involved in your tests in a Browsers YAML in a smart and compact format, using the uniform aliases from above.
The ‘Settings File’ is the third link of the above chain. It is a JSON format file that is generated based on the Browsers YAML file (using cbtr-init). It eventually contains other details of your tests e.g. unit testing framework, the test file paths etc. It serves the following purposes:
- Provides test settings for the in-built JavaScript unit tests and Selenium tests runner
- Serves as the common input file to generate multiple third party unit tests runner settings file e.g. testem.json, using executable binaries provided by cross-browser-tests-runner. Since each third-party unit tests runner has their own configuration syntax, generating each of them from a common settings file helps reducing efforts.
7.1. Format¶
7.1.1. Examples¶
7.1.1.1. JavaScript Unit Testing¶
{
"framework": "jasmine",
"test_file": "tests/functional/code/tests/jasmine/html/tests.html",
"retries": 1,
"limit": "4mb",
"browsers": {
"BrowserStack": {
"JS": [
{
"os": "Windows",
"osVersion": "7",
"browser": "Chrome",
"browserVersion": "32.0"
},
{
"os": "iOS",
"osVersion": "6.0",
"browser": "Mobile Safari",
"browserVersion": null,
"device": "iPad 3rd (6.0)"
} ]
},
"SauceLabs": {
"JS": [
{
"os": "Windows",
"osVersion": "7",
"browser": "Chrome",
"browserVersion": "32.0"
},
{
"os": "OS X",
"osVersion": "Sierra",
"browser": "Mobile Safari",
"browserVersion": "10.3"
}
]
},
"CrossBrowserTesting": {
"JS": [
{
"os": "Android",
"osVersion": "7.0",
"browser": "Dolphin Mobile",
"browserVersion": "12.0"
},
{
"os": "OS X",
"osVersion": "Sierra",
"browser": "Safari",
"browserVersion": "10.0"
}
]
}
},
"capabilities": {
"BrowserStack": {
"local": true,
"localIdentifier": "native-functional-tests",
"build": "native-runner-build",
"test": "native-runner-functional-test",
"project": "cross-browser-tests-runner/cross-browser-tests-runner",
"screenshots": true
},
"SauceLabs": {
"local": true,
"localIdentifier": "native-functional-tests",
"build": "native-runner-build",
"test": "native-runner-functional-test",
"timeout": 90,
"screenshots": true
},
"CrossBrowserTesting": {
"local": true,
"localIdentifier": "native-functional-tests",
"build": "native-runner-build",
"test": "native-runner-functional-test",
"project": "cross-browser-tests-runner/cross-browser-tests-runner",
"screenshots": true,
"timeout": 90
}
},
"server": {
"port": 8000,
"host": "127.0.0.1"
},
"parallel": {
"BrowserStack": 2,
"SauceLabs": 5,
"CrossBrowserTesting": 5
}
}
7.1.1.2. Selenium Testing¶
{
"test_file": "tests/functional/code/tests/selenium/html/tests.html",
"test_script": "tests/functional/code/scripts/selenium/script-1.js",
"browsers": {
"BrowserStack": {
"Selenium": [
{
"os": "OS X",
"osVersion": "Mavericks",
"browser": "Firefox",
"browserVersion": "39.0"
},
{
"os": "Android",
"osVersion": "4.0",
"browser": "Android Browser",
"browserVersion": null,
"device": "Motorola Razr"
}
]
},
"SauceLabs": {
"Selenium": [
{
"os": "OS X",
"osVersion": "Mavericks",
"browser": "Chrome",
"browserVersion": "33.0"
},
{
"os": "Android",
"osVersion": "5.0",
"browser": "Android Browser",
"browserVersion": null,
"device": "Android Emulator"
},
{
"os": "iOS",
"osVersion": "8.3",
"browser": "Safari",
"browserVersion": null,
"device": "iPhone 6 Plus Simulator"
}
]
},
"CrossBrowserTesting": {
"Selenium": [
{
"os": "OS X",
"osVersion": "Yosemite",
"browser": "Firefox",
"browserVersion": "39.0"
},
{
"os": "Windows",
"osVersion": "8.1",
"browser": "Internet Explorer",
"browserVersion": "11.0"
}
]
}
},
"capabilities": {
"BrowserStack": {
"local": true,
"localIdentifier": "native-functional-tests",
"build": "native-runner-build",
"test": "native-runner-functional-test",
"project": "cross-browser-tests-runner/cross-browser-tests-runner",
"screenshots": true,
"timeout": 120
},
"SauceLabs": {
"local": true,
"localIdentifier": "native-functional-tests",
"build": "native-runner-build",
"test": "native-runner-functional-test",
"project": "cross-browser-tests-runner/cross-browser-tests-runner",
"timeout": 120
},
"CrossBrowserTesting": {
"local": true,
"localIdentifier": "native-functional-tests",
"build": "native-runner-build",
"test": "native-runner-functional-test",
"project": "cross-browser-tests-runner/cross-browser-tests-runner",
"screenshots": true,
"timeout": 120
}
},
"server": {
"port": 7883,
"host": "127.0.0.1"
},
"parallel": {
"BrowserStack": 2,
"SauceLabs": 5,
"CrossBrowserTesting": 5
}
}
7.1.2. Parameters¶
Parameter | Applies To | Description | Possible Values | Default |
---|---|---|---|---|
framework |
JavaScript unit tests using the in-built native runner | It is the name of the JavaScript unit tests framework used in your tests | jasmine |
jasmine |
retries |
JavaScript unit tests using the in-built native runner | Number of retries to try a test once it fails (See Test Retries) | >=0 | 0 |
limit |
JavaScript unit tests using the in-built native runner | Size limit of data (test reports, and code coverage data) sent by a browser to accept | See Request Size Limit | "4mb" |
test_file |
JavaScript unit tests and Selenium tests using the in-built native runner | Path of test HTML file(s), must be relative to root directory of your project (See Test HTML for details) | A string or an array of strings - one per test html file | |
test_script |
Selenium tests | Path of Selenium test script(s), must be relative to root directory of your project (See Test Script for details) | A string or an array of strings - one per test script file | |
browsers |
All testing | The set of browsers to use for your tests - generated by cbtr-init based on browsers specified in your Browsers YAML file | ||
capabilities |
All testing | Testing “capabilities” other than browser details - generated with defaults by cbtr-init | See capabilities | See capabilities |
server |
All testing | Server host and port - generated with defaults by cbtr-init | See server | See server |
parallel |
All testing | Number of sessions to run in parallel on a cross-browser testing platform - generated with defaults by cbtr-init | See parallel | See parallel |
7.1.2.1. capabilities¶
Parameter | Platforms | Description | Possible Values | Default |
---|---|---|---|---|
local |
All | Enforces testing of local pages
if set to true |
true , false |
true |
localIdentifier |
All | Identifier for tunnel used for local testing | string type |
|
screenshots |
All | Enables taking screenshots if set
to true , disables if set to
false |
true , false |
true for BrowserStack |
video |
All | Enables capturing a video of test
if set to true , disables if
set to false |
true , false |
|
timeout |
All | Timeout for a browser/test session in seconds (note: BrowserStack has a minimum 60s timeout requirement) | >0 | 120 |
project |
BrowserStack |
username/repo slug of your
project, automatically set if
running on Travis, Circle or
Appveyor |
string type |
"anonymous/anonymous" |
build |
All | build id for your tests, automatically set if running on Travis, Circle of Appveyor to commit SHA1 | string type |
Output of git rev-parse HEAD |
test |
All | Name for your test session (applies to all browser sessions in your tests), automatically set to a unique identifier if running on Travis, Circle or Appveyor | string type |
A uuid |
tags |
SauceLabs |
An array of arbitrary tags for a test | array type |
|
customData |
SauceLabs |
An object with arbitrary key values | object type |
|
priority |
SauceLabs |
To assign higher/lower priority to a test as compared to others | number type |
|
parentTunnel |
SauceLabs |
While using sub-accounts, use this to use parent user’s tunnel | string type |
|
noServerFailureScreenshots |
SauceLabs |
Do not take screenshots of selenium script failure points | true , false |
true |
videoUploadOnPass |
SauceLabs |
Upload video even if a test passes | true , false |
true |
seleniumVersion |
BrowserStack SauceLabs |
Selenium version to use | string type |
|
appiumVersion |
BrowserStack SauceLabs |
Appium version to use | string type |
|
timezone |
BrowserStack SauceLabs |
Time zone to use for a test | string type |
|
captureConsole |
BrowserStack |
Capture console logs of a test | string type |
|
captureNetwork |
BrowserStack
CrossBrowserTesting |
Capture network packets of a test | true , false |
false |
captureLogs |
SauceLabs |
Record logs of a test | true , false |
true |
captureHtml |
SauceLabs |
Capture HTML output of a test | true , false |
false |
ieNoFlash |
BrowserStack |
Do not use Flash in Internet Explorer | true , false |
false |
ieDriver |
BrowserStack SauceLabs |
Version of Internet Explorer webdriver | string type |
|
ieCompat |
BrowserStack |
Compatibility level of Internet Explorer | number type |
|
iePopups |
BrowserStack |
Enable pop-ups in Internet Explorer | true , false |
false |
edgePopups |
BrowserStack |
Enable pop-ups in Edge | true , false |
false |
safariPopups |
BrowserStack |
Enable pop-ups in Safari | true , false |
false |
safariAllCookies |
BrowserStack |
Allow all cookies in Safari | true , false |
true |
safariDriver |
BrowserStack |
Version of Safari webdriver | string type |
|
geckoDriver |
BrowserStack |
Version of gecko (Firefox) driver | string type |
|
chromeDriver |
SauceLabs |
Version of Chrome webdriver | string type |
|
automationEngine |
SauceLabs |
Automation engine to use on devices | string type |
Appium |
autoAcceptAlerts |
SauceLabs |
Automatically accept JavaScript created alerts | true , false |
false |
prerun |
SauceLabs |
An object that describes an executable to run before the test | object type |
7.1.2.2. server¶
Parameter | Description | Possible Values | Default |
---|---|---|---|
host |
The IP address on which the test server listens | ip address, host name | "127.0.0.1" |
port |
The port on which the test server listens | number type |
7982 |
No changes in this section are needed unless:
- You need to connect to the server from a separate machine (probably
for your local testing) and not using a tunnel. In such case you may
want to change it to
"0.0.0.0"
. - The port
7982
is in use by some other process
7.1.2.3. parallel¶
Parameter | Description | Possible Values | Default |
---|---|---|---|
BrowserStack |
Number of sessions that can be run in parallel on BrowserStack | >0 | 2 |
SauceLabs |
Number of sessions that can be run in parallel on SauceLabs | >0 | 5 |
CrossBrowserTesting |
Number of sessions that can be run in parallel on CrossBrowserTesting | >0 | 5 |
7.2. Multiple Copies¶
You can have more than one test settings files, if you need to break your work down into multiple tests.
7.3. Samples¶
./node_modules/cross-browser-tests-runner/samples/cbtr/*/*.json