Merge pull request #14920 from NissimHadar/21193-updateNitpickReadme

Case 21193: update nitpick readme

tools/nitpick/README.md

@@ -1,271 +1,330 @@
-# nitpick
+# nitpick
-
+
-Nitpick is a stand alone application that provides a mechanism for regression testing.  The general idea is simple: 
+Nitpick is a stand alone application that provides a mechanism for regression testing.  The general idea is simple:  
-* Each test folder has a script that produces a set of snapshots.
+* Each test folder has a script that produces a set of snapshots.  
-* The snapshots are compared to a 'canonical' set of images that have been produced beforehand.
+* The snapshots are compared to a 'canonical' set of images that have been produced beforehand.  
-* The result, if any test failed, is a zipped folder describing the failure.
+* The result, if any test failed, is a zipped folder describing the failure.
-
+
-Nitpick has 5 functions, separated into separate tabs:
+Nitpick has 6 functions, separated into separate tabs:
-
+
-1. Creating tests, MD files and recursive scripts
+1. Creating tests, MD files and recursive scripts
-1. Windows task bar utility (Windows only)
+1. Windows task bar utility (Windows only)
-1. Running tests
+1. Running tests on desktop (i.e. locally)
-1. Evaluating the results of running tests
+1. Running tests on an attached device
-1. Web interface
+1. Evaluating the results of running tests
-
+1. Web interface
-## Installation
+## Creating Installers (for developers)
-`nitpick` is packaged with High Fidelity PR and Development builds.
+Nitpick is built as part of the High Fidelity build.
-### Windows
+### Creating installers
-1.  (First time) download and install Python 3 from https://hifi-qa.s3.amazonaws.com/nitpick/Windows/python-3.7.0-amd64.exe (also located at https://www.python.org/downloads/)
+#### Windows
-    1. Click the "add python to path" checkbox on the python installer  
+Note that X.X.X is the latest version.
-    1. After installation - add the path to python.exe to the Windows PATH environment variable.
+1.  Verify that 7Zip is installed.
-1.  (First time) download and install AWS CLI from https://hifi-qa.s3.amazonaws.com/nitpick/Windows/AWSCLI64PY3.msi (also available at https://aws.amazon.com/cli/
+1.  cd to the `build\tools\nitpick\Release` directory
-    1.  Open a new command prompt and run  
+1.  Delete any existing installers (named nitpick-installer-vX.X.X.exe)
-        `aws configure`
+1.  Select all, right-click and select 7-Zip->Add to archive...
-    1.  Enter the AWS account number
+1.  Set Archive format to 7z
-    1.  Enter the secret key
+1.  Check "Create SFX archive
-    1.  Leave region name and ouput format as default [None]
+1.  Enter installer name (i.e. `nitpick-installer-vX.X.X.exe`)
-    1.  Install the latest release of Boto3 via pip:  
+1.  Click "OK"
-        `pip install boto3`
+1.  Copy created installer to https://hifi-qa.s3.amazonaws.com/nitpick/Windows/nitpick-installer-vX.X.X.exe: aws s3 cp nitpick-installer-v1.2.exe s3://hifi-qa/nitpick/Mac/nitpick-installer-vX.X.X.exe
-
+#### Mac
-1.  (First time) Download adb (Android Debug Bridge) from *https://dl.google.com/android/repository/platform-tools-latest-windows.zip*
+These steps assume the hifi repository has been cloned to `~/hifi`.
-    1.  Copy the downloaded file to (for example) **C:\adb** and extract in place.  
+1.  (first time) Install brew  
-        Verify you see *adb.exe* in **C:\adb\platform-tools\\**.
+    In a terminal: `/usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"`
-    1.  After installation - add the path to adb.exe to the Windows PATH environment variable (note that it is in *adb\platform-tools*).
+1.  (First time) install create-dmg:
-### Mac
+    In a terminal: `brew install create-dmg`
-1.  (first time) Install brew
+1.  In a terminal: cd to the `build/tools/nitpick/Release` folder
-    In a terminal:  
+1.  Copy the quazip dynamic library (note final period):
-  `/usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"`  
+    In a terminal: `cp ~/hifi/build/ext/Xcode/quazip/project/lib/libquazip5.1.dylib .`
-    Note that you will need to press RETURN again, and will then be asked for your password.
+1.  Change the loader instruction to find the dynamic library locally
-1.  (First time) install Qt:
+    In a terminal: `install_name_tool -change ~/hifi/build/ext/Xcode/quazip/project/lib/libquazip5.1.dylib libquazip5.1.dylib nitpick`
-    In a terminal:  
+1.  Delete any existing disk images. In a terminal: `rm *.dmg`
-`brew install qt`
+1.  Create installer (note final period).In a terminal: `create-dmg --volname nitpick-installer-vX.X.X nitpick-installer-vX.X.X.dmg .`  
-1.  (First time) install Python from https://www.python.org/downloads/release/python-370/ (*macOS 64-bit installer* or *macOS 64-bit/32-bit installer*)
+    Make sure to wait for completion.
-    1. After installation - In a terminal: run  
+1.  Copy created installer to AWS: `~/Library/Python/3.7/bin/aws s3 cp nitpick-installer-vX.X.X.dmg s3://hifi-qa/nitpick/Mac/nitpick-installer-vX.X.X.dmg`
-      `open "/Applications/Python 3.7/Install Certificates.command"`.  
+## Installation
-This is needed because the Mac Python supplied no longer links with the deprecated Apple-supplied system OpenSSL libraries but rather supplies a private copy of OpenSSL 1.0.2 which does not automatically access the system default root certificates.  
+`nitpick` is packaged with High Fidelity PR and Development builds.
-    1. Verify that `/usr/local/bin/python3` exists.  
+### Windows
-1.  (First time - AWS interface) Install pip with the script provided by the Python Packaging Authority:  
+1.  (First time) download and install Python 3 from https://hifi-qa.s3.amazonaws.com/nitpick/Windows/python-3.7.0-amd64.exe (also located at https://www.python.org/downloads/)
-    In a terminal:  
+    1. Click the "add python to path" checkbox on the python installer  
-    `curl -O https://bootstrap.pypa.io/get-pip.py`  
+    1. After installation - add the path to python.exe to the Windows PATH environment variable.
-    In a terminal:  
+1.  (First time) download and install AWS CLI from https://hifi-qa.s3.amazonaws.com/nitpick/Windows/AWSCLI64PY3.msi (also available at https://aws.amazon.com/cli/
-    `python3 get-pip.py --user`  
+    1.  Open a new command prompt and run  
-    1.  Use pip to install the AWS CLI.  
+        `aws configure`
-        `pip3 install awscli --upgrade --user`  
+    1.  Enter the AWS account number
-        This will install aws in your user.  For user XXX, aws will be located in ~/Library/Python/3.7/bin  
+    1.  Enter the secret key
-    1.  Open a new command prompt and run  
+    1.  Leave region name and ouput format as default [None]
-        `~/Library/Python/3.7/bin/aws configure`  
+    1.  Install the latest release of Boto3 via pip:  
-    1.  Enter the AWS account number
+        `pip install boto3`
-    1.  Enter the secret key
+
-    1.  Leave region name and ouput format as default [None]
+1.  (First time) Download adb (Android Debug Bridge) from *https://dl.google.com/android/repository/platform-tools-latest-windows.zip*
-    1.  Install the latest release of Boto3 via pip:  pip3 install boto3  
+    1.  Copy the downloaded file to (for example) **C:\adb** and extract in place.  
-1.  (First time)Install adb (the Android Debug Bridge) - in a terminal:  
+        Verify you see *adb.exe* in **C:\adb\platform-tools\\**.
-    `brew cask install android-platform-tools`
+    1.  After installation - add the path to adb.exe to the Windows PATH environment variable (note that it is in *adb\platform-tools*).
-# Usage
+1.  `nitpick` is included in the High Fidelity installer but can also be downloaded from:  
-## Create
+[here](<https://hifi-qa.s3.amazonaws.com/nitpick/Windows/nitpick-installer-vX.X.X.dmg>).* 
-![](./Create.PNG)
+### Mac
-
+1.  (first time) Install brew
-The Create tab provides functions to create tests from snapshots, MD files, a test outline and recursive scripts.
+    In a terminal:  
-### Create Tests
+  `/usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"`  
-#### Usage
+    Note that you will need to press RETURN again, and will then be asked for your password.
-This function is used to create/update Expected Images after a successful run of a test, or multiple tests.
+1.  (First time) install Qt:
-
+    In a terminal:  
-The user will be asked for the snapshot folder and then the tests root folder.  All snapshots located in the snapshot folder will be used to create or update the expected images in the relevant tests.
+`brew install qt`
-#### Details
+1.  (First time) install Python from https://www.python.org/downloads/release/python-370/ (*macOS 64-bit installer* or *macOS 64-bit/32-bit installer*)
-As an example - if the snapshots folder contains an image named `tests.content.entity.zone.zoneOrientation.00003.png`, then this file will be copied to `tests/contente/enity/zone/zoneOrientation/ExpectedImage0003.png`.
+    1. After installation - In a terminal: run  
-### Create Tests Outline
+      `open "/Applications/Python 3.7/Install Certificates.command"`.  
-#### Usage
+This is needed because the Mac Python supplied no longer links with the deprecated Apple-supplied system OpenSSL libraries but rather supplies a private copy of OpenSSL 1.0.2 which does not automatically access the system default root certificates.  
-This function creates an MD file in the (user-selected) tests root folder.  The file provides links to both the tests and the MD files.
+    1. Verify that `/usr/local/bin/python3` exists.  
-### Create MD file
+1.  (First time - AWS interface) Install pip with the script provided by the Python Packaging Authority:  
-#### Usage
+    In a terminal:  
-This function creates a file named `test.md` from a `test.js` script.  The user will be asked for the folder containing the test script:
+    `curl -O https://bootstrap.pypa.io/get-pip.py`  
-#### Details
+    In a terminal:  
-The process to produce the MD file is a simplistic parse of the test script.
+    `python3 get-pip.py --user`  
-- The string in the `nitpick.perform(...)` function call will be the title of the file
+    1.  Use pip to install the AWS CLI.  
-
+        `pip3 install awscli --upgrade --user`  
-- Instructions to run the script are then provided: 
+        This will install aws in your user.  For user XXX, aws will be located in ~/Library/Python/3.7/bin  
-
+    1.  Open a new command prompt and run  
-**Run this script URL: [Manual]()   [Auto]()(from menu/Edit/Open and Run scripts from URL...).**
+        `~/Library/Python/3.7/bin/aws configure`  
-
+    1.  Enter the AWS account number
-- The step description is the string in the addStep/addStepStepSnapshot commands
+    1.  Enter the secret key
-
+    1.  Leave region name and ouput format as default [None]
-- Image links are provided where applicable to the local Expected Images files
+    1.  Install the latest release of Boto3 via pip:  pip3 install boto3  
-### Create all MD files
+1.  (First time)Install adb (the Android Debug Bridge) - in a terminal:  
-#### Usage
+    `brew cask install android-platform-tools`
-This function creates all MD files recursively from the user-selected root folder.  This can be any folder in the tests hierarchy (e.g. all engine\material tests).
+1.  `nitpick` is included in the High Fidelity installer but can also be downloaded from:  
-
+[here](<https://hifi-qa.s3.amazonaws.com/nitpick/Mac/nitpick-installer-vX.X.X.dmg>).* 
-The file provides a hierarchal list of all the tests 
+# Usage
-### Create testAuto script
+## Menu
-#### Usage
+File->Close: Closes `nitpick`  
-This function creates a script named `testAuto.js` in a user-selected test folder.
+Help->About: Provides the build date  
-#### Details
+Help->Online readme: Links to this file on GitHub
-The script created runs the `test.js` script in the folder in automatic mode.  The script is the same for all tests.
+## Create
-### Create all testAuto scripts
+![](./Create.PNG)
-#### Usage
+
-This function creates all testAuto scripts recursively from the user-selected root folder.  This can be any folder in the tests hierarchy (e.g. all engine\material tests).
+The Create tab provides functions to create tests from snapshots, MD files, a test outline and recursive scripts.
-
+### Create Tests
-The file provides a hierarchical list of all the tests 
+#### Usage
-### Create Recursive Script
+This function is used to create/update Expected Images after a successful run of a test, or multiple tests.
-#### Usage
+
-After the user selects a folder within the tests hierarchy, a script is created, named `testRecursive.js`.  This script calls all `test.js` scripts in the sub-folders.
+The user will be asked for the snapshot folder and then the tests root folder.  All snapshots located in the snapshot folder will be used to create or update the expected images in the relevant tests.
-#### Details
+#### Details
-The various scripts are called in alphabetical order.
+As an example - if the snapshots folder contains an image named `tests.content.entity.zone.zoneOrientation.00003.png`, then this file will be copied to `tests/contente/enity/zone/zoneOrientation/ExpectedImage0003.png`.
-
+### Create Tests Outline
-An example of a recursive script is as follows:  
+#### Usage
-```
+This function creates an MD file in the (user-selected) tests root folder.  The file provides links to both the tests and the MD files.
-// This is an automatically generated file, created by nitpick on Jul 5 2018, 10:19
+### Create MD file
-
+#### Usage
-PATH_TO_THE_REPO_PATH_UTILS_FILE = "https://raw.githubusercontent.com/highfidelity/hifi_tests/master/tests/utils/branchUtils.js";
+This function creates a file named `test.md` from a `test.js` script.  The user will be asked for the folder containing the test script:
-Script.include(PATH_TO_THE_REPO_PATH_UTILS_FILE);
+#### Details
-var nitpick = createNitpick(Script.resolvePath("."));
+The process to produce the MD file is a simplistic parse of the test script.
-
+- The string in the `nitpick.perform(...)` function call will be the title of the file
-var testsRootPath = nitpick.getTestsRootPath();
+
-
+- Instructions to run the script are then provided: 
-if (typeof Test !== 'undefined') {
+
-    Test.wait(10000);
+**Run this script URL: [Manual]()   [Auto]()(from menu/Edit/Open and Run scripts from URL...).**
-};
+
-
+- The step description is the string in the addStep/addStepStepSnapshot commands
-nitpick.enableRecursive();
+
-nitpick.enableAuto();
+- Image links are provided where applicable to the local Expected Images files
-
+### Create all MD files
-Script.include(testsRootPath + "content/overlay/layer/drawInFront/shape/test.js");
+#### Usage
-Script.include(testsRootPath + "content/overlay/layer/drawInFront/model/test.js");
+This function creates all MD files recursively from the user-selected root folder.  This can be any folder in the tests hierarchy (e.g. all engine\material tests).
-Script.include(testsRootPath + "content/overlay/layer/drawHUDLayer/test.js");
+
-
+The file provides a hierarchal list of all the tests 
-nitpick.runRecursive();
+### Create testAuto script
-```
+#### Usage
-### Create all Recursive Scripts
+This function creates a script named `testAuto.js` in a user-selected test folder.
-#### Usage
+#### Details
-In this case all recursive scripts, from the selected folder down, are created.
+The script created runs the `test.js` script in the folder in automatic mode.  The script is the same for all tests.
-
+### Create all testAuto scripts
-Running this function in the tests root folder will create (or update) all the recursive scripts.
+#### Usage
-## Windows (only)
+This function creates all testAuto scripts recursively from the user-selected root folder.  This can be any folder in the tests hierarchy (e.g. all engine\material tests).
-![](./Windows.PNG)
+
-
+The file provides a hierarchical list of all the tests 
-This tab is Windows-specific.  It provides buttons to hide and show the task bar.
+### Create Recursive Script
-
+#### Usage
-The task bar should be hidden for all tests that use the primary camera.  This is required to ensure that the snapshots are the right size.
+After the user selects a folder within the tests hierarchy, a script is created, named `testRecursive.js`.  This script calls all `test.js` scripts in the sub-folders.
-## Run
+#### Details
-![](./Run.PNG)
+The various scripts are called in alphabetical order.
-The run tab is used to run tests in automatic mode.  The tests require the location of a folder to store files in; this folder can safely be re-used for any number of runs (the "Working Folder"). 
+
-The test script that is run is `https://github.com/highfidelity/hifi_tests/blob/master/tests/testRecursive.js`.  The user can use a different branch' or even repository, if required.  
+An example of a recursive script is as follows:  
-Tests can be run server-less, or with the local host.  In the second case, the domain-server and assignment-clients are run before starting Interface.  
+```
-The default is to run the latest build.  The user can select a specific build or PR if desired.
+// This is an automatically generated file, created by nitpick on Jul 5 2018, 10:19
-Testing can be started immediately, or run on a schedule.  
+
-
+PATH_TO_THE_REPO_PATH_UTILS_FILE = "https://raw.githubusercontent.com/highfidelity/hifi_tests/master/tests/utils/branchUtils.js";
-A test run is performed in a number of steps:  
+Script.include(PATH_TO_THE_REPO_PATH_UTILS_FILE);
-1.  If the latest run has been selected then the `dev-builds.xml` file is downloaded to identify the latest build.  
+var nitpick = createNitpick(Script.resolvePath("."));
-1.  The installer is then downloaded.  
+
-1.  After downloading the High Fidelity application is installed.  This requires that UAC be disabled!
+var testsRootPath = nitpick.getTestsRootPath();
-1.  Any instances of the server-console, assignment-client, domain-server or Interface are killed before running the tests.  
+
-1.  Interface is run with the appropriate command line parameters.  
+if (typeof Test !== 'undefined') {
-1.  The expected images are then downloaded from GitHub and compared to the actual images from the tests.
+    Test.wait(10000);
-
+};
-The working folder will ultimately contain the following:  
+
-1.  A folder named `High Fidelity`.  This is where High Fidelity is installed.  
+nitpick.enableRecursive();
-1.  A folder named `snapshots`.  This folder contains the zipped results folders (one for each run).  It also contains both the actual images and the expected images from the last run; note that these are deleted before running tests (All PNG files in the folder are deleted.  In addition - a text file named `tests_completed.txt` is created at the end of the test script - this signals that Interface did not crash during the test run.  
+nitpick.enableAuto();
-1.  The `dev-builds.xml` file, if it was downloaded.  
+
-1.  The HighFidelity installer.  Note that this is always named `HighFidelity-Beta-latest-dev` so as not to store too many installers over time.  
+Script.include(testsRootPath + "content/overlay/layer/drawInFront/shape/test.js");
-1.  A log file describing the runs.  This file is appended to after each run.
+Script.include(testsRootPath + "content/overlay/layer/drawInFront/model/test.js");
-## Evaluate
+Script.include(testsRootPath + "content/overlay/layer/drawHUDLayer/test.js");
-![](./Evaluate.PNG)
+
-
+nitpick.runRecursive();
-The Evaluate tab provides a single function - evaluating the results of a test run.
+```
-
+### Create all Recursive Scripts
-A checkbox (defaulting to checked) runs the evaluation in interactive mode.  In this mode - every failure is shown to the user, who can then decide whether to pass the test, fail it or abort the whole evaluation.
+#### Usage
-
+In this case all recursive scripts, from the selected folder down, are created.
-If any tests have failed, then a zipped folder will be created in the snapshots folder, with a description of each failed step in each test.
+
-#### Usage
+Running this function in the tests root folder will create (or update) all the recursive scripts.
-Before starting the evaluation, make sure the GitHub user and branch are set correctly.  The user should not normally be changed, but the branch may need to be set to the appropriate RC.
+## Windows (only on Windows)
-
+![](./Windows.PNG)
-After setting the check-box as required and pressing Evaluate - the user will be asked for the snapshots folder.
+
-#### Details
+This tab is Windows-specific.  It provides buttons to hide and show the task bar.
-Evaluation proceeds in a number of steps:
+
-
+The task bar should be hidden for all tests that use the primary camera.  This is required to ensure that the snapshots are the right size.
-1. A folder is created to store any failures
+## Test on Desktop
-
+![](./TestOnDesktop.PNG)
-1. The expected images are download from GitHub.  They are named slightly differently from the snapshots (e.g. `tests.engine.render.effect.highlight.coverage.00000.png` and `tests.engine.render.effect.highlight.coverage.00000_EI.png`).
+The TestOnDesktop tab is used to run tests on desktop in automatic mode.  The tests require the location of a folder to store files in; this folder can safely be re-used for any number of runs (the "Working Folder").  
-
+The test script that is run is `https://github.com/highfidelity/hifi_tests/blob/master/tests/testRecursive.js`.  The user can use a different branch' or even repository, if required.  
-1. The images are then pair-wise compared, using the SSIM algorithm.  A fixed threshold is used to define a mismatch.
+Tests can be run server-less, or with the local host.  In the second case, the domain-server and assignment-clients are run before starting Interface.  
-
+The default is to run the latest build.  The user can select a specific build or PR if desired.
-1.  In interactive mode - a window is opened showing the expected image, actual image, difference image and error:
+Testing can be started immediately, or run on a schedule.  
-![](./nitpickMismatchExample.PNG)
+
-
+A test run is performed in a number of steps:  
-1.  If not in interactive mode, or the user has defined the results as an error, an error is written into the error folder.  The error itself is a folder with the 3 images and a small text file containing details.
+1.  If the latest run has been selected then the `dev-builds.xml` file is downloaded to identify the latest build.  
-
+1.  The installer is then downloaded.  
-1.  At the end of the test, the folder is zipped and the original folder is deleted.  If there are no errors then the zipped folder will be empty.
+1.  After downloading the High Fidelity application is installed.  This requires that UAC be disabled!
-
+1.  Any instances of the server-console, assignment-client, domain-server or Interface are killed before running the tests.  
-## Web Interface
+1.  Interface is run with the appropriate command line parameters.  
-![](./WebInterface.PNG)
+1.  The expected images are then downloaded from GitHub and compared to the actual images from the tests.
-This tab has two functions:  updating the TestRail cases, runs and results, and creating web page reports that are stored on AWS.  
+
-
+The working folder will ultimately contain the following:  
-Before updating TestRail, make sure the GitHub user and branch are set correctly.  The user should not normally be changed, but the branch may need to be set to the appropriate RC.
+1.  A folder named `High Fidelity`.  This is where High Fidelity is installed.  
-
+1.  A folder named `snapshots`.  This folder contains the zipped results folders (one for each run).  It also contains both the actual images and the expected images from the last run; note that these are deleted before running tests (All PNG files in the folder are deleted.  In addition - a text file named `tests_completed.txt` is created at the end of the test script - this signals that Interface did not crash during the test run.  
-Any access to TestRail will require the TestRail account (default is High Fidelity's account), a user-name and a password:
+1.  The `dev-builds.xml` file, if it was downloaded.  
-
+1.  The HighFidelity installer.  Note that this is always named `HighFidelity-Beta-latest-dev` so as not to store too many installers over time.  
-![](./TestRailSelector.PNG)
+1.  A log file describing the runs.  This file is appended to after each run.
-
+## Test on Mobile
-- The default test rail user is shown, and can be changed as needed.
+![](./TestOnMobile.PNG)
-- The user-name is usually the user's email.
+The TestOnMobile tab is used to run tests on a mobile device connected to the desktop running `nitpick`.  
-- The Project ID defaults to 14 - Interface.
+The test script that is run is `https://github.com/highfidelity/hifi_tests/blob/master/tests/testRecursive.js`.  The user can use a different branch' or even repository, if required.  
-- The Suite ID defaults to 1147 - Rendering.
+Tests can be run server-less, or with the local host.  In the second case, the domain-server and assignment-clients are run before starting Interface.  
-- The TestRail page provides 3 functions for writing to TestRail.
+The default is to run the latest build.  The user can select a specific build or PR if desired.  
-### Create Test Cases
+### Set Working Folder  
-#### Usage
+This mode needs a working folder to store downloads and temporary files.  All other commands are disabled until this folder has been selected.
-This function can either create an XML file that can then be imported into TestRail through TestRail itself, or automatically create the appropriate TestRail Sections.
+### Connect Device
-
+Clicking this button will initiate and attempt to connect to a connected device.  If successful, the following will appear (as appropriate for the device):
-The user will be first asked for the tests root folder and a folder to store temporary files (this is the output folder).
+![](./ConnectedDevice.PNG)
-
+
-If XML has been selected, then the XML file will be created in the output folder.
+### Download APK
-
+Leaving the Run Latest checkbox checked will download the latest APK.  Unchecking will enable entering a URL to a desired APK (such as a PR build).  
-If Python is selected, the user will then be prompted for TestRail data.  After pressing `Accept` - the Release combo will be populated (as it needs to be read from TestRail).
+Clicking the Download APK button will set the status to *Downloading installer*; this status will change to *Installer Download complete" when the download has completed.  The APK will be located in the working folder.
-
+### Installing APK
-After selecting the appropriate Release, press OK.  The Python script will be created in the output folder, and the user will be prompted to run it.
+After download it is possible to install the APK on the selected device.  
-
+When installation completes, the status will be *Installation complete*
-A busy window will appear until the process is complete.
+### Run Interface
-#### Details
+Pressing this button will run the full test suite on the device.  Snapshots will be stored on the device in */sdcard/DCIM/TEST.
-A number of Python scripts are created:
+### Pull Folder
-- `testrail.py` is the TestRail interface code.
+This button is used to copy the snapshots from a test to the local device for evaluation.  The default is the default snapshot folder on the device.
-- `stack.py` is a simple stack class
+## Evaluate
-- `getReleases.py` reads the release names from TestRail
+![](./Evaluate.PNG)
-- `addTestCases` is the script that writes to TestRail.
+
-
+The Evaluate tab provides a single function - evaluating the results of a test run.
-In addition - a file containing all the releases will be created - `releases.txt`
+
-### Create Run
+A checkbox (defaulting to checked) runs the evaluation in interactive mode.  In this mode - every failure is shown to the user, who can then decide whether to pass the test, fail it or abort the whole evaluation.
-A Run is created from previously created Test Cases.
+
-
+If any tests have failed, then a zipped folder will be created in the snapshots folder, with a description of each failed step in each test.
-The user will first be prompted for a temporary folder (for the Python scripts).
+#### Usage
-
+Before starting the evaluation, make sure the GitHub user and branch are set correctly.  The user should not normally be changed, but the branch may need to be set to the appropriate RC.
-After entering TestRail data and pressing `Accept` - the Sections combo will be populated (as it needs to be read from TestRail).
+
-
+After setting the check-box as required and pressing Evaluate - the user will be asked for the snapshots folder.
-After selecting the appropriate Section, press OK.  The Python script will be created in the output folder, and the user will be prompted to run it.
+#### Details
-
+Evaluation proceeds in a number of steps:
-A busy window will appear until the process is complete.
+
-#### Details
+1. A folder is created to store any failures
-A number of Python scripts are created:
+
-- `testrail.py` is the TestRail interface code.
+1. The expected images are download from GitHub.  They are named slightly differently from the snapshots (e.g. `tests.engine.render.effect.highlight.coverage.00000.png` and `tests.engine.render.effect.highlight.coverage.00000_EI.png`).
-- `stack.py` is a simple stack class
+
-- `getSections.py` reads the release names from TestRail
+1. The images are then pair-wise compared, using the SSIM algorithm.  A fixed threshold is used to define a mismatch.
-- `addRun` is the script that writes to TestRail.
+
-
+1.  In interactive mode - a window is opened showing the expected image, actual image, difference image and error:
-In addition - a file containing all the releases will be created - `sections.txt`
+![](./nitpickMismatchExample.PNG)
-### Update Run Results
+
-This function updates a Run with the results of an automated test.
+1.  If not in interactive mode, or the user has defined the results as an error, an error is written into the error folder.  The error itself is a folder with the 3 images and a small text file containing details.
-
+
-The user will first be prompted to enter the zipped results folder and a folder to store temporary files (this is the output folder).
+1.  At the end of the test, the folder is zipped and the original folder is deleted.  If there are no errors then the zipped folder will be empty.
-
+
-After entering TestRail data and pressing `Accept` - the Run combo will be populated (as it needs to be read from TestRail).
+## Web Interface
-
+![](./WebInterface.PNG)
-After selecting the appropriate Run, press OK.  The Python script will be created in the output folder, and the user will be prompted to run it.
+This tab has two functions:  updating the TestRail cases, runs and results, and creating web page reports that are stored on AWS.  
-
+
-A busy window will appear until the process is complete.
+Before updating TestRail, make sure the GitHub user and branch are set correctly.  The user should not normally be changed, but the branch may need to be set to the appropriate RC.
-#### Details
+
-A number of Python scripts are created:
+Any access to TestRail will require the TestRail account (default is High Fidelity's account), a user-name and a password:
-- `testrail.py` is the TestRail interface code.
+
-- `getRuns.py` reads the release names from TestRail
+![](./TestRailSelector.PNG)
-- `addRun` is the script that writes to TestRail.
+
-
+- The default test rail user is shown, and can be changed as needed.
-In addition - a file containing all the releases will be created - `runs.txt`.
+- The user-name is usually the user's email.
-### Create Web Page
+- The Project ID defaults to 14 - Interface.
-This function requests a zipped results folder and converts it to a web page.  The page is created in a user-selecetd working folder.  
+- The Suite ID defaults to 1147 - Rendering.
-If the `Update AWS` checkbox is checked then the page will also be copied to AWS, and the appropriate URL will be displayed in the window below the button.
+- The TestRail page provides 3 functions for writing to TestRail.
+### Create Test Cases
+#### Usage
+This function can either create an XML file that can then be imported into TestRail through TestRail itself, or automatically create the appropriate TestRail Sections.
+
+The user will be first asked for the tests root folder and a folder to store temporary files (this is the output folder).
+
+If XML has been selected, then the XML file will be created in the output folder.
+
+If Python is selected, the user will then be prompted for TestRail data.  After pressing `Accept` - the Release combo will be populated (as it needs to be read from TestRail).
+
+After selecting the appropriate Release, press OK.  The Python script will be created in the output folder, and the user will be prompted to run it.
+
+A busy window will appear until the process is complete.
+#### Details
+A number of Python scripts are created:
+- `testrail.py` is the TestRail interface code.
+- `stack.py` is a simple stack class
+- `getReleases.py` reads the release names from TestRail
+- `addTestCases` is the script that writes to TestRail.
+
+In addition - a file containing all the releases will be created - `releases.txt`
+### Create Run
+A Run is created from previously created Test Cases.
+
+The user will first be prompted for a temporary folder (for the Python scripts).
+
+After entering TestRail data and pressing `Accept` - the Sections combo will be populated (as it needs to be read from TestRail).
+
+After selecting the appropriate Section, press OK.  The Python script will be created in the output folder, and the user will be prompted to run it.
+
+A busy window will appear until the process is complete.
+#### Details
+A number of Python scripts are created:
+- `testrail.py` is the TestRail interface code.
+- `stack.py` is a simple stack class
+- `getSections.py` reads the release names from TestRail
+- `addRun` is the script that writes to TestRail.
+
+In addition - a file containing all the releases will be created - `sections.txt`
+### Update Run Results
+This function updates a Run with the results of an automated test.
+
+The user will first be prompted to enter the zipped results folder and a folder to store temporary files (this is the output folder).
+
+After entering TestRail data and pressing `Accept` - the Run combo will be populated (as it needs to be read from TestRail).
+
+After selecting the appropriate Run, press OK.  The Python script will be created in the output folder, and the user will be prompted to run it.
+
+A busy window will appear until the process is complete.
+#### Details
+A number of Python scripts are created:
+- `testrail.py` is the TestRail interface code.
+- `getRuns.py` reads the release names from TestRail
+- `addRun` is the script that writes to TestRail.
+
+In addition - a file containing all the releases will be created - `runs.txt`.
+### Create Web Page
+This function requests a zipped results folder and converts it to a web page.  The page is created in a user-selecetd working folder.  
+If the `Update AWS` checkbox is checked then the page will also be copied to AWS, and the appropriate URL will be displayed in the window below the button.