Merge pull request #14273 from NissimHadar/MAC

Case 19722 - Auto-Tester on Mac.
This commit is contained in:
Shannon Romano 2018-11-16 12:49:58 -08:00 committed by GitHub
commit ba699e3e9b
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
58 changed files with 561 additions and 243 deletions

View file

@ -198,4 +198,14 @@ void TestScriptingInterface::setOtherAvatarsReplicaCount(int count) {
int TestScriptingInterface::getOtherAvatarsReplicaCount() { int TestScriptingInterface::getOtherAvatarsReplicaCount() {
return qApp->getOtherAvatarsReplicaCount(); return qApp->getOtherAvatarsReplicaCount();
} }
QString TestScriptingInterface::getOperatingSystemType() {
#ifdef Q_OS_WIN
return "WINDOWS";
#elif defined Q_OS_MAC
return "MACOS";
#else
return "UNKNOWN";
#endif
}

View file

@ -163,6 +163,13 @@ public slots:
*/ */
Q_INVOKABLE int getOtherAvatarsReplicaCount(); Q_INVOKABLE int getOtherAvatarsReplicaCount();
/**jsdoc
* Returns the Operating Sytem type
* @function Test.getOperatingSystemType
* @returns {string} "WINDOWS", "MACOS" or "UNKNOWN"
*/
QString getOperatingSystemType();
private: private:
bool waitForCondition(qint64 maxWaitMs, std::function<bool()> condition); bool waitForCondition(qint64 maxWaitMs, std::function<bool()> condition);
QString _testResultsLocation; QString _testResultsLocation;

View file

@ -31,6 +31,6 @@ if (BUILD_TOOLS)
add_subdirectory(oven) add_subdirectory(oven)
set_target_properties(oven PROPERTIES FOLDER "Tools") set_target_properties(oven PROPERTIES FOLDER "Tools")
add_subdirectory(auto-tester) add_subdirectory(nitpick)
set_target_properties(auto-tester PROPERTIES FOLDER "Tools") set_target_properties(nitpick PROPERTIES FOLDER "Tools")
endif() endif()

View file

@ -1,4 +1,4 @@
set (TARGET_NAME auto-tester) set (TARGET_NAME nitpick)
project(${TARGET_NAME}) project(${TARGET_NAME})
# Automatically run UIC and MOC. This replaces the older WRAP macros # Automatically run UIC and MOC. This replaces the older WRAP macros
@ -22,7 +22,7 @@ set (QT_LIBRARIES Qt5::Core Qt5::Widgets QT::Gui Qt5::Xml)
if (WIN32) if (WIN32)
# Do not show Console # Do not show Console
set_property (TARGET auto-tester PROPERTY WIN32_EXECUTABLE true) set_property (TARGET nitpick PROPERTY WIN32_EXECUTABLE true)
endif() endif()
target_zlib() target_zlib()

View file

Before

Width:  |  Height:  |  Size: 17 KiB

After

Width:  |  Height:  |  Size: 17 KiB

View file

Before

Width:  |  Height:  |  Size: 13 KiB

After

Width:  |  Height:  |  Size: 13 KiB

View file

@ -1,11 +1,11 @@
# Auto Tester # nitpick
The auto-tester is a stand alone application that provides a mechanism for regression testing. The general idea is simple: Nitpick is a stand alone application that provides a mechanism for regression testing. The general idea is simple:
* Each test folder has a script that produces a set of snapshots. * Each test folder has a script that produces a set of snapshots.
* The snapshots are compared to a 'canonical' set of images that have been produced beforehand. * The snapshots are compared to a 'canonical' set of images that have been produced beforehand.
* The result, if any test failed, is a zipped folder describing the failure. * The result, if any test failed, is a zipped folder describing the failure.
Auto-tester has 5 functions, separated into 4 tabs: Nitpick has 5 functions, separated into 4 tabs:
1. Creating tests, MD files and recursive scripts 1. Creating tests, MD files and recursive scripts
1. Windows task bar utility (Windows only) 1. Windows task bar utility (Windows only)
1. Running tests 1. Running tests
@ -14,19 +14,25 @@ Auto-tester has 5 functions, separated into 4 tabs:
## Installation ## Installation
### Executable ### Executable
1. Download the installer by browsing to [here](<https://hifi-content.s3.amazonaws.com/nissim/autoTester/AutoTester-Installer-v6.6.exe>). 1. On Windows: download the installer by browsing to [here](<https://hifi-content.s3.amazonaws.com/nissim/nitpick/nitpick-installer-v1.0.exe>).
2. Double click on the installer and install to a convenient location 2. Double click on the installer and install to a convenient location
![](./setup_7z.PNG) ![](./setup_7z.PNG)
3. To run the auto-tester, double click **auto-tester.exe**. 3. To run nitpick, double click **nitpick.exe**.
### Python ### Python
The TestRail interface requires Python 3 to be installed. Auto-Tester has been tested with Python 3.7.0 but should work with newer versions. The TestRail interface requires Python 3 to be installed. Nitpick has been tested with Python 3.7.0 but should work with newer versions.
Python 3 can be downloaded from: Python 3 can be downloaded from:
1. Windows installer <https://www.python.org/downloads/> 1. Windows installer <https://www.python.org/downloads/>
2. Linux (source) <https://www.python.org/downloads/release/python-370/> (**Gzipped source tarball**) 2. Linux (source) <https://www.python.org/downloads/release/python-370/> (**Gzipped source tarball**)
3. Mac <https://www.python.org/downloads/release/python-370/> (**macOS 64-bit/32-bit installer** or **macOS 64-bit/32-bit installer**) 3. Mac <https://www.python.org/downloads/release/python-370/> (**macOS 64-bit/32-bit installer** or **macOS 64-bit/32-bit installer**)
#### Windows
After installation - create an environment variable called PYTHON_PATH and set it to the folder containing the Python executable. After installation - create an environment variable called PYTHON_PATH and set it to the folder containing the Python executable.
#### Mac
After installation - run `open "/Applications/Python 3.6/Install Certificates.command"`. This is needed because the Mac Python supplied no longer links with the deprecated Apple-supplied system OpenSSL libraries but rather supplies a private copy of OpenSSL 1.0.2 which does not automatically access the system default root certificates.
Verify that `/usr/local/bin/python3` exists.
### AWS interface ### AWS interface
#### Windows #### Windows
1. Download the AWS CLI from `https://aws.amazon.com/cli/` 1. Download the AWS CLI from `https://aws.amazon.com/cli/`
@ -35,9 +41,23 @@ After installation - create an environment variable called PYTHON_PATH and set i
1. Enter the AWS account number 1. Enter the AWS account number
1. Enter the secret key 1. Enter the secret key
1. Leave region name and ouput format as default [None] 1. Leave region name and ouput format as default [None]
1. Install the latest release of Boto3 via pip:
pip install boto3
#### Mac
1. Install pip with the script provided by the Python Packaging Authority:
$ curl -O https://bootstrap.pypa.io/get-pip.py
$ python3 get-pip.py --user
1. Use pip to install the AWS CLI.
$ pip3 install awscli --upgrade --user
This will install aws in your user. For user XXX, aws will be located in /Users/XXX/Library/Python/3.7/bin
1. Open a new command prompt and run `aws configure`
1. Enter the AWS account number
1. Enter the secret key
1. Leave region name and ouput format as default [None]
1. Install the latest release of Boto3 via pip:
pip3 install boto3
1. Install the latest release of Boto3 via pip:
>pip install boto3
# Create # Create
![](./Create.PNG) ![](./Create.PNG)
@ -57,7 +77,7 @@ This function creates an MD file in the (user-selected) tests root folder. The
This function creates a file named `test.md` from a `test.js` script. The user will be asked for the folder containing the test script: This function creates a file named `test.md` from a `test.js` script. The user will be asked for the folder containing the test script:
### Details ### Details
The process to produce the MD file is a simplistic parse of the test script. The process to produce the MD file is a simplistic parse of the test script.
- The string in the `autoTester.perform(...)` function call will be the title of the file - The string in the `nitpick.perform(...)` function call will be the title of the file
- Instructions to run the script are then provided: - Instructions to run the script are then provided:
@ -89,26 +109,26 @@ The various scripts are called in alphabetical order.
An example of a recursive script is as follows: An example of a recursive script is as follows:
``` ```
// This is an automatically generated file, created by auto-tester on Jul 5 2018, 10:19 // This is an automatically generated file, created by nitpick on Jul 5 2018, 10:19
PATH_TO_THE_REPO_PATH_UTILS_FILE = "https://raw.githubusercontent.com/highfidelity/hifi_tests/master/tests/utils/branchUtils.js"; PATH_TO_THE_REPO_PATH_UTILS_FILE = "https://raw.githubusercontent.com/highfidelity/hifi_tests/master/tests/utils/branchUtils.js";
Script.include(PATH_TO_THE_REPO_PATH_UTILS_FILE); Script.include(PATH_TO_THE_REPO_PATH_UTILS_FILE);
var autoTester = createAutoTester(Script.resolvePath(".")); var nitpick = createNitpick(Script.resolvePath("."));
var testsRootPath = autoTester.getTestsRootPath(); var testsRootPath = nitpick.getTestsRootPath();
if (typeof Test !== 'undefined') { if (typeof Test !== 'undefined') {
Test.wait(10000); Test.wait(10000);
}; };
autoTester.enableRecursive(); nitpick.enableRecursive();
autoTester.enableAuto(); nitpick.enableAuto();
Script.include(testsRootPath + "content/overlay/layer/drawInFront/shape/test.js"); Script.include(testsRootPath + "content/overlay/layer/drawInFront/shape/test.js");
Script.include(testsRootPath + "content/overlay/layer/drawInFront/model/test.js"); Script.include(testsRootPath + "content/overlay/layer/drawInFront/model/test.js");
Script.include(testsRootPath + "content/overlay/layer/drawHUDLayer/test.js"); Script.include(testsRootPath + "content/overlay/layer/drawHUDLayer/test.js");
autoTester.runRecursive(); nitpick.runRecursive();
``` ```
## Create all Recursive Scripts ## Create all Recursive Scripts
### Usage ### Usage
@ -165,7 +185,7 @@ Evaluation proceeds in a number of steps:
1. The images are then pair-wise compared, using the SSIM algorithm. A fixed threshold is used to define a mismatch. 1. The images are then pair-wise compared, using the SSIM algorithm. A fixed threshold is used to define a mismatch.
1. In interactive mode - a window is opened showing the expected image, actual image, difference image and error: 1. In interactive mode - a window is opened showing the expected image, actual image, difference image and error:
![](./autoTesterMismatchExample.PNG) ![](./nitpickMismatchExample.PNG)
1. If not in interactive mode, or the user has defined the results as an error, an error is written into the error folder. The error itself is a folder with the 3 images and a small text file containing details. 1. If not in interactive mode, or the user has defined the results as an error, an error is written into the error folder. The error itself is a folder with the 3 images and a small text file containing details.

View file

Before

Width:  |  Height:  |  Size: 20 KiB

After

Width:  |  Height:  |  Size: 20 KiB

View file

Before

Width:  |  Height:  |  Size: 10 KiB

After

Width:  |  Height:  |  Size: 10 KiB

View file

Before

Width:  |  Height:  |  Size: 12 KiB

After

Width:  |  Height:  |  Size: 12 KiB

View file

Before

Width:  |  Height:  |  Size: 17 KiB

After

Width:  |  Height:  |  Size: 17 KiB

View file

Before

Width:  |  Height:  |  Size: 14 KiB

After

Width:  |  Height:  |  Size: 14 KiB

View file

Before

Width:  |  Height:  |  Size: 476 KiB

After

Width:  |  Height:  |  Size: 476 KiB

View file

Before

Width:  |  Height:  |  Size: 4 KiB

After

Width:  |  Height:  |  Size: 4 KiB

View file

@ -22,11 +22,11 @@ AWSInterface::AWSInterface(QObject* parent) : QObject(parent) {
} }
void AWSInterface::createWebPageFromResults(const QString& testResults, void AWSInterface::createWebPageFromResults(const QString& testResults,
const QString& workingDirectory, const QString& snapshotDirectory,
QCheckBox* updateAWSCheckBox, QCheckBox* updateAWSCheckBox,
QLineEdit* urlLineEdit) { QLineEdit* urlLineEdit) {
_testResults = testResults; _testResults = testResults;
_workingDirectory = workingDirectory; _snapshotDirectory = snapshotDirectory;
_urlLineEdit = urlLineEdit; _urlLineEdit = urlLineEdit;
_urlLineEdit->setEnabled(false); _urlLineEdit->setEnabled(false);
@ -42,16 +42,15 @@ void AWSInterface::createWebPageFromResults(const QString& testResults,
void AWSInterface::extractTestFailuresFromZippedFolder() { void AWSInterface::extractTestFailuresFromZippedFolder() {
// For a test results zip file called `D:/tt/TestResults--2018-10-02_16-54-11(9426)[DESKTOP-PMKNLSQ].zip` // For a test results zip file called `D:/tt/TestResults--2018-10-02_16-54-11(9426)[DESKTOP-PMKNLSQ].zip`
// the folder will be called `TestResults--2018-10-02_16-54-11(9426)[DESKTOP-PMKNLSQ]` // the folder will be called `TestResults--2018-10-02_16-54-11(9426)[DESKTOP-PMKNLSQ]`
// and, this folder will be in the workign directory // and, this folder will be in the working directory
QStringList parts =_testResults.split('/'); QStringList parts =_testResults.split('/');
QString zipFolderName = _workingDirectory + "/" + parts[parts.length() - 1].split('.')[0]; QString zipFolderName = _snapshotDirectory + "/" + parts[parts.length() - 1].split('.')[0];
if (QDir(zipFolderName).exists()) { if (QDir(zipFolderName).exists()) {
QDir dir = zipFolderName; QDir dir = zipFolderName;
dir.removeRecursively(); dir.removeRecursively();
} }
QDir().mkdir(_workingDirectory); JlCompress::extractDir(_testResults, _snapshotDirectory);
JlCompress::extractDir(_testResults, _workingDirectory);
} }
void AWSInterface::createHTMLFile() { void AWSInterface::createHTMLFile() {
@ -61,7 +60,7 @@ void AWSInterface::createHTMLFile() {
QString filename = pathComponents[pathComponents.length() - 1]; QString filename = pathComponents[pathComponents.length() - 1];
_resultsFolder = filename.left(filename.length() - 4); _resultsFolder = filename.left(filename.length() - 4);
QString resultsPath = _workingDirectory + "/" + _resultsFolder + "/"; QString resultsPath = _snapshotDirectory + "/" + _resultsFolder + "/";
QDir().mkdir(resultsPath); QDir().mkdir(resultsPath);
_htmlFilename = resultsPath + HTML_FILENAME; _htmlFilename = resultsPath + HTML_FILENAME;
@ -157,7 +156,7 @@ void AWSInterface::writeTable(QTextStream& stream) {
// Note that failures are processed first, then successes // Note that failures are processed first, then successes
QStringList originalNamesFailures; QStringList originalNamesFailures;
QStringList originalNamesSuccesses; QStringList originalNamesSuccesses;
QDirIterator it1(_workingDirectory.toStdString().c_str()); QDirIterator it1(_snapshotDirectory.toStdString().c_str());
while (it1.hasNext()) { while (it1.hasNext()) {
QString nextDirectory = it1.next(); QString nextDirectory = it1.next();
@ -191,10 +190,10 @@ void AWSInterface::writeTable(QTextStream& stream) {
newNamesSuccesses.append(originalNamesSuccesses[i].split("--tests.")[1]); newNamesSuccesses.append(originalNamesSuccesses[i].split("--tests.")[1]);
} }
_htmlFailuresFolder = _workingDirectory + "/" + _resultsFolder + "/" + FAILURES_FOLDER; _htmlFailuresFolder = _snapshotDirectory + "/" + _resultsFolder + "/" + FAILURES_FOLDER;
QDir().mkdir(_htmlFailuresFolder); QDir().mkdir(_htmlFailuresFolder);
_htmlSuccessesFolder = _workingDirectory + "/" + _resultsFolder + "/" + SUCCESSES_FOLDER; _htmlSuccessesFolder = _snapshotDirectory + "/" + _resultsFolder + "/" + SUCCESSES_FOLDER;
QDir().mkdir(_htmlSuccessesFolder); QDir().mkdir(_htmlSuccessesFolder);
for (int i = 0; i < newNamesFailures.length(); ++i) { for (int i = 0; i < newNamesFailures.length(); ++i) {
@ -321,7 +320,7 @@ void AWSInterface::createEntry(int index, const QString& testResult, QTextStream
} }
void AWSInterface::updateAWS() { void AWSInterface::updateAWS() {
QString filename = _workingDirectory + "/updateAWS.py"; QString filename = _snapshotDirectory + "/updateAWS.py";
if (QFile::exists(filename)) { if (QFile::exists(filename)) {
QFile::remove(filename); QFile::remove(filename);
} }
@ -352,14 +351,23 @@ void AWSInterface::updateAWS() {
QStringList parts = nextDirectory.split('/'); QStringList parts = nextDirectory.split('/');
QString filename = parts[parts.length() - 3] + "/" + parts[parts.length() - 2] + "/" + parts[parts.length() - 1]; QString filename = parts[parts.length() - 3] + "/" + parts[parts.length() - 2] + "/" + parts[parts.length() - 1];
stream << "data = open('" << _workingDirectory << "/" << filename << "/" << "Actual Image.png" << "', 'rb')\n"; stream << "data = open('" << _snapshotDirectory << "/" << filename << "/"
<< "Actual Image.png"
<< "', 'rb')\n";
stream << "s3.Bucket('hifi-content').put_object(Bucket='" << AWS_BUCKET << "', Key='" << filename << "/" << "Actual Image.png" << "', Body=data)\n\n"; stream << "s3.Bucket('hifi-content').put_object(Bucket='" << AWS_BUCKET << "', Key='" << filename << "/" << "Actual Image.png" << "', Body=data)\n\n";
stream << "data = open('" << _workingDirectory << "/" << filename << "/" << "Expected Image.png" << "', 'rb')\n"; stream << "data = open('" << _snapshotDirectory << "/" << filename << "/"
<< "Expected Image.png"
<< "', 'rb')\n";
stream << "s3.Bucket('hifi-content').put_object(Bucket='" << AWS_BUCKET << "', Key='" << filename << "/" << "Expected Image.png" << "', Body=data)\n\n"; stream << "s3.Bucket('hifi-content').put_object(Bucket='" << AWS_BUCKET << "', Key='" << filename << "/" << "Expected Image.png" << "', Body=data)\n\n";
if (QFile::exists(_htmlFailuresFolder + "/" + parts[parts.length() - 1] + "/Difference Image.png")) { if (QFile::exists(_htmlFailuresFolder + "/" + parts[parts.length() - 1] + "/Difference Image.png")) {
stream << "data = open('" << _workingDirectory << "/" << filename << "/" << "Difference Image.png" << "', 'rb')\n"; stream << "data = open('" << _snapshotDirectory << "/" << filename << "/"
<< "Difference Image.png"
<< "', 'rb')\n";
stream << "s3.Bucket('hifi-content').put_object(Bucket='" << AWS_BUCKET << "', Key='" << filename << "/" << "Difference Image.png" << "', Body=data)\n\n"; stream << "s3.Bucket('hifi-content').put_object(Bucket='" << AWS_BUCKET << "', Key='" << filename << "/" << "Difference Image.png" << "', Body=data)\n\n";
} }
} }
@ -378,19 +386,28 @@ void AWSInterface::updateAWS() {
QStringList parts = nextDirectory.split('/'); QStringList parts = nextDirectory.split('/');
QString filename = parts[parts.length() - 3] + "/" + parts[parts.length() - 2] + "/" + parts[parts.length() - 1]; QString filename = parts[parts.length() - 3] + "/" + parts[parts.length() - 2] + "/" + parts[parts.length() - 1];
stream << "data = open('" << _workingDirectory << "/" << filename << "/" << "Actual Image.png" << "', 'rb')\n"; stream << "data = open('" << _snapshotDirectory << "/" << filename << "/"
<< "Actual Image.png"
<< "', 'rb')\n";
stream << "s3.Bucket('hifi-content').put_object(Bucket='" << AWS_BUCKET << "', Key='" << filename << "/" << "Actual Image.png" << "', Body=data)\n\n"; stream << "s3.Bucket('hifi-content').put_object(Bucket='" << AWS_BUCKET << "', Key='" << filename << "/" << "Actual Image.png" << "', Body=data)\n\n";
stream << "data = open('" << _workingDirectory << "/" << filename << "/" << "Expected Image.png" << "', 'rb')\n"; stream << "data = open('" << _snapshotDirectory << "/" << filename << "/"
<< "Expected Image.png"
<< "', 'rb')\n";
stream << "s3.Bucket('hifi-content').put_object(Bucket='" << AWS_BUCKET << "', Key='" << filename << "/" << "Expected Image.png" << "', Body=data)\n\n"; stream << "s3.Bucket('hifi-content').put_object(Bucket='" << AWS_BUCKET << "', Key='" << filename << "/" << "Expected Image.png" << "', Body=data)\n\n";
if (QFile::exists(_htmlSuccessesFolder + "/" + parts[parts.length() - 1] + "/Difference Image.png")) { if (QFile::exists(_htmlSuccessesFolder + "/" + parts[parts.length() - 1] + "/Difference Image.png")) {
stream << "data = open('" << _workingDirectory << "/" << filename << "/" << "Difference Image.png" << "', 'rb')\n"; stream << "data = open('" << _snapshotDirectory << "/" << filename << "/"
<< "Difference Image.png"
<< "', 'rb')\n";
stream << "s3.Bucket('hifi-content').put_object(Bucket='" << AWS_BUCKET << "', Key='" << filename << "/" << "Difference Image.png" << "', Body=data)\n\n"; stream << "s3.Bucket('hifi-content').put_object(Bucket='" << AWS_BUCKET << "', Key='" << filename << "/" << "Difference Image.png" << "', Body=data)\n\n";
} }
} }
stream << "data = open('" << _workingDirectory << "/" << _resultsFolder << "/" << HTML_FILENAME << "', 'rb')\n"; stream << "data = open('" << _snapshotDirectory << "/" << _resultsFolder << "/" << HTML_FILENAME << "', 'rb')\n";
stream << "s3.Bucket('hifi-content').put_object(Bucket='" << AWS_BUCKET << "', Key='" << _resultsFolder << "/" stream << "s3.Bucket('hifi-content').put_object(Bucket='" << AWS_BUCKET << "', Key='" << _resultsFolder << "/"
<< HTML_FILENAME << "', Body=data, ContentType='text/html')\n"; << HTML_FILENAME << "', Body=data, ContentType='text/html')\n";
@ -408,6 +425,11 @@ void AWSInterface::updateAWS() {
connect(process, static_cast<void (QProcess::*)(int, QProcess::ExitStatus)>(&QProcess::finished), this, connect(process, static_cast<void (QProcess::*)(int, QProcess::ExitStatus)>(&QProcess::finished), this,
[=](int exitCode, QProcess::ExitStatus exitStatus) { _busyWindow.hide(); }); [=](int exitCode, QProcess::ExitStatus exitStatus) { _busyWindow.hide(); });
#ifdef Q_OS_WIN
QStringList parameters = QStringList() << filename ; QStringList parameters = QStringList() << filename ;
process->start(_pythonCommand, parameters); process->start(_pythonCommand, parameters);
#elif defined Q_OS_MAC
QStringList parameters = QStringList() << "-c" << _pythonCommand + " " + filename;
process->start("sh", parameters);
#endif
} }

View file

@ -26,7 +26,7 @@ public:
explicit AWSInterface(QObject* parent = 0); explicit AWSInterface(QObject* parent = 0);
void createWebPageFromResults(const QString& testResults, void createWebPageFromResults(const QString& testResults,
const QString& workingDirectory, const QString& snapshotDirectory,
QCheckBox* updateAWSCheckBox, QCheckBox* updateAWSCheckBox,
QLineEdit* urlLineEdit); QLineEdit* urlLineEdit);
@ -49,7 +49,7 @@ public:
private: private:
QString _testResults; QString _testResults;
QString _workingDirectory; QString _snapshotDirectory;
QString _resultsFolder; QString _resultsFolder;
QString _htmlFailuresFolder; QString _htmlFailuresFolder;
QString _htmlSuccessesFolder; QString _htmlSuccessesFolder;

View file

@ -17,8 +17,7 @@ Downloader::Downloader(QUrl fileURL, QObject *parent) : QObject(parent) {
this, SLOT (fileDownloaded(QNetworkReply*)) this, SLOT (fileDownloaded(QNetworkReply*))
); );
QNetworkRequest request(fileURL); _networkAccessManager.get(QNetworkRequest(fileURL));
_networkAccessManager.get(request);
} }
void Downloader::fileDownloaded(QNetworkReply* reply) { void Downloader::fileDownloaded(QNetworkReply* reply) {
@ -37,4 +36,4 @@ void Downloader::fileDownloaded(QNetworkReply* reply) {
QByteArray Downloader::downloadedData() const { QByteArray Downloader::downloadedData() const {
return _downloadedData; return _downloadedData;
} }

View file

@ -37,7 +37,7 @@ public:
signals: signals:
void downloaded(); void downloaded();
private slots: private slots:
void fileDownloaded(QNetworkReply* pReply); void fileDownloaded(QNetworkReply* pReply);
private: private:

View file

@ -14,17 +14,28 @@
#include <QProcess> #include <QProcess>
PythonInterface::PythonInterface() { PythonInterface::PythonInterface() {
#ifdef Q_OS_WIN
if (QProcessEnvironment::systemEnvironment().contains("PYTHON_PATH")) { if (QProcessEnvironment::systemEnvironment().contains("PYTHON_PATH")) {
QString _pythonPath = QProcessEnvironment::systemEnvironment().value("PYTHON_PATH"); QString _pythonPath = QProcessEnvironment::systemEnvironment().value("PYTHON_PATH");
if (!QFile::exists(_pythonPath + "/" + _pythonExe)) { if (!QFile::exists(_pythonPath + "/" + _pythonExe)) {
QMessageBox::critical(0, _pythonExe, QString("Python executable not found in ") + _pythonPath); QMessageBox::critical(0, _pythonExe, QString("Python executable not found in ") + _pythonPath);
exit(-1);
} }
_pythonCommand = _pythonPath + "/" + _pythonExe; _pythonCommand = _pythonPath + "/" + _pythonExe;
} else { } else {
QMessageBox::critical(0, "PYTHON_PATH not defined", QMessageBox::critical(0, "PYTHON_PATH not defined",
"Please set PYTHON_PATH to directory containing the Python executable"); "Please set PYTHON_PATH to directory containing the Python executable");
exit(-1); exit(-1);
} }
#elif defined Q_OS_MAC
_pythonCommand = "/usr/local/bin/python3";
if (!QFile::exists(_pythonCommand)) {
QMessageBox::critical(0, "PYTHON_PATH not defined",
"python3 not found at " + _pythonCommand);
exit(-1);
}
#endif
} }
QString PythonInterface::getPythonCommand() { QString PythonInterface::getPythonCommand() {

View file

@ -19,7 +19,13 @@ public:
QString getPythonCommand(); QString getPythonCommand();
private: private:
#ifdef Q_OS_WIN
const QString _pythonExe{ "python.exe" }; const QString _pythonExe{ "python.exe" };
#else
// Both Mac and Linux use "python"
const QString _pythonExe{ "python" };
#endif
QString _pythonCommand; QString _pythonCommand;
}; };

View file

@ -19,8 +19,8 @@
#include <quazip5/quazip.h> #include <quazip5/quazip.h>
#include <quazip5/JlCompress.h> #include <quazip5/JlCompress.h>
#include "ui/AutoTester.h" #include "ui/Nitpick.h"
extern AutoTester* autoTester; extern Nitpick* nitpick;
#include <math.h> #include <math.h>
@ -30,9 +30,9 @@ Test::Test(QProgressBar* progressBar, QCheckBox* checkBoxInteractiveMode) {
_mismatchWindow.setModal(true); _mismatchWindow.setModal(true);
if (autoTester) { if (nitpick) {
autoTester->setUserText(GIT_HUB_DEFAULT_USER); nitpick->setUserText(GIT_HUB_DEFAULT_USER);
autoTester->setBranchText(GIT_HUB_DEFAULT_BRANCH); nitpick->setBranchText(GIT_HUB_DEFAULT_BRANCH);
} }
} }
@ -167,7 +167,7 @@ void Test::appendTestResultsToFile(const QString& _testResultsFolderPath, TestRe
// Create text file describing the failure // Create text file describing the failure
QTextStream stream(&descriptionFile); QTextStream stream(&descriptionFile);
stream << "Test failed in folder " << testResult._pathname.left(testResult._pathname.length() - 1) << endl; // remove trailing '/' stream << "Test in folder " << testResult._pathname.left(testResult._pathname.length() - 1) << endl; // remove trailing '/'
stream << "Expected image was " << testResult._expectedImageFilename << endl; stream << "Expected image was " << testResult._expectedImageFilename << endl;
stream << "Actual image was " << testResult._actualImageFilename << endl; stream << "Actual image was " << testResult._actualImageFilename << endl;
stream << "Similarity index was " << testResult._error << endl; stream << "Similarity index was " << testResult._error << endl;
@ -240,8 +240,8 @@ void Test::startTestsEvaluation(const bool isRunningFromCommandLine,
_expectedImagesFilenames.clear(); _expectedImagesFilenames.clear();
_expectedImagesFullFilenames.clear(); _expectedImagesFullFilenames.clear();
QString branch = (branchFromCommandLine.isNull()) ? autoTester->getSelectedBranch() : branchFromCommandLine; QString branch = (branchFromCommandLine.isNull()) ? nitpick->getSelectedBranch() : branchFromCommandLine;
QString user = (userFromCommandLine.isNull()) ? autoTester->getSelectedUser() : userFromCommandLine; QString user = (userFromCommandLine.isNull()) ? nitpick->getSelectedUser() : userFromCommandLine;
foreach(QString currentFilename, sortedTestResultsFilenames) { foreach(QString currentFilename, sortedTestResultsFilenames) {
QString fullCurrentFilename = _snapshotDirectory + "/" + currentFilename; QString fullCurrentFilename = _snapshotDirectory + "/" + currentFilename;
@ -268,7 +268,7 @@ void Test::startTestsEvaluation(const bool isRunningFromCommandLine,
} }
} }
autoTester->downloadFiles(expectedImagesURLs, _snapshotDirectory, _expectedImagesFilenames, (void *)this); nitpick->downloadFiles(expectedImagesURLs, _snapshotDirectory, _expectedImagesFilenames, (void *)this);
} }
void Test::finishTestsEvaluation() { void Test::finishTestsEvaluation() {
int numberOfFailures = compareImageLists(); int numberOfFailures = compareImageLists();
@ -288,7 +288,7 @@ void Test::finishTestsEvaluation() {
} }
if (_isRunningInAutomaticTestRun) { if (_isRunningInAutomaticTestRun) {
autoTester->automaticTestRunEvaluationComplete(zippedFolderName, numberOfFailures); nitpick->automaticTestRunEvaluationComplete(zippedFolderName, numberOfFailures);
} }
} }
@ -429,22 +429,22 @@ ExtractedText Test::getTestScriptLines(QString testFileName) {
QString line = stream.readLine(); QString line = stream.readLine();
// Name of test is the string in the following line: // Name of test is the string in the following line:
// autoTester.perform("Apply Material Entities to Avatars", Script.resolvePath("."), function(testType) {... // nitpick.perform("Apply Material Entities to Avatars", Script.resolvePath("."), function(testType) {...
const QString ws("\\h*"); //white-space character const QString ws("\\h*"); //white-space character
const QString functionPerformName(ws + "autoTester" + ws + "\\." + ws + "perform"); const QString functionPerformName(ws + "nitpick" + ws + "\\." + ws + "perform");
const QString quotedString("\\\".+\\\""); const QString quotedString("\\\".+\\\"");
QString regexTestTitle(ws + functionPerformName + "\\(" + quotedString); QString regexTestTitle(ws + functionPerformName + "\\(" + quotedString);
QRegularExpression lineContainingTitle = QRegularExpression(regexTestTitle); QRegularExpression lineContainingTitle = QRegularExpression(regexTestTitle);
// Each step is either of the following forms: // Each step is either of the following forms:
// autoTester.addStepSnapshot("Take snapshot"... // nitpick.addStepSnapshot("Take snapshot"...
// autoTester.addStep("Clean up after test"... // nitpick.addStep("Clean up after test"...
const QString functionAddStepSnapshotName(ws + "autoTester" + ws + "\\." + ws + "addStepSnapshot"); const QString functionAddStepSnapshotName(ws + "nitpick" + ws + "\\." + ws + "addStepSnapshot");
const QString regexStepSnapshot(ws + functionAddStepSnapshotName + ws + "\\(" + ws + quotedString + ".*"); const QString regexStepSnapshot(ws + functionAddStepSnapshotName + ws + "\\(" + ws + quotedString + ".*");
const QRegularExpression lineStepSnapshot = QRegularExpression(regexStepSnapshot); const QRegularExpression lineStepSnapshot = QRegularExpression(regexStepSnapshot);
const QString functionAddStepName(ws + "autoTester" + ws + "\\." + ws + "addStep"); const QString functionAddStepName(ws + "nitpick" + ws + "\\." + ws + "addStep");
const QString regexStep(ws + functionAddStepName + ws + "\\(" + ws + quotedString + ".*"); const QString regexStep(ws + functionAddStepName + ws + "\\(" + ws + quotedString + ".*");
const QRegularExpression lineStep = QRegularExpression(regexStep); const QRegularExpression lineStep = QRegularExpression(regexStep);
@ -620,7 +620,7 @@ void Test::createTestAutoScript() {
} }
if (createTestAutoScript(_testDirectory)) { if (createTestAutoScript(_testDirectory)) {
QMessageBox::information(0, "Success", "'autoTester.js` script has been created"); QMessageBox::information(0, "Success", "'nitpick.js` script has been created");
} }
} }
@ -653,7 +653,7 @@ void Test::createAllTestAutoScripts() {
} }
} }
QMessageBox::information(0, "Success", "'autoTester.js' scripts have been created"); QMessageBox::information(0, "Success", "'nitpick.js' scripts have been created");
} }
bool Test::createTestAutoScript(const QString& directory) { bool Test::createTestAutoScript(const QString& directory) {
@ -677,8 +677,8 @@ bool Test::createTestAutoScript(const QString& directory) {
stream << "if (typeof PATH_TO_THE_REPO_PATH_UTILS_FILE === 'undefined') PATH_TO_THE_REPO_PATH_UTILS_FILE = 'https://raw.githubusercontent.com/highfidelity/hifi_tests/master/tests/utils/branchUtils.js';\n"; stream << "if (typeof PATH_TO_THE_REPO_PATH_UTILS_FILE === 'undefined') PATH_TO_THE_REPO_PATH_UTILS_FILE = 'https://raw.githubusercontent.com/highfidelity/hifi_tests/master/tests/utils/branchUtils.js';\n";
stream << "Script.include(PATH_TO_THE_REPO_PATH_UTILS_FILE);\n"; stream << "Script.include(PATH_TO_THE_REPO_PATH_UTILS_FILE);\n";
stream << "var autoTester = createAutoTester(Script.resolvePath('.'));\n\n"; stream << "var nitpick = createAutoTester(Script.resolvePath('.'));\n\n";
stream << "autoTester.enableAuto();\n\n"; stream << "nitpick.enableAuto();\n\n";
stream << "Script.include('./test.js?raw=true');\n"; stream << "Script.include('./test.js?raw=true');\n";
testAutoScriptFile.close(); testAutoScriptFile.close();
@ -751,29 +751,29 @@ void Test::createRecursiveScript(const QString& topLevelDirectory, bool interact
textStream << "// This is an automatically generated file, created by auto-tester" << endl; textStream << "// This is an automatically generated file, created by auto-tester" << endl;
// Include 'autoTest.js' // Include 'autoTest.js'
QString branch = autoTester->getSelectedBranch(); QString branch = nitpick->getSelectedBranch();
QString user = autoTester->getSelectedUser(); QString user = nitpick->getSelectedUser();
textStream << "PATH_TO_THE_REPO_PATH_UTILS_FILE = \"https://raw.githubusercontent.com/" + user + "/hifi_tests/" + branch + textStream << "PATH_TO_THE_REPO_PATH_UTILS_FILE = \"https://raw.githubusercontent.com/" + user + "/hifi_tests/" + branch +
"/tests/utils/branchUtils.js\";" "/tests/utils/branchUtils.js\";"
<< endl; << endl;
textStream << "Script.include(PATH_TO_THE_REPO_PATH_UTILS_FILE);" << endl; textStream << "Script.include(PATH_TO_THE_REPO_PATH_UTILS_FILE);" << endl;
textStream << "var autoTester = createAutoTester(Script.resolvePath(\".\"));" << endl << endl; textStream << "var nitpick = createAutoTester(Script.resolvePath(\".\"));" << endl << endl;
textStream << "var testsRootPath = autoTester.getTestsRootPath();" << endl << endl; textStream << "var testsRootPath = nitpick.getTestsRootPath();" << endl << endl;
// Wait 10 seconds before starting // Wait 10 seconds before starting
textStream << "if (typeof Test !== 'undefined') {" << endl; textStream << "if (typeof Test !== 'undefined') {" << endl;
textStream << " Test.wait(10000);" << endl; textStream << " Test.wait(10000);" << endl;
textStream << "};" << endl << endl; textStream << "};" << endl << endl;
textStream << "autoTester.enableRecursive();" << endl; textStream << "nitpick.enableRecursive();" << endl;
textStream << "autoTester.enableAuto();" << endl << endl; textStream << "nitpick.enableAuto();" << endl << endl;
// This is used to verify that the recursive test contains at least one test // This is used to verify that the recursive test contains at least one test
bool testFound{ false }; bool testFound{ false };
// Directories are included in reverse order. The autoTester scripts use a stack mechanism, // Directories are included in reverse order. The nitpick scripts use a stack mechanism,
// so this ensures that the tests run in alphabetical order (a convenience when debugging) // so this ensures that the tests run in alphabetical order (a convenience when debugging)
QStringList directories; QStringList directories;
@ -819,7 +819,7 @@ void Test::createRecursiveScript(const QString& topLevelDirectory, bool interact
} }
textStream << endl; textStream << endl;
textStream << "autoTester.runRecursive();" << endl; textStream << "nitpick.runRecursive();" << endl;
allTestsFilename.close(); allTestsFilename.close();
} }
@ -881,7 +881,7 @@ void Test::createTestsOutline() {
// The directory name appears after the last slash (we are assured there is at least 1). // The directory name appears after the last slash (we are assured there is at least 1).
QString directoryName = directory.right(directory.length() - directory.lastIndexOf("/") - 1); QString directoryName = directory.right(directory.length() - directory.lastIndexOf("/") - 1);
// autoTester is run on a clone of the repository. We use relative paths, so we can use both local disk and GitHub // nitpick is run on a clone of the repository. We use relative paths, so we can use both local disk and GitHub
// For a test in "D:/GitHub/hifi_tests/tests/content/entity/zone/ambientLightInheritance" the // For a test in "D:/GitHub/hifi_tests/tests/content/entity/zone/ambientLightInheritance" the
// GitHub URL is "./content/entity/zone/ambientLightInheritance?raw=true" // GitHub URL is "./content/entity/zone/ambientLightInheritance?raw=true"
QString partialPath = directory.right(directory.length() - (directory.lastIndexOf("/tests/") + QString("/tests").length() + 1)); QString partialPath = directory.right(directory.length() - (directory.lastIndexOf("/tests/") + QString("/tests").length() + 1));
@ -937,11 +937,11 @@ void Test::createTestRailTestCases() {
} }
if (_testRailCreateMode == PYTHON) { if (_testRailCreateMode == PYTHON) {
_testRailInterface.createTestSuitePython(_testDirectory, outputDirectory, autoTester->getSelectedUser(), _testRailInterface.createTestSuitePython(_testDirectory, outputDirectory, nitpick->getSelectedUser(),
autoTester->getSelectedBranch()); nitpick->getSelectedBranch());
} else { } else {
_testRailInterface.createTestSuiteXML(_testDirectory, outputDirectory, autoTester->getSelectedUser(), _testRailInterface.createTestSuiteXML(_testDirectory, outputDirectory, nitpick->getSelectedUser(),
autoTester->getSelectedBranch()); nitpick->getSelectedBranch());
} }
} }
@ -1052,11 +1052,11 @@ void Test::createWebPage(QCheckBox* updateAWSCheckBox, QLineEdit* urlLineEdit) {
return; return;
} }
QString tempDirectory = QFileDialog::getExistingDirectory(nullptr, "Please select a folder to store temporary files in", QString snapshotDirectory = QFileDialog::getExistingDirectory(nullptr, "Please select a folder to store temporary files in",
nullptr, QFileDialog::ShowDirsOnly); nullptr, QFileDialog::ShowDirsOnly);
if (tempDirectory.isNull()) { if (snapshotDirectory.isNull()) {
return; return;
} }
_awsInterface.createWebPageFromResults(testResults, tempDirectory, updateAWSCheckBox, urlLineEdit); _awsInterface.createWebPageFromResults(testResults, snapshotDirectory, updateAWSCheckBox, urlLineEdit);
} }

View file

@ -146,7 +146,7 @@ private:
const QString DATETIME_FORMAT{ "yyyy-MM-dd_hh-mm-ss" }; const QString DATETIME_FORMAT{ "yyyy-MM-dd_hh-mm-ss" };
// NOTE: these need to match the appropriate var's in autoTester.js // NOTE: these need to match the appropriate var's in nitpick.js
// var advanceKey = "n"; // var advanceKey = "n";
// var pathSeparator = "."; // var pathSeparator = ".";
const QString ADVANCE_KEY{ "n" }; const QString ADVANCE_KEY{ "n" };

View file

@ -357,8 +357,13 @@ void TestRailInterface::createAddTestCasesPythonScript(const QString& testDirect
connect(process, static_cast<void (QProcess::*)(int, QProcess::ExitStatus)>(&QProcess::finished), this, connect(process, static_cast<void (QProcess::*)(int, QProcess::ExitStatus)>(&QProcess::finished), this,
[=](int exitCode, QProcess::ExitStatus exitStatus) { _busyWindow.hide(); }); [=](int exitCode, QProcess::ExitStatus exitStatus) { _busyWindow.hide(); });
#ifdef Q_OS_WIN
QStringList parameters = QStringList() << _outputDirectory + "/addTestCases.py"; QStringList parameters = QStringList() << _outputDirectory + "/addTestCases.py";
process->start(_pythonCommand, parameters); process->start(_pythonCommand, parameters);
#elif defined Q_OS_MAC
QStringList parameters = QStringList() << "-c" << _pythonCommand + " " + _outputDirectory + "/addTestCases.py";
process->start("sh", parameters);
#endif
} }
} }
@ -482,8 +487,13 @@ void TestRailInterface::addRun() {
connect(process, static_cast<void (QProcess::*)(int, QProcess::ExitStatus)>(&QProcess::finished), this, connect(process, static_cast<void (QProcess::*)(int, QProcess::ExitStatus)>(&QProcess::finished), this,
[=](int exitCode, QProcess::ExitStatus exitStatus) { _busyWindow.hide(); }); [=](int exitCode, QProcess::ExitStatus exitStatus) { _busyWindow.hide(); });
#ifdef Q_OS_WIN
QStringList parameters = QStringList() << _outputDirectory + "/addRun.py"; QStringList parameters = QStringList() << _outputDirectory + "/addRun.py";
process->start(_pythonCommand, parameters); process->start(_pythonCommand, parameters);
#elif defined Q_OS_MAC
QStringList parameters = QStringList() << "-c" << _pythonCommand + " " + _outputDirectory + "/addRun.py";
process->start("sh", parameters);
#endif
} }
} }
@ -586,8 +596,13 @@ void TestRailInterface::updateRunWithResults() {
connect(process, static_cast<void (QProcess::*)(int, QProcess::ExitStatus)>(&QProcess::finished), this, connect(process, static_cast<void (QProcess::*)(int, QProcess::ExitStatus)>(&QProcess::finished), this,
[=](int exitCode, QProcess::ExitStatus exitStatus) { _busyWindow.hide(); }); [=](int exitCode, QProcess::ExitStatus exitStatus) { _busyWindow.hide(); });
#ifdef Q_OS_WIN
QStringList parameters = QStringList() << _outputDirectory + "/updateRunWithResults.py"; QStringList parameters = QStringList() << _outputDirectory + "/updateRunWithResults.py";
process->start(_pythonCommand, parameters); process->start(_pythonCommand, parameters);
#elif defined Q_OS_MAC
QStringList parameters = QStringList() << "-c" << _pythonCommand + " " + _outputDirectory + "/updateRunWithResults.py";
process->start("sh", parameters);
#endif
} }
} }
@ -759,8 +774,13 @@ void TestRailInterface::getReleasesFromTestRail() {
[=](int exitCode, QProcess::ExitStatus exitStatus) { updateReleasesComboData(exitCode, exitStatus); }); [=](int exitCode, QProcess::ExitStatus exitStatus) { updateReleasesComboData(exitCode, exitStatus); });
connect(process, SIGNAL(finished(int)), process, SLOT(deleteLater())); connect(process, SIGNAL(finished(int)), process, SLOT(deleteLater()));
#ifdef Q_OS_WIN
QStringList parameters = QStringList() << filename; QStringList parameters = QStringList() << filename;
process->start(_pythonCommand, parameters); process->start(_pythonCommand, parameters);
#elif defined Q_OS_MAC
QStringList parameters = QStringList() << "-c" << _pythonCommand + " " + filename;
process->start("sh", parameters);
#endif
} }
void TestRailInterface::createTestSuitePython(const QString& testDirectory, void TestRailInterface::createTestSuitePython(const QString& testDirectory,
@ -1078,8 +1098,13 @@ void TestRailInterface::getTestSectionsFromTestRail() {
[=](int exitCode, QProcess::ExitStatus exitStatus) { updateSectionsComboData(exitCode, exitStatus); }); [=](int exitCode, QProcess::ExitStatus exitStatus) { updateSectionsComboData(exitCode, exitStatus); });
connect(process, SIGNAL(finished(int)), process, SLOT(deleteLater())); connect(process, SIGNAL(finished(int)), process, SLOT(deleteLater()));
#ifdef Q_OS_WIN
QStringList parameters = QStringList() << filename; QStringList parameters = QStringList() << filename;
process->start(_pythonCommand, parameters); process->start(_pythonCommand, parameters);
#elif defined Q_OS_MAC
QStringList parameters = QStringList() << "-c" << _pythonCommand + " " + filename;
process->start("sh", parameters);
#endif
} }
void TestRailInterface::getRunsFromTestRail() { void TestRailInterface::getRunsFromTestRail() {
@ -1117,9 +1142,13 @@ void TestRailInterface::getRunsFromTestRail() {
[=](int exitCode, QProcess::ExitStatus exitStatus) { updateRunsComboData(exitCode, exitStatus); }); [=](int exitCode, QProcess::ExitStatus exitStatus) { updateRunsComboData(exitCode, exitStatus); });
connect(process, SIGNAL(finished(int)), process, SLOT(deleteLater())); connect(process, SIGNAL(finished(int)), process, SLOT(deleteLater()));
#ifdef Q_OS_WIN
QStringList parameters = QStringList() << filename; QStringList parameters = QStringList() << filename;
process->start(_pythonCommand, parameters); process->start(_pythonCommand, parameters);
#elif defined Q_OS_MAC
QStringList parameters = QStringList() << "-c" << _pythonCommand + " " + filename;
process->start("sh", parameters);
#endif
} }
void TestRailInterface::createTestRailRun(const QString& outputDirectory) { void TestRailInterface::createTestRailRun(const QString& outputDirectory) {
@ -1164,4 +1193,4 @@ void TestRailInterface::extractTestFailuresFromZippedFolder(const QString& testR
QDir dir = tempSubDirectory; QDir dir = tempSubDirectory;
dir.mkdir(tempSubDirectory); dir.mkdir(tempSubDirectory);
JlCompress::extractDir(testResults, tempSubDirectory); JlCompress::extractDir(testResults, tempSubDirectory);
} }

View file

@ -13,14 +13,17 @@
#include <QtWidgets/QMessageBox> #include <QtWidgets/QMessageBox>
#include <QtWidgets/QFileDialog> #include <QtWidgets/QFileDialog>
#include "ui/AutoTester.h" #include "ui/Nitpick.h"
extern AutoTester* autoTester; extern Nitpick* nitpick;
#ifdef Q_OS_WIN #ifdef Q_OS_WIN
#include <windows.h> #include <windows.h>
#include <tlhelp32.h> #include <tlhelp32.h>
#endif #endif
// TODO: for debug
#include <iostream>
TestRunner::TestRunner(std::vector<QCheckBox*> dayCheckboxes, TestRunner::TestRunner(std::vector<QCheckBox*> dayCheckboxes,
std::vector<QCheckBox*> timeEditCheckboxes, std::vector<QCheckBox*> timeEditCheckboxes,
std::vector<QTimeEdit*> timeEdits, std::vector<QTimeEdit*> timeEdits,
@ -40,27 +43,29 @@ TestRunner::TestRunner(std::vector<QCheckBox*> dayCheckboxes,
_url = url; _url = url;
_runNow = runNow; _runNow = runNow;
installerThread = new QThread(); _installerThread = new QThread();
installerWorker = new Worker(); _installerWorker = new Worker();
installerWorker->moveToThread(installerThread);
installerThread->start(); _installerWorker->moveToThread(_installerThread);
connect(this, SIGNAL(startInstaller()), installerWorker, SLOT(runCommand())); _installerThread->start();
connect(installerWorker, SIGNAL(commandComplete()), this, SLOT(installationComplete())); connect(this, SIGNAL(startInstaller()), _installerWorker, SLOT(runCommand()));
connect(_installerWorker, SIGNAL(commandComplete()), this, SLOT(installationComplete()));
interfaceThread = new QThread(); _interfaceThread = new QThread();
interfaceWorker = new Worker(); _interfaceWorker = new Worker();
interfaceThread->start();
interfaceWorker->moveToThread(interfaceThread); _interfaceThread->start();
connect(this, SIGNAL(startInterface()), interfaceWorker, SLOT(runCommand())); _interfaceWorker->moveToThread(_interfaceThread);
connect(interfaceWorker, SIGNAL(commandComplete()), this, SLOT(interfaceExecutionComplete())); connect(this, SIGNAL(startInterface()), _interfaceWorker, SLOT(runCommand()));
connect(_interfaceWorker, SIGNAL(commandComplete()), this, SLOT(interfaceExecutionComplete()));
} }
TestRunner::~TestRunner() { TestRunner::~TestRunner() {
delete installerThread; delete _installerThread;
delete interfaceThread; delete _installerWorker;
delete interfaceThread; delete _interfaceThread;
delete interfaceWorker; delete _interfaceWorker;
if (_timer) { if (_timer) {
delete _timer; delete _timer;
@ -84,15 +89,96 @@ void TestRunner::setWorkingFolder() {
return; return;
} }
#ifdef Q_OS_WIN
_installationFolder = _workingFolder + "/High Fidelity"; _installationFolder = _workingFolder + "/High Fidelity";
#elif defined Q_OS_MAC
_installationFolder = _workingFolder + "/High_Fidelity";
#endif
_logFile.setFileName(_workingFolder + "/log.txt"); _logFile.setFileName(_workingFolder + "/log.txt");
autoTester->enableRunTabControls(); nitpick->enableRunTabControls();
_workingFolderLabel->setText(QDir::toNativeSeparators(_workingFolder)); _workingFolderLabel->setText(QDir::toNativeSeparators(_workingFolder));
_timer = new QTimer(this); _timer = new QTimer(this);
connect(_timer, SIGNAL(timeout()), this, SLOT(checkTime())); connect(_timer, SIGNAL(timeout()), this, SLOT(checkTime()));
_timer->start(30 * 1000); //time specified in ms _timer->start(30 * 1000); //time specified in ms
#ifdef Q_OS_MAC
// Create MAC shell scripts
QFile script;
// This script waits for a process to start
script.setFileName(_workingFolder + "/waitForStart.sh");
if (!script.open(QIODevice::WriteOnly | QIODevice::Text)) {
QMessageBox::critical(0, "Internal error: " + QString(__FILE__) + ":" + QString::number(__LINE__),
"Could not open 'waitForStart.sh'");
exit(-1);
}
script.write("#!/bin/sh\n\n");
script.write("PROCESS=\"$1\"\n");
script.write("until (pgrep -x $PROCESS >nul)\n");
script.write("do\n");
script.write("\techo waiting for \"$1\" to start\n");
script.write("\tsleep 2\n");
script.write("done\n");
script.write("echo \"$1\" \"started\"\n");
script.close();
script.setPermissions(QFile::ReadOwner | QFile::WriteOwner | QFile::ExeOwner);
// The Mac shell command returns immediately. This little script waits for a process to finish
script.setFileName(_workingFolder + "/waitForFinish.sh");
if (!script.open(QIODevice::WriteOnly | QIODevice::Text)) {
QMessageBox::critical(0, "Internal error: " + QString(__FILE__) + ":" + QString::number(__LINE__),
"Could not open 'waitForFinish.sh'");
exit(-1);
}
script.write("#!/bin/sh\n\n");
script.write("PROCESS=\"$1\"\n");
script.write("while (pgrep -x $PROCESS >nul)\n");
script.write("do\n");
script.write("\techo waiting for \"$1\" to finish\n");
script.write("\tsleep 2\n");
script.write("done\n");
script.write("echo \"$1\" \"finished\"\n");
script.close();
script.setPermissions(QFile::ReadOwner | QFile::WriteOwner | QFile::ExeOwner);
// Create an AppleScript to resize Interface. This is needed so that snapshots taken
// with the primary camera will be the correct size.
// This will be run from a normal shell script
script.setFileName(_workingFolder + "/setInterfaceSizeAndPosition.scpt");
if (!script.open(QIODevice::WriteOnly | QIODevice::Text)) {
QMessageBox::critical(0, "Internal error: " + QString(__FILE__) + ":" + QString::number(__LINE__),
"Could not open 'setInterfaceSizeAndPosition.scpt'");
exit(-1);
}
script.write("set width to 960\n");
script.write("set height to 540\n");
script.write("set x to 100\n");
script.write("set y to 100\n\n");
script.write("tell application \"System Events\" to tell application process \"interface\" to tell window 1 to set {size, position} to {{width, height}, {x, y}}\n");
script.close();
script.setPermissions(QFile::ReadOwner | QFile::WriteOwner | QFile::ExeOwner);
script.setFileName(_workingFolder + "/setInterfaceSizeAndPosition.sh");
if (!script.open(QIODevice::WriteOnly | QIODevice::Text)) {
QMessageBox::critical(0, "Internal error: " + QString(__FILE__) + ":" + QString::number(__LINE__),
"Could not open 'setInterfaceSizeAndPosition.sh'");
exit(-1);
}
script.write("#!/bin/sh\n\n");
script.write("echo resizing interface\n");
script.write(("osascript " + _workingFolder + "/setInterfaceSizeAndPosition.scpt\n").toStdString().c_str());
script.write("echo resize complete\n");
script.close();
script.setPermissions(QFile::ReadOwner | QFile::WriteOwner | QFile::ExeOwner);
#endif
} }
void TestRunner::run() { void TestRunner::run() {
@ -102,8 +188,8 @@ void TestRunner::run() {
_automatedTestIsRunning = true; _automatedTestIsRunning = true;
// Initial setup // Initial setup
_branch = autoTester->getSelectedBranch(); _branch = nitpick->getSelectedBranch();
_user = autoTester->getSelectedUser(); _user = nitpick->getSelectedUser();
// This will be restored at the end of the tests // This will be restored at the end of the tests
saveExistingHighFidelityAppDataFolder(); saveExistingHighFidelityAppDataFolder();
@ -120,7 +206,7 @@ void TestRunner::run() {
updateStatusLabel("Downloading Build XML"); updateStatusLabel("Downloading Build XML");
buildXMLDownloaded = false; buildXMLDownloaded = false;
autoTester->downloadFiles(urls, _workingFolder, filenames, (void*)this); nitpick->downloadFiles(urls, _workingFolder, filenames, (void*)this);
// `downloadComplete` will run after download has completed // `downloadComplete` will run after download has completed
} }
@ -149,7 +235,7 @@ void TestRunner::downloadComplete() {
updateStatusLabel("Downloading installer"); updateStatusLabel("Downloading installer");
autoTester->downloadFiles(urls, _workingFolder, filenames, (void*)this); nitpick->downloadFiles(urls, _workingFolder, filenames, (void*)this);
// `downloadComplete` will run again after download has completed // `downloadComplete` will run again after download has completed
@ -176,10 +262,43 @@ void TestRunner::runInstaller() {
QString installerFullPath = _workingFolder + "/" + _installerFilename; QString installerFullPath = _workingFolder + "/" + _installerFilename;
QString commandLine = QString commandLine;
"\"" + QDir::toNativeSeparators(installerFullPath) + "\"" + " /S /D=" + QDir::toNativeSeparators(_installationFolder); #ifdef Q_OS_WIN
commandLine = "\"" + QDir::toNativeSeparators(installerFullPath) + "\"" + " /S /D=" + QDir::toNativeSeparators(_installationFolder);
#elif defined Q_OS_MAC
// Create installation shell script
QFile script;
script.setFileName(_workingFolder + "/install_app.sh");
if (!script.open(QIODevice::WriteOnly | QIODevice::Text)) {
QMessageBox::critical(0, "Internal error: " + QString(__FILE__) + ":" + QString::number(__LINE__),
"Could not open 'install_app.sh'");
exit(-1);
}
if (!QDir().exists(_installationFolder)) {
QDir().mkdir(_installationFolder);
}
// This script installs High Fidelity. It is run as "yes | install_app.sh... so "yes" is killed at the end
script.write("#!/bin/sh\n\n");
script.write("VOLUME=`hdiutil attach \"$1\" | grep Volumes | awk '{print $3}'`\n");
QString folderName {"High Fidelity"};
if (!_runLatest->isChecked()) {
folderName += QString(" - ") + getPRNumberFromURL(_url->text());
}
installerWorker->setCommandLine(commandLine); script.write((QString("cp -rf \"$VOLUME/") + folderName + "/interface.app\" \"" + _workingFolder + "/High_Fidelity/\"\n").toStdString().c_str());
script.write((QString("cp -rf \"$VOLUME/") + folderName + "/Sandbox.app\" \"" + _workingFolder + "/High_Fidelity/\"\n").toStdString().c_str());
script.write("hdiutil detach \"$VOLUME\"\n");
script.write("killall yes\n");
script.close();
script.setPermissions(QFile::ReadOwner | QFile::WriteOwner | QFile::ExeOwner);
commandLine = "yes | " + _workingFolder + "/install_app.sh " + installerFullPath;
#endif
appendLog(commandLine);
_installerWorker->setCommandLine(commandLine);
emit startInstaller(); emit startInstaller();
} }
@ -213,11 +332,10 @@ void TestRunner::verifyInstallationSucceeded() {
} }
void TestRunner::saveExistingHighFidelityAppDataFolder() { void TestRunner::saveExistingHighFidelityAppDataFolder() {
#ifdef Q_OS_WIN
QString dataDirectory{ "NOT FOUND" }; QString dataDirectory{ "NOT FOUND" };
#ifdef Q_OS_WIN
dataDirectory = qgetenv("USERPROFILE") + "\\AppData\\Roaming"; dataDirectory = qgetenv("USERPROFILE") + "\\AppData\\Roaming";
#endif
if (_runLatest->isChecked()) { if (_runLatest->isChecked()) {
_appDataFolder = dataDirectory + "\\High Fidelity"; _appDataFolder = dataDirectory + "\\High Fidelity";
@ -238,6 +356,9 @@ void TestRunner::saveExistingHighFidelityAppDataFolder() {
// Copy an "empty" AppData folder (i.e. no entities) // Copy an "empty" AppData folder (i.e. no entities)
copyFolder(QDir::currentPath() + "/AppDataHighFidelity", _appDataFolder.path()); copyFolder(QDir::currentPath() + "/AppDataHighFidelity", _appDataFolder.path());
#elif defined Q_OS_MAC
// TODO: find Mac equivalent of AppData
#endif
} }
void TestRunner::createSnapshotFolder() { void TestRunner::createSnapshotFolder() {
@ -307,44 +428,110 @@ void TestRunner::killProcesses() {
QMessageBox::critical(0, "Internal error: " + QString(__FILE__) + ":" + QString::number(__LINE__), "unknown error"); QMessageBox::critical(0, "Internal error: " + QString(__FILE__) + ":" + QString::number(__LINE__), "unknown error");
exit(-1); exit(-1);
} }
#elif defined Q_OS_MAC
QString commandLine;
commandLine = QString("killall interface") + "; " + _workingFolder +"/waitForFinish.sh interface";
system(commandLine.toStdString().c_str());
commandLine = QString("killall Sandbox") + "; " + _workingFolder +"/waitForFinish.sh Sandbox";
system(commandLine.toStdString().c_str());
commandLine = QString("killall Console") + "; " + _workingFolder +"/waitForFinish.sh Console";
system(commandLine.toStdString().c_str());
#endif #endif
} }
void TestRunner::startLocalServerProcesses() { void TestRunner::startLocalServerProcesses() {
#ifdef Q_OS_WIN
QString commandLine; QString commandLine;
commandLine = "start \"domain-server.exe\" \"" + QDir::toNativeSeparators(_installationFolder) + "\\domain-server.exe\""; #ifdef Q_OS_WIN
commandLine =
"start \"domain-server.exe\" \"" + QDir::toNativeSeparators(_installationFolder) + "\\domain-server.exe\"";
system(commandLine.toStdString().c_str()); system(commandLine.toStdString().c_str());
commandLine = commandLine =
"start \"assignment-client.exe\" \"" + QDir::toNativeSeparators(_installationFolder) + "\\assignment-client.exe\" -n 6"; "start \"assignment-client.exe\" \"" + QDir::toNativeSeparators(_installationFolder) + "\\assignment-client.exe\" -n 6";
system(commandLine.toStdString().c_str()); system(commandLine.toStdString().c_str());
#elif defined Q_OS_MAC
commandLine = "open \"" +_installationFolder + "/Sandbox.app\"";
system(commandLine.toStdString().c_str());
#endif #endif
// Give server processes time to stabilize // Give server processes time to stabilize
QThread::sleep(20); QThread::sleep(20);
} }
void TestRunner::runInterfaceWithTestScript() { void TestRunner::runInterfaceWithTestScript() {
QString exeFile = QString("\"") + QDir::toNativeSeparators(_installationFolder) + "\\interface.exe\"";
QString snapshotFolder = QString("\"") + QDir::toNativeSeparators(_snapshotFolder) + "\"";
QString url = QString("hifi://localhost"); QString url = QString("hifi://localhost");
if (_runServerless->isChecked()) { if (_runServerless->isChecked()) {
// Move to an empty area // Move to an empty area
url = "file:///~serverless/tutorial.json"; url = "file:///~serverless/tutorial.json";
} else { } else {
#ifdef Q_OS_WIN
url = "hifi://localhost"; url = "hifi://localhost";
#elif defined Q_OS_MAC
// TODO: Find out Mac equivalent of AppData, then this won't be needed
url = "hifi://localhost/9999,9999,9999";
#endif
} }
QString testScript = QString testScript =
QString("https://raw.githubusercontent.com/") + _user + "/hifi_tests/" + _branch + "/tests/testRecursive.js"; QString("https://raw.githubusercontent.com/") + _user + "/hifi_tests/" + _branch + "/tests/testRecursive.js";
QString commandLine = exeFile + " --url " + url + " --no-updater --no-login-suggestion" + " --testScript " + testScript + QString commandLine;
" quitWhenFinished --testResultsLocation " + snapshotFolder; #ifdef Q_OS_WIN
QString exeFile = QString("\"") + QDir::toNativeSeparators(_installationFolder) + "\\interface.exe\"";
commandLine = exeFile +
" --url " + url +
" --no-updater" +
" --no-login-suggestion"
" --testScript " + testScript + " quitWhenFinished" +
" --testResultsLocation " + _snapshotFolder;
interfaceWorker->setCommandLine(commandLine); _interfaceWorker->setCommandLine(commandLine);
emit startInterface(); emit startInterface();
#elif defined Q_OS_MAC
// On The Mac, we need to resize Interface. The Interface window opens a few seconds after the process
// has started.
// Before starting interface, start a process that will resize interface 10s after it opens
// This is performed by creating a bash script that runs to processes
QFile script;
script.setFileName(_workingFolder + "/runInterfaceTests.sh");
if (!script.open(QIODevice::WriteOnly | QIODevice::Text)) {
QMessageBox::critical(0, "Internal error: " + QString(__FILE__) + ":" + QString::number(__LINE__),
"Could not open 'runInterfaceTests.sh'");
exit(-1);
}
script.write("#!/bin/sh\n\n");
commandLine = _workingFolder +"/waitForStart.sh interface && sleep 10 && " + _workingFolder +"/setInterfaceSizeAndPosition.sh &\n";
script.write(commandLine.toStdString().c_str());
commandLine =
"open \"" +_installationFolder + "/interface.app\" --args" +
" --url " + url +
" --no-updater" +
" --no-login-suggestion"
" --testScript " + testScript + " quitWhenFinished" +
" --testResultsLocation " + _snapshotFolder +
" && " + _workingFolder +"/waitForFinish.sh interface";
script.write(commandLine.toStdString().c_str());
script.close();
script.setPermissions(QFile::ReadOwner | QFile::WriteOwner | QFile::ExeOwner);
commandLine = _workingFolder + "/runInterfaceTests.sh";
_interfaceWorker->setCommandLine(commandLine);
emit startInterface();
#endif
// Helpful for debugging
appendLog(commandLine);
} }
void TestRunner::interfaceExecutionComplete() { void TestRunner::interfaceExecutionComplete() {
@ -352,9 +539,7 @@ void TestRunner::interfaceExecutionComplete() {
QFileInfo testCompleted(QDir::toNativeSeparators(_snapshotFolder) +"/tests_completed.txt"); QFileInfo testCompleted(QDir::toNativeSeparators(_snapshotFolder) +"/tests_completed.txt");
if (!testCompleted.exists()) { if (!testCompleted.exists()) {
QMessageBox::critical(0, "Tests not completed", "Interface seems to have crashed before completion of the test scripts"); QMessageBox::critical(0, "Tests not completed", "Interface seems to have crashed before completion of the test scripts\nExisting images will be evaluated");
_runNow->setEnabled(true);
return;
} }
evaluateResults(); evaluateResults();
@ -364,7 +549,7 @@ void TestRunner::interfaceExecutionComplete() {
void TestRunner::evaluateResults() { void TestRunner::evaluateResults() {
updateStatusLabel("Evaluating results"); updateStatusLabel("Evaluating results");
autoTester->startTestsEvaluation(false, true, _snapshotFolder, _branch, _user); nitpick->startTestsEvaluation(false, true, _snapshotFolder, _branch, _user);
} }
void TestRunner::automaticTestRunEvaluationComplete(QString zippedFolder, int numberOfFailures) { void TestRunner::automaticTestRunEvaluationComplete(QString zippedFolder, int numberOfFailures) {
@ -404,11 +589,15 @@ void TestRunner::addBuildNumberToResults(QString zippedFolderName) {
} }
void TestRunner::restoreHighFidelityAppDataFolder() { void TestRunner::restoreHighFidelityAppDataFolder() {
#ifdef Q_OS_WIN
_appDataFolder.removeRecursively(); _appDataFolder.removeRecursively();
if (_savedAppDataFolder != QDir()) { if (_savedAppDataFolder != QDir()) {
_appDataFolder.rename(_savedAppDataFolder.path(), _appDataFolder.path()); _appDataFolder.rename(_savedAppDataFolder.path(), _appDataFolder.path());
} }
#elif defined Q_OS_MAC
// TODO: find Mac equivalent of AppData
#endif
} }
// Copies a folder recursively // Copies a folder recursively
@ -473,7 +662,7 @@ void TestRunner::checkTime() {
} }
void TestRunner::updateStatusLabel(const QString& message) { void TestRunner::updateStatusLabel(const QString& message) {
autoTester->updateStatusLabel(message); nitpick->updateStatusLabel(message);
} }
void TestRunner::appendLog(const QString& message) { void TestRunner::appendLog(const QString& message) {
@ -487,11 +676,12 @@ void TestRunner::appendLog(const QString& message) {
_logFile.write("\n"); _logFile.write("\n");
_logFile.close(); _logFile.close();
autoTester->appendLogWindow(message); nitpick->appendLogWindow(message);
} }
QString TestRunner::getInstallerNameFromURL(const QString& url) { QString TestRunner::getInstallerNameFromURL(const QString& url) {
// An example URL: https://deployment.highfidelity.com/jobs/pr-build/label%3Dwindows/13023/HighFidelity-Beta-Interface-PR14006-be76c43.exe // An example URL: https://deployment.highfidelity.com/jobs/pr-build/label%3Dwindows/13023/HighFidelity-Beta-Interface-PR14006-be76c43.exe
// On Mac, replace `exe` with `dmg`
try { try {
QStringList urlParts = url.split("/"); QStringList urlParts = url.split("/");
return urlParts[urlParts.size() - 1]; return urlParts[urlParts.size() - 1];
@ -509,7 +699,11 @@ QString TestRunner::getPRNumberFromURL(const QString& url) {
QStringList urlParts = url.split("/"); QStringList urlParts = url.split("/");
QStringList filenameParts = urlParts[urlParts.size() - 1].split("-"); QStringList filenameParts = urlParts[urlParts.size() - 1].split("-");
if (filenameParts.size() <= 3) { if (filenameParts.size() <= 3) {
#ifdef Q_OS_WIN
throw "URL not in expected format, should look like `https://deployment.highfidelity.com/jobs/pr-build/label%3Dwindows/13023/HighFidelity-Beta-Interface-PR14006-be76c43.exe`"; throw "URL not in expected format, should look like `https://deployment.highfidelity.com/jobs/pr-build/label%3Dwindows/13023/HighFidelity-Beta-Interface-PR14006-be76c43.exe`";
#elif defined Q_OS_MAC
throw "URL not in expected format, should look like `https://deployment.highfidelity.com/jobs/pr-build/label%3Dwindows/13023/HighFidelity-Beta-Interface-PR14006-be76c43.dmg`";
#endif
} }
return filenameParts[filenameParts.size() - 2]; return filenameParts[filenameParts.size() - 2];
} catch (QString errorMessage) { } catch (QString errorMessage) {
@ -536,6 +730,7 @@ void TestRunner::parseBuildInformation() {
#elif defined(Q_OS_MAC) #elif defined(Q_OS_MAC)
platformOfInterest = "mac"; platformOfInterest = "mac";
#endif #endif
QDomElement element = domDocument.documentElement(); QDomElement element = domDocument.documentElement();
// Verify first element is "projects" // Verify first element is "projects"
@ -552,42 +747,48 @@ void TestRunner::parseBuildInformation() {
throw("File is not from 'interface' build"); throw("File is not from 'interface' build");
} }
// Now loop over the platforms // Now loop over the platforms, looking for ours
bool platformFound{ false };
element = element.firstChild().toElement();
while (!element.isNull()) { while (!element.isNull()) {
element = element.firstChild().toElement(); if (element.attribute("name") == platformOfInterest) {
if (element.tagName() != "platform" || element.attribute("name") != platformOfInterest) { platformFound = true;
continue; break;
} }
// Next element should be the build
element = element.firstChild().toElement();
if (element.tagName() != "build") {
throw("File seems to be in wrong format");
}
// Next element should be the version
element = element.firstChild().toElement();
if (element.tagName() != "version") {
throw("File seems to be in wrong format");
}
// Add the build number to the end of the filename
_buildInformation.build = element.text();
// First sibling should be stable_version
element = element.nextSibling().toElement(); element = element.nextSibling().toElement();
if (element.tagName() != "stable_version") {
throw("File seems to be in wrong format");
}
// Next sibling should be url
element = element.nextSibling().toElement();
if (element.tagName() != "url") {
throw("File seems to be in wrong format");
}
_buildInformation.url = element.text();
} }
if (!platformFound) {
throw("File seems to be in wrong format - platform " + platformOfInterest + " not found");
}
element = element.firstChild().toElement();
if (element.tagName() != "build") {
throw("File seems to be in wrong format");
}
// Next element should be the version
element = element.firstChild().toElement();
if (element.tagName() != "version") {
throw("File seems to be in wrong format");
}
// Add the build number to the end of the filename
_buildInformation.build = element.text();
// First sibling should be stable_version
element = element.nextSibling().toElement();
if (element.tagName() != "stable_version") {
throw("File seems to be in wrong format");
}
// Next sibling should be url
element = element.nextSibling().toElement();
if (element.tagName() != "url") {
throw("File seems to be in wrong format");
}
_buildInformation.url = element.text();
} catch (QString errorMessage) { } catch (QString errorMessage) {
QMessageBox::critical(0, "Internal error: " + QString(__FILE__) + ":" + QString::number(__LINE__), errorMessage); QMessageBox::critical(0, "Internal error: " + QString(__FILE__) + ":" + QString::number(__LINE__), errorMessage);
exit(-1); exit(-1);
@ -605,4 +806,4 @@ int Worker::runCommand() {
int result = system(_commandLine.toStdString().c_str()); int result = system(_commandLine.toStdString().c_str());
emit commandComplete(); emit commandComplete();
return result; return result;
} }

View file

@ -84,11 +84,18 @@ private slots:
signals: signals:
void startInstaller(); void startInstaller();
void startInterface(); void startInterface();
void startResize();
private: private:
bool _automatedTestIsRunning{ false }; bool _automatedTestIsRunning{ false };
#ifdef Q_OS_WIN
const QString INSTALLER_FILENAME_LATEST{ "HighFidelity-Beta-latest-dev.exe" }; const QString INSTALLER_FILENAME_LATEST{ "HighFidelity-Beta-latest-dev.exe" };
#elif defined(Q_OS_MAC)
const QString INSTALLER_FILENAME_LATEST{ "HighFidelity-Beta-latest-dev.dmg" };
#else
const QString INSTALLER_FILENAME_LATEST{ "" };
#endif
QString _installerURL; QString _installerURL;
QString _installerFilename; QString _installerFilename;
@ -124,11 +131,12 @@ private:
QDateTime _testStartDateTime; QDateTime _testStartDateTime;
QThread* installerThread; QThread* _installerThread;
QThread* interfaceThread; QThread* _interfaceThread;
Worker* installerWorker;
Worker* interfaceWorker;
Worker* _installerWorker;
Worker* _interfaceWorker;
BuildInformation _buildInformation; BuildInformation _buildInformation;
}; };
@ -144,8 +152,8 @@ signals:
void commandComplete(); void commandComplete();
void startInstaller(); void startInstaller();
void startInterface(); void startInterface();
private: private:
QString _commandLine; QString _commandLine;
}; };
#endif // hifi_testRunner_h #endif // hifi_testRunner_h

View file

@ -8,11 +8,11 @@
// See the accompanying file LICENSE or http://www.apache.org/licenses/LICENSE-2.0.html // See the accompanying file LICENSE or http://www.apache.org/licenses/LICENSE-2.0.html
// //
#include <QtWidgets/QApplication> #include <QtWidgets/QApplication>
#include "ui/AutoTester.h" #include "ui/Nitpick.h"
#include <iostream> #include <iostream>
AutoTester* autoTester; Nitpick* nitpick;
int main(int argc, char *argv[]) { int main(int argc, char *argv[]) {
// If no parameters then run in interactive mode // If no parameters then run in interactive mode
@ -62,13 +62,13 @@ int main(int argc, char *argv[]) {
QApplication application(argc, argv); QApplication application(argc, argv);
autoTester = new AutoTester(); nitpick = new Nitpick();
autoTester->setup(); nitpick->setup();
if (!testFolder.isNull()) { if (!testFolder.isNull()) {
autoTester->startTestsEvaluation(true ,false, testFolder, branch, user); nitpick->startTestsEvaluation(true ,false, testFolder, branch, user);
} else { } else {
autoTester->show(); nitpick->show();
} }
return application.exec(); return application.exec();

View file

@ -14,7 +14,7 @@
</rect> </rect>
</property> </property>
<property name="windowTitle"> <property name="windowTitle">
<string>AutoTester Help</string> <string>Nitpick Help</string>
</property> </property>
<widget class="QTextEdit" name="textEdit"> <widget class="QTextEdit" name="textEdit">
<property name="geometry"> <property name="geometry">

View file

@ -1,5 +1,5 @@
// //
// AutoTester.cpp // Nitpick.cpp
// zone/ambientLightInheritence // zone/ambientLightInheritence
// //
// Created by Nissim Hadar on 2 Nov 2017. // Created by Nissim Hadar on 2 Nov 2017.
@ -8,14 +8,14 @@
// Distributed under the Apache License, Version 2.0. // Distributed under the Apache License, Version 2.0.
// See the accompanying file LICENSE or http://www.apache.org/licenses/LICENSE-2.0.html // See the accompanying file LICENSE or http://www.apache.org/licenses/LICENSE-2.0.html
// //
#include "AutoTester.h" #include "Nitpick.h"
#ifdef Q_OS_WIN #ifdef Q_OS_WIN
#include <windows.h> #include <windows.h>
#include <shellapi.h> #include <shellapi.h>
#endif #endif
AutoTester::AutoTester(QWidget* parent) : QMainWindow(parent) { Nitpick::Nitpick(QWidget* parent) : QMainWindow(parent) {
_ui.setupUi(this); _ui.setupUi(this);
_ui.checkBoxInteractiveMode->setChecked(true); _ui.checkBoxInteractiveMode->setChecked(true);
@ -24,9 +24,9 @@ AutoTester::AutoTester(QWidget* parent) : QMainWindow(parent) {
_signalMapper = new QSignalMapper(); _signalMapper = new QSignalMapper();
connect(_ui.actionClose, &QAction::triggered, this, &AutoTester::on_closeButton_clicked); connect(_ui.actionClose, &QAction::triggered, this, &Nitpick::on_closeButton_clicked);
connect(_ui.actionAbout, &QAction::triggered, this, &AutoTester::about); connect(_ui.actionAbout, &QAction::triggered, this, &Nitpick::about);
connect(_ui.actionContent, &QAction::triggered, this, &AutoTester::content); connect(_ui.actionContent, &QAction::triggered, this, &Nitpick::content);
// The second tab hides and shows the Windows task bar // The second tab hides and shows the Windows task bar
#ifndef Q_OS_WIN #ifndef Q_OS_WIN
@ -36,13 +36,13 @@ AutoTester::AutoTester(QWidget* parent) : QMainWindow(parent) {
_ui.statusLabel->setText(""); _ui.statusLabel->setText("");
_ui.plainTextEdit->setReadOnly(true); _ui.plainTextEdit->setReadOnly(true);
setWindowTitle("Auto Tester - v6.7"); setWindowTitle("Nitpick - v1.0");
// Coming soon to an auto-tester near you... // Coming soon to a nitpick near you...
//// _helpWindow.textBrowser->setText() //// _helpWindow.textBrowser->setText()
} }
AutoTester::~AutoTester() { Nitpick::~Nitpick() {
delete _signalMapper; delete _signalMapper;
if (_test) { if (_test) {
@ -54,7 +54,7 @@ AutoTester::~AutoTester() {
} }
} }
void AutoTester::setup() { void Nitpick::setup() {
if (_test) { if (_test) {
delete _test; delete _test;
} }
@ -87,7 +87,7 @@ void AutoTester::setup() {
_testRunner = new TestRunner(dayCheckboxes, timeEditCheckboxes, timeEdits, _ui.workingFolderLabel, _ui.checkBoxServerless, _ui.checkBoxRunLatest, _ui.urlLineEdit, _ui.runNowButton); _testRunner = new TestRunner(dayCheckboxes, timeEditCheckboxes, timeEdits, _ui.workingFolderLabel, _ui.checkBoxServerless, _ui.checkBoxRunLatest, _ui.urlLineEdit, _ui.runNowButton);
} }
void AutoTester::startTestsEvaluation(const bool isRunningFromCommandLine, void Nitpick::startTestsEvaluation(const bool isRunningFromCommandLine,
const bool isRunningInAutomaticTestRun, const bool isRunningInAutomaticTestRun,
const QString& snapshotDirectory, const QString& snapshotDirectory,
const QString& branch, const QString& branch,
@ -96,8 +96,13 @@ void AutoTester::startTestsEvaluation(const bool isRunningFromCommandLine,
_test->startTestsEvaluation(isRunningFromCommandLine, isRunningInAutomaticTestRun, snapshotDirectory, branch, user); _test->startTestsEvaluation(isRunningFromCommandLine, isRunningInAutomaticTestRun, snapshotDirectory, branch, user);
} }
void AutoTester::on_tabWidget_currentChanged(int index) { void Nitpick::on_tabWidget_currentChanged(int index) {
// Enable the GitHub edit boxes as required
#ifdef Q_OS_WIN
if (index == 0 || index == 2 || index == 3) { if (index == 0 || index == 2 || index == 3) {
#else
if (index == 0 || index == 1 || index == 2) {
#endif
_ui.userLineEdit->setDisabled(false); _ui.userLineEdit->setDisabled(false);
_ui.branchLineEdit->setDisabled(false); _ui.branchLineEdit->setDisabled(false);
} else { } else {
@ -106,73 +111,73 @@ void AutoTester::on_tabWidget_currentChanged(int index) {
} }
} }
void AutoTester::on_evaluateTestsButton_clicked() { void Nitpick::on_evaluateTestsButton_clicked() {
_test->startTestsEvaluation(false, false); _test->startTestsEvaluation(false, false);
} }
void AutoTester::on_createRecursiveScriptButton_clicked() { void Nitpick::on_createRecursiveScriptButton_clicked() {
_test->createRecursiveScript(); _test->createRecursiveScript();
} }
void AutoTester::on_createAllRecursiveScriptsButton_clicked() { void Nitpick::on_createAllRecursiveScriptsButton_clicked() {
_test->createAllRecursiveScripts(); _test->createAllRecursiveScripts();
} }
void AutoTester::on_createTestsButton_clicked() { void Nitpick::on_createTestsButton_clicked() {
_test->createTests(); _test->createTests();
} }
void AutoTester::on_createMDFileButton_clicked() { void Nitpick::on_createMDFileButton_clicked() {
_test->createMDFile(); _test->createMDFile();
} }
void AutoTester::on_createAllMDFilesButton_clicked() { void Nitpick::on_createAllMDFilesButton_clicked() {
_test->createAllMDFiles(); _test->createAllMDFiles();
} }
void AutoTester::on_createTestAutoScriptButton_clicked() { void Nitpick::on_createTestAutoScriptButton_clicked() {
_test->createTestAutoScript(); _test->createTestAutoScript();
} }
void AutoTester::on_createAllTestAutoScriptsButton_clicked() { void Nitpick::on_createAllTestAutoScriptsButton_clicked() {
_test->createAllTestAutoScripts(); _test->createAllTestAutoScripts();
} }
void AutoTester::on_createTestsOutlineButton_clicked() { void Nitpick::on_createTestsOutlineButton_clicked() {
_test->createTestsOutline(); _test->createTestsOutline();
} }
void AutoTester::on_createTestRailTestCasesButton_clicked() { void Nitpick::on_createTestRailTestCasesButton_clicked() {
_test->createTestRailTestCases(); _test->createTestRailTestCases();
} }
void AutoTester::on_createTestRailRunButton_clicked() { void Nitpick::on_createTestRailRunButton_clicked() {
_test->createTestRailRun(); _test->createTestRailRun();
} }
void AutoTester::on_setWorkingFolderButton_clicked() { void Nitpick::on_setWorkingFolderButton_clicked() {
_testRunner->setWorkingFolder(); _testRunner->setWorkingFolder();
} }
void AutoTester::enableRunTabControls() { void Nitpick::enableRunTabControls() {
_ui.runNowButton->setEnabled(true); _ui.runNowButton->setEnabled(true);
_ui.daysGroupBox->setEnabled(true); _ui.daysGroupBox->setEnabled(true);
_ui.timesGroupBox->setEnabled(true); _ui.timesGroupBox->setEnabled(true);
} }
void AutoTester::on_runNowButton_clicked() { void Nitpick::on_runNowButton_clicked() {
_testRunner->run(); _testRunner->run();
} }
void AutoTester::on_checkBoxRunLatest_clicked() { void Nitpick::on_checkBoxRunLatest_clicked() {
_ui.urlLineEdit->setEnabled(!_ui.checkBoxRunLatest->isChecked()); _ui.urlLineEdit->setEnabled(!_ui.checkBoxRunLatest->isChecked());
} }
void AutoTester::automaticTestRunEvaluationComplete(QString zippedFolderName, int numberOfFailures) { void Nitpick::automaticTestRunEvaluationComplete(QString zippedFolderName, int numberOfFailures) {
_testRunner->automaticTestRunEvaluationComplete(zippedFolderName, numberOfFailures); _testRunner->automaticTestRunEvaluationComplete(zippedFolderName, numberOfFailures);
} }
void AutoTester::on_updateTestRailRunResultsButton_clicked() { void Nitpick::on_updateTestRailRunResultsButton_clicked() {
_test->updateTestRailRunResult(); _test->updateTestRailRunResult();
} }
@ -180,7 +185,7 @@ void AutoTester::on_updateTestRailRunResultsButton_clicked() {
// if (uState & ABS_AUTOHIDE) on_showTaskbarButton_clicked(); // if (uState & ABS_AUTOHIDE) on_showTaskbarButton_clicked();
// else on_hideTaskbarButton_clicked(); // else on_hideTaskbarButton_clicked();
// //
void AutoTester::on_hideTaskbarButton_clicked() { void Nitpick::on_hideTaskbarButton_clicked() {
#ifdef Q_OS_WIN #ifdef Q_OS_WIN
APPBARDATA abd = { sizeof abd }; APPBARDATA abd = { sizeof abd };
UINT uState = (UINT)SHAppBarMessage(ABM_GETSTATE, &abd); UINT uState = (UINT)SHAppBarMessage(ABM_GETSTATE, &abd);
@ -190,7 +195,7 @@ void AutoTester::on_hideTaskbarButton_clicked() {
#endif #endif
} }
void AutoTester::on_showTaskbarButton_clicked() { void Nitpick::on_showTaskbarButton_clicked() {
#ifdef Q_OS_WIN #ifdef Q_OS_WIN
APPBARDATA abd = { sizeof abd }; APPBARDATA abd = { sizeof abd };
UINT uState = (UINT)SHAppBarMessage(ABM_GETSTATE, &abd); UINT uState = (UINT)SHAppBarMessage(ABM_GETSTATE, &abd);
@ -200,23 +205,23 @@ void AutoTester::on_showTaskbarButton_clicked() {
#endif #endif
} }
void AutoTester::on_closeButton_clicked() { void Nitpick::on_closeButton_clicked() {
exit(0); exit(0);
} }
void AutoTester::on_createPythonScriptRadioButton_clicked() { void Nitpick::on_createPythonScriptRadioButton_clicked() {
_test->setTestRailCreateMode(PYTHON); _test->setTestRailCreateMode(PYTHON);
} }
void AutoTester::on_createXMLScriptRadioButton_clicked() { void Nitpick::on_createXMLScriptRadioButton_clicked() {
_test->setTestRailCreateMode(XML); _test->setTestRailCreateMode(XML);
} }
void AutoTester::on_createWebPagePushButton_clicked() { void Nitpick::on_createWebPagePushButton_clicked() {
_test->createWebPage(_ui.updateAWSCheckBox, _ui.awsURLLineEdit); _test->createWebPage(_ui.updateAWSCheckBox, _ui.awsURLLineEdit);
} }
void AutoTester::downloadFile(const QUrl& url) { void Nitpick::downloadFile(const QUrl& url) {
_downloaders.emplace_back(new Downloader(url, this)); _downloaders.emplace_back(new Downloader(url, this));
connect(_downloaders[_index], SIGNAL(downloaded()), _signalMapper, SLOT(map())); connect(_downloaders[_index], SIGNAL(downloaded()), _signalMapper, SLOT(map()));
@ -225,7 +230,7 @@ void AutoTester::downloadFile(const QUrl& url) {
++_index; ++_index;
} }
void AutoTester::downloadFiles(const QStringList& URLs, const QString& directoryName, const QStringList& filenames, void *caller) { void Nitpick::downloadFiles(const QStringList& URLs, const QString& directoryName, const QStringList& filenames, void *caller) {
connect(_signalMapper, SIGNAL(mapped(int)), this, SLOT(saveFile(int))); connect(_signalMapper, SIGNAL(mapped(int)), this, SLOT(saveFile(int)));
_directoryName = directoryName; _directoryName = directoryName;
@ -251,7 +256,7 @@ void AutoTester::downloadFiles(const QStringList& URLs, const QString& directory
} }
} }
void AutoTester::saveFile(int index) { void Nitpick::saveFile(int index) {
try { try {
QFile file(_directoryName + "/" + _filenames[index]); QFile file(_directoryName + "/" + _filenames[index]);
file.open(QIODevice::WriteOnly); file.open(QIODevice::WriteOnly);
@ -277,34 +282,34 @@ void AutoTester::saveFile(int index) {
} }
} }
void AutoTester::about() { void Nitpick::about() {
QMessageBox::information(0, "About", QString("Built ") + __DATE__ + ", " + __TIME__); QMessageBox::information(0, "About", QString("Built ") + __DATE__ + ", " + __TIME__);
} }
void AutoTester::content() { void Nitpick::content() {
_helpWindow.show(); _helpWindow.show();
} }
void AutoTester::setUserText(const QString& user) { void Nitpick::setUserText(const QString& user) {
_ui.userLineEdit->setText(user); _ui.userLineEdit->setText(user);
} }
QString AutoTester::getSelectedUser() { QString Nitpick::getSelectedUser() {
return _ui.userLineEdit->text(); return _ui.userLineEdit->text();
} }
void AutoTester::setBranchText(const QString& branch) { void Nitpick::setBranchText(const QString& branch) {
_ui.branchLineEdit->setText(branch); _ui.branchLineEdit->setText(branch);
} }
QString AutoTester::getSelectedBranch() { QString Nitpick::getSelectedBranch() {
return _ui.branchLineEdit->text(); return _ui.branchLineEdit->text();
} }
void AutoTester::updateStatusLabel(const QString& status) { void Nitpick::updateStatusLabel(const QString& status) {
_ui.statusLabel->setText(status); _ui.statusLabel->setText(status);
} }
void AutoTester::appendLogWindow(const QString& message) { void Nitpick::appendLogWindow(const QString& message) {
_ui.plainTextEdit->appendPlainText(message); _ui.plainTextEdit->appendPlainText(message);
} }

View file

@ -1,5 +1,5 @@
// //
// AutoTester.h // Nitpick.h
// //
// Created by Nissim Hadar on 2 Nov 2017. // Created by Nissim Hadar on 2 Nov 2017.
// Copyright 2013 High Fidelity, Inc. // Copyright 2013 High Fidelity, Inc.
@ -7,13 +7,13 @@
// Distributed under the Apache License, Version 2.0. // Distributed under the Apache License, Version 2.0.
// See the accompanying file LICENSE or http://www.apache.org/licenses/LICENSE-2.0.html // See the accompanying file LICENSE or http://www.apache.org/licenses/LICENSE-2.0.html
// //
#ifndef hifi_AutoTester_h #ifndef hifi_Nitpick_h
#define hifi_AutoTester_h #define hifi_Nitpick_h
#include <QtWidgets/QMainWindow> #include <QtWidgets/QMainWindow>
#include <QSignalMapper> #include <QSignalMapper>
#include <QTextEdit> #include <QTextEdit>
#include "ui_AutoTester.h" #include "ui_Nitpick.h"
#include "../Downloader.h" #include "../Downloader.h"
#include "../Test.h" #include "../Test.h"
@ -22,12 +22,12 @@
#include "../TestRunner.h" #include "../TestRunner.h"
#include "../AWSInterface.h" #include "../AWSInterface.h"
class AutoTester : public QMainWindow { class Nitpick : public QMainWindow {
Q_OBJECT Q_OBJECT
public: public:
AutoTester(QWidget* parent = Q_NULLPTR); Nitpick(QWidget* parent = Q_NULLPTR);
~AutoTester(); ~Nitpick();
void setup(); void setup();
@ -95,7 +95,7 @@ private slots:
void content(); void content();
private: private:
Ui::AutoTesterClass _ui; Ui::NitpickClass _ui;
Test* _test{ nullptr }; Test* _test{ nullptr };
TestRunner* _testRunner{ nullptr }; TestRunner* _testRunner{ nullptr };
@ -121,4 +121,4 @@ private:
void* _caller; void* _caller;
}; };
#endif // hifi_AutoTester_h #endif // hifi_Nitpick_h

View file

@ -1,7 +1,7 @@
<?xml version="1.0" encoding="UTF-8"?> <?xml version="1.0" encoding="UTF-8"?>
<ui version="4.0"> <ui version="4.0">
<class>AutoTesterClass</class> <class>NitpickClass</class>
<widget class="QMainWindow" name="AutoTesterClass"> <widget class="QMainWindow" name="NitpickClass">
<property name="geometry"> <property name="geometry">
<rect> <rect>
<x>0</x> <x>0</x>
@ -17,7 +17,7 @@
</sizepolicy> </sizepolicy>
</property> </property>
<property name="windowTitle"> <property name="windowTitle">
<string>AutoTester</string> <string>Nitpick</string>
</property> </property>
<widget class="QWidget" name="centralWidget"> <widget class="QWidget" name="centralWidget">
<widget class="QPushButton" name="closeButton"> <widget class="QPushButton" name="closeButton">
@ -198,7 +198,7 @@
<x>10</x> <x>10</x>
<y>160</y> <y>160</y>
<width>161</width> <width>161</width>
<height>28</height> <height>51</height>
</rect> </rect>
</property> </property>
<property name="text"> <property name="text">
@ -525,7 +525,7 @@
<rect> <rect>
<x>128</x> <x>128</x>
<y>95</y> <y>95</y>
<width>21</width> <width>31</width>
<height>31</height> <height>31</height>
</rect> </rect>
</property> </property>
@ -539,7 +539,7 @@
</property> </property>
<property name="geometry"> <property name="geometry">
<rect> <rect>
<x>160</x> <x>170</x>
<y>100</y> <y>100</y>
<width>451</width> <width>451</width>
<height>21</height> <height>21</height>
@ -554,9 +554,9 @@
<widget class="QCheckBox" name="checkBoxInteractiveMode"> <widget class="QCheckBox" name="checkBoxInteractiveMode">
<property name="geometry"> <property name="geometry">
<rect> <rect>
<x>200</x> <x>190</x>
<y>180</y> <y>180</y>
<width>120</width> <width>131</width>
<height>20</height> <height>20</height>
</rect> </rect>
</property> </property>
@ -572,8 +572,8 @@
<rect> <rect>
<x>330</x> <x>330</x>
<y>170</y> <y>170</y>
<width>101</width> <width>181</width>
<height>40</height> <height>51</height>
</rect> </rect>
</property> </property>
<property name="text"> <property name="text">
@ -684,10 +684,10 @@
</property> </property>
<property name="geometry"> <property name="geometry">
<rect> <rect>
<x>240</x> <x>270</x>
<y>30</y> <y>30</y>
<width>160</width> <width>160</width>
<height>40</height> <height>51</height>
</rect> </rect>
</property> </property>
<property name="text"> <property name="text">
@ -699,7 +699,7 @@
<rect> <rect>
<x>150</x> <x>150</x>
<y>42</y> <y>42</y>
<width>81</width> <width>111</width>
<height>17</height> <height>17</height>
</rect> </rect>
</property> </property>
@ -803,7 +803,7 @@
<x>0</x> <x>0</x>
<y>0</y> <y>0</y>
<width>720</width> <width>720</width>
<height>21</height> <height>22</height>
</rect> </rect>
</property> </property>
<widget class="QMenu" name="menuFile"> <widget class="QMenu" name="menuFile">