Merge pull request #14273 from NissimHadar/MAC

Case 19722 - Auto-Tester on Mac.
This commit is contained in:
Shannon Romano 2018-11-16 12:49:58 -08:00 committed by GitHub
commit ba699e3e9b
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
58 changed files with 561 additions and 243 deletions

View file

@ -198,4 +198,14 @@ void TestScriptingInterface::setOtherAvatarsReplicaCount(int count) {
int TestScriptingInterface::getOtherAvatarsReplicaCount() {
return qApp->getOtherAvatarsReplicaCount();
}
}
QString TestScriptingInterface::getOperatingSystemType() {
#ifdef Q_OS_WIN
return "WINDOWS";
#elif defined Q_OS_MAC
return "MACOS";
#else
return "UNKNOWN";
#endif
}

View file

@ -163,6 +163,13 @@ public slots:
*/
Q_INVOKABLE int getOtherAvatarsReplicaCount();
/**jsdoc
* Returns the Operating Sytem type
* @function Test.getOperatingSystemType
* @returns {string} "WINDOWS", "MACOS" or "UNKNOWN"
*/
QString getOperatingSystemType();
private:
bool waitForCondition(qint64 maxWaitMs, std::function<bool()> condition);
QString _testResultsLocation;

View file

@ -31,6 +31,6 @@ if (BUILD_TOOLS)
add_subdirectory(oven)
set_target_properties(oven PROPERTIES FOLDER "Tools")
add_subdirectory(auto-tester)
set_target_properties(auto-tester PROPERTIES FOLDER "Tools")
add_subdirectory(nitpick)
set_target_properties(nitpick PROPERTIES FOLDER "Tools")
endif()

View file

@ -1,4 +1,4 @@
set (TARGET_NAME auto-tester)
set (TARGET_NAME nitpick)
project(${TARGET_NAME})
# Automatically run UIC and MOC. This replaces the older WRAP macros
@ -22,7 +22,7 @@ set (QT_LIBRARIES Qt5::Core Qt5::Widgets QT::Gui Qt5::Xml)
if (WIN32)
# Do not show Console
set_property (TARGET auto-tester PROPERTY WIN32_EXECUTABLE true)
set_property (TARGET nitpick PROPERTY WIN32_EXECUTABLE true)
endif()
target_zlib()

View file

Before

Width:  |  Height:  |  Size: 17 KiB

After

Width:  |  Height:  |  Size: 17 KiB

View file

Before

Width:  |  Height:  |  Size: 13 KiB

After

Width:  |  Height:  |  Size: 13 KiB

View file

@ -1,11 +1,11 @@
# Auto Tester
# nitpick
The auto-tester is a stand alone application that provides a mechanism for regression testing. The general idea is simple:
Nitpick is a stand alone application that provides a mechanism for regression testing. The general idea is simple:
* Each test folder has a script that produces a set of snapshots.
* The snapshots are compared to a 'canonical' set of images that have been produced beforehand.
* The result, if any test failed, is a zipped folder describing the failure.
Auto-tester has 5 functions, separated into 4 tabs:
Nitpick has 5 functions, separated into 4 tabs:
1. Creating tests, MD files and recursive scripts
1. Windows task bar utility (Windows only)
1. Running tests
@ -14,19 +14,25 @@ Auto-tester has 5 functions, separated into 4 tabs:
## Installation
### Executable
1. Download the installer by browsing to [here](<https://hifi-content.s3.amazonaws.com/nissim/autoTester/AutoTester-Installer-v6.6.exe>).
1. On Windows: download the installer by browsing to [here](<https://hifi-content.s3.amazonaws.com/nissim/nitpick/nitpick-installer-v1.0.exe>).
2. Double click on the installer and install to a convenient location
![](./setup_7z.PNG)
3. To run the auto-tester, double click **auto-tester.exe**.
3. To run nitpick, double click **nitpick.exe**.
### Python
The TestRail interface requires Python 3 to be installed. Auto-Tester has been tested with Python 3.7.0 but should work with newer versions.
The TestRail interface requires Python 3 to be installed. Nitpick has been tested with Python 3.7.0 but should work with newer versions.
Python 3 can be downloaded from:
1. Windows installer <https://www.python.org/downloads/>
2. Linux (source) <https://www.python.org/downloads/release/python-370/> (**Gzipped source tarball**)
3. Mac <https://www.python.org/downloads/release/python-370/> (**macOS 64-bit/32-bit installer** or **macOS 64-bit/32-bit installer**)
#### Windows
After installation - create an environment variable called PYTHON_PATH and set it to the folder containing the Python executable.
#### Mac
After installation - run `open "/Applications/Python 3.6/Install Certificates.command"`. This is needed because the Mac Python supplied no longer links with the deprecated Apple-supplied system OpenSSL libraries but rather supplies a private copy of OpenSSL 1.0.2 which does not automatically access the system default root certificates.
Verify that `/usr/local/bin/python3` exists.
### AWS interface
#### Windows
1. Download the AWS CLI from `https://aws.amazon.com/cli/`
@ -35,9 +41,23 @@ After installation - create an environment variable called PYTHON_PATH and set i
1. Enter the AWS account number
1. Enter the secret key
1. Leave region name and ouput format as default [None]
1. Install the latest release of Boto3 via pip:
pip install boto3
#### Mac
1. Install pip with the script provided by the Python Packaging Authority:
$ curl -O https://bootstrap.pypa.io/get-pip.py
$ python3 get-pip.py --user
1. Use pip to install the AWS CLI.
$ pip3 install awscli --upgrade --user
This will install aws in your user. For user XXX, aws will be located in /Users/XXX/Library/Python/3.7/bin
1. Open a new command prompt and run `aws configure`
1. Enter the AWS account number
1. Enter the secret key
1. Leave region name and ouput format as default [None]
1. Install the latest release of Boto3 via pip:
pip3 install boto3
1. Install the latest release of Boto3 via pip:
>pip install boto3
# Create
![](./Create.PNG)
@ -57,7 +77,7 @@ This function creates an MD file in the (user-selected) tests root folder. The
This function creates a file named `test.md` from a `test.js` script. The user will be asked for the folder containing the test script:
### Details
The process to produce the MD file is a simplistic parse of the test script.
- The string in the `autoTester.perform(...)` function call will be the title of the file
- The string in the `nitpick.perform(...)` function call will be the title of the file
- Instructions to run the script are then provided:
@ -89,26 +109,26 @@ The various scripts are called in alphabetical order.
An example of a recursive script is as follows:
```
// This is an automatically generated file, created by auto-tester on Jul 5 2018, 10:19
// This is an automatically generated file, created by nitpick on Jul 5 2018, 10:19
PATH_TO_THE_REPO_PATH_UTILS_FILE = "https://raw.githubusercontent.com/highfidelity/hifi_tests/master/tests/utils/branchUtils.js";
Script.include(PATH_TO_THE_REPO_PATH_UTILS_FILE);
var autoTester = createAutoTester(Script.resolvePath("."));
var nitpick = createNitpick(Script.resolvePath("."));
var testsRootPath = autoTester.getTestsRootPath();
var testsRootPath = nitpick.getTestsRootPath();
if (typeof Test !== 'undefined') {
Test.wait(10000);
};
autoTester.enableRecursive();
autoTester.enableAuto();
nitpick.enableRecursive();
nitpick.enableAuto();
Script.include(testsRootPath + "content/overlay/layer/drawInFront/shape/test.js");
Script.include(testsRootPath + "content/overlay/layer/drawInFront/model/test.js");
Script.include(testsRootPath + "content/overlay/layer/drawHUDLayer/test.js");
autoTester.runRecursive();
nitpick.runRecursive();
```
## Create all Recursive Scripts
### Usage
@ -165,7 +185,7 @@ Evaluation proceeds in a number of steps:
1. The images are then pair-wise compared, using the SSIM algorithm. A fixed threshold is used to define a mismatch.
1. In interactive mode - a window is opened showing the expected image, actual image, difference image and error:
![](./autoTesterMismatchExample.PNG)
![](./nitpickMismatchExample.PNG)
1. If not in interactive mode, or the user has defined the results as an error, an error is written into the error folder. The error itself is a folder with the 3 images and a small text file containing details.

View file

Before

Width:  |  Height:  |  Size: 20 KiB

After

Width:  |  Height:  |  Size: 20 KiB

View file

Before

Width:  |  Height:  |  Size: 10 KiB

After

Width:  |  Height:  |  Size: 10 KiB

View file

Before

Width:  |  Height:  |  Size: 12 KiB

After

Width:  |  Height:  |  Size: 12 KiB

View file

Before

Width:  |  Height:  |  Size: 17 KiB

After

Width:  |  Height:  |  Size: 17 KiB

View file

Before

Width:  |  Height:  |  Size: 14 KiB

After

Width:  |  Height:  |  Size: 14 KiB

View file

Before

Width:  |  Height:  |  Size: 476 KiB

After

Width:  |  Height:  |  Size: 476 KiB

View file

Before

Width:  |  Height:  |  Size: 4 KiB

After

Width:  |  Height:  |  Size: 4 KiB

View file

@ -22,11 +22,11 @@ AWSInterface::AWSInterface(QObject* parent) : QObject(parent) {
}
void AWSInterface::createWebPageFromResults(const QString& testResults,
const QString& workingDirectory,
const QString& snapshotDirectory,
QCheckBox* updateAWSCheckBox,
QLineEdit* urlLineEdit) {
_testResults = testResults;
_workingDirectory = workingDirectory;
_snapshotDirectory = snapshotDirectory;
_urlLineEdit = urlLineEdit;
_urlLineEdit->setEnabled(false);
@ -42,16 +42,15 @@ void AWSInterface::createWebPageFromResults(const QString& testResults,
void AWSInterface::extractTestFailuresFromZippedFolder() {
// For a test results zip file called `D:/tt/TestResults--2018-10-02_16-54-11(9426)[DESKTOP-PMKNLSQ].zip`
// the folder will be called `TestResults--2018-10-02_16-54-11(9426)[DESKTOP-PMKNLSQ]`
// and, this folder will be in the workign directory
// and, this folder will be in the working directory
QStringList parts =_testResults.split('/');
QString zipFolderName = _workingDirectory + "/" + parts[parts.length() - 1].split('.')[0];
QString zipFolderName = _snapshotDirectory + "/" + parts[parts.length() - 1].split('.')[0];
if (QDir(zipFolderName).exists()) {
QDir dir = zipFolderName;
dir.removeRecursively();
}
QDir().mkdir(_workingDirectory);
JlCompress::extractDir(_testResults, _workingDirectory);
JlCompress::extractDir(_testResults, _snapshotDirectory);
}
void AWSInterface::createHTMLFile() {
@ -61,7 +60,7 @@ void AWSInterface::createHTMLFile() {
QString filename = pathComponents[pathComponents.length() - 1];
_resultsFolder = filename.left(filename.length() - 4);
QString resultsPath = _workingDirectory + "/" + _resultsFolder + "/";
QString resultsPath = _snapshotDirectory + "/" + _resultsFolder + "/";
QDir().mkdir(resultsPath);
_htmlFilename = resultsPath + HTML_FILENAME;
@ -157,7 +156,7 @@ void AWSInterface::writeTable(QTextStream& stream) {
// Note that failures are processed first, then successes
QStringList originalNamesFailures;
QStringList originalNamesSuccesses;
QDirIterator it1(_workingDirectory.toStdString().c_str());
QDirIterator it1(_snapshotDirectory.toStdString().c_str());
while (it1.hasNext()) {
QString nextDirectory = it1.next();
@ -191,10 +190,10 @@ void AWSInterface::writeTable(QTextStream& stream) {
newNamesSuccesses.append(originalNamesSuccesses[i].split("--tests.")[1]);
}
_htmlFailuresFolder = _workingDirectory + "/" + _resultsFolder + "/" + FAILURES_FOLDER;
_htmlFailuresFolder = _snapshotDirectory + "/" + _resultsFolder + "/" + FAILURES_FOLDER;
QDir().mkdir(_htmlFailuresFolder);
_htmlSuccessesFolder = _workingDirectory + "/" + _resultsFolder + "/" + SUCCESSES_FOLDER;
_htmlSuccessesFolder = _snapshotDirectory + "/" + _resultsFolder + "/" + SUCCESSES_FOLDER;
QDir().mkdir(_htmlSuccessesFolder);
for (int i = 0; i < newNamesFailures.length(); ++i) {
@ -321,7 +320,7 @@ void AWSInterface::createEntry(int index, const QString& testResult, QTextStream
}
void AWSInterface::updateAWS() {
QString filename = _workingDirectory + "/updateAWS.py";
QString filename = _snapshotDirectory + "/updateAWS.py";
if (QFile::exists(filename)) {
QFile::remove(filename);
}
@ -352,14 +351,23 @@ void AWSInterface::updateAWS() {
QStringList parts = nextDirectory.split('/');
QString filename = parts[parts.length() - 3] + "/" + parts[parts.length() - 2] + "/" + parts[parts.length() - 1];
stream << "data = open('" << _workingDirectory << "/" << filename << "/" << "Actual Image.png" << "', 'rb')\n";
stream << "data = open('" << _snapshotDirectory << "/" << filename << "/"
<< "Actual Image.png"
<< "', 'rb')\n";
stream << "s3.Bucket('hifi-content').put_object(Bucket='" << AWS_BUCKET << "', Key='" << filename << "/" << "Actual Image.png" << "', Body=data)\n\n";
stream << "data = open('" << _workingDirectory << "/" << filename << "/" << "Expected Image.png" << "', 'rb')\n";
stream << "data = open('" << _snapshotDirectory << "/" << filename << "/"
<< "Expected Image.png"
<< "', 'rb')\n";
stream << "s3.Bucket('hifi-content').put_object(Bucket='" << AWS_BUCKET << "', Key='" << filename << "/" << "Expected Image.png" << "', Body=data)\n\n";
if (QFile::exists(_htmlFailuresFolder + "/" + parts[parts.length() - 1] + "/Difference Image.png")) {
stream << "data = open('" << _workingDirectory << "/" << filename << "/" << "Difference Image.png" << "', 'rb')\n";
stream << "data = open('" << _snapshotDirectory << "/" << filename << "/"
<< "Difference Image.png"
<< "', 'rb')\n";
stream << "s3.Bucket('hifi-content').put_object(Bucket='" << AWS_BUCKET << "', Key='" << filename << "/" << "Difference Image.png" << "', Body=data)\n\n";
}
}
@ -378,19 +386,28 @@ void AWSInterface::updateAWS() {
QStringList parts = nextDirectory.split('/');
QString filename = parts[parts.length() - 3] + "/" + parts[parts.length() - 2] + "/" + parts[parts.length() - 1];
stream << "data = open('" << _workingDirectory << "/" << filename << "/" << "Actual Image.png" << "', 'rb')\n";
stream << "data = open('" << _snapshotDirectory << "/" << filename << "/"
<< "Actual Image.png"
<< "', 'rb')\n";
stream << "s3.Bucket('hifi-content').put_object(Bucket='" << AWS_BUCKET << "', Key='" << filename << "/" << "Actual Image.png" << "', Body=data)\n\n";
stream << "data = open('" << _workingDirectory << "/" << filename << "/" << "Expected Image.png" << "', 'rb')\n";
stream << "data = open('" << _snapshotDirectory << "/" << filename << "/"
<< "Expected Image.png"
<< "', 'rb')\n";
stream << "s3.Bucket('hifi-content').put_object(Bucket='" << AWS_BUCKET << "', Key='" << filename << "/" << "Expected Image.png" << "', Body=data)\n\n";
if (QFile::exists(_htmlSuccessesFolder + "/" + parts[parts.length() - 1] + "/Difference Image.png")) {
stream << "data = open('" << _workingDirectory << "/" << filename << "/" << "Difference Image.png" << "', 'rb')\n";
stream << "data = open('" << _snapshotDirectory << "/" << filename << "/"
<< "Difference Image.png"
<< "', 'rb')\n";
stream << "s3.Bucket('hifi-content').put_object(Bucket='" << AWS_BUCKET << "', Key='" << filename << "/" << "Difference Image.png" << "', Body=data)\n\n";
}
}
stream << "data = open('" << _workingDirectory << "/" << _resultsFolder << "/" << HTML_FILENAME << "', 'rb')\n";
stream << "data = open('" << _snapshotDirectory << "/" << _resultsFolder << "/" << HTML_FILENAME << "', 'rb')\n";
stream << "s3.Bucket('hifi-content').put_object(Bucket='" << AWS_BUCKET << "', Key='" << _resultsFolder << "/"
<< HTML_FILENAME << "', Body=data, ContentType='text/html')\n";
@ -408,6 +425,11 @@ void AWSInterface::updateAWS() {
connect(process, static_cast<void (QProcess::*)(int, QProcess::ExitStatus)>(&QProcess::finished), this,
[=](int exitCode, QProcess::ExitStatus exitStatus) { _busyWindow.hide(); });
#ifdef Q_OS_WIN
QStringList parameters = QStringList() << filename ;
process->start(_pythonCommand, parameters);
#elif defined Q_OS_MAC
QStringList parameters = QStringList() << "-c" << _pythonCommand + " " + filename;
process->start("sh", parameters);
#endif
}

View file

@ -26,7 +26,7 @@ public:
explicit AWSInterface(QObject* parent = 0);
void createWebPageFromResults(const QString& testResults,
const QString& workingDirectory,
const QString& snapshotDirectory,
QCheckBox* updateAWSCheckBox,
QLineEdit* urlLineEdit);
@ -49,7 +49,7 @@ public:
private:
QString _testResults;
QString _workingDirectory;
QString _snapshotDirectory;
QString _resultsFolder;
QString _htmlFailuresFolder;
QString _htmlSuccessesFolder;

View file

@ -17,8 +17,7 @@ Downloader::Downloader(QUrl fileURL, QObject *parent) : QObject(parent) {
this, SLOT (fileDownloaded(QNetworkReply*))
);
QNetworkRequest request(fileURL);
_networkAccessManager.get(request);
_networkAccessManager.get(QNetworkRequest(fileURL));
}
void Downloader::fileDownloaded(QNetworkReply* reply) {
@ -37,4 +36,4 @@ void Downloader::fileDownloaded(QNetworkReply* reply) {
QByteArray Downloader::downloadedData() const {
return _downloadedData;
}
}

View file

@ -37,7 +37,7 @@ public:
signals:
void downloaded();
private slots:
private slots:
void fileDownloaded(QNetworkReply* pReply);
private:

View file

@ -14,17 +14,28 @@
#include <QProcess>
PythonInterface::PythonInterface() {
#ifdef Q_OS_WIN
if (QProcessEnvironment::systemEnvironment().contains("PYTHON_PATH")) {
QString _pythonPath = QProcessEnvironment::systemEnvironment().value("PYTHON_PATH");
if (!QFile::exists(_pythonPath + "/" + _pythonExe)) {
QMessageBox::critical(0, _pythonExe, QString("Python executable not found in ") + _pythonPath);
exit(-1);
}
_pythonCommand = _pythonPath + "/" + _pythonExe;
} else {
QMessageBox::critical(0, "PYTHON_PATH not defined",
"Please set PYTHON_PATH to directory containing the Python executable");
exit(-1);
}
#elif defined Q_OS_MAC
_pythonCommand = "/usr/local/bin/python3";
if (!QFile::exists(_pythonCommand)) {
QMessageBox::critical(0, "PYTHON_PATH not defined",
"python3 not found at " + _pythonCommand);
exit(-1);
}
#endif
}
QString PythonInterface::getPythonCommand() {

View file

@ -19,7 +19,13 @@ public:
QString getPythonCommand();
private:
#ifdef Q_OS_WIN
const QString _pythonExe{ "python.exe" };
#else
// Both Mac and Linux use "python"
const QString _pythonExe{ "python" };
#endif
QString _pythonCommand;
};

View file

@ -19,8 +19,8 @@
#include <quazip5/quazip.h>
#include <quazip5/JlCompress.h>
#include "ui/AutoTester.h"
extern AutoTester* autoTester;
#include "ui/Nitpick.h"
extern Nitpick* nitpick;
#include <math.h>
@ -30,9 +30,9 @@ Test::Test(QProgressBar* progressBar, QCheckBox* checkBoxInteractiveMode) {
_mismatchWindow.setModal(true);
if (autoTester) {
autoTester->setUserText(GIT_HUB_DEFAULT_USER);
autoTester->setBranchText(GIT_HUB_DEFAULT_BRANCH);
if (nitpick) {
nitpick->setUserText(GIT_HUB_DEFAULT_USER);
nitpick->setBranchText(GIT_HUB_DEFAULT_BRANCH);
}
}
@ -167,7 +167,7 @@ void Test::appendTestResultsToFile(const QString& _testResultsFolderPath, TestRe
// Create text file describing the failure
QTextStream stream(&descriptionFile);
stream << "Test failed in folder " << testResult._pathname.left(testResult._pathname.length() - 1) << endl; // remove trailing '/'
stream << "Test in folder " << testResult._pathname.left(testResult._pathname.length() - 1) << endl; // remove trailing '/'
stream << "Expected image was " << testResult._expectedImageFilename << endl;
stream << "Actual image was " << testResult._actualImageFilename << endl;
stream << "Similarity index was " << testResult._error << endl;
@ -240,8 +240,8 @@ void Test::startTestsEvaluation(const bool isRunningFromCommandLine,
_expectedImagesFilenames.clear();
_expectedImagesFullFilenames.clear();
QString branch = (branchFromCommandLine.isNull()) ? autoTester->getSelectedBranch() : branchFromCommandLine;
QString user = (userFromCommandLine.isNull()) ? autoTester->getSelectedUser() : userFromCommandLine;
QString branch = (branchFromCommandLine.isNull()) ? nitpick->getSelectedBranch() : branchFromCommandLine;
QString user = (userFromCommandLine.isNull()) ? nitpick->getSelectedUser() : userFromCommandLine;
foreach(QString currentFilename, sortedTestResultsFilenames) {
QString fullCurrentFilename = _snapshotDirectory + "/" + currentFilename;
@ -268,7 +268,7 @@ void Test::startTestsEvaluation(const bool isRunningFromCommandLine,
}
}
autoTester->downloadFiles(expectedImagesURLs, _snapshotDirectory, _expectedImagesFilenames, (void *)this);
nitpick->downloadFiles(expectedImagesURLs, _snapshotDirectory, _expectedImagesFilenames, (void *)this);
}
void Test::finishTestsEvaluation() {
int numberOfFailures = compareImageLists();
@ -288,7 +288,7 @@ void Test::finishTestsEvaluation() {
}
if (_isRunningInAutomaticTestRun) {
autoTester->automaticTestRunEvaluationComplete(zippedFolderName, numberOfFailures);
nitpick->automaticTestRunEvaluationComplete(zippedFolderName, numberOfFailures);
}
}
@ -429,22 +429,22 @@ ExtractedText Test::getTestScriptLines(QString testFileName) {
QString line = stream.readLine();
// Name of test is the string in the following line:
// autoTester.perform("Apply Material Entities to Avatars", Script.resolvePath("."), function(testType) {...
// nitpick.perform("Apply Material Entities to Avatars", Script.resolvePath("."), function(testType) {...
const QString ws("\\h*"); //white-space character
const QString functionPerformName(ws + "autoTester" + ws + "\\." + ws + "perform");
const QString functionPerformName(ws + "nitpick" + ws + "\\." + ws + "perform");
const QString quotedString("\\\".+\\\"");
QString regexTestTitle(ws + functionPerformName + "\\(" + quotedString);
QRegularExpression lineContainingTitle = QRegularExpression(regexTestTitle);
// Each step is either of the following forms:
// autoTester.addStepSnapshot("Take snapshot"...
// autoTester.addStep("Clean up after test"...
const QString functionAddStepSnapshotName(ws + "autoTester" + ws + "\\." + ws + "addStepSnapshot");
// nitpick.addStepSnapshot("Take snapshot"...
// nitpick.addStep("Clean up after test"...
const QString functionAddStepSnapshotName(ws + "nitpick" + ws + "\\." + ws + "addStepSnapshot");
const QString regexStepSnapshot(ws + functionAddStepSnapshotName + ws + "\\(" + ws + quotedString + ".*");
const QRegularExpression lineStepSnapshot = QRegularExpression(regexStepSnapshot);
const QString functionAddStepName(ws + "autoTester" + ws + "\\." + ws + "addStep");
const QString functionAddStepName(ws + "nitpick" + ws + "\\." + ws + "addStep");
const QString regexStep(ws + functionAddStepName + ws + "\\(" + ws + quotedString + ".*");
const QRegularExpression lineStep = QRegularExpression(regexStep);
@ -620,7 +620,7 @@ void Test::createTestAutoScript() {
}
if (createTestAutoScript(_testDirectory)) {
QMessageBox::information(0, "Success", "'autoTester.js` script has been created");
QMessageBox::information(0, "Success", "'nitpick.js` script has been created");
}
}
@ -653,7 +653,7 @@ void Test::createAllTestAutoScripts() {
}
}
QMessageBox::information(0, "Success", "'autoTester.js' scripts have been created");
QMessageBox::information(0, "Success", "'nitpick.js' scripts have been created");
}
bool Test::createTestAutoScript(const QString& directory) {
@ -677,8 +677,8 @@ bool Test::createTestAutoScript(const QString& directory) {
stream << "if (typeof PATH_TO_THE_REPO_PATH_UTILS_FILE === 'undefined') PATH_TO_THE_REPO_PATH_UTILS_FILE = 'https://raw.githubusercontent.com/highfidelity/hifi_tests/master/tests/utils/branchUtils.js';\n";
stream << "Script.include(PATH_TO_THE_REPO_PATH_UTILS_FILE);\n";
stream << "var autoTester = createAutoTester(Script.resolvePath('.'));\n\n";
stream << "autoTester.enableAuto();\n\n";
stream << "var nitpick = createAutoTester(Script.resolvePath('.'));\n\n";
stream << "nitpick.enableAuto();\n\n";
stream << "Script.include('./test.js?raw=true');\n";
testAutoScriptFile.close();
@ -751,29 +751,29 @@ void Test::createRecursiveScript(const QString& topLevelDirectory, bool interact
textStream << "// This is an automatically generated file, created by auto-tester" << endl;
// Include 'autoTest.js'
QString branch = autoTester->getSelectedBranch();
QString user = autoTester->getSelectedUser();
QString branch = nitpick->getSelectedBranch();
QString user = nitpick->getSelectedUser();
textStream << "PATH_TO_THE_REPO_PATH_UTILS_FILE = \"https://raw.githubusercontent.com/" + user + "/hifi_tests/" + branch +
"/tests/utils/branchUtils.js\";"
<< endl;
textStream << "Script.include(PATH_TO_THE_REPO_PATH_UTILS_FILE);" << endl;
textStream << "var autoTester = createAutoTester(Script.resolvePath(\".\"));" << endl << endl;
textStream << "var nitpick = createAutoTester(Script.resolvePath(\".\"));" << endl << endl;
textStream << "var testsRootPath = autoTester.getTestsRootPath();" << endl << endl;
textStream << "var testsRootPath = nitpick.getTestsRootPath();" << endl << endl;
// Wait 10 seconds before starting
textStream << "if (typeof Test !== 'undefined') {" << endl;
textStream << " Test.wait(10000);" << endl;
textStream << "};" << endl << endl;
textStream << "autoTester.enableRecursive();" << endl;
textStream << "autoTester.enableAuto();" << endl << endl;
textStream << "nitpick.enableRecursive();" << endl;
textStream << "nitpick.enableAuto();" << endl << endl;
// This is used to verify that the recursive test contains at least one test
bool testFound{ false };
// Directories are included in reverse order. The autoTester scripts use a stack mechanism,
// Directories are included in reverse order. The nitpick scripts use a stack mechanism,
// so this ensures that the tests run in alphabetical order (a convenience when debugging)
QStringList directories;
@ -819,7 +819,7 @@ void Test::createRecursiveScript(const QString& topLevelDirectory, bool interact
}
textStream << endl;
textStream << "autoTester.runRecursive();" << endl;
textStream << "nitpick.runRecursive();" << endl;
allTestsFilename.close();
}
@ -881,7 +881,7 @@ void Test::createTestsOutline() {
// The directory name appears after the last slash (we are assured there is at least 1).
QString directoryName = directory.right(directory.length() - directory.lastIndexOf("/") - 1);
// autoTester is run on a clone of the repository. We use relative paths, so we can use both local disk and GitHub
// nitpick is run on a clone of the repository. We use relative paths, so we can use both local disk and GitHub
// For a test in "D:/GitHub/hifi_tests/tests/content/entity/zone/ambientLightInheritance" the
// GitHub URL is "./content/entity/zone/ambientLightInheritance?raw=true"
QString partialPath = directory.right(directory.length() - (directory.lastIndexOf("/tests/") + QString("/tests").length() + 1));
@ -937,11 +937,11 @@ void Test::createTestRailTestCases() {
}
if (_testRailCreateMode == PYTHON) {
_testRailInterface.createTestSuitePython(_testDirectory, outputDirectory, autoTester->getSelectedUser(),
autoTester->getSelectedBranch());
_testRailInterface.createTestSuitePython(_testDirectory, outputDirectory, nitpick->getSelectedUser(),
nitpick->getSelectedBranch());
} else {
_testRailInterface.createTestSuiteXML(_testDirectory, outputDirectory, autoTester->getSelectedUser(),
autoTester->getSelectedBranch());
_testRailInterface.createTestSuiteXML(_testDirectory, outputDirectory, nitpick->getSelectedUser(),
nitpick->getSelectedBranch());
}
}
@ -1052,11 +1052,11 @@ void Test::createWebPage(QCheckBox* updateAWSCheckBox, QLineEdit* urlLineEdit) {
return;
}
QString tempDirectory = QFileDialog::getExistingDirectory(nullptr, "Please select a folder to store temporary files in",
QString snapshotDirectory = QFileDialog::getExistingDirectory(nullptr, "Please select a folder to store temporary files in",
nullptr, QFileDialog::ShowDirsOnly);
if (tempDirectory.isNull()) {
if (snapshotDirectory.isNull()) {
return;
}
_awsInterface.createWebPageFromResults(testResults, tempDirectory, updateAWSCheckBox, urlLineEdit);
_awsInterface.createWebPageFromResults(testResults, snapshotDirectory, updateAWSCheckBox, urlLineEdit);
}

View file

@ -146,7 +146,7 @@ private:
const QString DATETIME_FORMAT{ "yyyy-MM-dd_hh-mm-ss" };
// NOTE: these need to match the appropriate var's in autoTester.js
// NOTE: these need to match the appropriate var's in nitpick.js
// var advanceKey = "n";
// var pathSeparator = ".";
const QString ADVANCE_KEY{ "n" };

View file

@ -357,8 +357,13 @@ void TestRailInterface::createAddTestCasesPythonScript(const QString& testDirect
connect(process, static_cast<void (QProcess::*)(int, QProcess::ExitStatus)>(&QProcess::finished), this,
[=](int exitCode, QProcess::ExitStatus exitStatus) { _busyWindow.hide(); });
#ifdef Q_OS_WIN
QStringList parameters = QStringList() << _outputDirectory + "/addTestCases.py";
process->start(_pythonCommand, parameters);
#elif defined Q_OS_MAC
QStringList parameters = QStringList() << "-c" << _pythonCommand + " " + _outputDirectory + "/addTestCases.py";
process->start("sh", parameters);
#endif
}
}
@ -482,8 +487,13 @@ void TestRailInterface::addRun() {
connect(process, static_cast<void (QProcess::*)(int, QProcess::ExitStatus)>(&QProcess::finished), this,
[=](int exitCode, QProcess::ExitStatus exitStatus) { _busyWindow.hide(); });
#ifdef Q_OS_WIN
QStringList parameters = QStringList() << _outputDirectory + "/addRun.py";
process->start(_pythonCommand, parameters);
#elif defined Q_OS_MAC
QStringList parameters = QStringList() << "-c" << _pythonCommand + " " + _outputDirectory + "/addRun.py";
process->start("sh", parameters);
#endif
}
}
@ -586,8 +596,13 @@ void TestRailInterface::updateRunWithResults() {
connect(process, static_cast<void (QProcess::*)(int, QProcess::ExitStatus)>(&QProcess::finished), this,
[=](int exitCode, QProcess::ExitStatus exitStatus) { _busyWindow.hide(); });
#ifdef Q_OS_WIN
QStringList parameters = QStringList() << _outputDirectory + "/updateRunWithResults.py";
process->start(_pythonCommand, parameters);
#elif defined Q_OS_MAC
QStringList parameters = QStringList() << "-c" << _pythonCommand + " " + _outputDirectory + "/updateRunWithResults.py";
process->start("sh", parameters);
#endif
}
}
@ -759,8 +774,13 @@ void TestRailInterface::getReleasesFromTestRail() {
[=](int exitCode, QProcess::ExitStatus exitStatus) { updateReleasesComboData(exitCode, exitStatus); });
connect(process, SIGNAL(finished(int)), process, SLOT(deleteLater()));
#ifdef Q_OS_WIN
QStringList parameters = QStringList() << filename;
process->start(_pythonCommand, parameters);
#elif defined Q_OS_MAC
QStringList parameters = QStringList() << "-c" << _pythonCommand + " " + filename;
process->start("sh", parameters);
#endif
}
void TestRailInterface::createTestSuitePython(const QString& testDirectory,
@ -1078,8 +1098,13 @@ void TestRailInterface::getTestSectionsFromTestRail() {
[=](int exitCode, QProcess::ExitStatus exitStatus) { updateSectionsComboData(exitCode, exitStatus); });
connect(process, SIGNAL(finished(int)), process, SLOT(deleteLater()));
#ifdef Q_OS_WIN
QStringList parameters = QStringList() << filename;
process->start(_pythonCommand, parameters);
#elif defined Q_OS_MAC
QStringList parameters = QStringList() << "-c" << _pythonCommand + " " + filename;
process->start("sh", parameters);
#endif
}
void TestRailInterface::getRunsFromTestRail() {
@ -1117,9 +1142,13 @@ void TestRailInterface::getRunsFromTestRail() {
[=](int exitCode, QProcess::ExitStatus exitStatus) { updateRunsComboData(exitCode, exitStatus); });
connect(process, SIGNAL(finished(int)), process, SLOT(deleteLater()));
#ifdef Q_OS_WIN
QStringList parameters = QStringList() << filename;
process->start(_pythonCommand, parameters);
#elif defined Q_OS_MAC
QStringList parameters = QStringList() << "-c" << _pythonCommand + " " + filename;
process->start("sh", parameters);
#endif
}
void TestRailInterface::createTestRailRun(const QString& outputDirectory) {
@ -1164,4 +1193,4 @@ void TestRailInterface::extractTestFailuresFromZippedFolder(const QString& testR
QDir dir = tempSubDirectory;
dir.mkdir(tempSubDirectory);
JlCompress::extractDir(testResults, tempSubDirectory);
}
}

View file

@ -13,14 +13,17 @@
#include <QtWidgets/QMessageBox>
#include <QtWidgets/QFileDialog>
#include "ui/AutoTester.h"
extern AutoTester* autoTester;
#include "ui/Nitpick.h"
extern Nitpick* nitpick;
#ifdef Q_OS_WIN
#include <windows.h>
#include <tlhelp32.h>
#endif
// TODO: for debug
#include <iostream>
TestRunner::TestRunner(std::vector<QCheckBox*> dayCheckboxes,
std::vector<QCheckBox*> timeEditCheckboxes,
std::vector<QTimeEdit*> timeEdits,
@ -40,27 +43,29 @@ TestRunner::TestRunner(std::vector<QCheckBox*> dayCheckboxes,
_url = url;
_runNow = runNow;
installerThread = new QThread();
installerWorker = new Worker();
installerWorker->moveToThread(installerThread);
installerThread->start();
connect(this, SIGNAL(startInstaller()), installerWorker, SLOT(runCommand()));
connect(installerWorker, SIGNAL(commandComplete()), this, SLOT(installationComplete()));
_installerThread = new QThread();
_installerWorker = new Worker();
_installerWorker->moveToThread(_installerThread);
_installerThread->start();
connect(this, SIGNAL(startInstaller()), _installerWorker, SLOT(runCommand()));
connect(_installerWorker, SIGNAL(commandComplete()), this, SLOT(installationComplete()));
interfaceThread = new QThread();
interfaceWorker = new Worker();
interfaceThread->start();
interfaceWorker->moveToThread(interfaceThread);
connect(this, SIGNAL(startInterface()), interfaceWorker, SLOT(runCommand()));
connect(interfaceWorker, SIGNAL(commandComplete()), this, SLOT(interfaceExecutionComplete()));
_interfaceThread = new QThread();
_interfaceWorker = new Worker();
_interfaceThread->start();
_interfaceWorker->moveToThread(_interfaceThread);
connect(this, SIGNAL(startInterface()), _interfaceWorker, SLOT(runCommand()));
connect(_interfaceWorker, SIGNAL(commandComplete()), this, SLOT(interfaceExecutionComplete()));
}
TestRunner::~TestRunner() {
delete installerThread;
delete interfaceThread;
delete _installerThread;
delete _installerWorker;
delete interfaceThread;
delete interfaceWorker;
delete _interfaceThread;
delete _interfaceWorker;
if (_timer) {
delete _timer;
@ -84,15 +89,96 @@ void TestRunner::setWorkingFolder() {
return;
}
#ifdef Q_OS_WIN
_installationFolder = _workingFolder + "/High Fidelity";
#elif defined Q_OS_MAC
_installationFolder = _workingFolder + "/High_Fidelity";
#endif
_logFile.setFileName(_workingFolder + "/log.txt");
autoTester->enableRunTabControls();
nitpick->enableRunTabControls();
_workingFolderLabel->setText(QDir::toNativeSeparators(_workingFolder));
_timer = new QTimer(this);
connect(_timer, SIGNAL(timeout()), this, SLOT(checkTime()));
_timer->start(30 * 1000); //time specified in ms
#ifdef Q_OS_MAC
// Create MAC shell scripts
QFile script;
// This script waits for a process to start
script.setFileName(_workingFolder + "/waitForStart.sh");
if (!script.open(QIODevice::WriteOnly | QIODevice::Text)) {
QMessageBox::critical(0, "Internal error: " + QString(__FILE__) + ":" + QString::number(__LINE__),
"Could not open 'waitForStart.sh'");
exit(-1);
}
script.write("#!/bin/sh\n\n");
script.write("PROCESS=\"$1\"\n");
script.write("until (pgrep -x $PROCESS >nul)\n");
script.write("do\n");
script.write("\techo waiting for \"$1\" to start\n");
script.write("\tsleep 2\n");
script.write("done\n");
script.write("echo \"$1\" \"started\"\n");
script.close();
script.setPermissions(QFile::ReadOwner | QFile::WriteOwner | QFile::ExeOwner);
// The Mac shell command returns immediately. This little script waits for a process to finish
script.setFileName(_workingFolder + "/waitForFinish.sh");
if (!script.open(QIODevice::WriteOnly | QIODevice::Text)) {
QMessageBox::critical(0, "Internal error: " + QString(__FILE__) + ":" + QString::number(__LINE__),
"Could not open 'waitForFinish.sh'");
exit(-1);
}
script.write("#!/bin/sh\n\n");
script.write("PROCESS=\"$1\"\n");
script.write("while (pgrep -x $PROCESS >nul)\n");
script.write("do\n");
script.write("\techo waiting for \"$1\" to finish\n");
script.write("\tsleep 2\n");
script.write("done\n");
script.write("echo \"$1\" \"finished\"\n");
script.close();
script.setPermissions(QFile::ReadOwner | QFile::WriteOwner | QFile::ExeOwner);
// Create an AppleScript to resize Interface. This is needed so that snapshots taken
// with the primary camera will be the correct size.
// This will be run from a normal shell script
script.setFileName(_workingFolder + "/setInterfaceSizeAndPosition.scpt");
if (!script.open(QIODevice::WriteOnly | QIODevice::Text)) {
QMessageBox::critical(0, "Internal error: " + QString(__FILE__) + ":" + QString::number(__LINE__),
"Could not open 'setInterfaceSizeAndPosition.scpt'");
exit(-1);
}
script.write("set width to 960\n");
script.write("set height to 540\n");
script.write("set x to 100\n");
script.write("set y to 100\n\n");
script.write("tell application \"System Events\" to tell application process \"interface\" to tell window 1 to set {size, position} to {{width, height}, {x, y}}\n");
script.close();
script.setPermissions(QFile::ReadOwner | QFile::WriteOwner | QFile::ExeOwner);
script.setFileName(_workingFolder + "/setInterfaceSizeAndPosition.sh");
if (!script.open(QIODevice::WriteOnly | QIODevice::Text)) {
QMessageBox::critical(0, "Internal error: " + QString(__FILE__) + ":" + QString::number(__LINE__),
"Could not open 'setInterfaceSizeAndPosition.sh'");
exit(-1);
}
script.write("#!/bin/sh\n\n");
script.write("echo resizing interface\n");
script.write(("osascript " + _workingFolder + "/setInterfaceSizeAndPosition.scpt\n").toStdString().c_str());
script.write("echo resize complete\n");
script.close();
script.setPermissions(QFile::ReadOwner | QFile::WriteOwner | QFile::ExeOwner);
#endif
}
void TestRunner::run() {
@ -102,8 +188,8 @@ void TestRunner::run() {
_automatedTestIsRunning = true;
// Initial setup
_branch = autoTester->getSelectedBranch();
_user = autoTester->getSelectedUser();
_branch = nitpick->getSelectedBranch();
_user = nitpick->getSelectedUser();
// This will be restored at the end of the tests
saveExistingHighFidelityAppDataFolder();
@ -120,7 +206,7 @@ void TestRunner::run() {
updateStatusLabel("Downloading Build XML");
buildXMLDownloaded = false;
autoTester->downloadFiles(urls, _workingFolder, filenames, (void*)this);
nitpick->downloadFiles(urls, _workingFolder, filenames, (void*)this);
// `downloadComplete` will run after download has completed
}
@ -149,7 +235,7 @@ void TestRunner::downloadComplete() {
updateStatusLabel("Downloading installer");
autoTester->downloadFiles(urls, _workingFolder, filenames, (void*)this);
nitpick->downloadFiles(urls, _workingFolder, filenames, (void*)this);
// `downloadComplete` will run again after download has completed
@ -176,10 +262,43 @@ void TestRunner::runInstaller() {
QString installerFullPath = _workingFolder + "/" + _installerFilename;
QString commandLine =
"\"" + QDir::toNativeSeparators(installerFullPath) + "\"" + " /S /D=" + QDir::toNativeSeparators(_installationFolder);
QString commandLine;
#ifdef Q_OS_WIN
commandLine = "\"" + QDir::toNativeSeparators(installerFullPath) + "\"" + " /S /D=" + QDir::toNativeSeparators(_installationFolder);
#elif defined Q_OS_MAC
// Create installation shell script
QFile script;
script.setFileName(_workingFolder + "/install_app.sh");
if (!script.open(QIODevice::WriteOnly | QIODevice::Text)) {
QMessageBox::critical(0, "Internal error: " + QString(__FILE__) + ":" + QString::number(__LINE__),
"Could not open 'install_app.sh'");
exit(-1);
}
if (!QDir().exists(_installationFolder)) {
QDir().mkdir(_installationFolder);
}
// This script installs High Fidelity. It is run as "yes | install_app.sh... so "yes" is killed at the end
script.write("#!/bin/sh\n\n");
script.write("VOLUME=`hdiutil attach \"$1\" | grep Volumes | awk '{print $3}'`\n");
QString folderName {"High Fidelity"};
if (!_runLatest->isChecked()) {
folderName += QString(" - ") + getPRNumberFromURL(_url->text());
}
installerWorker->setCommandLine(commandLine);
script.write((QString("cp -rf \"$VOLUME/") + folderName + "/interface.app\" \"" + _workingFolder + "/High_Fidelity/\"\n").toStdString().c_str());
script.write((QString("cp -rf \"$VOLUME/") + folderName + "/Sandbox.app\" \"" + _workingFolder + "/High_Fidelity/\"\n").toStdString().c_str());
script.write("hdiutil detach \"$VOLUME\"\n");
script.write("killall yes\n");
script.close();
script.setPermissions(QFile::ReadOwner | QFile::WriteOwner | QFile::ExeOwner);
commandLine = "yes | " + _workingFolder + "/install_app.sh " + installerFullPath;
#endif
appendLog(commandLine);
_installerWorker->setCommandLine(commandLine);
emit startInstaller();
}
@ -213,11 +332,10 @@ void TestRunner::verifyInstallationSucceeded() {
}
void TestRunner::saveExistingHighFidelityAppDataFolder() {
#ifdef Q_OS_WIN
QString dataDirectory{ "NOT FOUND" };
#ifdef Q_OS_WIN
dataDirectory = qgetenv("USERPROFILE") + "\\AppData\\Roaming";
#endif
if (_runLatest->isChecked()) {
_appDataFolder = dataDirectory + "\\High Fidelity";
@ -238,6 +356,9 @@ void TestRunner::saveExistingHighFidelityAppDataFolder() {
// Copy an "empty" AppData folder (i.e. no entities)
copyFolder(QDir::currentPath() + "/AppDataHighFidelity", _appDataFolder.path());
#elif defined Q_OS_MAC
// TODO: find Mac equivalent of AppData
#endif
}
void TestRunner::createSnapshotFolder() {
@ -307,44 +428,110 @@ void TestRunner::killProcesses() {
QMessageBox::critical(0, "Internal error: " + QString(__FILE__) + ":" + QString::number(__LINE__), "unknown error");
exit(-1);
}
#elif defined Q_OS_MAC
QString commandLine;
commandLine = QString("killall interface") + "; " + _workingFolder +"/waitForFinish.sh interface";
system(commandLine.toStdString().c_str());
commandLine = QString("killall Sandbox") + "; " + _workingFolder +"/waitForFinish.sh Sandbox";
system(commandLine.toStdString().c_str());
commandLine = QString("killall Console") + "; " + _workingFolder +"/waitForFinish.sh Console";
system(commandLine.toStdString().c_str());
#endif
}
void TestRunner::startLocalServerProcesses() {
#ifdef Q_OS_WIN
QString commandLine;
commandLine = "start \"domain-server.exe\" \"" + QDir::toNativeSeparators(_installationFolder) + "\\domain-server.exe\"";
#ifdef Q_OS_WIN
commandLine =
"start \"domain-server.exe\" \"" + QDir::toNativeSeparators(_installationFolder) + "\\domain-server.exe\"";
system(commandLine.toStdString().c_str());
commandLine =
"start \"assignment-client.exe\" \"" + QDir::toNativeSeparators(_installationFolder) + "\\assignment-client.exe\" -n 6";
system(commandLine.toStdString().c_str());
#elif defined Q_OS_MAC
commandLine = "open \"" +_installationFolder + "/Sandbox.app\"";
system(commandLine.toStdString().c_str());
#endif
// Give server processes time to stabilize
QThread::sleep(20);
}
void TestRunner::runInterfaceWithTestScript() {
QString exeFile = QString("\"") + QDir::toNativeSeparators(_installationFolder) + "\\interface.exe\"";
QString snapshotFolder = QString("\"") + QDir::toNativeSeparators(_snapshotFolder) + "\"";
QString url = QString("hifi://localhost");
if (_runServerless->isChecked()) {
// Move to an empty area
url = "file:///~serverless/tutorial.json";
} else {
#ifdef Q_OS_WIN
url = "hifi://localhost";
#elif defined Q_OS_MAC
// TODO: Find out Mac equivalent of AppData, then this won't be needed
url = "hifi://localhost/9999,9999,9999";
#endif
}
QString testScript =
QString("https://raw.githubusercontent.com/") + _user + "/hifi_tests/" + _branch + "/tests/testRecursive.js";
QString commandLine = exeFile + " --url " + url + " --no-updater --no-login-suggestion" + " --testScript " + testScript +
" quitWhenFinished --testResultsLocation " + snapshotFolder;
QString commandLine;
#ifdef Q_OS_WIN
QString exeFile = QString("\"") + QDir::toNativeSeparators(_installationFolder) + "\\interface.exe\"";
commandLine = exeFile +
" --url " + url +
" --no-updater" +
" --no-login-suggestion"
" --testScript " + testScript + " quitWhenFinished" +
" --testResultsLocation " + _snapshotFolder;
interfaceWorker->setCommandLine(commandLine);
_interfaceWorker->setCommandLine(commandLine);
emit startInterface();
#elif defined Q_OS_MAC
// On The Mac, we need to resize Interface. The Interface window opens a few seconds after the process
// has started.
// Before starting interface, start a process that will resize interface 10s after it opens
// This is performed by creating a bash script that runs to processes
QFile script;
script.setFileName(_workingFolder + "/runInterfaceTests.sh");
if (!script.open(QIODevice::WriteOnly | QIODevice::Text)) {
QMessageBox::critical(0, "Internal error: " + QString(__FILE__) + ":" + QString::number(__LINE__),
"Could not open 'runInterfaceTests.sh'");
exit(-1);
}
script.write("#!/bin/sh\n\n");
commandLine = _workingFolder +"/waitForStart.sh interface && sleep 10 && " + _workingFolder +"/setInterfaceSizeAndPosition.sh &\n";
script.write(commandLine.toStdString().c_str());
commandLine =
"open \"" +_installationFolder + "/interface.app\" --args" +
" --url " + url +
" --no-updater" +
" --no-login-suggestion"
" --testScript " + testScript + " quitWhenFinished" +
" --testResultsLocation " + _snapshotFolder +
" && " + _workingFolder +"/waitForFinish.sh interface";
script.write(commandLine.toStdString().c_str());
script.close();
script.setPermissions(QFile::ReadOwner | QFile::WriteOwner | QFile::ExeOwner);
commandLine = _workingFolder + "/runInterfaceTests.sh";
_interfaceWorker->setCommandLine(commandLine);
emit startInterface();
#endif
// Helpful for debugging
appendLog(commandLine);
}
void TestRunner::interfaceExecutionComplete() {
@ -352,9 +539,7 @@ void TestRunner::interfaceExecutionComplete() {
QFileInfo testCompleted(QDir::toNativeSeparators(_snapshotFolder) +"/tests_completed.txt");
if (!testCompleted.exists()) {
QMessageBox::critical(0, "Tests not completed", "Interface seems to have crashed before completion of the test scripts");
_runNow->setEnabled(true);
return;
QMessageBox::critical(0, "Tests not completed", "Interface seems to have crashed before completion of the test scripts\nExisting images will be evaluated");
}
evaluateResults();
@ -364,7 +549,7 @@ void TestRunner::interfaceExecutionComplete() {
void TestRunner::evaluateResults() {
updateStatusLabel("Evaluating results");
autoTester->startTestsEvaluation(false, true, _snapshotFolder, _branch, _user);
nitpick->startTestsEvaluation(false, true, _snapshotFolder, _branch, _user);
}
void TestRunner::automaticTestRunEvaluationComplete(QString zippedFolder, int numberOfFailures) {
@ -404,11 +589,15 @@ void TestRunner::addBuildNumberToResults(QString zippedFolderName) {
}
void TestRunner::restoreHighFidelityAppDataFolder() {
#ifdef Q_OS_WIN
_appDataFolder.removeRecursively();
if (_savedAppDataFolder != QDir()) {
_appDataFolder.rename(_savedAppDataFolder.path(), _appDataFolder.path());
}
#elif defined Q_OS_MAC
// TODO: find Mac equivalent of AppData
#endif
}
// Copies a folder recursively
@ -473,7 +662,7 @@ void TestRunner::checkTime() {
}
void TestRunner::updateStatusLabel(const QString& message) {
autoTester->updateStatusLabel(message);
nitpick->updateStatusLabel(message);
}
void TestRunner::appendLog(const QString& message) {
@ -487,11 +676,12 @@ void TestRunner::appendLog(const QString& message) {
_logFile.write("\n");
_logFile.close();
autoTester->appendLogWindow(message);
nitpick->appendLogWindow(message);
}
QString TestRunner::getInstallerNameFromURL(const QString& url) {
// An example URL: https://deployment.highfidelity.com/jobs/pr-build/label%3Dwindows/13023/HighFidelity-Beta-Interface-PR14006-be76c43.exe
// On Mac, replace `exe` with `dmg`
try {
QStringList urlParts = url.split("/");
return urlParts[urlParts.size() - 1];
@ -509,7 +699,11 @@ QString TestRunner::getPRNumberFromURL(const QString& url) {
QStringList urlParts = url.split("/");
QStringList filenameParts = urlParts[urlParts.size() - 1].split("-");
if (filenameParts.size() <= 3) {
#ifdef Q_OS_WIN
throw "URL not in expected format, should look like `https://deployment.highfidelity.com/jobs/pr-build/label%3Dwindows/13023/HighFidelity-Beta-Interface-PR14006-be76c43.exe`";
#elif defined Q_OS_MAC
throw "URL not in expected format, should look like `https://deployment.highfidelity.com/jobs/pr-build/label%3Dwindows/13023/HighFidelity-Beta-Interface-PR14006-be76c43.dmg`";
#endif
}
return filenameParts[filenameParts.size() - 2];
} catch (QString errorMessage) {
@ -536,6 +730,7 @@ void TestRunner::parseBuildInformation() {
#elif defined(Q_OS_MAC)
platformOfInterest = "mac";
#endif
QDomElement element = domDocument.documentElement();
// Verify first element is "projects"
@ -552,42 +747,48 @@ void TestRunner::parseBuildInformation() {
throw("File is not from 'interface' build");
}
// Now loop over the platforms
// Now loop over the platforms, looking for ours
bool platformFound{ false };
element = element.firstChild().toElement();
while (!element.isNull()) {
element = element.firstChild().toElement();
if (element.tagName() != "platform" || element.attribute("name") != platformOfInterest) {
continue;
if (element.attribute("name") == platformOfInterest) {
platformFound = true;
break;
}
// Next element should be the build
element = element.firstChild().toElement();
if (element.tagName() != "build") {
throw("File seems to be in wrong format");
}
// Next element should be the version
element = element.firstChild().toElement();
if (element.tagName() != "version") {
throw("File seems to be in wrong format");
}
// Add the build number to the end of the filename
_buildInformation.build = element.text();
// First sibling should be stable_version
element = element.nextSibling().toElement();
if (element.tagName() != "stable_version") {
throw("File seems to be in wrong format");
}
// Next sibling should be url
element = element.nextSibling().toElement();
if (element.tagName() != "url") {
throw("File seems to be in wrong format");
}
_buildInformation.url = element.text();
}
if (!platformFound) {
throw("File seems to be in wrong format - platform " + platformOfInterest + " not found");
}
element = element.firstChild().toElement();
if (element.tagName() != "build") {
throw("File seems to be in wrong format");
}
// Next element should be the version
element = element.firstChild().toElement();
if (element.tagName() != "version") {
throw("File seems to be in wrong format");
}
// Add the build number to the end of the filename
_buildInformation.build = element.text();
// First sibling should be stable_version
element = element.nextSibling().toElement();
if (element.tagName() != "stable_version") {
throw("File seems to be in wrong format");
}
// Next sibling should be url
element = element.nextSibling().toElement();
if (element.tagName() != "url") {
throw("File seems to be in wrong format");
}
_buildInformation.url = element.text();
} catch (QString errorMessage) {
QMessageBox::critical(0, "Internal error: " + QString(__FILE__) + ":" + QString::number(__LINE__), errorMessage);
exit(-1);
@ -605,4 +806,4 @@ int Worker::runCommand() {
int result = system(_commandLine.toStdString().c_str());
emit commandComplete();
return result;
}
}

View file

@ -84,11 +84,18 @@ private slots:
signals:
void startInstaller();
void startInterface();
void startResize();
private:
bool _automatedTestIsRunning{ false };
#ifdef Q_OS_WIN
const QString INSTALLER_FILENAME_LATEST{ "HighFidelity-Beta-latest-dev.exe" };
#elif defined(Q_OS_MAC)
const QString INSTALLER_FILENAME_LATEST{ "HighFidelity-Beta-latest-dev.dmg" };
#else
const QString INSTALLER_FILENAME_LATEST{ "" };
#endif
QString _installerURL;
QString _installerFilename;
@ -124,11 +131,12 @@ private:
QDateTime _testStartDateTime;
QThread* installerThread;
QThread* interfaceThread;
Worker* installerWorker;
Worker* interfaceWorker;
QThread* _installerThread;
QThread* _interfaceThread;
Worker* _installerWorker;
Worker* _interfaceWorker;
BuildInformation _buildInformation;
};
@ -144,8 +152,8 @@ signals:
void commandComplete();
void startInstaller();
void startInterface();
private:
QString _commandLine;
};
#endif // hifi_testRunner_h
#endif // hifi_testRunner_h

View file

@ -8,11 +8,11 @@
// See the accompanying file LICENSE or http://www.apache.org/licenses/LICENSE-2.0.html
//
#include <QtWidgets/QApplication>
#include "ui/AutoTester.h"
#include "ui/Nitpick.h"
#include <iostream>
AutoTester* autoTester;
Nitpick* nitpick;
int main(int argc, char *argv[]) {
// If no parameters then run in interactive mode
@ -62,13 +62,13 @@ int main(int argc, char *argv[]) {
QApplication application(argc, argv);
autoTester = new AutoTester();
autoTester->setup();
nitpick = new Nitpick();
nitpick->setup();
if (!testFolder.isNull()) {
autoTester->startTestsEvaluation(true ,false, testFolder, branch, user);
nitpick->startTestsEvaluation(true ,false, testFolder, branch, user);
} else {
autoTester->show();
nitpick->show();
}
return application.exec();

View file

@ -14,7 +14,7 @@
</rect>
</property>
<property name="windowTitle">
<string>AutoTester Help</string>
<string>Nitpick Help</string>
</property>
<widget class="QTextEdit" name="textEdit">
<property name="geometry">

View file

@ -1,5 +1,5 @@
//
// AutoTester.cpp
// Nitpick.cpp
// zone/ambientLightInheritence
//
// Created by Nissim Hadar on 2 Nov 2017.
@ -8,14 +8,14 @@
// Distributed under the Apache License, Version 2.0.
// See the accompanying file LICENSE or http://www.apache.org/licenses/LICENSE-2.0.html
//
#include "AutoTester.h"
#include "Nitpick.h"
#ifdef Q_OS_WIN
#include <windows.h>
#include <shellapi.h>
#endif
AutoTester::AutoTester(QWidget* parent) : QMainWindow(parent) {
Nitpick::Nitpick(QWidget* parent) : QMainWindow(parent) {
_ui.setupUi(this);
_ui.checkBoxInteractiveMode->setChecked(true);
@ -24,9 +24,9 @@ AutoTester::AutoTester(QWidget* parent) : QMainWindow(parent) {
_signalMapper = new QSignalMapper();
connect(_ui.actionClose, &QAction::triggered, this, &AutoTester::on_closeButton_clicked);
connect(_ui.actionAbout, &QAction::triggered, this, &AutoTester::about);
connect(_ui.actionContent, &QAction::triggered, this, &AutoTester::content);
connect(_ui.actionClose, &QAction::triggered, this, &Nitpick::on_closeButton_clicked);
connect(_ui.actionAbout, &QAction::triggered, this, &Nitpick::about);
connect(_ui.actionContent, &QAction::triggered, this, &Nitpick::content);
// The second tab hides and shows the Windows task bar
#ifndef Q_OS_WIN
@ -36,13 +36,13 @@ AutoTester::AutoTester(QWidget* parent) : QMainWindow(parent) {
_ui.statusLabel->setText("");
_ui.plainTextEdit->setReadOnly(true);
setWindowTitle("Auto Tester - v6.7");
setWindowTitle("Nitpick - v1.0");
// Coming soon to an auto-tester near you...
// Coming soon to a nitpick near you...
//// _helpWindow.textBrowser->setText()
}
AutoTester::~AutoTester() {
Nitpick::~Nitpick() {
delete _signalMapper;
if (_test) {
@ -54,7 +54,7 @@ AutoTester::~AutoTester() {
}
}
void AutoTester::setup() {
void Nitpick::setup() {
if (_test) {
delete _test;
}
@ -87,7 +87,7 @@ void AutoTester::setup() {
_testRunner = new TestRunner(dayCheckboxes, timeEditCheckboxes, timeEdits, _ui.workingFolderLabel, _ui.checkBoxServerless, _ui.checkBoxRunLatest, _ui.urlLineEdit, _ui.runNowButton);
}
void AutoTester::startTestsEvaluation(const bool isRunningFromCommandLine,
void Nitpick::startTestsEvaluation(const bool isRunningFromCommandLine,
const bool isRunningInAutomaticTestRun,
const QString& snapshotDirectory,
const QString& branch,
@ -96,8 +96,13 @@ void AutoTester::startTestsEvaluation(const bool isRunningFromCommandLine,
_test->startTestsEvaluation(isRunningFromCommandLine, isRunningInAutomaticTestRun, snapshotDirectory, branch, user);
}
void AutoTester::on_tabWidget_currentChanged(int index) {
void Nitpick::on_tabWidget_currentChanged(int index) {
// Enable the GitHub edit boxes as required
#ifdef Q_OS_WIN
if (index == 0 || index == 2 || index == 3) {
#else
if (index == 0 || index == 1 || index == 2) {
#endif
_ui.userLineEdit->setDisabled(false);
_ui.branchLineEdit->setDisabled(false);
} else {
@ -106,73 +111,73 @@ void AutoTester::on_tabWidget_currentChanged(int index) {
}
}
void AutoTester::on_evaluateTestsButton_clicked() {
void Nitpick::on_evaluateTestsButton_clicked() {
_test->startTestsEvaluation(false, false);
}
void AutoTester::on_createRecursiveScriptButton_clicked() {
void Nitpick::on_createRecursiveScriptButton_clicked() {
_test->createRecursiveScript();
}
void AutoTester::on_createAllRecursiveScriptsButton_clicked() {
void Nitpick::on_createAllRecursiveScriptsButton_clicked() {
_test->createAllRecursiveScripts();
}
void AutoTester::on_createTestsButton_clicked() {
void Nitpick::on_createTestsButton_clicked() {
_test->createTests();
}
void AutoTester::on_createMDFileButton_clicked() {
void Nitpick::on_createMDFileButton_clicked() {
_test->createMDFile();
}
void AutoTester::on_createAllMDFilesButton_clicked() {
void Nitpick::on_createAllMDFilesButton_clicked() {
_test->createAllMDFiles();
}
void AutoTester::on_createTestAutoScriptButton_clicked() {
void Nitpick::on_createTestAutoScriptButton_clicked() {
_test->createTestAutoScript();
}
void AutoTester::on_createAllTestAutoScriptsButton_clicked() {
void Nitpick::on_createAllTestAutoScriptsButton_clicked() {
_test->createAllTestAutoScripts();
}
void AutoTester::on_createTestsOutlineButton_clicked() {
void Nitpick::on_createTestsOutlineButton_clicked() {
_test->createTestsOutline();
}
void AutoTester::on_createTestRailTestCasesButton_clicked() {
void Nitpick::on_createTestRailTestCasesButton_clicked() {
_test->createTestRailTestCases();
}
void AutoTester::on_createTestRailRunButton_clicked() {
void Nitpick::on_createTestRailRunButton_clicked() {
_test->createTestRailRun();
}
void AutoTester::on_setWorkingFolderButton_clicked() {
void Nitpick::on_setWorkingFolderButton_clicked() {
_testRunner->setWorkingFolder();
}
void AutoTester::enableRunTabControls() {
void Nitpick::enableRunTabControls() {
_ui.runNowButton->setEnabled(true);
_ui.daysGroupBox->setEnabled(true);
_ui.timesGroupBox->setEnabled(true);
}
void AutoTester::on_runNowButton_clicked() {
void Nitpick::on_runNowButton_clicked() {
_testRunner->run();
}
void AutoTester::on_checkBoxRunLatest_clicked() {
void Nitpick::on_checkBoxRunLatest_clicked() {
_ui.urlLineEdit->setEnabled(!_ui.checkBoxRunLatest->isChecked());
}
void AutoTester::automaticTestRunEvaluationComplete(QString zippedFolderName, int numberOfFailures) {
void Nitpick::automaticTestRunEvaluationComplete(QString zippedFolderName, int numberOfFailures) {
_testRunner->automaticTestRunEvaluationComplete(zippedFolderName, numberOfFailures);
}
void AutoTester::on_updateTestRailRunResultsButton_clicked() {
void Nitpick::on_updateTestRailRunResultsButton_clicked() {
_test->updateTestRailRunResult();
}
@ -180,7 +185,7 @@ void AutoTester::on_updateTestRailRunResultsButton_clicked() {
// if (uState & ABS_AUTOHIDE) on_showTaskbarButton_clicked();
// else on_hideTaskbarButton_clicked();
//
void AutoTester::on_hideTaskbarButton_clicked() {
void Nitpick::on_hideTaskbarButton_clicked() {
#ifdef Q_OS_WIN
APPBARDATA abd = { sizeof abd };
UINT uState = (UINT)SHAppBarMessage(ABM_GETSTATE, &abd);
@ -190,7 +195,7 @@ void AutoTester::on_hideTaskbarButton_clicked() {
#endif
}
void AutoTester::on_showTaskbarButton_clicked() {
void Nitpick::on_showTaskbarButton_clicked() {
#ifdef Q_OS_WIN
APPBARDATA abd = { sizeof abd };
UINT uState = (UINT)SHAppBarMessage(ABM_GETSTATE, &abd);
@ -200,23 +205,23 @@ void AutoTester::on_showTaskbarButton_clicked() {
#endif
}
void AutoTester::on_closeButton_clicked() {
void Nitpick::on_closeButton_clicked() {
exit(0);
}
void AutoTester::on_createPythonScriptRadioButton_clicked() {
void Nitpick::on_createPythonScriptRadioButton_clicked() {
_test->setTestRailCreateMode(PYTHON);
}
void AutoTester::on_createXMLScriptRadioButton_clicked() {
void Nitpick::on_createXMLScriptRadioButton_clicked() {
_test->setTestRailCreateMode(XML);
}
void AutoTester::on_createWebPagePushButton_clicked() {
void Nitpick::on_createWebPagePushButton_clicked() {
_test->createWebPage(_ui.updateAWSCheckBox, _ui.awsURLLineEdit);
}
void AutoTester::downloadFile(const QUrl& url) {
void Nitpick::downloadFile(const QUrl& url) {
_downloaders.emplace_back(new Downloader(url, this));
connect(_downloaders[_index], SIGNAL(downloaded()), _signalMapper, SLOT(map()));
@ -225,7 +230,7 @@ void AutoTester::downloadFile(const QUrl& url) {
++_index;
}
void AutoTester::downloadFiles(const QStringList& URLs, const QString& directoryName, const QStringList& filenames, void *caller) {
void Nitpick::downloadFiles(const QStringList& URLs, const QString& directoryName, const QStringList& filenames, void *caller) {
connect(_signalMapper, SIGNAL(mapped(int)), this, SLOT(saveFile(int)));
_directoryName = directoryName;
@ -251,7 +256,7 @@ void AutoTester::downloadFiles(const QStringList& URLs, const QString& directory
}
}
void AutoTester::saveFile(int index) {
void Nitpick::saveFile(int index) {
try {
QFile file(_directoryName + "/" + _filenames[index]);
file.open(QIODevice::WriteOnly);
@ -277,34 +282,34 @@ void AutoTester::saveFile(int index) {
}
}
void AutoTester::about() {
void Nitpick::about() {
QMessageBox::information(0, "About", QString("Built ") + __DATE__ + ", " + __TIME__);
}
void AutoTester::content() {
void Nitpick::content() {
_helpWindow.show();
}
void AutoTester::setUserText(const QString& user) {
void Nitpick::setUserText(const QString& user) {
_ui.userLineEdit->setText(user);
}
QString AutoTester::getSelectedUser() {
QString Nitpick::getSelectedUser() {
return _ui.userLineEdit->text();
}
void AutoTester::setBranchText(const QString& branch) {
void Nitpick::setBranchText(const QString& branch) {
_ui.branchLineEdit->setText(branch);
}
QString AutoTester::getSelectedBranch() {
QString Nitpick::getSelectedBranch() {
return _ui.branchLineEdit->text();
}
void AutoTester::updateStatusLabel(const QString& status) {
void Nitpick::updateStatusLabel(const QString& status) {
_ui.statusLabel->setText(status);
}
void AutoTester::appendLogWindow(const QString& message) {
void Nitpick::appendLogWindow(const QString& message) {
_ui.plainTextEdit->appendPlainText(message);
}
}

View file

@ -1,5 +1,5 @@
//
// AutoTester.h
// Nitpick.h
//
// Created by Nissim Hadar on 2 Nov 2017.
// Copyright 2013 High Fidelity, Inc.
@ -7,13 +7,13 @@
// Distributed under the Apache License, Version 2.0.
// See the accompanying file LICENSE or http://www.apache.org/licenses/LICENSE-2.0.html
//
#ifndef hifi_AutoTester_h
#define hifi_AutoTester_h
#ifndef hifi_Nitpick_h
#define hifi_Nitpick_h
#include <QtWidgets/QMainWindow>
#include <QSignalMapper>
#include <QTextEdit>
#include "ui_AutoTester.h"
#include "ui_Nitpick.h"
#include "../Downloader.h"
#include "../Test.h"
@ -22,12 +22,12 @@
#include "../TestRunner.h"
#include "../AWSInterface.h"
class AutoTester : public QMainWindow {
class Nitpick : public QMainWindow {
Q_OBJECT
public:
AutoTester(QWidget* parent = Q_NULLPTR);
~AutoTester();
Nitpick(QWidget* parent = Q_NULLPTR);
~Nitpick();
void setup();
@ -95,7 +95,7 @@ private slots:
void content();
private:
Ui::AutoTesterClass _ui;
Ui::NitpickClass _ui;
Test* _test{ nullptr };
TestRunner* _testRunner{ nullptr };
@ -121,4 +121,4 @@ private:
void* _caller;
};
#endif // hifi_AutoTester_h
#endif // hifi_Nitpick_h

View file

@ -1,7 +1,7 @@
<?xml version="1.0" encoding="UTF-8"?>
<ui version="4.0">
<class>AutoTesterClass</class>
<widget class="QMainWindow" name="AutoTesterClass">
<class>NitpickClass</class>
<widget class="QMainWindow" name="NitpickClass">
<property name="geometry">
<rect>
<x>0</x>
@ -17,7 +17,7 @@
</sizepolicy>
</property>
<property name="windowTitle">
<string>AutoTester</string>
<string>Nitpick</string>
</property>
<widget class="QWidget" name="centralWidget">
<widget class="QPushButton" name="closeButton">
@ -198,7 +198,7 @@
<x>10</x>
<y>160</y>
<width>161</width>
<height>28</height>
<height>51</height>
</rect>
</property>
<property name="text">
@ -525,7 +525,7 @@
<rect>
<x>128</x>
<y>95</y>
<width>21</width>
<width>31</width>
<height>31</height>
</rect>
</property>
@ -539,7 +539,7 @@
</property>
<property name="geometry">
<rect>
<x>160</x>
<x>170</x>
<y>100</y>
<width>451</width>
<height>21</height>
@ -554,9 +554,9 @@
<widget class="QCheckBox" name="checkBoxInteractiveMode">
<property name="geometry">
<rect>
<x>200</x>
<x>190</x>
<y>180</y>
<width>120</width>
<width>131</width>
<height>20</height>
</rect>
</property>
@ -572,8 +572,8 @@
<rect>
<x>330</x>
<y>170</y>
<width>101</width>
<height>40</height>
<width>181</width>
<height>51</height>
</rect>
</property>
<property name="text">
@ -684,10 +684,10 @@
</property>
<property name="geometry">
<rect>
<x>240</x>
<x>270</x>
<y>30</y>
<width>160</width>
<height>40</height>
<height>51</height>
</rect>
</property>
<property name="text">
@ -699,7 +699,7 @@
<rect>
<x>150</x>
<y>42</y>
<width>81</width>
<width>111</width>
<height>17</height>
</rect>
</property>
@ -803,7 +803,7 @@
<x>0</x>
<y>0</y>
<width>720</width>
<height>21</height>
<height>22</height>
</rect>
</property>
<widget class="QMenu" name="menuFile">