Merge branch 'exportToTestRail1' of https://github.com/NissimHadar/hifi into exportToTestRail1
Before Width: | Height: | Size: 12 KiB After Width: | Height: | Size: 14 KiB |
Before Width: | Height: | Size: 9.1 KiB After Width: | Height: | Size: 11 KiB |
|
@ -36,6 +36,9 @@ This function is used to create/update Expected Images after a successful run of
|
|||
The user will be asked for the snapshot folder and then the tests root folder. All snapshots located in the snapshot folder will be used to create or update the expected images in the relevant tests.
|
||||
### Details
|
||||
As an example - if the snapshots folder contains an image named `tests.content.entity.zone.zoneOrientation.00003.png`, then this file will be copied to `tests/contente/enity/zone/zoneOrientation/ExpectedImage0003.png`.
|
||||
## Create Tests Outline
|
||||
### Usage
|
||||
This function creates an MD file in the (user-selected) tests root folder. The file provides links to both the tests and the MD files.
|
||||
## Create MD file
|
||||
### Usage
|
||||
This function creates a file named `test.md` from a `test.js` script. The user will be asked for the folder containing the test script:
|
||||
|
@ -54,13 +57,20 @@ The process to produce the MD file is a simplistic parse of the test script.
|
|||
### Usage
|
||||
This function creates all MD files recursively from the user-selected root folder. This can be any folder in the tests hierarchy (e.g. all engine\material tests).
|
||||
|
||||
The file provides a hierarchial list of all the tests
|
||||
## Create Tests Outline
|
||||
The file provides a hierarchal list of all the tests
|
||||
## Create testAuto script
|
||||
### Usage
|
||||
This function creates an MD file in the (user-selected) tests root folder. The file provides links to both the tests and the MD files.
|
||||
This function creates a script named `testAuto.js` in a user-selected test folder.
|
||||
### Details
|
||||
The script created runs the `test.js` script in the folder in automatic mode. The script is the same for all tests.
|
||||
## Create all testAuto scripts
|
||||
### Usage
|
||||
This function creates all testAuto scripts recursively from the user-selected root folder. This can be any folder in the tests hierarchy (e.g. all engine\material tests).
|
||||
|
||||
The file provides a hierarchical list of all the tests
|
||||
## Create Recursive Script
|
||||
### Usage
|
||||
After the user selects a folder within the tests hierarchy, a script is created, named `testRecursive.js`. This script calls all `test.js` scripts in the subfolders.
|
||||
After the user selects a folder within the tests hierarchy, a script is created, named `testRecursive.js`. This script calls all `test.js` scripts in the sub-folders.
|
||||
### Details
|
||||
The various scripts are called in alphabetical order.
|
||||
|
||||
|
@ -103,13 +113,13 @@ If any tests have failed, then a zipped folder will be created in the snapshots
|
|||
### Usage
|
||||
Before starting the evaluation, make sure the GitHub user and branch are set correctly. The user should not normally be changed, but the branch may need to be set to the appropriate RC.
|
||||
|
||||
After setting the checkbox as required and pressing Evaluate - the user will be asked for the snapshots folder.
|
||||
After setting the check-box as required and pressing Evaluate - the user will be asked for the snapshots folder.
|
||||
### Details
|
||||
Evaluation proceeds in a number of steps:
|
||||
|
||||
1. A folder is created to store any failures
|
||||
|
||||
1. The expecetd images are download from GitHub. They are named slightly differently from the snapshots (e.g. `tests.engine.render.effect.highlight.coverage.00000.png` and `tests.engine.render.effect.highlight.coverage.00000_EI.png`).
|
||||
1. The expected images are download from GitHub. They are named slightly differently from the snapshots (e.g. `tests.engine.render.effect.highlight.coverage.00000.png` and `tests.engine.render.effect.highlight.coverage.00000_EI.png`).
|
||||
|
||||
1. The images are then pair-wise compared, using the SSIM algorithm. A fixed threshold is used to define a mismatch.
|
||||
|
||||
|
@ -129,9 +139,9 @@ Any access to TestRail will require the TestRail account (default is High Fideli
|
|||

|
||||
|
||||
- The default test rail user is shown, and can be changed as needed.
|
||||
- The username is usually the user's email.
|
||||
- The user-name is usually the user's email.
|
||||
- The Project ID defaults to 14 - Interface.
|
||||
- The Suite ID defaults to 1147 - Renderong.
|
||||
- The Suite ID defaults to 1147 - Rendering.
|
||||
- The TestRail page provides 3 functions for writing to TestRail.
|
||||
## Create Test Cases
|
||||
### Usage
|
||||
|
|
Before Width: | Height: | Size: 11 KiB After Width: | Height: | Size: 12 KiB |
Before Width: | Height: | Size: 9.6 KiB After Width: | Height: | Size: 11 KiB |
|
@ -244,7 +244,7 @@ void Test::startTestsEvaluation(const QString& testFolder, const QString& branch
|
|||
void Test::finishTestsEvaluation(bool isRunningFromCommandline, bool interactiveMode, QProgressBar* progressBar) {
|
||||
bool success = compareImageLists((!isRunningFromCommandline && interactiveMode), progressBar);
|
||||
|
||||
if (interactiveMode && !isRunningFromCommandline) {
|
||||
if (!isRunningFromCommandline) {
|
||||
if (success) {
|
||||
QMessageBox::information(0, "Success", "All images are as expected");
|
||||
} else {
|
||||
|
@ -302,174 +302,6 @@ void Test::includeTest(QTextStream& textStream, const QString& testPathname) {
|
|||
textStream << "Script.include(testsRootPath + \"" << partialPathWithoutTests + "\");" << endl;
|
||||
}
|
||||
|
||||
// Creates a single script in a user-selected folder.
|
||||
// This script will run all text.js scripts in every applicable sub-folder
|
||||
void Test::createRecursiveScript() {
|
||||
// Select folder to start recursing from
|
||||
QString previousSelection = _testDirectory;
|
||||
QString parent = previousSelection.left(previousSelection.lastIndexOf('/'));
|
||||
if (!parent.isNull() && parent.right(1) != "/") {
|
||||
parent += "/";
|
||||
}
|
||||
|
||||
_testDirectory =
|
||||
QFileDialog::getExistingDirectory(nullptr, "Please select folder that will contain the top level test script", parent,
|
||||
QFileDialog::ShowDirsOnly);
|
||||
|
||||
// If user cancelled then restore previous selection and return
|
||||
if (_testDirectory == "") {
|
||||
_testDirectory = previousSelection;
|
||||
return;
|
||||
}
|
||||
|
||||
createRecursiveScript(_testDirectory, true);
|
||||
}
|
||||
|
||||
// This method creates a `testRecursive.js` script in every sub-folder.
|
||||
void Test::createAllRecursiveScripts() {
|
||||
// Select folder to start recursing from
|
||||
QString previousSelection = _testsRootDirectory;
|
||||
QString parent = previousSelection.left(previousSelection.lastIndexOf('/'));
|
||||
if (!parent.isNull() && parent.right(1) != "/") {
|
||||
parent += "/";
|
||||
}
|
||||
|
||||
_testsRootDirectory = QFileDialog::getExistingDirectory(nullptr, "Please select the root folder for the recursive scripts",
|
||||
parent, QFileDialog::ShowDirsOnly);
|
||||
|
||||
// If user cancelled then restore previous selection and return
|
||||
if (_testsRootDirectory == "") {
|
||||
_testsRootDirectory = previousSelection;
|
||||
return;
|
||||
}
|
||||
|
||||
createRecursiveScript(_testsRootDirectory, false);
|
||||
|
||||
QDirIterator it(_testsRootDirectory.toStdString().c_str(), QDirIterator::Subdirectories);
|
||||
while (it.hasNext()) {
|
||||
QString directory = it.next();
|
||||
|
||||
// Only process directories
|
||||
QDir dir;
|
||||
if (!isAValidDirectory(directory)) {
|
||||
continue;
|
||||
}
|
||||
|
||||
// Only process directories that have sub-directories
|
||||
bool hasNoSubDirectories{ true };
|
||||
QDirIterator it2(directory.toStdString().c_str(), QDirIterator::Subdirectories);
|
||||
while (it2.hasNext()) {
|
||||
QString directory2 = it2.next();
|
||||
|
||||
// Only process directories
|
||||
QDir dir;
|
||||
if (isAValidDirectory(directory2)) {
|
||||
hasNoSubDirectories = false;
|
||||
break;
|
||||
}
|
||||
}
|
||||
|
||||
if (!hasNoSubDirectories) {
|
||||
createRecursiveScript(directory, false);
|
||||
}
|
||||
}
|
||||
|
||||
QMessageBox::information(0, "Success", "Scripts have been created");
|
||||
}
|
||||
|
||||
void Test::createRecursiveScript(const QString& topLevelDirectory, bool interactiveMode) {
|
||||
const QString recursiveTestsFilename("testRecursive.js");
|
||||
QFile allTestsFilename(topLevelDirectory + "/" + recursiveTestsFilename);
|
||||
if (!allTestsFilename.open(QIODevice::WriteOnly | QIODevice::Text)) {
|
||||
QMessageBox::critical(0,
|
||||
"Internal error: " + QString(__FILE__) + ":" + QString::number(__LINE__),
|
||||
"Failed to create \"" + recursiveTestsFilename + "\" in directory \"" + topLevelDirectory + "\""
|
||||
);
|
||||
|
||||
exit(-1);
|
||||
}
|
||||
|
||||
QTextStream textStream(&allTestsFilename);
|
||||
|
||||
const QString DATE_TIME_FORMAT("MMM d yyyy, h:mm");
|
||||
textStream << "// This is an automatically generated file, created by auto-tester on " << QDateTime::currentDateTime().toString(DATE_TIME_FORMAT) << endl << endl;
|
||||
|
||||
// Include 'autoTest.js'
|
||||
QString branch = autoTester->getSelectedBranch();
|
||||
QString user = autoTester->getSelectedUser();
|
||||
|
||||
textStream << "PATH_TO_THE_REPO_PATH_UTILS_FILE = \"https://raw.githubusercontent.com/" + user + "/hifi_tests/" + branch + "/tests/utils/branchUtils.js\";" << endl;
|
||||
textStream << "Script.include(PATH_TO_THE_REPO_PATH_UTILS_FILE);" << endl;
|
||||
textStream << "var autoTester = createAutoTester(Script.resolvePath(\".\"));" << endl << endl;
|
||||
|
||||
textStream << "var testsRootPath = autoTester.getTestsRootPath();" << endl << endl;
|
||||
|
||||
// Wait 10 seconds before starting
|
||||
textStream << "if (typeof Test !== 'undefined') {" << endl;
|
||||
textStream << " Test.wait(10000);" << endl;
|
||||
textStream << "};" << endl << endl;
|
||||
|
||||
textStream << "autoTester.enableRecursive();" << endl;
|
||||
textStream << "autoTester.enableAuto();" << endl << endl;
|
||||
|
||||
// This is used to verify that the recursive test contains at least one test
|
||||
bool testFound{ false };
|
||||
|
||||
// Directories are included in reverse order. The autoTester scripts use a stack mechanism,
|
||||
// so this ensures that the tests run in alphabetical order (a convenience when debugging)
|
||||
QStringList directories;
|
||||
|
||||
// First test if top-level folder has a test.js file
|
||||
const QString testPathname{ topLevelDirectory + "/" + TEST_FILENAME };
|
||||
QFileInfo fileInfo(testPathname);
|
||||
if (fileInfo.exists()) {
|
||||
// Current folder contains a test
|
||||
directories.push_front(testPathname);
|
||||
|
||||
testFound = true;
|
||||
}
|
||||
|
||||
QDirIterator it(topLevelDirectory.toStdString().c_str(), QDirIterator::Subdirectories);
|
||||
while (it.hasNext()) {
|
||||
QString directory = it.next();
|
||||
|
||||
// Only process directories
|
||||
QDir dir(directory);
|
||||
if (!isAValidDirectory(directory)) {
|
||||
continue;
|
||||
}
|
||||
|
||||
const QString testPathname { directory + "/" + TEST_FILENAME };
|
||||
QFileInfo fileInfo(testPathname);
|
||||
if (fileInfo.exists()) {
|
||||
// Current folder contains a test
|
||||
directories.push_front(testPathname);
|
||||
|
||||
testFound = true;
|
||||
}
|
||||
}
|
||||
|
||||
if (interactiveMode && !testFound) {
|
||||
QMessageBox::information(0, "Failure", "No \"" + TEST_FILENAME + "\" files found");
|
||||
allTestsFilename.close();
|
||||
return;
|
||||
}
|
||||
|
||||
// Now include the test scripts
|
||||
for (int i = 0; i < directories.length(); ++i) {
|
||||
includeTest(textStream, directories.at(i));
|
||||
}
|
||||
|
||||
textStream << endl;
|
||||
textStream << "autoTester.runRecursive();" << endl;
|
||||
|
||||
allTestsFilename.close();
|
||||
|
||||
if (interactiveMode) {
|
||||
QMessageBox::information(0, "Success", "Script has been created");
|
||||
}
|
||||
}
|
||||
|
||||
void Test::createTests() {
|
||||
// Rename files sequentially, as ExpectedResult_00000.jpeg, ExpectedResult_00001.jpg and so on
|
||||
// Any existing expected result images will be deleted
|
||||
|
@ -613,9 +445,7 @@ ExtractedText Test::getTestScriptLines(QString testFileName) {
|
|||
return relevantTextFromTest;
|
||||
}
|
||||
|
||||
// Create an MD file for a user-selected test.
|
||||
// The folder selected must contain a script named "test.js", the file produced is named "test.md"
|
||||
void Test::createMDFile() {
|
||||
bool Test::createFileSetup() {
|
||||
// Folder selection
|
||||
QString previousSelection = _testDirectory;
|
||||
QString parent = previousSelection.left(previousSelection.lastIndexOf('/'));
|
||||
|
@ -624,20 +454,18 @@ void Test::createMDFile() {
|
|||
}
|
||||
|
||||
_testDirectory = QFileDialog::getExistingDirectory(nullptr, "Please select folder containing the test", parent,
|
||||
QFileDialog::ShowDirsOnly);
|
||||
QFileDialog::ShowDirsOnly);
|
||||
|
||||
// If user cancelled then restore previous selection and return
|
||||
if (_testDirectory == "") {
|
||||
_testDirectory = previousSelection;
|
||||
return;
|
||||
return false;
|
||||
}
|
||||
|
||||
createMDFile(_testDirectory);
|
||||
|
||||
QMessageBox::information(0, "Success", "MD file has been created");
|
||||
return true;
|
||||
}
|
||||
|
||||
void Test::createAllMDFiles() {
|
||||
bool Test::createAllFilesSetup() {
|
||||
// Select folder to start recursing from
|
||||
QString previousSelection = _testsRootDirectory;
|
||||
QString parent = previousSelection.left(previousSelection.lastIndexOf('/'));
|
||||
|
@ -646,11 +474,31 @@ void Test::createAllMDFiles() {
|
|||
}
|
||||
|
||||
_testsRootDirectory = QFileDialog::getExistingDirectory(nullptr, "Please select the root folder for the MD files", parent,
|
||||
QFileDialog::ShowDirsOnly);
|
||||
QFileDialog::ShowDirsOnly);
|
||||
|
||||
// If user cancelled then restore previous selection and return
|
||||
if (_testsRootDirectory == "") {
|
||||
_testsRootDirectory = previousSelection;
|
||||
return false;
|
||||
}
|
||||
|
||||
return true;
|
||||
}
|
||||
|
||||
// Create an MD file for a user-selected test.
|
||||
// The folder selected must contain a script named "test.js", the file produced is named "test.md"
|
||||
void Test::createMDFile() {
|
||||
if (!createFileSetup()) {
|
||||
return;
|
||||
}
|
||||
|
||||
if (createMDFile(_testDirectory)) {
|
||||
QMessageBox::information(0, "Success", "MD file has been created");
|
||||
}
|
||||
}
|
||||
|
||||
void Test::createAllMDFiles() {
|
||||
if (!createAllFilesSetup()) {
|
||||
return;
|
||||
}
|
||||
|
||||
|
@ -681,18 +529,18 @@ void Test::createAllMDFiles() {
|
|||
QMessageBox::information(0, "Success", "MD files have been created");
|
||||
}
|
||||
|
||||
void Test::createMDFile(const QString& _testDirectory) {
|
||||
bool Test::createMDFile(const QString& directory) {
|
||||
// Verify folder contains test.js file
|
||||
QString testFileName(_testDirectory + "/" + TEST_FILENAME);
|
||||
QString testFileName(directory + "/" + TEST_FILENAME);
|
||||
QFileInfo testFileInfo(testFileName);
|
||||
if (!testFileInfo.exists()) {
|
||||
QMessageBox::critical(0, "Error", "Could not find file: " + TEST_FILENAME);
|
||||
return;
|
||||
return false;
|
||||
}
|
||||
|
||||
ExtractedText testScriptLines = getTestScriptLines(testFileName);
|
||||
|
||||
QString mdFilename(_testDirectory + "/" + "test.md");
|
||||
QString mdFilename(directory + "/" + "test.md");
|
||||
QFile mdFile(mdFilename);
|
||||
if (!mdFile.open(QIODevice::WriteOnly)) {
|
||||
QMessageBox::critical(0, "Internal error: " + QString(__FILE__) + ":" + QString::number(__LINE__), "Failed to create file " + mdFilename);
|
||||
|
@ -727,6 +575,217 @@ void Test::createMDFile(const QString& _testDirectory) {
|
|||
}
|
||||
|
||||
mdFile.close();
|
||||
return true;
|
||||
}
|
||||
|
||||
void Test::createTestAutoScript() {
|
||||
if (!createFileSetup()) {
|
||||
return;
|
||||
}
|
||||
|
||||
if (createTestAutoScript(_testDirectory)) {
|
||||
QMessageBox::information(0, "Success", "'autoTester.js` script has been created");
|
||||
}
|
||||
}
|
||||
|
||||
void Test::createAllTestAutoScripts() {
|
||||
if (!createAllFilesSetup()) {
|
||||
return;
|
||||
}
|
||||
|
||||
// First test if top-level folder has a test.js file
|
||||
const QString testPathname{ _testsRootDirectory + "/" + TEST_FILENAME };
|
||||
QFileInfo fileInfo(testPathname);
|
||||
if (fileInfo.exists()) {
|
||||
createTestAutoScript(_testsRootDirectory);
|
||||
}
|
||||
|
||||
QDirIterator it(_testsRootDirectory.toStdString().c_str(), QDirIterator::Subdirectories);
|
||||
while (it.hasNext()) {
|
||||
QString directory = it.next();
|
||||
|
||||
// Only process directories
|
||||
QDir dir;
|
||||
if (!isAValidDirectory(directory)) {
|
||||
continue;
|
||||
}
|
||||
|
||||
const QString testPathname{ directory + "/" + TEST_FILENAME };
|
||||
QFileInfo fileInfo(testPathname);
|
||||
if (fileInfo.exists()) {
|
||||
createTestAutoScript(directory);
|
||||
}
|
||||
}
|
||||
|
||||
QMessageBox::information(0, "Success", "'autoTester.js' scripts have been created");
|
||||
}
|
||||
|
||||
bool Test::createTestAutoScript(const QString& directory) {
|
||||
// Verify folder contains test.js file
|
||||
QString testFileName(directory + "/" + TEST_FILENAME);
|
||||
QFileInfo testFileInfo(testFileName);
|
||||
if (!testFileInfo.exists()) {
|
||||
QMessageBox::critical(0, "Error", "Could not find file: " + TEST_FILENAME);
|
||||
return false;
|
||||
}
|
||||
|
||||
QString testAutoScriptFilename(directory + "/" + "testAuto.js");
|
||||
QFile testAutoScriptFile(testAutoScriptFilename);
|
||||
if (!testAutoScriptFile.open(QIODevice::WriteOnly)) {
|
||||
QMessageBox::critical(0, "Internal error: " + QString(__FILE__) + ":" + QString::number(__LINE__),
|
||||
"Failed to create file " + testAutoScriptFilename);
|
||||
exit(-1);
|
||||
}
|
||||
|
||||
QTextStream stream(&testAutoScriptFile);
|
||||
|
||||
stream << "if (typeof PATH_TO_THE_REPO_PATH_UTILS_FILE === 'undefined') PATH_TO_THE_REPO_PATH_UTILS_FILE = 'https://raw.githubusercontent.com/highfidelity/hifi_tests/master/tests/utils/branchUtils.js';\n";
|
||||
stream << "Script.include(PATH_TO_THE_REPO_PATH_UTILS_FILE);\n";
|
||||
stream << "var autoTester = createAutoTester(Script.resolvePath('.'));\n\n";
|
||||
stream << "autoTester.enableAuto();\n\n";
|
||||
stream << "Script.include('./test.js?raw=true');\n";
|
||||
|
||||
testAutoScriptFile.close();
|
||||
return true;
|
||||
}
|
||||
|
||||
// Creates a single script in a user-selected folder.
|
||||
// This script will run all text.js scripts in every applicable sub-folder
|
||||
void Test::createRecursiveScript() {
|
||||
if (!createFileSetup()) {
|
||||
return;
|
||||
}
|
||||
|
||||
createRecursiveScript(_testDirectory, true);
|
||||
QMessageBox::information(0, "Success", "'testRecursive.js` script has been created");
|
||||
}
|
||||
|
||||
// This method creates a `testRecursive.js` script in every sub-folder.
|
||||
void Test::createAllRecursiveScripts() {
|
||||
if (!createAllFilesSetup()) {
|
||||
return;
|
||||
}
|
||||
|
||||
createRecursiveScript(_testsRootDirectory, false);
|
||||
|
||||
QDirIterator it(_testsRootDirectory.toStdString().c_str(), QDirIterator::Subdirectories);
|
||||
while (it.hasNext()) {
|
||||
QString directory = it.next();
|
||||
|
||||
// Only process directories
|
||||
QDir dir;
|
||||
if (!isAValidDirectory(directory)) {
|
||||
continue;
|
||||
}
|
||||
|
||||
// Only process directories that have sub-directories
|
||||
bool hasNoSubDirectories{ true };
|
||||
QDirIterator it2(directory.toStdString().c_str(), QDirIterator::Subdirectories);
|
||||
while (it2.hasNext()) {
|
||||
QString directory2 = it2.next();
|
||||
|
||||
// Only process directories
|
||||
QDir dir;
|
||||
if (isAValidDirectory(directory2)) {
|
||||
hasNoSubDirectories = false;
|
||||
break;
|
||||
}
|
||||
}
|
||||
|
||||
if (!hasNoSubDirectories) {
|
||||
createRecursiveScript(directory, false);
|
||||
}
|
||||
}
|
||||
|
||||
QMessageBox::information(0, "Success", "Scripts have been created");
|
||||
}
|
||||
|
||||
void Test::createRecursiveScript(const QString& topLevelDirectory, bool interactiveMode) {
|
||||
const QString recursiveTestsFilename("testRecursive.js");
|
||||
QFile allTestsFilename(topLevelDirectory + "/" + recursiveTestsFilename);
|
||||
if (!allTestsFilename.open(QIODevice::WriteOnly | QIODevice::Text)) {
|
||||
QMessageBox::critical(0, "Internal error: " + QString(__FILE__) + ":" + QString::number(__LINE__),
|
||||
"Failed to create \"" + recursiveTestsFilename + "\" in directory \"" + topLevelDirectory + "\"");
|
||||
|
||||
exit(-1);
|
||||
}
|
||||
|
||||
QTextStream textStream(&allTestsFilename);
|
||||
|
||||
textStream << "// This is an automatically generated file, created by auto-tester" << endl;
|
||||
|
||||
// Include 'autoTest.js'
|
||||
QString branch = autoTester->getSelectedBranch();
|
||||
QString user = autoTester->getSelectedUser();
|
||||
|
||||
textStream << "PATH_TO_THE_REPO_PATH_UTILS_FILE = \"https://raw.githubusercontent.com/" + user + "/hifi_tests/" + branch +
|
||||
"/tests/utils/branchUtils.js\";"
|
||||
<< endl;
|
||||
textStream << "Script.include(PATH_TO_THE_REPO_PATH_UTILS_FILE);" << endl;
|
||||
textStream << "var autoTester = createAutoTester(Script.resolvePath(\".\"));" << endl << endl;
|
||||
|
||||
textStream << "var testsRootPath = autoTester.getTestsRootPath();" << endl << endl;
|
||||
|
||||
// Wait 10 seconds before starting
|
||||
textStream << "if (typeof Test !== 'undefined') {" << endl;
|
||||
textStream << " Test.wait(10000);" << endl;
|
||||
textStream << "};" << endl << endl;
|
||||
|
||||
textStream << "autoTester.enableRecursive();" << endl;
|
||||
textStream << "autoTester.enableAuto();" << endl << endl;
|
||||
|
||||
// This is used to verify that the recursive test contains at least one test
|
||||
bool testFound{ false };
|
||||
|
||||
// Directories are included in reverse order. The autoTester scripts use a stack mechanism,
|
||||
// so this ensures that the tests run in alphabetical order (a convenience when debugging)
|
||||
QStringList directories;
|
||||
|
||||
// First test if top-level folder has a test.js file
|
||||
const QString testPathname{ topLevelDirectory + "/" + TEST_FILENAME };
|
||||
QFileInfo fileInfo(testPathname);
|
||||
if (fileInfo.exists()) {
|
||||
// Current folder contains a test
|
||||
directories.push_front(testPathname);
|
||||
|
||||
testFound = true;
|
||||
}
|
||||
|
||||
QDirIterator it(topLevelDirectory.toStdString().c_str(), QDirIterator::Subdirectories);
|
||||
while (it.hasNext()) {
|
||||
QString directory = it.next();
|
||||
|
||||
// Only process directories
|
||||
QDir dir(directory);
|
||||
if (!isAValidDirectory(directory)) {
|
||||
continue;
|
||||
}
|
||||
|
||||
const QString testPathname{ directory + "/" + TEST_FILENAME };
|
||||
QFileInfo fileInfo(testPathname);
|
||||
if (fileInfo.exists()) {
|
||||
// Current folder contains a test
|
||||
directories.push_front(testPathname);
|
||||
|
||||
testFound = true;
|
||||
}
|
||||
}
|
||||
|
||||
if (interactiveMode && !testFound) {
|
||||
QMessageBox::information(0, "Failure", "No \"" + TEST_FILENAME + "\" files found");
|
||||
allTestsFilename.close();
|
||||
return;
|
||||
}
|
||||
|
||||
// Now include the test scripts
|
||||
for (int i = 0; i < directories.length(); ++i) {
|
||||
includeTest(textStream, directories.at(i));
|
||||
}
|
||||
|
||||
textStream << endl;
|
||||
textStream << "autoTester.runRecursive();" << endl;
|
||||
|
||||
allTestsFilename.close();
|
||||
}
|
||||
|
||||
void Test::createTestsOutline() {
|
||||
|
|
|
@ -46,22 +46,29 @@ public:
|
|||
void startTestsEvaluation(const QString& testFolder = QString(), const QString& branchFromCommandLine = QString(), const QString& userFromCommandLine = QString());
|
||||
void finishTestsEvaluation(bool isRunningFromCommandline, bool interactiveMode, QProgressBar* progressBar);
|
||||
|
||||
void createRecursiveScript();
|
||||
void createAllRecursiveScripts();
|
||||
void createRecursiveScript(const QString& topLevelDirectory, bool interactiveMode);
|
||||
|
||||
void createTests();
|
||||
|
||||
void createTestsOutline();
|
||||
|
||||
bool createFileSetup();
|
||||
bool createAllFilesSetup();
|
||||
|
||||
void createMDFile();
|
||||
void createAllMDFiles();
|
||||
void createMDFile(const QString& topLevelDirectory);
|
||||
bool createMDFile(const QString& directory);
|
||||
|
||||
void createTestAutoScript();
|
||||
void createAllTestAutoScripts();
|
||||
bool createTestAutoScript(const QString& directory);
|
||||
|
||||
void createTestRailTestCases();
|
||||
void createTestRailRun();
|
||||
void updateTestRailRunResult();
|
||||
|
||||
void createRecursiveScript();
|
||||
void createAllRecursiveScripts();
|
||||
void createRecursiveScript(const QString& topLevelDirectory, bool interactiveMode);
|
||||
|
||||
bool compareImageLists(bool isInteractiveMode, QProgressBar* progressBar);
|
||||
|
||||
QStringList createListOfAll_imagesInDirectory(const QString& imageFormat, const QString& pathToImageDirectory);
|
||||
|
|
|
@ -25,7 +25,7 @@ TestRailInterface::TestRailInterface() {
|
|||
_testRailTestCasesSelectorWindow.setUser("@highfidelity.io");
|
||||
////_testRailTestCasesSelectorWindow.setUser("nissim.hadar@gmail.com");
|
||||
|
||||
_testRailTestCasesSelectorWindow.setProjectID(INTERFACE_PROJECT_ID);
|
||||
_testRailTestCasesSelectorWindow.setProjectID(INTERFACE_AUTOMATION_PROJECT_ID);
|
||||
////_testRailTestCasesSelectorWindow.setProjectID(2);
|
||||
|
||||
_testRailTestCasesSelectorWindow.setSuiteID(INTERFACE_SUITE_ID);
|
||||
|
@ -36,7 +36,7 @@ TestRailInterface::TestRailInterface() {
|
|||
_testRailRunSelectorWindow.setUser("@highfidelity.io");
|
||||
////_testRailRunSelectorWindow.setUser("nissim.hadar@gmail.com");
|
||||
|
||||
_testRailRunSelectorWindow.setProjectID(INTERFACE_PROJECT_ID);
|
||||
_testRailRunSelectorWindow.setProjectID(INTERFACE_AUTOMATION_PROJECT_ID);
|
||||
////_testRailRunSelectorWindow.setProjectID(2);
|
||||
|
||||
_testRailRunSelectorWindow.setSuiteID(INTERFACE_SUITE_ID);
|
||||
|
@ -47,7 +47,7 @@ TestRailInterface::TestRailInterface() {
|
|||
_testRailResultsSelectorWindow.setUser("@highfidelity.io");
|
||||
////_testRailResultsSelectorWindow.setUser("nissim.hadar@gmail.com");
|
||||
|
||||
_testRailResultsSelectorWindow.setProjectID(INTERFACE_PROJECT_ID);
|
||||
_testRailResultsSelectorWindow.setProjectID(INTERFACE_AUTOMATION_PROJECT_ID);
|
||||
////_testRailResultsSelectorWindow.setProjectID(2);
|
||||
|
||||
_testRailResultsSelectorWindow.setSuiteID(INTERFACE_SUITE_ID);
|
||||
|
@ -908,7 +908,7 @@ QDomElement TestRailInterface::processTestXML(const QString& fullDirectory,
|
|||
++i;
|
||||
QString title{ words[i] };
|
||||
for (++i; i < words.length() - 1; ++i) {
|
||||
title += " / " + words[i];
|
||||
title += "/" + words[i];
|
||||
}
|
||||
|
||||
QDomElement titleElement = _document.createElement("title");
|
||||
|
|
|
@ -92,10 +92,10 @@ public:
|
|||
|
||||
private:
|
||||
// HighFidelity Interface project ID in TestRail
|
||||
const int INTERFACE_PROJECT_ID{ 24 };
|
||||
const int INTERFACE_AUTOMATION_PROJECT_ID{ 26 };
|
||||
|
||||
// Rendering suite ID
|
||||
const int INTERFACE_SUITE_ID{ 1147 };
|
||||
const int INTERFACE_SUITE_ID{ 1312 };
|
||||
|
||||
QDomDocument _document;
|
||||
|
||||
|
|
|
@ -78,6 +78,14 @@ void AutoTester::on_createAllMDFilesButton_clicked() {
|
|||
_test->createAllMDFiles();
|
||||
}
|
||||
|
||||
void AutoTester::on_createTestAutoScriptButton_clicked() {
|
||||
_test->createTestAutoScript();
|
||||
}
|
||||
|
||||
void AutoTester::on_createAllTestAutoScriptsButton_clicked() {
|
||||
_test->createAllTestAutoScripts();
|
||||
}
|
||||
|
||||
void AutoTester::on_createTestsOutlineButton_clicked() {
|
||||
_test->createTestsOutline();
|
||||
}
|
||||
|
|
|
@ -50,6 +50,9 @@ private slots:
|
|||
void on_createMDFileButton_clicked();
|
||||
void on_createAllMDFilesButton_clicked();
|
||||
|
||||
void on_createTestAutoScriptButton_clicked();
|
||||
void on_createAllTestAutoScriptsButton_clicked();
|
||||
|
||||
void on_createTestsOutlineButton_clicked();
|
||||
|
||||
void on_createTestRailTestCasesButton_clicked();
|
||||
|
|
|
@ -6,7 +6,7 @@
|
|||
<rect>
|
||||
<x>0</x>
|
||||
<y>0</y>
|
||||
<width>432</width>
|
||||
<width>582</width>
|
||||
<height>734</height>
|
||||
</rect>
|
||||
</property>
|
||||
|
@ -23,8 +23,8 @@
|
|||
<widget class="QPushButton" name="closeButton">
|
||||
<property name="geometry">
|
||||
<rect>
|
||||
<x>166</x>
|
||||
<y>610</y>
|
||||
<x>235</x>
|
||||
<y>620</y>
|
||||
<width>100</width>
|
||||
<height>40</height>
|
||||
</rect>
|
||||
|
@ -36,9 +36,9 @@
|
|||
<widget class="QTabWidget" name="tabWidget">
|
||||
<property name="geometry">
|
||||
<rect>
|
||||
<x>12</x>
|
||||
<x>30</x>
|
||||
<y>140</y>
|
||||
<width>408</width>
|
||||
<width>521</width>
|
||||
<height>461</height>
|
||||
</rect>
|
||||
</property>
|
||||
|
@ -52,7 +52,7 @@
|
|||
<widget class="QPushButton" name="createTestsButton">
|
||||
<property name="geometry">
|
||||
<rect>
|
||||
<x>96</x>
|
||||
<x>145</x>
|
||||
<y>20</y>
|
||||
<width>220</width>
|
||||
<height>40</height>
|
||||
|
@ -65,8 +65,8 @@
|
|||
<widget class="QPushButton" name="createMDFileButton">
|
||||
<property name="geometry">
|
||||
<rect>
|
||||
<x>96</x>
|
||||
<y>100</y>
|
||||
<x>20</x>
|
||||
<y>140</y>
|
||||
<width>220</width>
|
||||
<height>40</height>
|
||||
</rect>
|
||||
|
@ -78,8 +78,8 @@
|
|||
<widget class="QPushButton" name="createAllMDFilesButton">
|
||||
<property name="geometry">
|
||||
<rect>
|
||||
<x>96</x>
|
||||
<y>150</y>
|
||||
<x>270</x>
|
||||
<y>140</y>
|
||||
<width>220</width>
|
||||
<height>40</height>
|
||||
</rect>
|
||||
|
@ -91,8 +91,8 @@
|
|||
<widget class="QPushButton" name="createTestsOutlineButton">
|
||||
<property name="geometry">
|
||||
<rect>
|
||||
<x>96</x>
|
||||
<y>230</y>
|
||||
<x>145</x>
|
||||
<y>80</y>
|
||||
<width>220</width>
|
||||
<height>40</height>
|
||||
</rect>
|
||||
|
@ -104,8 +104,8 @@
|
|||
<widget class="QPushButton" name="createRecursiveScriptButton">
|
||||
<property name="geometry">
|
||||
<rect>
|
||||
<x>96</x>
|
||||
<y>310</y>
|
||||
<x>20</x>
|
||||
<y>260</y>
|
||||
<width>220</width>
|
||||
<height>40</height>
|
||||
</rect>
|
||||
|
@ -117,8 +117,8 @@
|
|||
<widget class="QPushButton" name="createAllRecursiveScriptsButton">
|
||||
<property name="geometry">
|
||||
<rect>
|
||||
<x>96</x>
|
||||
<y>360</y>
|
||||
<x>270</x>
|
||||
<y>260</y>
|
||||
<width>220</width>
|
||||
<height>40</height>
|
||||
</rect>
|
||||
|
@ -127,6 +127,32 @@
|
|||
<string>Create all Recursive Scripts</string>
|
||||
</property>
|
||||
</widget>
|
||||
<widget class="QPushButton" name="createTestAutoScriptButton">
|
||||
<property name="geometry">
|
||||
<rect>
|
||||
<x>20</x>
|
||||
<y>200</y>
|
||||
<width>220</width>
|
||||
<height>40</height>
|
||||
</rect>
|
||||
</property>
|
||||
<property name="text">
|
||||
<string>Create testAuto script</string>
|
||||
</property>
|
||||
</widget>
|
||||
<widget class="QPushButton" name="createAllTestAutoScriptsButton">
|
||||
<property name="geometry">
|
||||
<rect>
|
||||
<x>270</x>
|
||||
<y>200</y>
|
||||
<width>220</width>
|
||||
<height>40</height>
|
||||
</rect>
|
||||
</property>
|
||||
<property name="text">
|
||||
<string>Create all testAuto scripts</string>
|
||||
</property>
|
||||
</widget>
|
||||
</widget>
|
||||
<widget class="QWidget" name="tab_2">
|
||||
<attribute name="title">
|
||||
|
@ -135,8 +161,8 @@
|
|||
<widget class="QProgressBar" name="progressBar">
|
||||
<property name="geometry">
|
||||
<rect>
|
||||
<x>90</x>
|
||||
<y>100</y>
|
||||
<x>150</x>
|
||||
<y>230</y>
|
||||
<width>255</width>
|
||||
<height>23</height>
|
||||
</rect>
|
||||
|
@ -148,8 +174,8 @@
|
|||
<widget class="QCheckBox" name="checkBoxInteractiveMode">
|
||||
<property name="geometry">
|
||||
<rect>
|
||||
<x>90</x>
|
||||
<y>50</y>
|
||||
<x>150</x>
|
||||
<y>180</y>
|
||||
<width>131</width>
|
||||
<height>20</height>
|
||||
</rect>
|
||||
|
@ -164,8 +190,8 @@
|
|||
<widget class="QPushButton" name="evaluateTestsButton">
|
||||
<property name="geometry">
|
||||
<rect>
|
||||
<x>200</x>
|
||||
<y>40</y>
|
||||
<x>260</x>
|
||||
<y>170</y>
|
||||
<width>101</width>
|
||||
<height>40</height>
|
||||
</rect>
|
||||
|
@ -182,8 +208,8 @@
|
|||
<widget class="QPushButton" name="updateTestRailRunResultsButton">
|
||||
<property name="geometry">
|
||||
<rect>
|
||||
<x>180</x>
|
||||
<y>160</y>
|
||||
<x>210</x>
|
||||
<y>230</y>
|
||||
<width>161</width>
|
||||
<height>40</height>
|
||||
</rect>
|
||||
|
@ -195,8 +221,8 @@
|
|||
<widget class="QRadioButton" name="createPythonScriptRadioButton">
|
||||
<property name="geometry">
|
||||
<rect>
|
||||
<x>80</x>
|
||||
<y>40</y>
|
||||
<x>110</x>
|
||||
<y>110</y>
|
||||
<width>95</width>
|
||||
<height>20</height>
|
||||
</rect>
|
||||
|
@ -211,8 +237,8 @@
|
|||
<widget class="QPushButton" name="createTestRailRunButton">
|
||||
<property name="geometry">
|
||||
<rect>
|
||||
<x>180</x>
|
||||
<y>100</y>
|
||||
<x>210</x>
|
||||
<y>170</y>
|
||||
<width>161</width>
|
||||
<height>40</height>
|
||||
</rect>
|
||||
|
@ -224,8 +250,8 @@
|
|||
<widget class="QPushButton" name="createTestRailTestCasesButton">
|
||||
<property name="geometry">
|
||||
<rect>
|
||||
<x>180</x>
|
||||
<y>40</y>
|
||||
<x>210</x>
|
||||
<y>110</y>
|
||||
<width>161</width>
|
||||
<height>40</height>
|
||||
</rect>
|
||||
|
@ -237,8 +263,8 @@
|
|||
<widget class="QRadioButton" name="createXMLScriptRadioButton">
|
||||
<property name="geometry">
|
||||
<rect>
|
||||
<x>80</x>
|
||||
<y>60</y>
|
||||
<x>110</x>
|
||||
<y>130</y>
|
||||
<width>95</width>
|
||||
<height>20</height>
|
||||
</rect>
|
||||
|
@ -255,8 +281,8 @@
|
|||
<widget class="QPushButton" name="hideTaskbarButton">
|
||||
<property name="geometry">
|
||||
<rect>
|
||||
<x>100</x>
|
||||
<y>100</y>
|
||||
<x>160</x>
|
||||
<y>130</y>
|
||||
<width>211</width>
|
||||
<height>40</height>
|
||||
</rect>
|
||||
|
@ -268,8 +294,8 @@
|
|||
<widget class="QPushButton" name="showTaskbarButton">
|
||||
<property name="geometry">
|
||||
<rect>
|
||||
<x>100</x>
|
||||
<y>170</y>
|
||||
<x>160</x>
|
||||
<y>200</y>
|
||||
<width>211</width>
|
||||
<height>40</height>
|
||||
</rect>
|
||||
|
@ -283,8 +309,8 @@
|
|||
<widget class="QLabel" name="label_3">
|
||||
<property name="geometry">
|
||||
<rect>
|
||||
<x>110</x>
|
||||
<y>90</y>
|
||||
<x>160</x>
|
||||
<y>80</y>
|
||||
<width>81</width>
|
||||
<height>16</height>
|
||||
</rect>
|
||||
|
@ -301,8 +327,8 @@
|
|||
<widget class="QTextEdit" name="branchTextEdit">
|
||||
<property name="geometry">
|
||||
<rect>
|
||||
<x>200</x>
|
||||
<y>85</y>
|
||||
<x>250</x>
|
||||
<y>75</y>
|
||||
<width>140</width>
|
||||
<height>24</height>
|
||||
</rect>
|
||||
|
@ -311,8 +337,8 @@
|
|||
<widget class="QTextEdit" name="userTextEdit">
|
||||
<property name="geometry">
|
||||
<rect>
|
||||
<x>200</x>
|
||||
<y>47</y>
|
||||
<x>250</x>
|
||||
<y>37</y>
|
||||
<width>140</width>
|
||||
<height>24</height>
|
||||
</rect>
|
||||
|
@ -321,8 +347,8 @@
|
|||
<widget class="QLabel" name="label_4">
|
||||
<property name="geometry">
|
||||
<rect>
|
||||
<x>110</x>
|
||||
<y>50</y>
|
||||
<x>160</x>
|
||||
<y>40</y>
|
||||
<width>81</width>
|
||||
<height>16</height>
|
||||
</rect>
|
||||
|
@ -342,7 +368,7 @@
|
|||
<rect>
|
||||
<x>0</x>
|
||||
<y>0</y>
|
||||
<width>432</width>
|
||||
<width>582</width>
|
||||
<height>21</height>
|
||||
</rect>
|
||||
</property>
|
||||
|
|