dEQP
====

These steps are specifically for testing SwiftShader's Vulkan implementation using dEQP on Windows (steps for Linux below the Windows instructions).

Prerequisites
-------------

1. Install the latest [Python 3](https://www.python.org/downloads/)
2. Install [Visual Studio](https://visualstudio.microsoft.com/vs/community/)
3. Install [CMake](https://cmake.org/download/)
4. Install [Go](https://golang.org/doc/install)
5. Install [MinGW-W64](http://mingw-w64.org/doku.php/download)
  * Select 'x86_64' as Architecture during setup
6. Install [Git](https://git-scm.com/download/win)
7. Set environment variables: Config Panel -> System and Security -> System -> Advanced system settigns -> Environment Variables
  * Add `` to your PATH environment variable
  * Add `\bin` to your PATH environment variable

8. (Optional) Install [TortoiseGit](https://tortoisegit.org/)

Getting the Code
----------------

12. Get dEQP (either in 'cmd' or by using TortoiseGit):

    `git clone https://github.com/KhronosGroup/VK-GL-CTS`

    You may wish to check out a stable vulkan-cts-* branch.

13. Get dEQP's dependencies. In your dEQP root directory, open 'cmd' and run:

    `python3 external\fetch_sources.py`

14. Get Cherry (either in 'cmd' or by using TortoiseGit):

    `git clone https://android.googlesource.com/platform/external/cherry`

15. Set environment variable (see point 9):

    Add new variable GOPATH='``'

Building the code
-----------------

16. Build dEQP's Visual Studio files using the CMake GUI, or, in the dEQP root dir, run:
    ```
    mkdir build
    cd build
    cmake ..
    ```
    Note: don't call 'cmake .' directly in the root directory. It will make things fails later on. If you do, simply erase the files created by CMake and follow the steps above.

17. Build dEQP:

    Open `\build\dEQP-Core-default.sln` in Visual Studio and Build Solution

    Note: Choose a 'Debug' build.

18. Generate test cases:
    ```
    mkdir \data
    cd 
    python3 scripts\build_caselists.py \data
    ```

    Note: you need to run `python3 scripts\build_caselists.py \data` every time you update dEQP.

Preparing the server
--------------------

19. Edit `\cherry\data.go`
* Search for `../candy-build/deqp-wgl` and replace that by `/build`
* Just above, add an option to CommandLine: `--deqp-gl-context-type=egl`
* Remove `--deqp-watchdog=enable` to avoid timeouts during debugging.

  Note: If you chose a Release build at step 17, modify the BinaryPath from 'Debug' to 'Release'.

Testing Vulkan
--------------

20. Assuming you already built SwiftShader, copy and rename this file:

    `\build\Release_x64\vk_swiftshader.dll` or\
    `\build\Debug_x64\vk_swiftshader.dll`

    To:

    `\build\external\vulkancts\modules\vulkan\Debug\vulkan-1.dll`

    This will cause dEQP to load SwiftShader's Vulkan implementatin directly, without going through a system-provided [loader](https://github.com/KhronosGroup/Vulkan-Loader/blob/master/loader/LoaderAndLayerInterface.md#the-loader) library or any layers.

     This step can also be automated by setting the `SWIFTSHADER_VULKAN_API_LIBRARY_INSTALL_PATH` environment variable to a path where we'd like the drop-in API library to be installed. For example `/build/external/vulkancts/modules/vulkan/Debug/`.

    To use SwiftShader as an [Installable Client Driver](https://github.com/KhronosGroup/Vulkan-Loader/blob/master/loader/LoaderAndLayerInterface.md#installable-client-drivers) (ICD) instead:
    * Edit environment variables:
      * Define VK_ICD_FILENAMES to `\src\Vulkan\vk_swiftshader_icd.json`
    * If the location of `vk_swiftshader.dll` you're using is different than the one specified in `src\Vulkan\vk_swiftshader_icd.json`, modify it to point to the `vk_swiftshader.dll` file you want to use.

Running the tests
-----------------

21. Start the test server. Go to `` and run:

    `go run server.go`

22. Open your favorite browser and navigate to `localhost:8080`

    Get Started -> Choose Device 'localhost' -> Select Tests 'dEQP-VK' -> Execute tests!

Mustpass sets
-------------

dEQP contains more tests than what is expected to pass by a conformant implementation (e.g. some tests are considered too strict, or assume certain undefined behavior). The [android/cts/master/vk-master.txt](https://android.googlesource.com/platform/external/deqp/+/master/android/cts/master/vk-master.txt) text file which can be loaded in Cherry's 'Test sets' tab to only run the latest tests expected to pass by certified Android devices.

Linux
-----

The Linux process is similar to Windows. However it doesn't use Release or Debug variants, paths use forward slashes, and it uses shared object files instead of DLLs.

1. Install the latest [Python 3](https://www.python.org/downloads/)
2. Install GCC and Make. In a terminal, run:

    `sudo apt-get install gcc make`

3. Install [CMake](https://cmake.org/download/)
4. Install [Go](https://golang.org/doc/install)
5. Install Git. In a terminal, run:

    `sudo apt-get install git`

6. Download the [Vulkan SDK](https://vulkan.lunarg.com/) and unpack it into a location you like.

Getting the Code
----------------

7. Get Swiftshader. In a terminal, go to the location you want to keep Swiftshader, and run:

    ```
    git clone https://swiftshader.googlesource.com/SwiftShader && (cd SwiftShader && curl -Lo `git rev-parse --git-dir`/hooks/commit-msg https://gerrit-review.googlesource.com/tools/hooks/commit-msg ; chmod +x `git rev-parse --git-dir`/hooks/commit-msg)
    ```

    This will also install the commit hooks you need for committing to SwiftShader.

8. Get dEQP:

   `git clone https://github.com/KhronosGroup/VK-GL-CTS`

9. Get dEQP's dependencies. In your dEQP root directory, run:

    `python3 external/fetch_sources.py`

10. Get Cherry, similar to step 8:

    `git clone https://android.googlesource.com/platform/external/cherry`

11. Set environment variable. Open ~/.bashrc in your preferred editor and add the following line:

    GOPATH='``'

Building the code
-----------------

12. Build Swiftshader. In the Swiftshader root dir, run:
    ```
    cd build
    cmake ..
    make --jobs=$(nproc)
    ```

13. Set your environment variables. In the terminal in which you'll be building dEQP, run the following commands:

    ```
    export LD_LIBRARY_PATH="/x86_64/lib:$LD_LIBRARY_PATH"
    export LD_LIBRARY_PATH="/build:$LD_LIBRARY_PATH"
    ```

14. Build dEQP. In the dEQP root dir, run:
    ```
    mkdir build
    cd build
    cmake ..
    make --jobs=$(nproc)
    ```

    Also: don't call 'cmake .' directly in the root directory. It will make things fails later on. If you do, simply erase the files created by CMake and follow the steps above.

15. Generate test cases:
    ```
    mkdir /data
    cd 
    python3 scripts/build_caselists.py /data
    ```

    Note: you need to run `python3 scripts/build_caselists.py /data` every time you update dEQP.

Preparing the server
--------------------

16. Edit `/cherry/data.go`
* Search for ".exe" and remove all instances.
* Search for `../candy-build/deqp-wgl/execserver/Release` and replace that by `/build/execserver/execserver`
* Just above, add an option to CommandLine: `--deqp-gl-context-type=egl`
* Just below, remove 'Debug/' from the BinaryPath.
* Just one more line below, replace `../candy-build/deqp-wgl/` with `/build/modules/${TestPackageDir}`.
* Remove `--deqp-watchdog=enable` to avoid timeouts during debugging.

Testing Vulkan
--------------

17. Use SwiftShader as an [Installable Client Driver](https://github.com/KhronosGroup/Vulkan-Loader/blob/master/loader/LoaderAndLayerInterface.md#installable-client-drivers) (ICD). Add the following line to your `~/.bashrc`:

      `export VK_ICD_FILENAMES="/build/Linux/vk_swiftshader_icd.json"`

    Then run `source ~/.bashrc` in the terminal(s) you'll be running tests from.


Running the tests
-----------------

18. Start the test server. Go to `` and run:

    `go run server.go`

19. Open your favorite browser and navigate to `localhost:8080`

    Get Started -> Choose Device 'localhost' -> Select Tests 'dEQP-VK' -> Execute tests!

20. To make sure that you're running SwiftShader's drivers, select only the dEQP-VK->info->device test. In the next window, click on these tests in the left pane. If you see SwiftShader in the deviceName field, then you've set your suite up properly.

21. If you want to run Vulkan tests in the command line, go to the build directory in dEQP root. Then run the following command:

    `external/vulkanacts/modules/vulkan/deqp-vk`

    You can also run individual tests with:

    `external/vulkanacts/modules/vulkan/deqp-vk --deqp-case=`

    And you can find a list of the test names in `/tests/regres/testlists/vk-master.txt` However, deqp-vk will cease upon the first failure. It's recommended that you use cherry for your testing needs unless you know what you're doing.

22. To check that you're running SwiftShader in cherry, start the server 

Mustpass sets
-------------

dEQP contains more tests than what is expected to pass by a conformant implementation (e.g. some tests are considered too strict, or assume certain undefined behavior). The [android/cts/master/vk-master.txt](https://android.googlesource.com/platform/external/deqp/+/master/android/cts/master/vk-master.txt) text file which can be loaded in Cherry's 'Test sets' tab to only run the latest tests expected to pass by certified Android devices.