CUDA-Notes-Part1-Set up CUDA computation for Tensorflow and Tensorflow2

I have to solve the GPU computing toolkits problem several times since different project frequently changes between Tensorflow and Tensorflow2. This tutorial concludes the details in setting up the GPU environment for Tensorflow. I won’t introduce how to set up python environment and anything about hardware in this blog. This tutorial is for Windows OS (The logic also works in other OSs, but the links will not be available anymore).

1. Prerequisites

In this blog, I will discuss about how to setup GPU environment for Tensorflow and Tensorflow2. Let’s create two virtual environment for DL research first. In this blog, I will use conda as the environment management tool.

1.1 Create virtual environment

In Anaconda Prompt, create two new virtual environment for python. For example,

create create -n ENV_NAME python=3.7

For Tensorflow, I created a new virtual env called DLrshenv_tf1; for Tensorflow2, I created a new virtual env called DLrshenv.

1.2 Software requirement

1. Visual Studio 2015. Note that CUDA only support VS2015. Any other version will lead to a failed installation for CUDA Toolkits. May need a subscription (it’s free).

2. CUDA Toolkit 10.1 for Tensorflow2 and CUDA Toolkit 10.0 for Tensorflow. The difference is ‘cudart64_101.dll’ and ‘cudart64_100.dll’. In CUDA Toolkit 10.1, the ‘cudart64_100.dll’ was updated to ‘cudart64_101.dll’.

3. cuDNN for CUDA 10.1 and cuDNN for CUDA 10.0

4. NVIDIA driver. (As a GPU user, I think you are very familiar with it.)

1.3 How to test the prerequisites?

Open cmd, and try

nvcc -V

Note that if you installed two CUDA version like 10.1 and 10.0, the return will be the last one your installed, which should be consistent with the system variable CUDA_PATH.

2. Python env setup

Install Tensorflow and Tensorflow2 in two virtual envs.

For Tensorflow2:

activate DLrshenv
pip install tensorflow
pip install tensorflow-gpu

For Tensorflow:

activate DLrshenv_tf1
pip install tensorflow==1.15
pip install tensorflow-gpu==1.15

3. cuDNN

Copy all the file under CUDA into the path you install CUDA. Generally, it should under

C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.0

C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.1

4. Environment variables

Add bins and libnvvp in two paths to the User’s variables. Under Path, there should be four new variables include:

C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.0\libnvvp

C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.0\bin

C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.1\libnvvp

C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.1\bin