Машинное обучение и Большие данные: различия между версиями

Материал из Artem Aleksashkin's Wiki
Перейти к навигации Перейти к поиску
Строка 98: Строка 98:
</pre>
</pre>
* https://www.tensorflow.org/install/source
* https://www.tensorflow.org/install/source
** You have to build it from source due this error: 2021-03-29 00:19:23.703486: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1657] Ignoring visible gpu device (device: 0, name: GeForce GTX 760, pci bus id: 0000:04:00.0, compute capability: 3.0) with Cuda compute capability 3.0. The minimum required Cuda capability is 3.5.
** In 3.0 won't work XLA: https://www.tensorflow.org/xla
<pre>
<pre>
git clone https://github.com/tensorflow/tensorflow.git
git clone https://github.com/tensorflow/tensorflow.git

Версия от 22:27, 30 марта 2021

Ai-brain.jpg

Hardware

Software

wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/cuda-ubuntu2004.pin
sudo mv cuda-ubuntu2004.pin /etc/apt/preferences.d/cuda-repository-pin-600
wget https://developer.download.nvidia.com/compute/cuda/11.2.2/local_installers/cuda-repo-ubuntu2004-11-2-local_11.2.2-460.32.03-1_amd64.deb
sudo dpkg -i cuda-repo-ubuntu2004-11-2-local_11.2.2-460.32.03-1_amd64.deb
sudo apt-key add /var/cuda-repo-ubuntu2004-11-2-local/7fa2af80.pub
sudo apt-get update
sudo apt-get -y install cuda nvidia-cuda-toolkit

Change Default Python

sudo update-alternatives --install /usr/bin/python python /usr/bin/python2 1
sudo update-alternatives --install /usr/bin/python python /usr/bin/python3 2
sudo update-alternatives --config python

Define Your Software Versions

  • Ubuntu 20.04. Kernel 5.4.0-70-generic
    • All downloads I took for ubuntu 18.04.
  • Nvidia GeForce GTX 760 4gb -> Nvidia Kepler
  • Nvidia Kepler -> CUDA SDK 10.0 – 10.2 support for compute capability 3.0 – 7.5 (Kepler, Maxwell, Pascal, Volta, Turing). Last version with support for compute capability 3.x (Kepler). 10.2 is the last official release for macOS, as support will not be available for macOS in newer releases.
  • Check all possible TensorFlow and Cuda versions here: https://www.tensorflow.org/install/source#gpu
  • For me - tenorflow-2.3.0, cuda 10.2, nvidia-440.33.0, cuDNN 7.6, Bazel 3.1.0, GCC 7.3.1, TensorRT 6.0
    • Cuda 10.1 won't install due 418 driver is not comportable with new kernel 5.4.0-70-generic

Installation

wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/cuda-ubuntu1804.pin
sudo mv cuda-ubuntu1804.pin /etc/apt/preferences.d/cuda-repository-pin-600
wget https://developer.download.nvidia.com/compute/cuda/10.2/Prod/local_installers/cuda-repo-ubuntu1804-10-2-local-10.2.89-440.33.01_1.0-1_amd64.deb
sudo dpkg -i cuda-repo-ubuntu1804-10-2-local-10.2.89-440.33.01_1.0-1_amd64.deb
sudo apt-key add /var/cuda-repo-10-2-local-10.2.89-440.33.01/7fa2af80.pub
sudo apt-get update
sudo apt-get -y install cuda

sudo dpkg -i libcudnn7_7.6.5.32-1+cuda10.2_amd64.deb
sudo dpkg -i libcudnn7-dev_7.6.5.32-1+cuda10.2_amd64.deb
sudo dpkg -i libcudnn7-doc_7.6.5.32-1+cuda10.2_amd64.deb

sudo dpkg -i nv-tensorrt-repo-ubuntu1804-cuda10.2-trt6.0.1.8-ga-20191108_1-1_amd64.deb
sudo apt-get update
sudo apt-key add /var/nv-tensorrt-repo-cuda10.2-trt6.0.1.8-ga-20191108/7fa2af80.pub
sudo apt-get install tensorrt
  • ccache won't work. Do not install it or disable.
    • Or you'll ger error on build: C++ compilation of rule '@com_google_protobuf//:protobuf' failed (Exit 1): crosstool_wrapper_driver_is_not_gcc failed: error executing command external/local_config_cuda/crosstool/clang/bin/crosstool_wrapper_driver_is_not_gcc -MD -MF bazel-out/host/bin/external/com_google_protobuf/_objs/protobuf/descriptor_database.d ... (remaining 48 argument(s) skipped) ccache: error: invalid size: D
  • GCC 7
    • Only 7 or you'll get error on build: # 138 | #error -- unsupported GNU version! gcc versions later than 8 are not supported!
sudo apt install gcc-7 g++-7
sudo update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-9 100 --slave /usr/bin/g++ g++ /usr/bin/g++-9 --slave /usr/bin/gcov gcov /usr/bin/gcov-9
sudo update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-7 90 --slave /usr/bin/g++ g++ /usr/bin/g++-7 --slave /usr/bin/gcov gcov /usr/bin/gcov-7
sudo update-alternatives --config gcc # gcc (Ubuntu 7.5.0-6ubuntu2) 7.5.0

sudo apt install apt-transport-https curl gnupg
curl -fsSL https://bazel.build/bazel-release.pub.gpg | gpg --dearmor > bazel.gpg
sudo mv bazel.gpg /etc/apt/trusted.gpg.d/
echo "deb [arch=amd64] https://storage.googleapis.com/bazel-apt stable jdk1.8" | sudo tee /etc/apt/sources.list.d/bazel.list
sudo apt update
sudo apt install bazel-3.2.0
sudo ln -s /usr/bin/bazel-3.2.0 /usr/bin/bazel
bazel --version  # 3.2.0
  • https://www.tensorflow.org/install/source
    • You have to build it from source due this error: 2021-03-29 00:19:23.703486: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1657] Ignoring visible gpu device (device: 0, name: GeForce GTX 760, pci bus id: 0000:04:00.0, compute capability: 3.0) with Cuda compute capability 3.0. The minimum required Cuda capability is 3.5.
    • In 3.0 won't work XLA: https://www.tensorflow.org/xla
git clone https://github.com/tensorflow/tensorflow.git
cd tensorflow
git checkout r2.3
./configure
bazel build --config=cuda --config=opt //tensorflow/tools/pip_package:build_pip_package

Testing

>>> import tensorflow as tf
>>> tf.__version__
'2.3.0'
>>> tf.test.is_built_with_cuda()
True

$ nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2019 NVIDIA Corporation
Built on Sun_Jul_28_19:07:16_PDT_2019
Cuda compilation tools, release 10.1, V10.1.243

Python 3.8.5 (default, Jan 27 2021, 15:41:15) 
[GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import tensorflow as tf
>>> tf.config.list_physical_devices("GPU")
2021-03-29 00:19:23.520023: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1
2021-03-29 00:19:23.575281: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-03-29 00:19:23.575801: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: 
pciBusID: 0000:04:00.0 name: GeForce GTX 760 computeCapability: 3.0
coreClock: 1.15GHz coreCount: 6 deviceMemorySize: 3.94GiB deviceMemoryBandwidth: 179.05GiB/s
2021-03-29 00:19:23.576789: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1
2021-03-29 00:19:23.581937: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10
2021-03-29 00:19:23.583502: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10
2021-03-29 00:19:23.585776: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10
2021-03-29 00:19:23.591336: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10
2021-03-29 00:19:23.593271: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10
2021-03-29 00:19:23.701034: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2021-03-29 00:19:23.701468: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-03-29 00:19:23.702637: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-03-29 00:19:23.703486: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1657] Ignoring visible gpu device (device: 0, name: GeForce GTX 760, pci bus id: 0000:04:00.0, compute capability: 3.0) with Cuda compute capability 3.0. The minimum required Cuda capability is 3.5.
[]

git clone https://github.com/tensorflow/tensorflow.git
cd ./tensorflow
git checkout r2.2

sudo apt install apt-transport-https curl gnupg
curl -fsSL https://bazel.build/bazel-release.pub.gpg | gpg --dearmor > bazel.gpg
sudo mv bazel.gpg /etc/apt/trusted.gpg.d/
echo "deb [arch=amd64] https://storage.googleapis.com/bazel-apt stable jdk1.8" | sudo tee /etc/apt/sources.list.d/bazel.list
sudo apt update && sudo apt install bazel-2.0.0

Курсы

Большие данные

Методы

  • Теорема Байеса
  • Функции ошибки и регуляризация
  • Расстояние Кульбака-Лейблера и перекрестная энтропия
  • Градиентный спуск: основы
  • Граф вычислений и дифференцирование на нем
  • Перцептрон
  • Глубокие нейронные сети
  • Классификация
  • Кластеризация
  • Регрессия
  • Машинное зрение
  • Метод к-средних
  • word2vec

Библиотеки

Датасеты

Железо и драйверы

Темы

Face Recognition

Speech Recognition

Image Object Recognition

Anomaly Detection

Prediction

StereoVision

Некоторые полезные ресурсы

Источник — http://wiki.aleksashkin.net/index.php?title=Машинное_обучение_и_Большие_данные&oldid=1558