Машинное обучение и Большие данные

Hardware

• Lenovo x230 + eGPU
  • Expresscard V8.0 EXP GDC Beast PCIe PCI-E
  • Блок питания на 350-600 ватт
  • Nvidia GeForce 760 4gb
  • Similar setup
  • Troubleshooting
  • 16 GB of mem will produce lags. Remove 1 stick of mem to 8 GB
  • Be sure that you GPU conneted to power fully(8+6 or 8+8) - it can produce 43 error
  • eGPU setup Lenovo Thinkpad x230 with GTX 760 Part 1 ( setup )
  • eGPU setup Lenovo Thinkpad x230 with GTX 760 Part 2 ( Fixing Error 12)
  • eGPU setup Lenovo Thinkpad x230 with GTX 760 Part 3 ( Gameplay )
  • In Windows go to Control panel, Hardware setup, Nvidia settings, 3d graphics, There you can select default video adapter
    • But it won't help to use GPU in games - you need to connect external screen to GPU and disable laptop screen. Then games will run on eGPU.

Software

• NVIDIA CUDA Installation Guide for Linux

wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/cuda-ubuntu2004.pin
sudo mv cuda-ubuntu2004.pin /etc/apt/preferences.d/cuda-repository-pin-600
wget https://developer.download.nvidia.com/compute/cuda/11.2.2/local_installers/cuda-repo-ubuntu2004-11-2-local_11.2.2-460.32.03-1_amd64.deb
sudo dpkg -i cuda-repo-ubuntu2004-11-2-local_11.2.2-460.32.03-1_amd64.deb
sudo apt-key add /var/cuda-repo-ubuntu2004-11-2-local/7fa2af80.pub
sudo apt-get update
sudo apt-get -y install cuda nvidia-cuda-toolkit

• TensorFlow for GPU
• cuDNN SDK
• TensorRT
• Installing TensorFlow GPU in Ubuntu 20.04
• https://developer.nvidia.com/cuda-gpus

Change Default Python

sudo update-alternatives --install /usr/bin/python python /usr/bin/python2 1
sudo update-alternatives --install /usr/bin/python python /usr/bin/python3 2
sudo update-alternatives --config python

Define Your Software Versions

• Nvidia GeForce GTX 760 4gb -> Nvidia Kepler
• Nvidia Kepler -> CUDA SDK 10.0 – 10.2 support for compute capability 3.0 – 7.5 (Kepler, Maxwell, Pascal, Volta, Turing). Last version with support for compute capability 3.x (Kepler). 10.2 is the last official release for macOS, as support will not be available for macOS in newer releases.
• Check all possible TensorFlow and Cuda versions here: https://www.tensorflow.org/install/source#gpu
• For me - tenorflow-2.3.0, cuda 10.2, nvidia-440.33.0, cuDNN 7.6, Bazel 3.1.0, GCC 7.3.1, TensorRT 6.0
  • PS cuda 10.1 won't install due 418 driver is not comportable with new kernel 5.4.0-70-generic

Installation

• https://developer.nvidia.com/cuda-10.2-download-archive?target_os=Linux&target_arch=x86_64&target_distro=Ubuntu&target_version=1804&target_type=deblocal

wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/cuda-ubuntu1804.pin
sudo mv cuda-ubuntu1804.pin /etc/apt/preferences.d/cuda-repository-pin-600
wget https://developer.download.nvidia.com/compute/cuda/10.2/Prod/local_installers/cuda-repo-ubuntu1804-10-2-local-10.2.89-440.33.01_1.0-1_amd64.deb
sudo dpkg -i cuda-repo-ubuntu1804-10-2-local-10.2.89-440.33.01_1.0-1_amd64.deb
sudo apt-key add /var/cuda-repo-10-2-local-10.2.89-440.33.01/7fa2af80.pub
sudo apt-get update
sudo apt-get -y install cuda

• https://developer.nvidia.com/rdp/cudnn-archive -> Download cuDNN v7.6.5 (November 18th, 2019), for CUDA 10.2
  • https://developer.nvidia.com/compute/machine-learning/cudnn/secure/7.6.5.32/Production/10.2_20191118/Ubuntu18_04-x64/libcudnn7_7.6.5.32-1%2Bcuda10.2_amd64.deb
  • https://developer.nvidia.com/compute/machine-learning/cudnn/secure/7.6.5.32/Production/10.2_20191118/Ubuntu18_04-x64/libcudnn7-dev_7.6.5.32-1%2Bcuda10.2_amd64.deb
  • https://developer.nvidia.com/compute/machine-learning/cudnn/secure/7.6.5.32/Production/10.2_20191118/Ubuntu18_04-x64/libcudnn7-doc_7.6.5.32-1%2Bcuda10.2_amd64.deb

sudo dpkg -i libcudnn7_7.6.5.32-1+cuda10.2_amd64.deb
sudo dpkg -i libcudnn7-dev_7.6.5.32-1+cuda10.2_amd64.deb
sudo dpkg -i libcudnn7-doc_7.6.5.32-1+cuda10.2_amd64.deb

• https://developer.nvidia.com/nvidia-tensorrt-6x-download

sudo dpkg -i nv-tensorrt-repo-ubuntu1804-cuda10.2-trt6.0.1.8-ga-20191108_1-1_amd64.deb
sudo apt-get update
sudo apt-key add /var/nv-tensorrt-repo-cuda10.2-trt6.0.1.8-ga-20191108/7fa2af80.pub
sudo apt-get install tensorrt

• https://docs.bazel.build/versions/master/install-ubuntu.html

sudo apt install apt-transport-https curl gnupg
curl -fsSL https://bazel.build/bazel-release.pub.gpg | gpg --dearmor > bazel.gpg
sudo mv bazel.gpg /etc/apt/trusted.gpg.d/
echo "deb [arch=amd64] https://storage.googleapis.com/bazel-apt stable jdk1.8" | sudo tee /etc/apt/sources.list.d/bazel.list
sudo apt update
sudo apt install bazel-3.2.0
sudo ln -s /usr/bin/bazel-3.2.0 /usr/bin/bazel
bazel --version  # 3.2.0

• https://www.tensorflow.org/install/source

git clone https://github.com/tensorflow/tensorflow.git
cd tensorflow
git checkout r2.3
./configure
bazel build --config=cuda --config=opt //tensorflow/tools/pip_package:build_pip_package

Testing

>>> import tensorflow as tf
>>> tf.__version__
'2.3.0'
>>> tf.test.is_built_with_cuda()
True

$ nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2019 NVIDIA Corporation
Built on Sun_Jul_28_19:07:16_PDT_2019
Cuda compilation tools, release 10.1, V10.1.243

Python 3.8.5 (default, Jan 27 2021, 15:41:15) 
[GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import tensorflow as tf
>>> tf.config.list_physical_devices("GPU")
2021-03-29 00:19:23.520023: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1
2021-03-29 00:19:23.575281: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-03-29 00:19:23.575801: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: 
pciBusID: 0000:04:00.0 name: GeForce GTX 760 computeCapability: 3.0
coreClock: 1.15GHz coreCount: 6 deviceMemorySize: 3.94GiB deviceMemoryBandwidth: 179.05GiB/s
2021-03-29 00:19:23.576789: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1
2021-03-29 00:19:23.581937: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10
2021-03-29 00:19:23.583502: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10
2021-03-29 00:19:23.585776: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10
2021-03-29 00:19:23.591336: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10
2021-03-29 00:19:23.593271: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10
2021-03-29 00:19:23.701034: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2021-03-29 00:19:23.701468: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-03-29 00:19:23.702637: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-03-29 00:19:23.703486: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1657] Ignoring visible gpu device (device: 0, name: GeForce GTX 760, pci bus id: 0000:04:00.0, compute capability: 3.0) with Cuda compute capability 3.0. The minimum required Cuda capability is 3.5.
[]

• https://medium.com/@mccann.matt/compiling-tensorflow-with-cuda-3-0-support-42d8fe0bf3b5

git clone https://github.com/tensorflow/tensorflow.git
cd ./tensorflow
git checkout r2.2

sudo apt install apt-transport-https curl gnupg
curl -fsSL https://bazel.build/bazel-release.pub.gpg | gpg --dearmor > bazel.gpg
sudo mv bazel.gpg /etc/apt/trusted.gpg.d/
echo "deb [arch=amd64] https://storage.googleapis.com/bazel-apt stable jdk1.8" | sudo tee /etc/apt/sources.list.d/bazel.list
sudo apt update && sudo apt install bazel-2.0.0

Курсы

• https://medium.com/nuances-of-programming/%D1%82%D0%BE%D0%BF-10-%D0%BA%D1%83%D1%80%D1%81%D0%BE%D0%B2-%D0%BF%D0%BE-%D0%BC%D0%B0%D1%88%D0%B8%D0%BD%D0%BD%D0%BE%D0%BC%D1%83-%D0%B8-%D0%B3%D0%BB%D1%83%D0%B1%D0%BE%D0%BA%D0%BE%D0%BC%D1%83-%D0%BE%D0%B1%D1%83%D1%87%D0%B5%D0%BD%D0%B8%D1%8E-%D0%B2-2020-1e1d870a24b7
• https://skillbox.ru/course/profession-data-scientist/
• https://new.geekbrains.ru/machine-learning
• https://praktikum.yandex.ru/data-scientist/
• https://www.coursera.org/learn/machine-learning
• https://www.coursera.org/specializations/deep-learning
• https://ru.coursera.org/specializations/machine-learning-data-analysis
• https://www.udacity.com/course/machine-learning--ud262
• https://www.udacity.com/course/intro-to-machine-learning--ud120

Большие данные

• Лучшие в своём деле: Артур Хачуян | Большие данные — Big Data | ЛСД #7 https://www.youtube.com/watch?v=frLydE1UCvA
• «На чём корпорации вертели вашу приватность», Артур Хачуян (Tazeros Global) https://www.youtube.com/watch?v=8IJxO44kq24

Методы

• Теорема Байеса
• Функции ошибки и регуляризация
• Расстояние Кульбака-Лейблера и перекрестная энтропия
• Градиентный спуск: основы
• Граф вычислений и дифференцирование на нем
• Перцептрон
• Глубокие нейронные сети
• Классификация
• Кластеризация
• Регрессия
• Машинное зрение
• Метод к-средних
• word2vec

Библиотеки

• NumPy - https://www.numpy.org/
• SciPy - https://www.scipy.org/
• Pandas - https://pandas.pydata.org/
• Scikit-learn - https://scikit-learn.org/stable/
• Matplotlib - https://matplotlib.org/
• Gensim - https://radimrehurek.com/gensim/
• H2O - https://github.com/h2oai/h2o-tutorials
• XGBoost - https://xgboost.readthedocs.io/en/latest/
• CatBoost - https://yandex.ru/dev/catboost/
• Theano - http://deeplearning.net/software/theano/
• TensorFlow - https://www.tensorflow.org/
• sknn - https://scikit-neuralnetwork.readthedocs.io/en/latest/
• Theanets - https://github.com/lmjohns3/theanets
• Keras - https://keras.io/
• OpenCV - https://opencv.org/

Датасеты

• http://yann.lecun.com/exdb/mnist/
• http://www.image-net.org/
• https://en.wikipedia.org/wiki/List_of_datasets_for_machine_learning_research
• https://storage.googleapis.com/openimages/web/index.html
• https://github.com/philipperemy/timit
• https://www.nist.gov/programs-projects/face-recognition-grand-challenge-frgc

Железо и драйверы

• https://developer.nvidia.com/cuda-downloads?target_os=Linux&target_arch=x86_64&target_distro=Ubuntu&target_version=1604&target_type=deblocal
• https://www.tensorflow.org/install/gpu
• https://cloud.google.com/tpu/

Темы

• https://en.wikipedia.org/wiki/Outline_of_artificial_intelligence#Branches_of_artificial_intelligence

Face Recognition

• https://github.com/ageitgey/face_recognition
• https://cmusatyalab.github.io/openface/

Speech Recognition

• https://pypi.org/project/SpeechRecognition/

Image Object Recognition

• https://imageai.readthedocs.io/en/latest/
  • https://stackabuse.com/object-detection-with-imageai-in-python/
  • https://towardsdatascience.com/object-detection-with-10-lines-of-code-d6cb4d86f606

StereoVision

• https://medium.com/stereopi/opencv-and-depth-map-on-stereopi-tutorial-62cb6792bbed

Некоторые полезные ресурсы

• https://ods.ai/
• https://en.wikipedia.org/wiki/Outline_of_artificial_intelligence
• https://en.wikipedia.org/wiki/List_of_artificial_intelligence_projects
• https://en.wikipedia.org/wiki/List_of_datasets_for_machine_learning_research
• http://wordnet.princeton.edu/
• http://project.phil.spbu.ru/RussNet/index_ru.shtml
• http://wordnet.ru/
• http://www.vision.caltech.edu/Image_Datasets/Caltech101/
• https://www.scipy.org/
• http://www.numpy.org/
• http://leenissen.dk/fann/html/files/fann_train-h.html
• https://archive.ics.uci.edu/ml/datasets/mushroom
• https://archive.ics.uci.edu/ml/index.php
• https://www.kaggle.com/
• https://keras.io/
• https://xgboost.readthedocs.io/en/latest/
• https://lasagne.readthedocs.io/en/latest/
• http://deeplearning.net/software/theano/index.html
• https://catboost.yandex/
• https://arxiv.org/
• https://arxiv.org/list/cs.CV/recent
• https://yandex.ru/dev/catboost/