Машинное обучение и Большие данные

Hardware

• Lenovo x230 + eGPU
  • Expresscard V8.0 EXP GDC Beast PCIe PCI-E
  • Блок питания на 350-600 ватт
  • Nvidia GeForce 760 4gb
  • Similar setup
  • Troubleshooting
  • 16 GB of mem will produce lags. Remove 1 stick of mem to 8 GB
  • Be sure that you GPU conneted to power fully(8+6 or 8+8) - it can produce 43 error
  • eGPU setup Lenovo Thinkpad x230 with GTX 760 Part 1 ( setup )
  • eGPU setup Lenovo Thinkpad x230 with GTX 760 Part 2 ( Fixing Error 12)
  • eGPU setup Lenovo Thinkpad x230 with GTX 760 Part 3 ( Gameplay )
  • In Windows go to Control panel, Hardware setup, Nvidia settings, 3d graphics, There you can select default video adapter
    • But it won't help to use GPU in games - you need to connect external screen to GPU and disable laptop screen. Then games will run on eGPU.

Software

• NVIDIA CUDA Installation Guide for Linux

wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/cuda-ubuntu2004.pin
sudo mv cuda-ubuntu2004.pin /etc/apt/preferences.d/cuda-repository-pin-600
wget https://developer.download.nvidia.com/compute/cuda/11.2.2/local_installers/cuda-repo-ubuntu2004-11-2-local_11.2.2-460.32.03-1_amd64.deb
sudo dpkg -i cuda-repo-ubuntu2004-11-2-local_11.2.2-460.32.03-1_amd64.deb
sudo apt-key add /var/cuda-repo-ubuntu2004-11-2-local/7fa2af80.pub
sudo apt-get update
sudo apt-get -y install cuda nvidia-cuda-toolkit

• TensorFlow for GPU
• cuDNN SDK
• TensorRT
• Installing TensorFlow GPU in Ubuntu 20.04
• https://developer.nvidia.com/cuda-gpus

Change Default Python

sudo update-alternatives --install /usr/bin/python python /usr/bin/python2 1
sudo update-alternatives --install /usr/bin/python python /usr/bin/python3 2
sudo update-alternatives --config python

Define Your Cuda Version

• Nvidia GeForce GTX 760 4gb -> Nvidia Kepler
• Nvidia Kepler -> CUDA SDK 10.0 – 10.2 support for compute capability 3.0 – 7.5 (Kepler, Maxwell, Pascal, Volta, Turing). Last version with support for compute capability 3.x (Kepler). 10.2 is the last official release for macOS, as support will not be available for macOS in newer releases.

Define Your Tensorflow Version

• Check all possible TensorFlow and Cuda versions here: https://www.tensorflow.org/install/source#gpu
• For me - tenorflow-2.3.0, cuda 10.2, nvidia-440.33.0, cuDNN 7.6, Bazel 3.1.0, GCC 7.3.1
  • PS cuda 10.1 won't install due 418 driver is not comportable with new kernel 5.4.0-70-generic

wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/cuda-ubuntu1804.pin
sudo mv cuda-ubuntu1804.pin /etc/apt/preferences.d/cuda-repository-pin-600
wget https://developer.download.nvidia.com/compute/cuda/10.2/Prod/local_installers/cuda-repo-ubuntu1804-10-2-local-10.2.89-440.33.01_1.0-1_amd64.deb
sudo dpkg -i cuda-repo-ubuntu1804-10-2-local-10.2.89-440.33.01_1.0-1_amd64.deb
sudo apt-key add /var/cuda-repo-10-2-local-10.2.89-440.33.01/7fa2af80.pub
sudo apt-get update
sudo apt-get -y install cuda

• https://developer.nvidia.com/rdp/cudnn-archive -> Download cuDNN v7.6.5 (November 18th, 2019), for CUDA 10.2
  • https://developer.nvidia.com/compute/machine-learning/cudnn/secure/7.6.5.32/Production/10.2_20191118/Ubuntu18_04-x64/libcudnn7_7.6.5.32-1%2Bcuda10.2_amd64.deb
  • https://developer.nvidia.com/compute/machine-learning/cudnn/secure/7.6.5.32/Production/10.2_20191118/Ubuntu18_04-x64/libcudnn7-dev_7.6.5.32-1%2Bcuda10.2_amd64.deb
  • https://developer.nvidia.com/compute/machine-learning/cudnn/secure/7.6.5.32/Production/10.2_20191118/Ubuntu18_04-x64/libcudnn7-doc_7.6.5.32-1%2Bcuda10.2_amd64.deb

>>> import tensorflow as tf
>>> tf.__version__
'2.3.0'
>>> tf.test.is_built_with_cuda()
True

$ nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2019 NVIDIA Corporation
Built on Sun_Jul_28_19:07:16_PDT_2019
Cuda compilation tools, release 10.1, V10.1.243

Python 3.8.5 (default, Jan 27 2021, 15:41:15) 
[GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import tensorflow as tf
>>> tf.config.list_physical_devices("GPU")
2021-03-29 00:19:23.520023: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1
2021-03-29 00:19:23.575281: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-03-29 00:19:23.575801: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties: 
pciBusID: 0000:04:00.0 name: GeForce GTX 760 computeCapability: 3.0
coreClock: 1.15GHz coreCount: 6 deviceMemorySize: 3.94GiB deviceMemoryBandwidth: 179.05GiB/s
2021-03-29 00:19:23.576789: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1
2021-03-29 00:19:23.581937: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10
2021-03-29 00:19:23.583502: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10
2021-03-29 00:19:23.585776: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10
2021-03-29 00:19:23.591336: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10
2021-03-29 00:19:23.593271: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10
2021-03-29 00:19:23.701034: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2021-03-29 00:19:23.701468: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-03-29 00:19:23.702637: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-03-29 00:19:23.703486: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1657] Ignoring visible gpu device (device: 0, name: GeForce GTX 760, pci bus id: 0000:04:00.0, compute capability: 3.0) with Cuda compute capability 3.0. The minimum required Cuda capability is 3.5.
[]

• https://medium.com/@mccann.matt/compiling-tensorflow-with-cuda-3-0-support-42d8fe0bf3b5

git clone https://github.com/tensorflow/tensorflow.git
cd ./tensorflow
git checkout r2.2

sudo apt install apt-transport-https curl gnupg
curl -fsSL https://bazel.build/bazel-release.pub.gpg | gpg --dearmor > bazel.gpg
sudo mv bazel.gpg /etc/apt/trusted.gpg.d/
echo "deb [arch=amd64] https://storage.googleapis.com/bazel-apt stable jdk1.8" | sudo tee /etc/apt/sources.list.d/bazel.list
sudo apt update && sudo apt install bazel-2.0.0

Курсы

• https://medium.com/nuances-of-programming/%D1%82%D0%BE%D0%BF-10-%D0%BA%D1%83%D1%80%D1%81%D0%BE%D0%B2-%D0%BF%D0%BE-%D0%BC%D0%B0%D1%88%D0%B8%D0%BD%D0%BD%D0%BE%D0%BC%D1%83-%D0%B8-%D0%B3%D0%BB%D1%83%D0%B1%D0%BE%D0%BA%D0%BE%D0%BC%D1%83-%D0%BE%D0%B1%D1%83%D1%87%D0%B5%D0%BD%D0%B8%D1%8E-%D0%B2-2020-1e1d870a24b7
• https://skillbox.ru/course/profession-data-scientist/
• https://new.geekbrains.ru/machine-learning
• https://praktikum.yandex.ru/data-scientist/
• https://www.coursera.org/learn/machine-learning
• https://www.coursera.org/specializations/deep-learning
• https://ru.coursera.org/specializations/machine-learning-data-analysis
• https://www.udacity.com/course/machine-learning--ud262
• https://www.udacity.com/course/intro-to-machine-learning--ud120

Большие данные

• Лучшие в своём деле: Артур Хачуян | Большие данные — Big Data | ЛСД #7 https://www.youtube.com/watch?v=frLydE1UCvA
• «На чём корпорации вертели вашу приватность», Артур Хачуян (Tazeros Global) https://www.youtube.com/watch?v=8IJxO44kq24

Методы

• Теорема Байеса
• Функции ошибки и регуляризация
• Расстояние Кульбака-Лейблера и перекрестная энтропия
• Градиентный спуск: основы
• Граф вычислений и дифференцирование на нем
• Перцептрон
• Глубокие нейронные сети
• Классификация
• Кластеризация
• Регрессия
• Машинное зрение
• Метод к-средних
• word2vec

Библиотеки

• NumPy - https://www.numpy.org/
• SciPy - https://www.scipy.org/
• Pandas - https://pandas.pydata.org/
• Scikit-learn - https://scikit-learn.org/stable/
• Matplotlib - https://matplotlib.org/
• Gensim - https://radimrehurek.com/gensim/
• H2O - https://github.com/h2oai/h2o-tutorials
• XGBoost - https://xgboost.readthedocs.io/en/latest/
• CatBoost - https://yandex.ru/dev/catboost/
• Theano - http://deeplearning.net/software/theano/
• TensorFlow - https://www.tensorflow.org/
• sknn - https://scikit-neuralnetwork.readthedocs.io/en/latest/
• Theanets - https://github.com/lmjohns3/theanets
• Keras - https://keras.io/
• OpenCV - https://opencv.org/

Датасеты

• http://yann.lecun.com/exdb/mnist/
• http://www.image-net.org/
• https://en.wikipedia.org/wiki/List_of_datasets_for_machine_learning_research
• https://storage.googleapis.com/openimages/web/index.html
• https://github.com/philipperemy/timit
• https://www.nist.gov/programs-projects/face-recognition-grand-challenge-frgc

Железо и драйверы

• https://developer.nvidia.com/cuda-downloads?target_os=Linux&target_arch=x86_64&target_distro=Ubuntu&target_version=1604&target_type=deblocal
• https://www.tensorflow.org/install/gpu
• https://cloud.google.com/tpu/

Темы

• https://en.wikipedia.org/wiki/Outline_of_artificial_intelligence#Branches_of_artificial_intelligence

Face Recognition

• https://github.com/ageitgey/face_recognition
• https://cmusatyalab.github.io/openface/

Speech Recognition

• https://pypi.org/project/SpeechRecognition/

Image Object Recognition

• https://imageai.readthedocs.io/en/latest/
  • https://stackabuse.com/object-detection-with-imageai-in-python/
  • https://towardsdatascience.com/object-detection-with-10-lines-of-code-d6cb4d86f606

StereoVision

• https://medium.com/stereopi/opencv-and-depth-map-on-stereopi-tutorial-62cb6792bbed

Некоторые полезные ресурсы

• https://ods.ai/
• https://en.wikipedia.org/wiki/Outline_of_artificial_intelligence
• https://en.wikipedia.org/wiki/List_of_artificial_intelligence_projects
• https://en.wikipedia.org/wiki/List_of_datasets_for_machine_learning_research
• http://wordnet.princeton.edu/
• http://project.phil.spbu.ru/RussNet/index_ru.shtml
• http://wordnet.ru/
• http://www.vision.caltech.edu/Image_Datasets/Caltech101/
• https://www.scipy.org/
• http://www.numpy.org/
• http://leenissen.dk/fann/html/files/fann_train-h.html
• https://archive.ics.uci.edu/ml/datasets/mushroom
• https://archive.ics.uci.edu/ml/index.php
• https://www.kaggle.com/
• https://keras.io/
• https://xgboost.readthedocs.io/en/latest/
• https://lasagne.readthedocs.io/en/latest/
• http://deeplearning.net/software/theano/index.html
• https://catboost.yandex/
• https://arxiv.org/
• https://arxiv.org/list/cs.CV/recent
• https://yandex.ru/dev/catboost/