Wednesday, December 12, 2018

Fedora 29 : Using pytorch on Fedora distro.

The goal of this tutorial is about how to install and start using the pytorch python module.
Another part is to show tensors without using matplotlib python module.
The reason I wrote this simple tutorial and not on my python blogger is Fedora distro.
The python module named pytorch is based on Torch, used for applications such as natural language processing.
The installation of pytorch into many operating systems can be tricky.
Let's start this tutorial using GitHub clone commands:
[mythcat@desk ~]$  git clone --recursive https://github.com/pytorch/pytorch
...
running install_scripts
Installing convert-caffe2-to-onnx script to /home/mythcat/.local/bin
Installing convert-onnx-to-caffe2 script to /home/mythcat/.local/bin
Using this commands un Fedora linux shell will install easy this python module:
[mythcat@desk ~]$  cd pytorch/
[mythcat@desk ~]$ pip install typing
[mythcat@desk ~]$ python setup.py install --user
[mythcat@desk ~]$ pip install torchvision --user
Collecting torchvision
...
You cannot use the pytorch into pytorch folder.
[mythcat@desk pytorch]$ cd ..
[mythcat@desk ~]$ python -c "import torch; print(torch.__version__)"
1.0.0a0+bf1d411
The result of this output is not an common error. You can fix if you set the paths for pytorch installation. Let's test the pytorch installation:
[mythcat@desk ~]$ python
Python 2.7.15 (default, Oct 15 2018, 15:26:09) 
[GCC 8.2.1 20180801 (Red Hat 8.2.1-2)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch 
>>> import torchvision
>>> import torchvision.dataset as datasets
Traceback (most recent call last):
  File "", line 1, in 
ImportError: No module named dataset
>>> import torchvision.datasets as datasets
>>> print(dir(torch))
['Argument', 'ArgumentSpec', 'Block', 'BoolType', 'ByteStorage', 'ByteTensor', 'CharStorage', 'CharTensor', 'Code', 
'CompleteArgumentSpec', 'DoubleStorage', 'DoubleTensor', 'DynamicType', 'ExecutionPlanState', 'FatalError', 'FloatStorage',
 'FloatTensor', 'FloatType', 'FunctionSchema', 'Future', 'Generator', 'Gradient', 'Graph', 'GraphExecutor', 
'GraphExecutorState', 'HalfStorage', 'HalfStorageBase', 'HalfTensor', 'IODescriptor', 'IntStorage', 'IntTensor', 'IntType',
 'JITException', 'ListType',
...
>>> print(dir(datasets))
['CIFAR10', 'CIFAR100', 'CocoCaptions', 'CocoDetection', 'DatasetFolder', 'EMNIST', 'FakeData', 'FashionMNIST', 
'ImageFolder', 'LSUN', 'LSUNClass', 'MNIST', 'Omniglot', 'PhotoTour', 'SEMEION', 'STL10', 'SVHN', '__all__', 
'__builtins__', '__doc__', '__file__', '__name__', '__package__', '__path__', 'cifar', 'coco', 'fakedata', 
'folder', 'lsun', 'mnist', 'omniglot', 'phototour', 'semeion', 'stl10', 'svhn', 'utils']
>>> x = torch.rand(76)
>>> x.size()
>>> print(x)
tensor([0.9839, 0.5844, 0.4347, 0.5883, 0.1383, 0.7701, 0.1879, 0.5604, 0.4486,
        0.6782, 0.5038, 0.1078, 0.1244, 0.0996, 0.0230, 0.5457, 0.8903, 0.7732,
        0.9948, 0.3201, 0.3149, 0.7180, 0.8811, 0.4468, 0.8169, 0.2998, 0.3900,
        0.8067, 0.0090, 0.6006, 0.8385, 0.8786, 0.3652, 0.5630, 0.1407, 0.7747,
        0.5734, 0.4998, 0.4056, 0.7473, 0.2797, 0.8852, 0.3563, 0.9421, 0.1136,
        0.7676, 0.4224, 0.4350, 0.4968, 0.4457, 0.3047, 0.6792, 0.1026, 0.3593,
        0.4147, 0.6517, 0.5916, 0.3567, 0.8584, 0.9421, 0.2091, 0.6339, 0.5428,
        0.3811, 0.9310, 0.8856, 0.0770, 0.7920, 0.4860, 0.4276, 0.4780, 0.8627,
        0.7287, 0.4340, 0.2859, 0.2213])
>>> from PIL import Image
>>> logo = np.array(Image.open('logo.png').resize((512,512)))
>>> logo_tensor = torch.from_numpy(logo)
>>> logo_tensor.size()
(512, 512, 4)
>>> img = Image.fromarray(logo)
>>> img.show()

Sunday, December 9, 2018

Fedora 29 : Python 3 and Jupyter notebook.

Today I tested the Jupyter Notebook with Fedora 29.
About the Jupyter Notebook the official website comes with this intro:
The Jupyter Notebook is an open-source web application that allows you to create and share documents that contain live code, equations, visualizations and narrative text. Uses include: data cleaning and transformation, numerical simulation, statistical modeling, data visualization, machine learning, and much more.
First I check with DNF tool the update and the upgrade of the Fedora 29 distro.
The next step was to install this:
# dnf install python3-pip
# dnf install python3-devel.x86_64
# pip3 install --upgrade pip
With my account shell I used this commands to create and run the Jupiter Notebook:
$ pip3 install --user virtualenv
$ mkdir my_project
$ cd my_project/
$ virtualenv my_project_env
$ source my_project_env/bin/activate
$ pip3 install jupiter
$ jupiter notebook
The last command will start your default browser and will see this:
You can see I created a new notebook with Python 3.
The result is shown into another tab webpage browser where I used few commands to install new module scipy and I check if this working well:
!pip3 install scipy
help
quickref
The result of this notebook looks like this image:

Thursday, December 6, 2018

Fedora 29 : Shotcut video editor.

Today I tested the new software video editor for Fedora 29.
This video editor is named Shotcut and come with a new released 18.11.18 available here.
This Linux tool come for many operating systems and is a good video editor
I download and unarchive this software from the official website.
The first error comes from libnsl packages but finally, all worked well.
[mythcat@desk Shotcut]$ cd Shotcut.app/
[mythcat@desk Shotcut.app]$ ll
total 80
drwxr-xr-x.  2 mythcat mythcat    89 Nov 19 01:08 bin
-rwxr-xr-x.  1 mythcat mythcat 35147 Nov 18 23:53 COPYING
-rwxr-xr-x.  1 mythcat mythcat   779 Nov 19 01:10 ffmpeg
-rwxr-xr-x.  1 mythcat mythcat   779 Nov 19 01:10 ffplay
-rwxr-xr-x.  1 mythcat mythcat   782 Nov 19 01:10 ffprobe
drwxr-xr-x.  8 mythcat mythcat  8192 Nov 19 01:10 lib
-rwxr-xr-x.  1 mythcat mythcat   773 Nov 19 01:10 melt
-rwxr-xr-x.  1 mythcat mythcat   776 Nov 19 01:10 qmelt
drwxr-xr-x. 13 mythcat mythcat   163 Nov 19 01:10 share
-rwxr-xr-x.  1 mythcat mythcat   759 Nov 19 01:10 shotcut
-rwxr-xr-x.  1 mythcat mythcat   899 Nov 19 01:10 source-me
-rw-r--r--.  1 mythcat mythcat   802 Nov 19 01:10 versions
[mythcat@desk Shotcut.app]$ ./shotcut 
bin/shotcut: error while loading shared libraries: libnsl.so.1: cannot open shared object file: No such file or directory
...
[root@desk Shotcut.app]# dnf install libnsl
Last metadata expiration check: 0:23:04 ago on Thu 06 Dec 2018 12:31:08 PM EET.
...
Installed:
  libnsl-2.28-23.fc29.x86_64                                                    

Complete!
The result can be see into the next screenshot:

Monday, December 3, 2018

The saga of build Librelancer over Mono, NuGET and Cake.

I wrote this article because it is a good way to understand beyond the errors encountered for Mono, NuGet and Cake.
I started the day with Fedora 29 installing the old Librelancer game:
Librelancer is a cross-platform, open source game engine re-implementing the 2003 space trading and combat game Freelancer. The engine comes with an editor for several of the game's file formats called LancerEdit.
See the official webpage.
Some errors are temporarily fixed, see:
TERM=xterm.
This error refers an issue open on Feb 13,2018,12:52 PM GMT+2, see here.
However, this is a great way to go through Fedora installations to avoid searching for GitHub issues and issues.
The default install of Cake with the version 0.30.0 will not solve the last error:
[root@desk Librelancer]# nuget  install Cake -Version 0.30.0 
Installing 'Cake 0.30.0'.
Successfully installed 'Cake 0.30.0'. 
Let's hope that problems will solve with time.
Below are the correct steps for going through the installation until the last error.
[mythcat@desk ~]$ git clone --depth=50 --branch=master https://github.com/Librelancer/Librelancer.git 
Cloning into 'Librelancer'...
remote: Enumerating objects: 3085, done.
remote: Counting objects: 100% (3085/3085), done.
remote: Compressing objects: 100% (1414/1414), done.
remote: Total 3085 (delta 2131), reused 2295 (delta 1639), pack-reused 0
Receiving objects: 100% (3085/3085), 7.97 MiB | 2.79 MiB/s, done.
Resolving deltas: 100% (2131/2131), done.
Checking out files: 100% (863/863), done.
[mythcat@desk ~]$ cd Librelancer/
[mythcat@desk Librelancer]$ ll
total 48
-rw-rw-r--.  1 mythcat mythcat 4912 Dec  3 11:23 build.cake
-rw-rw-r--.  1 mythcat mythcat 7439 Dec  3 11:23 build.ps1
-rwxrwxr-x.  1 mythcat mythcat 3210 Dec  3 11:23 build.sh
-rw-rw-r--.  1 mythcat mythcat   33 Dec  3 11:23 cake.config
-rw-rw-r--.  1 mythcat mythcat 1029 Dec  3 11:23 CMakeLists.txt
-rw-rw-r--.  1 mythcat mythcat 2768 Dec  3 11:23 Credits.txt
drwxrwxr-x.  4 mythcat mythcat   87 Dec  3 11:23 deps
drwxrwxr-x.  4 mythcat mythcat 4096 Dec  3 11:23 editoricons
drwxrwxr-x. 12 mythcat mythcat  208 Dec  3 11:23 extern
-rw-rw-r--.  1 mythcat mythcat 1166 Dec  3 11:23 LICENSE
-rw-rw-r--.  1 mythcat mythcat 1877 Dec  3 11:23 README.md
drwxrwxr-x.  2 mythcat mythcat   75 Dec  3 11:23 scripts
drwxrwxr-x. 15 mythcat mythcat 4096 Dec  3 11:23 src
drwxrwxr-x.  2 mythcat mythcat   29 Dec  3 11:23 tools

[mythcat@desk Librelancer]$ git submodule update --init --recursive
Submodule 'extern/BulletSharpPInvoke' (https://github.com/AndresTraks/BulletSharpPInvoke) registered for path 
'extern/BulletSharpPInvoke'
Submodule 'extern/Collada141' (https://github.com/Librelancer/Collada141) registered for path 
'extern/Collada141'
Submodule 'extern/FontConfigSharp' (https://github.com/CallumDev/FontConfigSharp.git) registered for path 
'extern/FontConfigSharp'
Submodule 'extern/ImGui.NET' (https://github.com/mellinoe/ImGui.NET) registered for path 'extern/ImGui.NET'
Submodule 'extern/SharpFont' (https://github.com/Robmaister/SharpFont.git) registered for path 'extern/SharpFont'
Submodule 'extern/StbSharp' (https://github.com/rds1983/StbSharp) registered for path 'extern/StbSharp'
Submodule 'extern/bullet3' (https://github.com/bulletphysics/bullet3) registered for path 'extern/bullet3'
Submodule 'extern/cimgui' (https://github.com/Extrawurst/cimgui) registered for path 'extern/cimgui'
Submodule 'extern/lidgren-network-gen3' (https://github.com/lidgren/lidgren-network-gen3) registered for path 
'extern/lidgren-network-gen3'
Submodule 'extern/nvidia-texture-tools' (https://github.com/castano/nvidia-texture-tools) registered for path 
'extern/nvidia-texture-tools'
Cloning into '/home/mythcat/Librelancer/extern/BulletSharpPInvoke'...
Cloning into '/home/mythcat/Librelancer/extern/Collada141'...
Cloning into '/home/mythcat/Librelancer/extern/FontConfigSharp'...
Cloning into '/home/mythcat/Librelancer/extern/ImGui.NET'...
Cloning into '/home/mythcat/Librelancer/extern/SharpFont'...
...
[mythcat@desk Librelancer]$ export GITHUB_TOKEN=[secure]
Check your mono version 
[mythcat@desk Librelancer]$ mono --version
Mono JIT compiler version 4.8.0 (Stable 4.8.0.520/8f6d0f6 Wed Sep 20 21:27:10 UTC 2017)
Copyright (C) 2002-2014 Novell, Inc, Xamarin Inc and Contributors. www.mono-project.com
    TLS:           __thread
    SIGSEGV:       normal
    Notifications: epoll
    Architecture:  amd64
    Disabled:      none
    Misc:          softdebug 
    LLVM:          supported, not enabled.
    GC:            sgen

[root@desk Librelancer]# rpm --import https://packages.microsoft.com/keys/microsoft.asc
[root@desk Librelancer]# wget -q https://packages.microsoft.com/config/fedora/27/prod.repo
[root@desk Librelancer]# ls
build.cake  cake.config     deps         LICENSE    scripts
build.ps1   CMakeLists.txt  editoricons  prod.repo  src
build.sh    Credits.txt     extern       README.md  tools
[root@desk Librelancer]# vim prod.repo 
[root@desk Librelancer]# mv prod.repo /etc/yum.repos.d/microsoft-prod.repo
[root@desk Librelancer]# chown root:root /etc/yum.repos.d/microsoft-prod.repo
[root@desk Librelancer]# dnf update
packages-microsoft-com-prod                      48 kB/s | 156 kB     00:03    
Last metadata expiration check: 0:00:01 ago on Mon 03 Dec 2018 11:52:11 AM EET.
Dependencies resolved.
Nothing to do.
Complete!

[root@desk Librelancer]# dnf search dotnet-sdk
Last metadata expiration check: 0:01:05 ago on Mon 03 Dec 2018 11:52:11 AM EET.
=========================== Name Matched: dotnet-sdk ===========================
dotnet-sdk-2.1.x86_64 : Microsoft .NET Core SDK 2.1.500 2.1.500
dotnet-sdk-2.1.200.x86_64 : Microsoft .NET Core SDK - 2.1.200 2.1.200
dotnet-sdk-2.1.201.x86_64 : Microsoft .NET Core SDK - 2.1.201 2.1.201
dotnet-sdk-2.1.202.x86_64 : Microsoft .NET Core SDK - 2.1.202 2.1.202
dotnet-sdk-2.1.300-rc1-008673.x86_64 : Microsoft .NET Core SDK 2.1.300 - rc1
                                     : 2.1.300-rc1-008673

[root@desk Librelancer]# dnf install dotnet-sdk-2.1.202.x86_64 
Last metadata expiration check: 0:02:18 ago on Mon 03 Dec 2018 11:52:11 AM EET.
Dependencies resolved.
================================================================================
 Package                 Arch    Version     Repository                    Size
================================================================================
Installing:
 dotnet-sdk-2.1.202      x86_64  2.1.202-1   packages-microsoft-com-prod   96 M
Installing dependencies:
 aspnetcore-store-2.0.0  x86_64  2.0.0-1     packages-microsoft-com-prod   24 M
 aspnetcore-store-2.0.3  x86_64  2.0.3-1     packages-microsoft-com-prod  7.9 M
 aspnetcore-store-2.0.5  x86_64  2.0.5-1     packages-microsoft-com-prod  1.6 M
 aspnetcore-store-2.0.6  x86_64  2.0.6-1     packages-microsoft-com-prod  9.3 M
 aspnetcore-store-2.0.7  x86_64  2.0.7-1     packages-microsoft-com-prod   24 k
 aspnetcore-store-2.0.8  x86_64  2.0.8-1     packages-microsoft-com-prod  8.5 M
 aspnetcore-store-2.0.9  x86_64  2.0.9-1     packages-microsoft-com-prod  956 k
 dotnet-host             x86_64  2.1.6-1     packages-microsoft-com-prod   45 k
 dotnet-hostfxr-2.0.9    x86_64  2.0.9-1     packages-microsoft-com-prod  182 k
 dotnet-runtime-2.0.9    x86_64  2.0.9-1     packages-microsoft-com-prod   24 M

Transaction Summary
================================================================================
Install  11 Packages

Total download size: 173 M
Installed size: 173 M
Is this ok [y/N]: y
...
Complete!

This will fix a bug : 
Unhandled Exception:
System.TypeInitializationException: The type initializer for 'System.Console' threw an exception.
[mythcat@desk Librelancer]$ TERM=xterm


[mythcat@desk Librelancer]$ ./build.sh 
Could not load file or assembly 'Microsoft.Build.Utilities.v4.0, Version=4.0.0.0, Culture=neutral, 
PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies.
Could not load file or assembly 'Microsoft.Build.Utilities.v4.0, Version=4.0.0.0, Culture=neutral, 
PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies.

[root@desk Librelancer]# dnf install nuget.x86_64 

The dotnet come with this:

[root@desk Librelancer]# dotnet --info
.NET Core SDK (reflecting any global.json):
 Version:   2.1.500
 Commit:    b68b931422

Runtime Environment:
 OS Name:     fedora
 OS Version:  29
 OS Platform: Linux
 RID:         fedora.29-x64
 Base Path:   /usr/share/dotnet/sdk/2.1.500/

Host (useful for support):
  Version: 2.1.6
  Commit:  3f4f8eebd8

.NET Core SDKs installed:
  2.1.202 [/usr/share/dotnet/sdk]
  2.1.500 [/usr/share/dotnet/sdk]

.NET Core runtimes installed:
  Microsoft.AspNetCore.All 2.1.6 [/usr/share/dotnet/shared/Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.App 2.1.6 [/usr/share/dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 2.0.9 [/usr/share/dotnet/shared/Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.1.6 [/usr/share/dotnet/shared/Microsoft.NETCore.App]

To install additional .NET Core runtimes or SDKs:
  https://aka.ms/dotnet-download

Let's test again:
[mythcat@desk Librelancer]$ ./build.sh 
Feeds used:
  https://api.nuget.org/v3/index.json

Restoring NuGet package Cake.0.30.0.
WARNING: Unable to find version '0.30.0' of package 'Cake'.
  https://api.nuget.org/v3/index.json: Unable to load the service index for source https://api.nuget.org/v3/index.json.
  An error occurred while sending the request
  Error: TrustFailure (Ssl error:1000007d:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED)
  Ssl error:1000007d:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED

Unable to find version '0.30.0' of package 'Cake'.
  https://api.nuget.org/v3/index.json: Unable to load the service index for source https://api.nuget.org/v3/index.json.
  An error occurred while sending the request
  Error: TrustFailure (Ssl error:1000007d:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED)
  Ssl error:1000007d:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED

Could not restore NuGet tools.

[root@desk Librelancer]# cat /etc/ssl/certs/* >ca-bundle.crt
[root@desk Librelancer]# TERM=xterm
[root@desk Librelancer]# cert-sync ca-bundle.crt
Mono Certificate Store Sync - version 4.8.0.0
Populate Mono certificate store from a concatenated list of certificates.
Copyright 2002, 2003 Motus Technologies. Copyright 2004-2008 Novell. BSD licensed.

Importing into legacy system store:
I already trust 0, your new list has 129
Certificate added: CN=ACCVRAIZ1, OU=PKIACCV, O=ACCV, C=ES
Certificate added: C=ES, O=FNMT-RCM, OU=AC RAIZ FNMT-RCM
Certificate added: C=IT, L=Milan, O=Actalis S.p.A./03358520967, CN=Actalis Authentication Root CA
Certificate added: C=SE, O=AddTrust AB, OU=AddTrust External TTP Network, CN=AddTrust External CA Root
...
129 new root certificates were added to your trust store.
Import process completed.
[root@desk Librelancer]# rm ca-bundle.crt
rm: remove regular file 'ca-bundle.crt'? y

[mythcat@desk Librelancer]$ ./build.sh 
Feeds used:
  https://api.nuget.org/v3/index.json

Restoring NuGet package Cake.0.30.0.
  GET https://api.nuget.org/v3-flatcontainer/cake/0.30.0/cake.0.30.0.nupkg
  OK https://api.nuget.org/v3-flatcontainer/cake/0.30.0/cake.0.30.0.nupkg 53ms
Installing Cake 0.30.0.
Adding package 'Cake.0.30.0' to folder '/home/mythcat/Librelancer/tools'
Added package 'Cake.0.30.0' to folder '/home/mythcat/Librelancer/tools'
Install failed. Rolling back...
Error: One or more errors occurred.
    Could not load type 'NuGet.Packaging.PackageArchiveReader' from assembly 'NuGet.Packaging, 
Version=4.7.0.5, Culture=neutral, PublicKeyToken=31bf3856ad364e35'.