2020년 3월 23일 월요일

Orange3, Orange3-Associate 설치 (ppc64le, AC922 환경)

지난번 포스트에 이어, 이번에는 PyPI에 올라와 있는 Orange3-Associate, Orange3을 소스에서 빌드하여 설치하는 방법을 기록해 둡니다.

Orange3-associate 관련 페이지는 아래와 같습니다.
https://pypi.org/project/Orange3-Associate/
https://orange3-associate.readthedocs.io/

우선 Pypi 페이지에서 다운로드 받을 수 있는 whl 파일을 찾아보았는데, 소스코드만 공개되어 있습니다. ppc64le 환경에서는 소스를 빌드하여 직접 설치해서 사용해야 합니다.

[root@p607-kvm1 opt]# gzip -d Orange3-Associate-1.1.5.tar.gz
[root@p607-kvm1 opt]# tar -xvf Orange3-Associate-1.1.5.tar
[root@p607-kvm1 opt]# ls -al
total 3093748
drwxr-xr-x.  4 root root        119 Mar 10 00:41 .
dr-xr-xr-x. 20 root root       4096 Mar  8 22:33 ..
drwxr-xr-x  20 root root        308 Feb 27 03:39 anaconda3
-rw-rw-r--   1 root root 3167160320 Mar  9 02:49 anaconda3_20200227.tar
drwxr-xr-x   6 1000 1000        172 Nov 15  2017 Orange3-Associate-1.1.5
-rw-r--r--   1 root root     829440 Feb 26 12:54 Orange3-Associate-1.1.5.tar
[root@p607-kvm1 opt]# cd Orange3-Associate-1.1.5/

지난번과 동일하게 서버에 gcc 환경을 구성합니다. 제가 사용할 Python 3.6 버전은 anaconda3에 포함되어 있는데, gcc 7.3.x 버전을 이용하여 빌드된 것입니다. 따라서, 이후 Orange3-associate를 빌드할때에도 gcc 7.3.x 버전이 필요합니다. 

[root@p607-kvm1 Orange3-Associate-1.1.5]# yum install devtoolset-7-gcc-c++.ppc64le
[root@p607-kvm1 Orange3-Associate-1.1.5]# export PATH=/opt/rh/devtoolset-7/root/usr/bin:$PATH

이제 Orange3-Associate 를 빌드합니다.
[root@p607-kvm1 Orange3-Associate-1.1.5]# /opt/anaconda3/bin/python3 setup.py build
running build
running build_py
running build_ext
building 'orangecontrib.associate._fpgrowth' extension
gcc -pthread -B /opt/anaconda3/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstri                                        ct-prototypes -fPIC -I/opt/anaconda3/include/python3.6m -c orangecontrib/associate/_fpgrowth.cpp -o build/temp.lin                                        ux-ppc64le-3.6/orangecontrib/associate/_fpgrowth.o -std=c++11 -O3
cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
g++ -pthread -shared -B /opt/anaconda3/compiler_compat -L/opt/anaconda3/lib -Wl,-rpath=/opt/anaconda3/lib -Wl,--no                                        -as-needed -Wl,--sysroot=/ build/temp.linux-ppc64le-3.6/orangecontrib/associate/_fpgrowth.o -o build/lib.linux-ppc                                        64le-3.6/orangecontrib/associate/_fpgrowth.cpython-36m-powerpc64le-linux-gnu.so
[root@p607-kvm1 Orange3-Associate-1.1.5]# /opt/anaconda3/bin/python3 setup.py install
running install
running bdist_egg
running egg_info
writing Orange3_Associate.egg-info/PKG-INFO
writing dependency_links to Orange3_Associate.egg-info/dependency_links.txt
writing entry points to Orange3_Associate.egg-info/entry_points.txt
writing namespace_packages to Orange3_Associate.egg-info/namespace_packages.txt
writing requirements to Orange3_Associate.egg-info/requires.txt
writing top-level names to Orange3_Associate.egg-info/top_level.txt
reading manifest file 'Orange3_Associate.egg-info/SOURCES.txt'
writing manifest file 'Orange3_Associate.egg-info/SOURCES.txt'
installing library code to build/bdist.linux-ppc64le/egg
running install_lib
running build_py
running build_ext
creating build/bdist.linux-ppc64le
creating build/bdist.linux-ppc64le/egg
creating build/bdist.linux-ppc64le/egg/orangecontrib
copying build/lib.linux-ppc64le-3.6/orangecontrib/__init__.py -> build/bdist.linux-ppc64le/egg/orangecontrib
creating build/bdist.linux-ppc64le/egg/orangecontrib/associate
copying build/lib.linux-ppc64le-3.6/orangecontrib/associate/fpgrowth.py -> build/bdist.linux-ppc64le/egg/orangecon                                        trib/associate
copying build/lib.linux-ppc64le-3.6/orangecontrib/associate/__init__.py -> build/bdist.linux-ppc64le/egg/orangecon                                        trib/associate
creating build/bdist.linux-ppc64le/egg/orangecontrib/associate/widgets
copying build/lib.linux-ppc64le-3.6/orangecontrib/associate/widgets/owitemsets.py -> build/bdist.linux-ppc64le/egg                                        /orangecontrib/associate/widgets
copying build/lib.linux-ppc64le-3.6/orangecontrib/associate/widgets/__init__.py -> build/bdist.linux-ppc64le/egg/o                                        rangecontrib/associate/widgets
copying build/lib.linux-ppc64le-3.6/orangecontrib/associate/widgets/owassociate.py -> build/bdist.linux-ppc64le/eg                                        g/orangecontrib/associate/widgets
creating build/bdist.linux-ppc64le/egg/orangecontrib/associate/widgets/icons
copying build/lib.linux-ppc64le-3.6/orangecontrib/associate/widgets/icons/AssociationRules.svg -> build/bdist.linu                                        x-ppc64le/egg/orangecontrib/associate/widgets/icons
copying build/lib.linux-ppc64le-3.6/orangecontrib/associate/widgets/icons/FrequentItemsets.svg.2016_01_06_16_55_23                                        .0.svg -> build/bdist.linux-ppc64le/egg/orangecontrib/associate/widgets/icons
copying build/lib.linux-ppc64le-3.6/orangecontrib/associate/widgets/icons/FrequentItemsets.svg -> build/bdist.linu                                        x-ppc64le/egg/orangecontrib/associate/widgets/icons
copying build/lib.linux-ppc64le-3.6/orangecontrib/associate/_fpgrowth.pyx -> build/bdist.linux-ppc64le/egg/orangec                                        ontrib/associate
copying build/lib.linux-ppc64le-3.6/orangecontrib/associate/_fpgrowth.cpython-36m-powerpc64le-linux-gnu.so -> buil                                        d/bdist.linux-ppc64le/egg/orangecontrib/associate
byte-compiling build/bdist.linux-ppc64le/egg/orangecontrib/__init__.py to __init__.cpython-36.pyc
byte-compiling build/bdist.linux-ppc64le/egg/orangecontrib/associate/fpgrowth.py to fpgrowth.cpython-36.pyc
byte-compiling build/bdist.linux-ppc64le/egg/orangecontrib/associate/__init__.py to __init__.cpython-36.pyc
byte-compiling build/bdist.linux-ppc64le/egg/orangecontrib/associate/widgets/owitemsets.py to owitemsets.cpython-3                                        6.pyc
byte-compiling build/bdist.linux-ppc64le/egg/orangecontrib/associate/widgets/__init__.py to __init__.cpython-36.py                                        c
byte-compiling build/bdist.linux-ppc64le/egg/orangecontrib/associate/widgets/owassociate.py to owassociate.cpython                                        -36.pyc
creating stub loader for orangecontrib/associate/_fpgrowth.cpython-36m-powerpc64le-linux-gnu.so
byte-compiling build/bdist.linux-ppc64le/egg/orangecontrib/associate/_fpgrowth.py to _fpgrowth.cpython-36.pyc
creating build/bdist.linux-ppc64le/egg/EGG-INFO
copying Orange3_Associate.egg-info/PKG-INFO -> build/bdist.linux-ppc64le/egg/EGG-INFO
copying Orange3_Associate.egg-info/SOURCES.txt -> build/bdist.linux-ppc64le/egg/EGG-INFO
copying Orange3_Associate.egg-info/dependency_links.txt -> build/bdist.linux-ppc64le/egg/EGG-INFO
copying Orange3_Associate.egg-info/entry_points.txt -> build/bdist.linux-ppc64le/egg/EGG-INFO
copying Orange3_Associate.egg-info/namespace_packages.txt -> build/bdist.linux-ppc64le/egg/EGG-INFO
copying Orange3_Associate.egg-info/not-zip-safe -> build/bdist.linux-ppc64le/egg/EGG-INFO
copying Orange3_Associate.egg-info/requires.txt -> build/bdist.linux-ppc64le/egg/EGG-INFO
copying Orange3_Associate.egg-info/top_level.txt -> build/bdist.linux-ppc64le/egg/EGG-INFO
writing build/bdist.linux-ppc64le/egg/EGG-INFO/native_libs.txt
creating dist
creating 'dist/Orange3_Associate-1.1.5-py3.6-linux-ppc64le.egg' and adding 'build/bdist.linux-ppc64le/egg' to it
removing 'build/bdist.linux-ppc64le/egg' (and everything under it)
Processing Orange3_Associate-1.1.5-py3.6-linux-ppc64le.egg
creating /opt/anaconda3/lib/python3.6/site-packages/Orange3_Associate-1.1.5-py3.6-linux-ppc64le.egg
Extracting Orange3_Associate-1.1.5-py3.6-linux-ppc64le.egg to /opt/anaconda3/lib/python3.6/site-packages
Adding Orange3-Associate 1.1.5 to easy-install.pth file

Installed /opt/anaconda3/lib/python3.6/site-packages/Orange3_Associate-1.1.5-py3.6-linux-ppc64le.egg
Processing dependencies for Orange3-Associate==1.1.5
Searching for scipy==1.3.0
Best match: scipy 1.3.0
Adding scipy 1.3.0 to easy-install.pth file

Using /opt/anaconda3/lib/python3.6/site-packages
Searching for numpy==1.17.2
Best match: numpy 1.17.2
Adding numpy 1.17.2 to easy-install.pth file
Installing f2py script to /opt/anaconda3/bin
Installing f2py3 script to /opt/anaconda3/bin
Installing f2py3.6 script to /opt/anaconda3/bin

Using /opt/anaconda3/lib/python3.6/site-packages
Finished processing dependencies for Orange3-Associate==1.1.5

정상적으로 빌드되었고, 아래 경로를 찾아보면 Orange3-Associate 관련 패키지가 생성된 것을 볼 수 있습니다.
[root@p607-kvm1 Orange3-Associate-1.1.5]# ls -al /opt/anaconda3/lib/python3.6/site-packages
(생략)
drwxr-xr-x   4 root root     43 Mar 10 00:46 Orange3_Associate-1.1.5-py3.6-linux-ppc64le.egg
(생략)

이제 예제 스크립트를 실행해봅니다. 예제는 아래 사이트에 나와 있는 것을 수행해봅니다.

[root@p607-kvm1 ~]# /opt/anaconda3/bin/python3
Python 3.6.9 |Anaconda, Inc.| (default, Jul 30 2019, 19:18:58)
[GCC 7.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> T = [[1,    3, 4   ],
...      [   2, 3,    5],
...      [1, 2, 3,    5],
...      [   2,       5]]
>>> from orangecontrib.associate.fpgrowth import *
>>> itemsets = frequent_itemsets(T, 2)
>>> itemsets
<generator object frequent_itemsets at 0x7fff9c1fde08>
>>> list(itemsets)
[(frozenset({1}), 2), (frozenset({2}), 3), (frozenset({3}), 3), (frozenset({1, 3}), 2), (frozenset({2, 3}), 2), (frozenset({5}), 3), (frozenset({2, 5}), 3), (frozenset({3, 5}), 2), (frozenset({2, 3, 5}), 2)]
>>> import Orange
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'Orange'
>>>

Orange 를 import 할 수 없다는 에러가 발생합니다.
아무래도 Orange3-associate 과는 별도의 Orange 패키지를 설치해야 하는 것 같습니다. 

pypi를 찾아보니, Orange3 패키지가 올라와 있습니다만 ppc64le 환경용 whl 파일은 따로 없습니다. pip 에서도 Oragne3 을 찾을 수 있습니다. 이번에는 pip install 을 수행해 보겠습니다.
[root@p639-kvm1 pytagcloud-0.3.5]# pip install orange3
Collecting orange3
  Downloading https://files.pythonhosted.org/packages/ac/87/0e1d5ff5d3644ab80c4e6c6a7861271d59277aab6db103                                                273114df82c041/Orange3-3.24.1.tar.gz (2.5MB)
     |████████████████████████████████| 2.5MB 5.6MB/s
  Installing build dependencies ... done
(생략)
  Stored in directory: /root/.cache/pip/wheels/93/a2/de/184ecdefdb91d760706d02389015dfa3b96e927eb1c42ace37
  Building wheel for openTSNE (setup.py) ... error
  ERROR: Command errored out with exit status 1:
   command: /opt/anaconda3/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip                                                -install-bydkbko3/openTSNE/setup.py'"'"'; __file__='"'"'/tmp/pip-install-bydkbko3/openTSNE/setup.py'"'"';f                                                =getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.c                                                lose();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /tmp/pip-wheel-bojfbwix --python-tag                                               

Orange3 을 pip로 설치하다보니 에러가 발생합니다. 에러가 가장 처음 발생한 부분부터 찾아서 하나씩 해결해보겠습니다.

우선 가장 처음 에러가 발생한 부분을 보니, openTSNE 를 빌드하던 중입니다. OpenTSNE를 Pypi에서 다운로드 받아서 소스에서 직접 빌드해봅니다.

[root@p639-kvm1 ~]# cd /root/openTSNE-0.3.12

현재 사용하는 플랫폼은 POWER9 cpu를 사용하는 AC922 서버이므로, setup.py 를 열어서 -march=native 를 -mcpu=power9 으로 변경해줍니다. -march=native 를 그대로 사용하면 Intel Xeon processor로 인식하여 에러가 발생합니다.

[root@p639-kvm1 openTSNE-0.3.12]# vi setup.py
#            "native": "-march=native",
            "native": "-mcpu=power9",

[root@p639-kvm1 openTSNE-0.3.12]# /opt/anaconda3/bin/python setup.py build
./tmpl9ju6hl_/fftw3.c:1:10: fatal error: fftw3.h: No such file or directory
 #include <fftw3.h>
          ^~~~~~~~~
compilation terminated.
FFTW3 header files not found. Using numpy implementation of FFT.
running build
running build_py
running build_ext
gcc -pthread -B /opt/anaconda3/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/opt/anaconda3/include -I/opt/anaconda3/Library/include -c ./tmpo1ebp1ta/omp.c -o ./tmpo1ebp1ta/omp.o
./tmpo1ebp1ta/omp.c:2:5: warning: function declaration isn’t a prototype [-Wstrict-prototypes]
 int main() {}
     ^~~~
(생략)
gcc -pthread -shared -B /opt/anaconda3/compiler_compat -L/opt/anaconda3/lib -Wl,-rpath=/opt/anaconda3/lib -Wl,--no-as-needed -Wl,--sysroot=/ build/temp.linux-ppc64le-3.6/openTSNE/_matrix_mul/matrix_mul_numpy.o -o build/lib.linux-ppc64le-3.6/openTSNE/_matrix_mul/matrix_mul.cpython-36m-powerpc64le-linux-gnu.so -lm -O3 -mcpu=power9 -fopenmp

빌드가 완료되었으니, install 을 해봅니다.
[root@p639-kvm1 openTSNE-0.3.12]# /opt/anaconda3/bin/python setup.py install
./tmplduur_i_/fftw3.c:1:10: fatal error: fftw3.h: No such file or directory
(생략)
Using /opt/anaconda3/lib/python3.6/site-packages
Searching for setuptools==41.4.0
Best match: setuptools 41.4.0
Adding setuptools 41.4.0 to easy-install.pth file
Installing easy_install script to /opt/anaconda3/bin

Using /opt/anaconda3/lib/python3.6/site-packages
Finished processing dependencies for openTSNE==0.3.12

conda 에서 확인해보니, openTSNE 가 잘 설치되었습니다. 
[root@p639-kvm1 openTSNE-0.3.12]# conda list openTSNE
# packages in environment at /opt/anaconda3:
#
# Name                    Version                   Build  Channel
opentsne                  0.3.12                   pypi_0    pypi

다시 orange3 를 설치 시도해보니, 이번엔 scikit-learn 에서 에러가 발생합니다. 로그를 읽어보면, orange3의 prerequisites인 일부 패키지에서 scikit-learn>=0.22.0 버전 이상을 요구하는데, 이걸 설치하려다 보니 Cython>=0.28.5 버전을 요구합니다. pip로 Cython 해당 버전을 빌드하는데 에러가 발생하는 것 같습니다.

[root@p639-kvm1 pytagcloud-0.3.5]# pip install orange3
Processing /root/.cache/pip/wheels/65/8d/95/66d4803abaff62bcc5dee7df1b43734c192b9bbcc5a94aa4df/Orange3-3.24.1-cp36-cp36m-linux_ppc64le.whl
Requirement already satisfied: python-louvain>=0.13 in /opt/anaconda3/lib/python3.6/site-packages (from orange3) (0.13)
Collecting scikit-learn>=0.22.0
  Using cached https://files.pythonhosted.org/packages/e4/40/8bc77d8f536be0a892b37fff19fd81f15935e24724303480f85238ec7f22/scikit-learn-0.22.2.post1.tar.gz
Building wheels for collected packages: scikit-learn
  Building wheel for scikit-learn (setup.py) ... error
  ERROR: Command errored out with exit status 1:
   command: /opt/anaconda3/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-ghyqlp20/scikit-learn/setup.py'"'"'; __file__='"'"'/tmp/pip-install-ghyqlp20/scikit-learn/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /tmp/pip-wheel-lyzjovl0 --python-tag cp36
       cwd: /tmp/pip-install-ghyqlp20/scikit-learn/
  ValueError: Please install Cython with a version >= 0.28.5 in order to build a scikit-learn from source. The current version of Cython is 0.28.1 installed in ['/opt/anaconda3/lib/python3.6/site-packages/Cython'].
  ----------------------------------------
  ERROR: Failed building wheel for scikit-learn

현재 Conda 에 설치된 Cython 버전과, 
[root@p639-kvm1 ~]# conda list Cython
# packages in environment at /opt/anaconda3:
#
# Name                    Version                   Build  Channel
cython                    0.28.1           py36h14c3975_0    https://repo.anaconda.com/pkgs/main

[root@p639-kvm1 ~]# conda install Cython==0.28.5
The following NEW packages will be INSTALLED:

  pandas             repos/powerai/1.6.1/conda/pkgs/main/linux-ppc64le::pandas-0.24.2-py36he6710b0_0

The following packages will be UPDATED:

  cython             pkgs/main::cython-0.28.1-py36h14c3975~ --> repos/powerai/1.6.1/conda/pkgs/main::cython-0.28.5-py36hf484d3e_0


Proceed ([y]/n)? y


Downloading and Extracting Packages
cython-0.28.5        | 3.0 MB    | ############################################################################################################### | 100%
pandas-0.24.2        | 10.3 MB   | ############################################################################################################### | 100%
Preparing transaction: done
Verifying transaction: done
Executing transaction: done

다시 pip install orange3 시도합니다.
[root@p639-kvm1 ~]# pip install orange3
Processing ./.cache/pip/wheels/65/8d/95/66d4803abaff62bcc5dee7df1b43734c192b9bbcc5a94aa4df/Orange3-3.24.1-cp36-cp36m-linux_ppc64le.whl
Collecting scikit-learn>=0.22.0
  Using cached https://files.pythonhosted.org/packages/e4/40/8bc77d8f536be0a892b37fff19fd81f15935e24724303480f85238ec7f22/scikit-learn-0.22.2.post1.tar.gz
Requirement already satisfied: xlsxwriter in /opt/anaconda3/lib/python3.6/site-packages (from orange3) (1.0.4)
Requirement already satisfied: matplotlib>=2.0.0 in /opt/anaconda3/lib/python3.6/site-packages (from orange3) (3.0.0)
Requirement already satisfied: numpy>=1.16.0 in /opt/anaconda3/lib/python3.6/site-packages (from orange3) (1.17.2)
Requirement already satisfied: scipy>=0.16.1 in /opt/anaconda3/lib/python3.6/site-packages (from orange3) (1.3.0)
Requirement already satisfied: openpyxl in /opt/anaconda3/lib/python3.6/site-packages (from orange3) (2.5.3)
Requirement already satisfied: setuptools>=36.3 in /opt/anaconda3/lib/python3.6/site-packages (from orange3) (41.4.0)
Requirement already satisfied: keyrings.alt in /opt/anaconda3/lib/python3.6/site-packages (from orange3) (3.4.0)
(생략)
  Building wheel for scikit-learn (setup.py): still running...
  Building wheel for scikit-learn (setup.py): finished with status 'done'
  Created wheel for scikit-learn: filename=scikit_learn-0.22.2.post1-cp36-cp36m-linux_ppc64le.whl size=18232672 sha256=86081cde15b4fb435825e06b301fa4a2c8fbf8f97dae66fe34e4b6691bb15f3c
  Stored in directory: /root/.cache/pip/wheels/9c/ab/08/f3ae672cb3582e5f5a1d366531f6b96115017c66d6f4f2f5f3
Successfully built scikit-learn
Installing collected packages: scikit-learn, orange-widget-base, orange3
  Found existing installation: scikit-learn 0.21.3
    Uninstalling scikit-learn-0.21.3:
      Successfully uninstalled scikit-learn-0.21.3
Successfully installed orange-widget-base-4.5.0 orange3-3.24.1 scikit-learn-0.22.2.post1

잘 완료되었고, conda list 로 확인해보면 orange3이 설치되어 있습니다.
[root@p639-kvm1 ~]# conda list orange3
# packages in environment at /opt/anaconda3:
#
# Name                    Version                   Build  Channel
orange3                   3.24.1                   pypi_0    pypi

다시 python을 실행하여 import Orange 가 잘 실행되는지 확인합니다.

** 위 테스트 시, 최신 버전인 Orange3 3.24.1 을 설치하면서 Pandas 도 1.0.3 으로 설치되었습니다. 어떤 이유에서인지 Pandas 1.0.3 은 import pandas 시 에러가 발생하여, pip uninstall pandas, pip uninstall Orange3 을 다시 삭제하고, Orange3 3.24.0 버전으로 낮추어 재설치했습니다.

** pip install Orange3==3.24.0 설치 결과
Successfully built Orange3
Installing collected packages: Orange3
Successfully installed Orange3-3.24.0

** pip uninstall pandas 후, 원래 설치되어있던 이전 버전의 pandas 정보 (0.24.2)
[root@p639-kvm1 ~]# conda list pandas
# packages in environment at /opt/anaconda3:
#
# Name                    Version                   Build  Channel
pandas                    0.24.2           py36he6710b0_0
[root@p639-kvm1 ~]# python
Python 3.6.9 |Anaconda, Inc.| (default, Jul 30 2019, 19:18:58)
[GCC 7.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pandas
>>> import Orange

댓글 없음:

댓글 쓰기