rocky系统编译ceph

Last updated on 8 months ago

不同系统编译ceph 遇到的问题大多情况是 缺包;缺包的话有两种方式

做不同系统编译ceph 遇到的问题大多情况是 缺包;缺包的话有两种方式

这两个方法可以解决大多数的缺包问题,当然也有些包没有适配操作系统的可能需要自己拉源码编译,这种比较繁琐,这个方法放在最后,实在不行才自己编;

还有一个问题则是编译时遇到的问题,由于gcc 版本不同(跨度很大),有些语法不是完全兼容的,这个时候需要分析 报错的内容,(优先谷歌下,绝大多数问题,已有前人踩坑)

环境配置

由于没有物理机,直接在用容器编

1
2
3
4
5
6
7
[root@9ebdf34fb3b7 ceph]# cat /etc/redhat-release Rocky Linux release 8.6 (Green Obsidian) [root@9ebdf34fb3b7 ceph]#  
[root@8dd40184b527 /]# cat /etc/rocky-release Rocky Linux release 9.3 (Blue Onyx)

[root@9ebdf34fb3b7 ceph]# ldd --version ldd (GNU libc) 2.28 Copyright (C) 2018 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. Written by Roland McGrath and Ulrich Drepper.

[root@8dd40184b527 build]# uname -a Linux 8dd40184b527 3.10.0-1160.21.1.el7.x86_64 #1 SMP Tue Mar 16 18:28:22 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
[root@8dd40184b527 build]#

安装必要的依赖

执行 install-deps.sh 脚本 ,由于是rocky,脚本里面没有涉及改系统的选项, 手动添加下,(rokcy也是 centos的分支)

1
2
3
4
5
6
[root@8dd40184b527 ceph_copy]# ./install-deps.sh rocky is unknown, dependencies will have to be installed manually.
[root@8dd40184b527 ceph_copy]#
[root@8dd40184b527 ceph_copy]# sed -i 's/virtuozzo/rocky/g' ./install-deps.sh
[root@8dd40184b527 ceph_copy]# ./install-deps.sh
Using dnf to install dependencies Last metadata expiration check: 0:10:34 ago on Sun Feb 18 07:28:20 2024. Dependencies resolved. =====================
...

上面的脚本安装的可能不全,下面是遇到编译问题,直接从yum源安装的

1
2
[root@8dd40184b527 ceph_copy]# echo "cmake gcc  python3-devel.x86_64  make wget ncurses-devel.x86_64  python snappy.x86_64  libaio-devel.x86_64 fuse3-devel.x86_64 openldap-devel.x86_64 python3-sphinx.noarch  libblkid-devel.x86_64 systemd-devel keyutils-libs-devel.x86_64  lz4-devel.x86_64  libcurl-devel.x86_64 nss-devel.x86_64 openssl-devel.x86_64 expat-devel.x86_64 liboath-devel.x86_64 python  lttng-ust-devel.x86_64  libbabeltrace-devel.x86_64  libcap-ng-devel.x86_64 librabbitmq-devel.x86_64 gperf libnl3-devel.x86_64  librdkafka-devel.x86_64   libcap-ng-devel.x86_64  keyutils-libs-devel.x86_64" > /tmp/install 
[root@8dd40184b527 ceph_copy]# for i in $(cat /tmp/install);do yum install -y $i;done

执行 do_cmake

do_cmake 会先检测一些必要的依赖是否已安装

1
2
3
4
5
CMake Error at CMakeLists.txt:63 (message):  Can't find sphinx-build.
-- Configuring incomplete, errors occurred!

yum install pip && pip install Sphinx -i https://pypi.tuna.tsinghua.edu.cn/simple
# 这个库可以不需要的,直接在 do_cmkae 注释掉也行
1
2
3
4
5
CMake Error at /usr/share/cmake/Modules/FindPackageHandleStandardArgs.cmake:230 (message):  
Could NOT find verbs (missing: VERBS_LIBRARIES VERBS_INCLUDE_DIR) Call Stack (most recent call first):

/usr/share/cmake/Modules/FindPackageHandleStandardArgs.cmake:594 (_FPHSA_FAILURE_MESSAGE) cmake/modules/Findverbs.cmake:13 (find_package_handle_standard_args) CMakeLists.txt:204 (find_package)
# RDMA 需要的用的库,目前我们产品没有用用到,不用加入编译,直接注释 后者 do_cmake 加入 -DWITH_RDMA=OFF
1
2
3
4
5
-- Found Backtrace: /usr/include  CMake Error at /usr/share/cmake/Modules/FindPackageHandleStandardArgs.cmake:230 (message): 

Could NOT find udev (missing: UDEV_LIBRARIES UDEV_INCLUDE_DIR) Call Stack (most recent call first): /usr/share/cmake/Modules/FindPackageHandleStandardArgs.cmake:594 (_FPHSA_FAILURE_MESSAGE) cmake/modules/Findudev.cmake:30 (find_package_handle_standard_args) CMakeLists.txt:217 (find_package)

yum install systemd-devel
1
2
3
4
5
-- Found udev: /usr/lib64/libudev.so   CMake Error at /usr/share/cmake/Modules/FindPackageHandleStandardArgs.cmake:230 (message):  
Could NOT find blkid (missing: BLKID_LIBRARIES BLKID_INCLUDE_DIR) Call Stack (most recent call first):
/usr/share/cmake/Modules/FindPackageHandleStandardArgs.cmake:594 (_FPHSA_FAILURE_MESSAGE) cmake/modules/Findblkid.cmake:31 (find_package_handle_standard_args) CMakeLists.txt:219 (find_package)

yum install libblkid-devel.x86_64
1
2
3
-- Found udev: /usr/lib64/libudev.so   CMake Error at /usr/share/cmake/Modules/FindPackageHandleStandardArgs.cmake:230 (message): 
Could NOT find blkid (missing: BLKID_LIBRARIES BLKID_INCLUDE_DIR) Call Stack (most recent call first): /usr/share/cmake/Modules/FindPackageHandleStandardArgs.cmake:594 (_FPHSA_FAILURE_MESSAGE) cmake/modules/Findblkid.cmake:31 (find_package_handle_standard_args) CMakeLists.txt:219 (find_package)
yum install libblkid-devel.x86_64
1
2
3
4
5
6
CMake Error at /usr/share/cmake/Modules/FindPackageHandleStandardArgs.cmake:230 (message):  
Could NOT find OpenLdap (missing: OPENLDAP_INCLUDE_DIR LDAP_LIBRARY LBER_LIBRARY) Call Stack (most recent call first):

/usr/share/cmake/Modules/FindPackageHandleStandardArgs.cmake:594 (_FPHSA_FAILURE_MESSAGE) cmake/modules/FindOpenLdap.cmake:16 (find_package_handle_standard_args) CMakeLists.txt:244 (find_package)

yum install -y openldap-devel.x86_64
1
2
3
4
5
6
7
8
缺少level 包 Call Stack (most recent call first):  /usr/share/cmake/Modules/FindPackageHandleStandardArgs.cmake:594 (_FPHSA_FAILURE_MESSAGE)  cmake/modules/Findleveldb.cmake:18  

yum 源码里没有,手动下载并安装

wget https://dl.fedoraproject.org/pub/epel/9/Everything/x86_64/Packages/l/leveldb-devel-1.23-4.el9.x86_64.rpm
wget https://dl.fedoraproject.org/pub/epel/9/Everything/x86_64/Packages/l/leveldb-1.23-4.el9.x86_64.rpm
rpm -ivh leveldb-1.23-4.el9.x86_64.rpm
rpm -ivh leveldb-devel-1.23-4.el9.x86_64.rpm
1
2
3
4
5
6
7
CMake Error at /usr/share/cmake/Modules/FindPackageHandleStandardArgs.cmake:230 (message):  
Could NOT find snappy (missing: SNAPPY_LIBRARIES SNAPPY_INCLUDE_DIR) Call Stack (most recent call first): /usr/share/cmake/Modules/FindPackageHandleStandardArgs.cmake:594 (_FPHSA_FAILURE_MESSAGE) cmake/modules/Findsnappy.cmake:25

(find_package_handle_standard_args) CMakeLists.txt:349 (find_package)
yum 源码里没有,手动下载并安装
wget https://dl.rockylinux.org/pub/rocky/9/devel/x86_64/os/Packages/s/snappy-devel-1.1.8-8.el9.x86_64.rpm
rpm -ivh snappy-devel-1.1.8-8.el9.x86_64.rpm
1
2
3
4
5
6
7
8
CMake Error at /usr/share/cmake/Modules/FindPackageHandleStandardArgs.cmake:230 (message):  
Could NOT find OATH (missing: OATH_LIBRARY OATH_INCLUDE_DIR) Call Stack (most recent call first): /usr/share/cmake/Modules/FindPackageHandleStandardArgs.cmake:594 (_FPHSA_FAILURE_MESSAGE) cmake/modules/FindOATH.cmake:17

(find_package_handle_standard_args) CMakeLists.txt:461 (find_package) yum 源码里没有,手动下载并安装
wget https://dl.fedoraproject.org/pub/epel/9/Everything/x86_64/Packages/l/liboath-2.6.7-2.el9.x86_64.rpm
wget https://dl.fedoraproject.org/pub/epel/9/Everything/x86_64/Packages/l/liboath-devel-2.6.7-2.el9.x86_64.rpm
rpm -ivh liboath-2.6.7-2.el9.x86_64.rpm
rpm -ivh liboath-devel-2.6.7-2.el9.x86_64.rpm
1
2
3
4
5
6
7
-- crypto soname: libcrypto.so.3 -- Found PythonInterp: /usr/bin/python (found suitable version "3.9.18", minimum required is "2.7") 

CMake Error at /usr/share/cmake/Modules/FindPackageHandleStandardArgs.cmake:230 (message):
Could NOT find PythonLibs (missing: PYTHON_LIBRARIES PYTHON_INCLUDE_DIRS)

(Required is at least version "2.7")
yum install python3-devel.x86_64
1
2
3
4
-- Found PythonLibs: /usr/lib64/libpython3.9.so (found suitable version "3.9.18", minimum required is "2.7") CMake Error at /usr/share/cmake/Modules/FindPackageHandleStandardArgs.cmake:230 (message):
Could NOT find ZLIB (missing: ZLIB_LIBRARY ZLIB_INCLUDE_DIR)

yum install zlib-devel.x86_64
1
2
3
4
5
6
Make Error at /usr/share/cmake/Modules/FindPackageHandleStandardArgs.cmake:230 (message):  
Could NOT find LTTngUST (missing: LTTNGUST_LIBRARIES LTTNGUST_INCLUDE_DIRS) Call Stack (most recent call first): /usr/share/cmake/Modules/FindPackageHandleStandardArgs.cmake:594 (_FPHSA_FAILURE_MESSAGE) cmake/modules/FindLTTngUST.cmake:107 (find_package_handle_standard_args) CMakeLists.txt:570 (find_package)

LTTN 用不到,编译选项关闭 -DWITH_LTTNG=OFF -DWITH_BABELTRACE=OFF
如果真的需要 安装 这两个包
lttng-ust-devel.x86_64 libbabeltrace-devel.x86_64
1
2
3
4
5
CMake Error at cmake/modules/FindCython.cmake:13 (message):  Could not find cython: /usr/bin/python: No module named cython Call Stack (most recent call first):  src/pybind/CMakeLists.txt:23 (find_package)
CMake Error at /usr/share/cmake/Modules/FindPackageHandleStandardArgs.cmake:230 (message): Could NOT find Cython (missing: CYTHON_VERSION)
Call Stack (most recent call first): /usr/share/cmake/Modules/FindPackageHandleStandardArgs.cmake:594 (_FPHSA_FAILURE_MESSAGE) cmake/modules/FindCython.cmake:16 (find_package_handle_standard_args) src/pybind/CMakeLists.txt:23 (find_package)

pip install cython -i https://pypi.tuna.tsinghua.edu.cn/simple
1
2
3
-- Found Cython: 0.29   CMake Error at src/pybind/CMakeLists.txt:68 (message):  mgr plugins require python3 binding ## force ceph to build against python3:

#指定pyton版本 添加编译选项 -DWITH_PYTHON3=ON
1
2
3
-- Could NOT find nl-genl-3 (missing: GENL_LIBRARIES GENL_INCLUDE_DIR) CMake Error at /usr/share/cmake/Modules/FindPackageHandleStandardArgs.cmake:230 (message):  
Could NOT find keyutils (missing: KEYUTILS_LIBRARIES KEYUTILS_INCLUDE_DIR) Call Stack (most recent call first): /usr/share/cmake/Modules/FindPackageHandleStandardArgs.cmake:594 (_FPHSA_FAILURE_MESSAGE) cmake/modules/Findkeyutils.cmake:16 (find_package_handle_standard_args) CMakeLists.txt:225 (find_package)
yum install keyutils-libs-devel.x86_64
1
2
3
4
Call Stack (most recent call first):  /usr/share/cmake/Modules/FindPackageHandleStandardArgs.cmake:594 (_FPHSA_FAILURE_MESSAGE)  cmake/modules/FindRabbitMQ.cmake:9 (find_package_handle_standard_args)  src/rgw/CMakeLists.txt:134 (find_package) 

rpm 官方下载
wget https://dl.rockylinux.org/pub/rocky/9/devel/x86_64/os/Packages/l/librabbitmq-devel-0.11.0-7.el9.x86_64.rpm rpm -ivh librabbitmq-devel-0.11.0-7.el9.x86_64.rpm
1
/boost/thread/pthread/thread_data.hpp:60:5: error: missing binary operator before token "("   60 | #if PTHREAD_STACK_MIN > 0    |   ^~~~~~~~~~~~~~~~~ ...skipped <pbin.v2/libs/coroutine/build/gcc-10.3.1/release/link-static/threadapi-pthread/threading-multi/visibility-hidden>libboost_coroutine.a(clean) for lack of <pbin.v2/libs/coroutine/build/gcc-10.3.1/release/link-static/threadapi-pthread/threading-multi/visibility-hidden>posix/stack_traits.o... ...skipped <pbin.v2/libs/coroutine/build/gcc-10.3.1/release/link-static/threadapi-pthread/threading-multi/visibility-hidde z 修改 boost 源文件的 /root/ceph/build/boost/src/Boost/boost/thread/pthread/thread_data.hpp   #ifdef PTHREAD_STACK_MIN    /data/rphuang/ceph_copy/build/boost/src/Boost/boost/thread/pthread 

高版本的gcc 不支持这个语法,

需要将 #if PTHREAD_STACK_MIN > 0 改为 #ifdef PTHREAD_STACK_MIN

缺少 librdkafka 这个再编译过程中才检测到的

1
2
3
4
5
6
7

wget https://dl.rockylinux.org/pub/rocky/9/devel/x86_64/os/Packages/l/librdkafka-1.6.1-102.el9.x86_64.rpm
wget https://dl.rockylinux.org/pub/rocky/9/devel/x86_64/os/Packages/l/librdkafka-devel-1.6.1-102.el9.x86_64.rpm
wget https://dl.rockylinux.org/pub/rocky/9/devel/x86_64/os/Packages/l/libzstd-devel-1.5.1-2.el9.x86_64.rpm
rpm -ivh libzstd-devel-1.5.1-2.el9.x86_64.rpm
rpm -ivh librdkafka-1.6.1-102.el9.x86_64.rpm
rpm -ivh librdkafka-devel-1.6.1-102.el9.x86_64.rpm

缺少 gperf

1
2
3
4
5
6
7
8
9
ir<std::__cxx11::basic_string<char>, bool>’} to make the copy explicit or ‘const std::pair<const std::__cxx11::basic_string<char>, bool>&’ to prevent copying /data/rphuang/ceph_rocky/ceph/src/rgw/rgw_rados.cc:15:10: fatal error: gperftools/heap-checker.h: No such file or directory   15 | #include "gperftools/heap-checker.h"    |      ^~~~~~~~~~~~~~~~~~~~~~~~~~~ compilation terminated.   

wget https://dl.fedoraproject.org/pub/epel/9/Everything/x86_64/Packages/g/gperftools-devel-2.9.1-2.el9.x86_64.rpm
wget https://dl.fedoraproject.org/pub/epel/9/Everything/x86_64/Packages/g/gperftools-libs-2.9.1-2.el9.x86_64.rpm
wget https://dl.fedoraproject.org/pub/epel/9/Everything/x86_64/Packages/l/libunwind-1.6.2-1.el9.x86_64.rpm
wget https://dl.fedoraproject.org/pub/epel/9/Everything/x86_64/Packages/g/gperftools-2.9.1-2.el9.x86_64.rpm
rpm -ivh libunwind-1.6.2-1.el9.x86_64.rpm
rpm -ivh libperf-devel-5.14.0-362.18.1.el9_3.0.1.x86_64.rpm
rpm -ivh gperftools-devel-2.9.1-2.el9.x86_64.rpm rpm -ivh gperftools-libs-2.9.1-2.el9.x86_64.rpm
1
2
3
4
5
6
`使用vstaart 脚本搭建集群  Populating config ... Traceback (most recent call last):
File "/data/rphuang/ceph_copy/build/bin/ceph", line 151, in <module>
from ceph_daemon import admin_socket, DaemonWatcher, Termsize File "/data/rphuang/ceph_copy/src/pybind/ceph_daemon.py", line 24,
in <module> from prettytable import PrettyTable, HEADER ModuleNotFoundError: No module named 'prettytable'

# 用vstart 启动集群的时候 提示没有 prettytable pip install PrettyTable ````