运行sched_ext调度器

系统版本信息该版本较低不支持sched_ext本文基于bclinux / RHEL / CentOS如果用的是ubuntu替换成对应的apt命令即可。[roothadoop02 ~]# uname -a Linux hadoop02 5.10.0-200.0.0.131.30.oe2203sp3.bclinux.x86_64 #1 SMP Tue Sep 24 08:43:22 CST 2024 x86_64 x86_64 x86_64 GNU/LinuxLinux内核从6.12版本开始正式支持sched_ext调度程序所以需升级内核至6.12或其他高版本。本文将内核升级至6.15.7版本sched_ext调度器基于6.15.7实现。一、pahole 版本 ≥ 1.16打开SCHED_CLASS_EXT配置项时依赖关系要求pahole的版本 ≥ 1.16。下载pahole源码[roothadoop02 ~]# git clone git://git.kernel.org/pub/scm/devel/pahole/pahole.git [roothadoop02 ~]# cd pahole/ [roothadoop02 ~]# git checkout v1.30在编译pahole的过程中pahole/.gitmodules文件配置了git libbpf的地址如果不能访问外网git hub可通过下面命令将地址换成国内镜像。pahole根目录执行 sed -i s|https://github.com/libbpf/libbpf|https://gitee.com/mirrors/libbpf.git| .gitmodules git submodule sync git submodule update --init安装cmake等工具[roothadoop02 ~]# yum install cmake gcc make elfutils-libelf-devel -y编译及安装[roothadoop02 pahole]# mkdir build cd build [roothadoop02 build]# cmake -D__LIBlib .. [roothadoop02 build]# make -j$(nproc) [ 1%] Building C object CMakeFiles/bpf.dir/lib/bpf/src/bpf.c.o [ 3%] Building C object CMakeFiles/bpf.dir/lib/bpf/src/bpf_prog_linfo.c.o [ 5%] Building C object CMakeFiles/bpf.dir/lib/bpf/src/btf.c.o [ 6%] Building C object CMakeFiles/bpf.dir/lib/bpf/src/btf_dump.c.o [ 8%] Building C object CMakeFiles/bpf.dir/lib/bpf/src/btf_iter.c.o [roothadoop02 build]# make install Consolidate compiler generated dependencies of target bpf [ 37%] Built target bpf Consolidate compiler generated dependencies of target dwarves [ 57%] Built target dwarves Consolidate compiler generated dependencies of target dwarves_emit安装[roothadoop02 build]# make install -- Installing: /usr/local/include/dwarves/elfcreator.h -- Installing: /usr/local/include/dwarves/elf_symtab.h -- Installing: /usr/local/include/dwarves/hash.h -- Installing: /usr/local/include/dwarves/libctf.h -- Installing: /usr/local/share/man/man1/pahole.1 -- Installing: /usr/local/bin/ostra-cg -- Installing: /usr/local/share/dwarves/runtime/python/ostra.py -- Installing: /usr/local/bin/btfdiff -- Installing: /usr/local/bin/fullcircle -- Installing: /usr/local/share/dwarves/runtime/Makefile -- Installing: /usr/local/share/dwarves/runtime/ctracer_relay.c -- Installing: /usr/local/share/dwarves/runtime/ctracer_relay.h -- Installing: /usr/local/share/dwarves/runtime/linux.blacklist.cu修改环境变量echo /usr/local/lib64 | sudo tee /etc/ld.so.conf.d/dwarves.conf sudo ldconfig安装完成[roothadoop02 linux-6.15.7]# pahole --version v1.30二、6.15.7内核编译选项sche_ext相关在开始设置config选项前一定要按第一节内容安装pahole否则部分config将不会显示。make menuconfig设置以下配置项务必注意需要按顺序设置CONFIG_LOCALVERSION填入自己设置的内核版本名称General setup - Local version - append to kernel releaseCONFIG_BPF_SYSCALLy允许用户态通过bpf()系统调用加载 eBPF 程序到内核General setup - BPF subsystem - Enable bpf() system callCONFIG_BPF_JITy启用BPF即时编译加速General setup - BPF subsystem - Enable BPF Just In Time compiler勾选后CONFIG_HAVE_EBPF_JIT 将自动激活依赖架构支持,x86_64/ARM64 通常支持CONFIG_DEBUG_INFOy生成完整的调试符号信息“Kernel hacking” - “Compile-time checks and compiler options” - “Debug information选择Generate DWARF Version 4 debuginfoCONFIG_DEBUG_INFO_SPLITn禁用调试信息拆分“Kernel hacking” - “Compile-time checks and compiler options” - Produce split debuginfo in .dwo filesCONFIG_DEBUG_INFO_REDUCEDn禁用缩减版调试信息Kernel hacking” - “Compile-time checks and compiler options” - Reduce debugging informationCONFIG_DEBUG_INFO_BTFy生成BPF类型格式(BTF)调试信息提供内核数据结构元数据“Kernel hacking” - “Compile-time checks and compiler options” - “Generate BTF type information ”CONFIG_SCHED_CLASS_EXTy启用可扩展的 BPF 调度类sched_extGeneral setup - Extensible Scheduling Class (NEW)”CONFIG_FUNCTION_ERROR_INJECTIONy开启故障注入测试“Kernel hacking” - Kernel Testing and Coverage - Fault-injections of functionsCONFIG_BPF_KPROBE_OVERRIDEy控制 eBPF 程序是否能够修改内核函数的返回值‌“Kernel hacking” - Tracers - enable BPF programs to override a kprobed function可通过检测脚本检查配置是否正确放在linux内核源码的根目录[roothadoop02 linux-6.15.7]# cat check_config.sh #!/bin/bash # check_config.sh # 功能验证内核配置是否满足 sched_ext 要求并检查 pahole 版本兼容性 CONFIG${1:-.config} # 支持自定义配置文件路径 # 检查配置项函数支持必须启用或禁用的选项 check_config() { local opt$1 local expected_val${2:-y} # 默认检查是否y if [ $expected_val n ]; then # 对于需要n的选项不存在或n都算通过 if ! grep -q ^$opt $CONFIG || grep -q ^$optn $CONFIG; then echo -e \033[32m[OK]\033[0m $opt not set or n else echo -e \033[31m[FAIL]\033[0m $opt is enabled (should be n) fi else # 对于需要y的选项必须明确存在且y if grep -q ^$opty $CONFIG; then echo -e \033[32m[OK]\033[0m $opt enabled else echo -e \033[31m[FAIL]\033[0m $opt not set fi fi } version_compare() { local need_ver$1 local current_ver$2 [ $(printf %s\n $need_ver $current_ver | sort -V | head -n1) $need_ver ] } echo Checking required kernel configs (must be y) check_config CONFIG_BPF_SYSCALL check_config CONFIG_BPF_JIT check_config CONFIG_DEBUG_INFO check_config CONFIG_DEBUG_INFO_BTF check_config CONFIG_SCHED_CLASS_EXT check_config CONFIG_FUNCTION_ERROR_INJECTION check_config CONFIG_BPF_KPROBE_OVERRIDE echo echo Checking conflict kernel configs (must be n or absent) check_config CONFIG_DEBUG_INFO_SPLIT n check_config CONFIG_DEBUG_INFO_REDUCED n echo echo Checking pahole version (need 1.16) if command -v pahole /dev/null 21; then ver$(pahole --version | grep -oE [0-9]\.[0-9]) echo Detected pahole version: $ver if version_compare 1.16 $ver; then echo -e \033[32m[OK]\033[0m pahole 1.16 else echo -e \033[31m[FAIL]\033[0m pahole too old (need 1.16) fi else echo -e \033[31m[FAIL]\033[0m pahole not installed fi echo echo Checking libdwarves_emit.so availability if ldconfig -p | grep -q libdwarves_emit.so; then echo -e \033[32m[OK]\033[0m libdwarves_emit.so found else echo -e \033[31m[WARN]\033[0m libdwarves_emit.so not in standard paths echo Try: export LD_LIBRARY_PATH/usr/local/lib64:\$LD_LIBRARY_PATH fi执行check_config.sh 确保各项检查pass编译后的内核启动时如果卡在Loading initial ramdisk ...用下面的config文件重新编译内核这个文件已经设置了sched_ext相关的配置项。6.15.7_sched_ext.config链接: https://pan.baidu.com/s/1-i5uhL7G5MXcdtnvY8S56w 提取码: htk5三、6.15.7内核编译选项签名相关不关闭签名可能产生下面的错误强烈建议关闭。- SSL error:0480006C:PEM routines::no start line: ../crypto/pem/pem_lib.c:763 extract-cert: certs/signing_key.pem make[3]: *** [certs/Makefile:74: certs/signing_key.x509] Error 1 make[2]: *** [scripts/Makefile.build:461: certs] Error 2 make[2]: *** Waiting for unfinished jobs.... make[1]: *** [/home/geshifei/linux-6.15.7/Makefile:2003: .] Error 2 make: *** [Makefile:248: __sub-make] Error 2CONFIG_MODULE_SIGn关闭模块签名“Enable loadable module support” - “Module signature verification”四、关闭证书linux-6.15.7源码目录执行scripts/config --disable SYSTEM_TRUSTED_KEYSscripts/config --disable SYSTEM_REVOCATION_KEYS五、编译6.15.7内核make -j$(nproc) make modules_install make install grub2-mkconfig -o /boot/grub2/grub.cfg六、重启进6.15.7重启启动项中选择刚刚编译的6.15.7 kernel。本文后续操作都是在6.15.7系统中七、安装工具包编译clang、全功能bpftool需要用到这些工具包。[roothadoop00 ~] yum install valgrind valgrind-devel [roothadoop00 ~] yum install libedit-devel [roothadoop00 ~] yum install elfutils-libelf-devel [roothadoop00 ~] yum install zlib-devel [roothadoop00 ~] yum install binutils-devel [roothadoop00 ~] yum install libcap-develubuntu[roothadoop00 ~] apt install valgrind [roothadoop00 ~] apt install libedit-dev [roothadoop00 ~] apt install libelf-dev [roothadoop00 ~] apt install zlib1g-dev [roothadoop00 ~] apt install binutils-dev [roothadoop00 ~] apt install libcap-dev八、升级clang≥16BPF程序在内核中运行时可以通过 BTF (BPF Type Format) 来获取类型信息这些信息是通过clang编码到.BTF section中的。低版本calng会出现不支持新标记的问题比如btf_type_tag、btf_decl_tag本文将clang升级到17.0.6。1下载及编译clang 17.0.6[roothadoop02 ~] cd /opt 下载代码 官方速度很慢git clone gitgithub.com:llvm/llvm-project.git 国内gitee速度很快git clone https://gitee.com/mirrors/llvm-project.git 国内清华需要排队git clone https://mirrors.tuna.tsinghua.edu.cn/git/llvm-project.git [roothadoop02 opt] cd llvm-project/ [roothadoop02 llvm-project] git checkout llvmorg-17.0.6 [roothadoop02 llvm-project] mkdir build cd build [roothadoop02 build] cmake -G Unix Makefiles -DLLVM_ENABLE_PROJECTSclang;lld;compiler-rt -DCMAKE_BUILD_TYPERelease ../llvm [roothadoop02 build] make -j$(nproc) [roothadoop02 build] make install2检查clang版本[roothadoop02 build]# which clang /usr/local/bin/clang [roothadoop02 build] /usr/local/bin/clang --version clang version 17.0.6 (gitgithub.com:llvm/llvm-project.git 6009708b4367171ccdbf4b5905cb6a803753fe18) Target: x86_64-unknown-linux-gnu Thread model: posix InstalledDir: /usr/local/bin九、安装libbpf一定要用内核源码自带的libbpf编译否则可能有兼容问题。[roothadoop02 ~]# cd /home/geshifei/linux-6.15.7/tools/lib/bpf [roothadoop02 bpf]# make -j$(nproc) make install十、安装libbpftool一定要用内核源码自带的bpftool编译否则可能有兼容问题。需要按照六七节安装工具包才能编译出全功能版的libpftool在编译bpftool时显示Auto-detecting system features: ... clang-bpf-co-re: [ on ] ... llvm: [ on ] ... libcap: [ on ] ... libbfd: [ on ]编译命令[roothadoop02 ~]# cd /home/geshifei/linux-6.15.7/tools/bpf/bpftool/ [roothadoop00 bpftool]# make -j$(nproc) [roothadoop00 bpftool]# make install十一、运行自带的sched_ext调度器a编译[roothadoop00 bpftool]# cd /home/geshifei/linux-6.15.7/tools/sched_ext/ [roothadoop00 sched_ext]# make -j$(nproc)b编译成功后调度器在build/bin目录下。[roothadoop00 sched_ext]# /home/geshifei/linux-6.15.7/tools/sched_ext/build/bin/scx_simple local1 global0 local143 global15 local275 global24 local558 global25 ^CEXIT: unregistered from user space [roothadoop00 sched_ext]#