Database startup crashes with Segmentation fault (exit code 139) during early initialization

[cms-cms]

Environment

• Build ID: 124847-eeddc17ae5df8e1d1d1e6ba28dcc4edebe43bdc4
• File Size: 2829058312 bytes
• SHA256: 3ac42f93fb621ed60ea744ae593495d1048579a700dc03342f851486ccd51670
• Previous Working Build: 124799/6a070199 (works fine on similar machines)
• Command to Reproduce: ./observer --version
• Result: Segmentation fault, exit code 139

Description

The observer binary crashes with a segmentation fault during the --version command execution, before entering the main() function. The crash occurs during early initialization.

Steps to Reproduce

1. Download the specific build using the provided script:

mkdir -p ./bin

wget \
  https://work.oceanbase-dev.com/obartifact/api/v1/archive/ob_compile/oceanbase-lite/master/el7/x86_64/sanity/latest_product \
  -o ./wget_observer.log \
  -O ./bin/observer

chmod +x ./bin/observer

grep 'Location:' ./wget_observer.log
sha256sum ./bin/observer
./bin/observer --version
echo "exit_code=$?"

2. Execute ./bin/observer --version.
3. The process crashes with a segmentation fault (exit code 139).

Logs & Core Dump

• Log Archive: /obdata/data/test0710/observer_crash_sqaappnoxdnv62s2011161204175.sa128_20260710_151550.tar.gz
• Core File: /obdata/data/core_bak/core-observer-3266851-1783667636

Stack Trace Analysis

__libc_csu_init
early_disable_hugepage_for_observer               main.cpp:401
disable_hugepage_for_self_text                     main.cpp:379
fopen
observer custom malloc hook                        malloc_hook.cpp:130
ob_malloc / ObTenantCtxAllocator
AObject::set_label                                 alloc_struct.h:338
Sanity overridden strncpy                          sanity.cpp:89
dlsym
_dlerror_run
SIGSEGV, accessing address 0x19

Root Cause Analysis

The crash occurs before the program enters main(). The sequence is:

1. Early hugepage initialization (early_disable_hugepage_for_observer) calls fopen.
2. fopen triggers the custom malloc hook.
3. During malloc initialization, it calls Sanity's overridden strncpy.
4. strncpy uses dlsym to look up the real function.
5. The crash happens inside the dynamic loader (_dlerror_run).

The likely cause is an initialization order problem or a malloc/sanity hook reentrancy issue, where early constructors trigger hooked malloc, which recursively enters dlsym.

Code Locations to Investigate

Focus on changes near:

• src/observer/main.cpp:379 / 401 (disable_hugepage_for_self_text, early_disable_hugepage_for_observer)
• src/sanity/sanity.cpp:89 (Sanity overridden strncpy)
• malloc_hook.cpp:130 (observer custom malloc hook)

The goal is to avoid recursive entry into dlsym via hooked malloc during the early constructor phase.

Additional Context

• Reproducibility: Consistent (必现)
• Test Changes: The issue appeared after replacing the version. The previous build (124799/6a070199) works normally on similar machines.
• Environment Anomalies: None reported.
• Is this an RD-provided temporary version? No.
• Type of Issue: This appears to be a product bug introduced in the new build.

[hnwyllmm]

dima personal assistant added

Issue identified and fixed: The newly added pre-main constructor (101) was calling fopen to scan /proc/self/maps before main, which triggered a malloc/Sanity/dlsym reentrancy and caused a SIGSEGV. The fix moves disable_hugepage_for_self_text to the first Linux statement in main, preserving the PR_SET_THP_DISABLE/MADV_NOHUGEPAGE behavior. Commit 3528f109b5a has been pushed to branch issue/2026071000117337332. The ob-make seekdb and --version checks pass. In two rounds of empty-load A/B testing for a standard release, the RssFile for both was around 117MB, with a comprehensive difference of about 0.16%, showing no stable memory benefit. Both sides show FilePmdMapped=0 and THP_enabled=0. Integration verification with the xSanity package is pending.