SEGV Thrown when DBD::Oracle built using Oracle >12.2

[ancientwizard]

UPDATED

The SEGV takes place when Perl is handling a SIGCHLD through I suspect other signals will also fault. I'll include a gdb trace later (example). It always happens within Perl but the gdb back-trace shows this happens when it's sleeping within Oracle. The bt will make this clearer.

One of my apps batches many child processes from a queue. The frequency makes this app highly susceptible to the bug.

The latest pull request incudes a test that reproduces this issue.

[ancientwizard]

The SEGV always looks like this within GDB.

Thread 2 "perl" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7ffff7e296c0 (LWP 3577801)]
0x000000000052b64e in Perl_csighandler (sig=17, sip=0x7ffff7e288b0, uap=0x7ffff7e28780) at mg.c:1526
1526         (PL_signals & PERL_SIGNALS_UNSAFE_FLAG))
(gdb) list
1521         sig == SIGBUS ||
1522 #endif
1523 #ifdef SIGSEGV
1524         sig == SIGSEGV ||
1525 #endif
1526         (PL_signals & PERL_SIGNALS_UNSAFE_FLAG))
1527      /* Call the perl level handler now--
1528      * with risk we may be in malloc() or being destructed etc. */
1529 #if defined(HAS_SIGACTION) && defined(SA_SIGINFO)
1530      (*PL_sighandlerp)(sig, NULL, NULL);
(gdb) bt
#0  0x000000000052b64e in Perl_csighandler (sig=17, sip=0x7ffff7e288b0, uap=0x7ffff7e28780) at mg.c:1526
#1  <signal handler called>
#2  futex_reltimed_wait_cancelable (private=0, reltime=0x7ffff7e28d70, expected=0, futex_word=0x26c7b08) at ../sysdeps/unix/sysv/linux/futex-internal.h:142
#3  __pthread_cond_wait_common (abstime=0x7ffff7e28e30, mutex=0x26610c0, cond=0x26c7ae0) at pthread_cond_wait.c:455
#4  __pthread_cond_timedwait (cond=0x26c7ae0, mutex=0x26610c0, abstime=0x7ffff7e28e30) at pthread_cond_wait.c:508
#5  0x00007fffeda2cd63 in sltspctimewait () from /usr/lib/oracle/23/client64/lib/libclntshcore.so.23.1
#6  0x00007fffeeff1ec9 in kpucpincrtime () from /usr/lib/oracle/23/client64/lib/libclntsh.so.23.1
#7  0x00007ffff7bb51ca in start_thread (arg=<optimized out>) at pthread_create.c:479
#8  0x00007ffff6e5d8d3 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
(gdb)

[Leont]

As Tony mentioned in the other threads:

It's only a problem when you mix library worker threads with signals.

Your backtrace:

#6  0x00007fffeeff1ec9 in kpucpincrtime () from /usr/lib/oracle/23/client64/lib/libclntsh.so.23.1
#7  0x00007ffff7bb51ca in start_thread (arg=<optimized out>) at pthread_create.c:479
#8  0x00007ffff6e5d8d3 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

My advice: make smart use of sigprocmask. The worker threads inherit their mask from their parent at creation time, but unlike the signal disposition it's not shared between threads. So you want to do something like:

use POSIX qw/sigprocmask SIGCHLD SIG_BLOCK SIG_SETMASK/;
my $new = POSIX::SigSet->new(SIGCHLD);
my $old = POSIX::SigSet->new;
sigprocmask(SIG_BLOCK, $new, $old);
start_threads();
sigprocmask(SIG_SETMASK, $old);

That way the SIGCHLD will not land on your worker thread, so it won't escalate to a SIGSEGV.

I have no idea how they start, but I'm guessing your start_threads would be something like require DBD::Oracle;.

[ancientwizard]

I'll certainly give this a try unfortunately the worker thread set is dynamic. The threads aren't created by simply loading DBD::Oracle. I see one new thread after a DB connect and it's closed. Apparently other threads are created by other OCI functions I have yet to discover (prove) though copilot insists they are created as needed on demand for various features. I figure this masking can't be a once and done. It may need to wrap all OCI functions that may create threads. -Yuck assuming this is the only Or best solution.

[Leont]

Alternatively, you could also have it blocked by default, and only lower the mask in some (preferably common enough) codepath where you are sure no new threads are started. You might find my Signal::Mask an easier interface to work with than raw sigprocmask.

Instead of lowering it you could also use sigtimedwait to explicitly consume a signal.

[ancientwizard]

I do not believe there is a reliable in Perl code Band-Aid for this issue, If I didn't mention this above this is more systemic than Oracle. All libraries bolted on to Perl that create and join threads outside of Perl's controls is subject to this SEGV. Example: libcurl also creates thread(s); perhaps its more predictable and such a band aid would work; not in my work-load.
They have authored a patch applied to "bleed" that may make it into a future Perl. I've manually applied the Patch to a Perl source and built it, in the office. It does avoid the SEGV.

It does so by:

1. Testing if the thread is a Perl-thread; if so continue normally.
2. If not it forces a Perl(context/interpreter) upon it and then lets it continue.
   ( this however could be risky by having two threads bumping around in the same part of Perl )
   hopefully the code set is short and sweet; a risk all the same. It did work well in my tests and application. I'll continue to test more over the coming week or two.
   UPDATE: While I can't say it's perfect it's better than a SEGV. Over the last week I believe it is reliable "enough"; as in no fallout detected thus far.

The patch applied into bleed.

if (!my_perl) {
  my_perl = PL_curinterp;
  PERL_SET_THX(my_perl);
}

There is an alternative, very similar to the above, which I have yet to SAND & TEST; Handling any error conditions needs to be handled in a Perl way. It will smell something like this. I've also tested this alternative, and it seems just as reliable.

if (! my_perl)
{
    /* Mask SIGCHLD for this thread
     * (we don’t want to see this thread here ever again!)
     */
    sigset_t set;
    sigemptyset(&set);
    sigaddset(&set, SIGCHLD);

    if (pthread_sigmask(SIG_BLOCK, &set, NULL) != 0) {
        perror("pthread_sigmask");
        return;
    }

    /* Re-raise the signal for another thread to handle */
    if (raise(SIGCHLD) != 0) {
        perror("raise");
    }

    return;
}