No stack backtraces or macOS crash reports for crashing plugins

[nickdowell]

If a plugin crashes while being tested by pluginval, all I see in the log is e.g.

pluginval received Abort trap: 6, exiting immediately

Finished validating: /Library/Audio/Plug-Ins/VST3/myplugin.vst3
*** FAILED WITH EXIT CODE: 9

This is not very helpful to a developer trying to figure out where a crash occurred.

I came across commit 4084b3b that made pluginval print stack backtraces if a crash occurs - a very useful feature - but this now appears to be broken.

---

As a macOS developer, I expect to find a crash report in ~/Library/Logs/DiagnosticReports for further investigation.

These reports contain more information than a simple backtrace, and the extra info can be useful in tracking down crashes.

pluginval is preventing Apple's crash reporter producing a crash report by installing a signal handler and offers no way to disable this (IMO harmful) behaviour.

Related:

• Feature request: add an option or a define to generate MDMP or macOS crashlogs after a crash  #120

I see there's a comment about the macOS crash dialog potentially causing the process to hang - maybe are other (better?) solutions to this like configuring Apple's crash reporter to show notifications instead of dialogs? (defaults write com.apple.CrashReporter UseUNC 1)

[drowaudio]

Is your binary built with debug symbols? They'll need these to print stack traces.

I don't think using the system diagnostic is particularly useful UX for pluginval, particularly as it's a cross platform tool, where possible I try to unify behaviour across platforms.

I could be convinced that an option to disable the internal signal handling could be useful though...

---

The most common way of using pluginval is to run it in release builds to validate a plugin and then if there are errors, run it under a debugger to get detailed information of what caused the crash. You can also then use tools like Asan and Tsan for extra info.

[nickdowell]

Is your binary built with debug symbols? They'll need these to print stack traces.

Yes.

By commenting out the call to setupSignalHandling() I get full crash reports with symbols.

The reason no stack traces are printed is that initialiseCrashHandler isn't called, and therefore handleCrash(void*) never gets a chance to log a backtrace.

I've noticed that handleCrash(void*) contains code that is not safe to run in a signal handler (e.g. juce::File::getSpecialLocation) and was almost certainly the cause of hangs when crashing. Maybe I can help provide a safer implementation... I maintained a crash reporting SDK for a couple of years, so have a bit of knowledge in this area including fixing deadlocks in crash handling 😄

Some things that are not safe to use in a signal handler include:

• backtrace_symbols (and therefore juce::SystemStats::getStackBacktrace)
• juce::File
• juce::String

dladdr is not 100% safe either, although hitting a deadlock there is more unlikely than the above which (in)directly use malloc.

---

The most common way of using pluginval is to run it in release builds to validate a plugin and then if there are errors, run it under a debugger to get detailed information of what caused the crash. You can also then use tools like Asan and Tsan for extra info.

99% of the time this is fine, but sometimes you run into crashes that are extremely hard to reproduce (even with the same random seed) or only occur in release builds. In these scenarios being able to get a system crash log from CI would be very helpful.

[nickdowell]

I've started work on a fix for the lack of stack traces, and for the async-signal-safe issue.

https://github.com/Tracktion/pluginval/compare/develop...nickdowell:pluginval:develop?expand=1

Let me know if this is a change that would be accepted as a PR and I'll clean it up a bit more.

[sudara]

sometimes you run into crashes that are extremely hard to reproduce (even with the same random seed) or only occur in release builds. In these scenarios being able to get a system crash log from CI would be very helpful.

I definitely agree that more is better re: context. I often feel like pluginval leaves me at a dead end vs. a "starting place," can feel daunting!

I maintained a crash reporting SDK for a couple of years

Hey cool! I've used bugsnag quite a bit (but on the web).

Let me know if this is a change that would be accepted as a PR

I would be interested. Could you include a bit of description about what's going on? I'd be happy to then try out the branch.

[drowaudio]

I've started work on a fix for the lack of stack traces, and for the async-signal-safe issue.

https://github.com/Tracktion/pluginval/compare/develop...nickdowell:pluginval:develop?expand=1

Let me know if this is a change that would be accepted as a PR and I'll clean it up a bit more.

Yes, I had a quick look and if that gives the desired behaviour and stack traces that would be great if you could contribute it.

Thanks!