I thought I found a really easy bug that was causing all the ringbuffer problems tonight. For a little while it seemed like I’d just made an arithmetic mistake and caused an overflow while reading from a buffer block used to copy between the realtime and non-realtime ringbuffers. I love those kind of bugs. They’re not always easy to find, but once you do they’re wonderfully easy to fix.
Debugging these issues with python in front of everything is really difficult, but I was able to reproduce what I think is the same intermittent error I’m getting in python scripts with a C test program. It looks like a buffer overflow in the coredump, but when re-running it inside gdb at the point of the overflow… the pointers are at the edge of the buffer but not overflowing…
Getting it to crash again with the ASAN (address sanitizer) tool was interesting. This showed some totally unrelated memory being freed along with the overflow error. I have no idea what’s going on still, but hunches are pointing toward the direction of the async cleanup thread which frees shared buffers having some subtle bug that blows everything up.
Not the kind of bug I want to be debugging days before an event, but I’m really determined to fix this ringbuffer issue if I can since it enables smooth live input processing, and that interactive aspect (people in the space making sounds and having them influence what’s happening in the system) feels kind of important to making the system part of its environment.
Edit: it’s late now, but I just realized I have no synchronization on the cleanup routines! I think they make some assumptions about when it’s OK to free things and maybe there are some rare race conditions. The crashes are so unpredictable that could make sense… at the same time, this might be tricky since the flow of this starts in the realtime thread and I can’t lock. When a buffer is done being streamed it’s moved into a new linked list and that list is periodically swept by another thread to reclaim the memory. Something weird is going on at the edges though, maybe…
There’s not a big chance I’ll be able to do anything proper about this issue this week (like refine the overall async flow in the render scheduler) but I wonder if a workaround could be to just… sweep and free memory less often. I have enough RAM and I’m using small enough buffers generally that I think that could be OK? I’m also realizing as I type that the problem seems to get worse when I use very very large buffers for renders… is the race condition somehow related to the size of the buffer? Maybe the time it takes to copy? I’m not sure!