Oh boy, more leak fixes this weekend!

• Forgetting to sem_close some of the sempahore guards on sampler memory was leaving file descriptors behind in a weird zombie DEL state (when looking at them in lsof) and eventually the hard limit of per-process file descriptors would get hit and everything would grind to a mysterious error-free halt.
• I wasn’t actually acquiring / releasing the semaphore when creating new shared memory blocks in the sampler! That’s fine for long-lived memory like the instrument ringbuffers, but after a while (between ~2 and ~10 minutes depending on the number of python render processes) luck would align the lpsampler_create calls and blow the process up.
  
  My initial fix was to add a foreground cleanup routine: using the async API for linenoise with a select() timeout yields the main console thread back to loop over the python processes and replace any that have become zombies or otherwise failed.
  
  The real fix was to just acquire the sempahore lock as soon as it’s opened in the lpsampler_create routine, to synchronize with any other processes that are still in the middle of reading from it.

I won’t claim there are no more leaks… (AddressSanitizer complains of some on shutdown, at least, so there must be some cleanup issues still) but running the python and C instruments for a very long time using the sampler & resampler interfaces is actually stable!

I was getting a bit worried at all the mysterious crashes. Debugging concurrent python programs is just as annoying as microcontroller debugging without any serial or hardware debugger interfaces: things just stop working silently and mysteriously…

Using coredumpctl list after a crash seems like a much easier way to get some clue than trying to dig through the system logs. I’m dumping way too many messages into the system log haha. Anyway, after seeing a new crash in the coredumpctl list (which lists the PID for the crashed process too) it’s possible to get a backtrace with actually useful info, even after the program has been killed, by using coredumpctl gdb <PID>.

I’ll be able to spend tonight and tomorrow (I decided to take an extra day of vacation for prep) on instrument design and starting on a realization score for one of the more complex pieces we’ll be trying in Littlefield next week…

I wish I was further along in instrument building, but I’m (knock on wood!) feeling pretty good about being able to rely on these new astrid interfaces to refine instruments during recording.

I’m going to do my best to get a set of base instruments carved out from all the tests, though.

There’s one more bit of plumbing to test, which is to have instruments sample from external resampler ring buffers. I’d like to have one sampler instrument which acts a bit more like a loop pedal of some kind, and can sample from the outputs of arbitrary instruments…

Running multiple instances of instruments is also untested after all the messaging changes. I’m expecting weird behavior so the plan is to just run one instance of any given instrument, and rely on the (now more robust) render pools for polyphony, or build it directly into the stream callback for C instruments.