Thursday June 26th

Oh my! It was an arithmetic mistake leading to an overflow. Some desperate print debugging last night finally revealed where the overflow was happening. Memory errors are so so weird. The overflow was actually happening when copying out a segment from the shared ringbuffer, not in the places I was looking and which were showing memory corruption.

(A ringbuffer is just a fancy sounding software version of a tape loop that you keep writing over – one tape head to write, another tape head or several others to read.)

A realtime-safe ringbuffer collects live input in the audio thread, and a non-realtime background thread (where it’s OK to block and stall sometimes) copies that into a shared memory ringbuffer, which is what the instruments actually read from when they ask for live input during renders. It’d been scratching my head trying to find problems with copying from the realtime-safe to non-realtime ringbuffer because that’s all new, but the problem was when copying portions out from the non-realtime buffer into a new buffer for the instrument to use. (Yes, there are a lot of copies!) With certain offset values I’d made an off-by-one error which read one element past the end of the new buffer being given to the instruments.

C gets a bad rap for these kinds of problems, which is fair! They’re easy mistakes to make and they can be really difficult to track down – all signs were pointing to either a problem with the shared ringbuffer itself – or even more weirdly – totally unrelated memory from renders being freed in a cleanup thread after playback is complete. Still, I love these kinds of problems because once you find them, like a mysterious but small leak, they’re easy to fix and they stay fixed.

Astrid ran all night without crashing. A relief!

Wednesday June 25th

I didn’t get anywhere with the odd (input ringbuffer? maybe the cleanup job thread?) problem that halts the program unexpectedly last night, but tonight it’s all running fine with slightly short renders and fewer layers. The pacing and density feels nice now, and fingers crossed I can defer actually figuring out that weird bug by keeping things at this activity level. :-)

The bluetooth speakers are back in the mix again - they sound nice inside flowerpots, much more localized in space and resonant. Still just going with six channels now, but two are going to the bluetooth speaker flower pots and four are going to six exciter resonators – two of the channels are just doubled up, I’m literally hooking two exciters up to each amp socket, which I think is OK because these are 4ohm exciters and the amps can handle 8ohms? I don’t really understand what ohms are. Nothing seems at risk of catching fire or anything though lol.

I’d love to spend another few weeks composing to play with the way things move over time… but I think it’s sounding OK and hopefully will be a reasonable balance of stimmy & slow moving for folks who want to meditate, etc. I’m excited that I didn’t have to disable the input ringbuffer! There will be a mic on a little stand with bell percussion, and one to travel around. Some of the computer routines use the mics for feedback, convolution, smearing, etc. The bells really come though, singing whistling etc mixes into everything else much more subtly. I hope people join in as they feel! I plan to make a little zine/flyer to give out encouraging that sort of thing, and giving permission to do things like move around, get close to the speakers, leave, etc. :-) (A little nudge I like getting in situations like these.)

Edit: hilariously (and distressingly) it all just crashed right after I wrote this after running for a few hours this evening. So it is… that means I still need to make a little emergency reconnect script, because it takes a few seconds to make the jack input/output connections and I’d prefer not to fiddle with that in the moment!

Tuesday June 24th

I thought I found a really easy bug that was causing all the ringbuffer problems tonight. For a little while it seemed like I’d just made an arithmetic mistake and caused an overflow while reading from a buffer block used to copy between the realtime and non-realtime ringbuffers. I love those kind of bugs. They’re not always easy to find, but once you do they’re wonderfully easy to fix.

Debugging these issues with python in front of everything is really difficult, but I was able to reproduce what I think is the same intermittent error I’m getting in python scripts with a C test program. It looks like a buffer overflow in the coredump, but when re-running it inside gdb at the point of the overflow… the pointers are at the edge of the buffer but not overflowing…

Getting it to crash again with the ASAN (address sanitizer) tool was interesting. This showed some totally unrelated memory being freed along with the overflow error. I have no idea what’s going on still, but hunches are pointing toward the direction of the async cleanup thread which frees shared buffers having some subtle bug that blows everything up.

Not the kind of bug I want to be debugging days before an event, but I’m really determined to fix this ringbuffer issue if I can since it enables smooth live input processing, and that interactive aspect (people in the space making sounds and having them influence what’s happening in the system) feels kind of important to making the system part of its environment.

Edit: it’s late now, but I just realized I have no synchronization on the cleanup routines! I think they make some assumptions about when it’s OK to free things and maybe there are some rare race conditions. The crashes are so unpredictable that could make sense… at the same time, this might be tricky since the flow of this starts in the realtime thread and I can’t lock. When a buffer is done being streamed it’s moved into a new linked list and that list is periodically swept by another thread to reclaim the memory. Something weird is going on at the edges though, maybe…

There’s not a big chance I’ll be able to do anything proper about this issue this week (like refine the overall async flow in the render scheduler) but I wonder if a workaround could be to just… sweep and free memory less often. I have enough RAM and I’m using small enough buffers generally that I think that could be OK? I’m also realizing as I type that the problem seems to get worse when I use very very large buffers for renders… is the race condition somehow related to the size of the buffer? Maybe the time it takes to copy? I’m not sure!

Monday June 23rd

Once again debugging show-stopping crashes days before the event. Mysterious issues with the input ringbuffer. :-/

I think I’ve decided to ditch the four bluetooth channels and just stick to the six exciters. The bluetooth speakers don’t add much. I thought they’d be a nice transparent compliment to the colorful exciters, but the bamboo lapdesk and cardboard resonators also sound pretty transparent, but a little more interesting. My favorite of the bunch right now is the metal suitcase with a bass exciter – it makes all sorts of cool creaks and rumbles as sounds travel through it.

I’d love to not have to touch the computer at all during the event, but it’s looking like I’ll probably need to restart the system after it crashes periodically… on the plus side without the bluetooth speakers, and switching to long wires instead of the wireless system, I think the batteries are going to easily last the whole day. I’m recharging the amplifiers now but after a full day of testing yesterday they still had ~30% remaining. With the backup battery for the laptop at least power shouldn’t be an issue!

Saturday June 21st

There’s still a number of things that could bring the whole system down haha, but I think it’ll be okay: if that happens, it’s not the end of the world to restart the system. The only things that need to fully last the whole four hours of pridefest are the laptop, the amplifiers the bluetooth speakers and the bluetooth dongle. I guess that’s everything, really.

I’m doing a full test tonight with fingers crossed. My guess is the bluetooth speakers will fail first, which isn’t a big deal. They can be recharged while the rest of the system runs, though I think they’ll make a bloop sound when they turn off haha. I have a backup battery bank for the laptop which might be the next thing to run out of battery, but with the backup I’m fairly confident I’ll be able to get 4 hours even if I’m pushing the laptop a lot? (I’m not, really, but I’ve only tested with 4 to 6 active channels so far, not all 10.)

These battery powered amplifier modules drained overnight when I left them basically idle but on, which isn’t super promising since the amount of power drawn really depends on the signal going through the amplifier, and they drained overnight with no signal at all… but I’m still cautiously optimistic I’ll be able to get 4 hours out of them.

I’m working on chilling out the sounds generated today, keeping in mind the whole point of this is to make a meditation zone where folks can relax and chill out, maybe engage in some mindful listening or activated listening, pick up an instrument to intone a sound, etc. Small, subtle, slow-moving things, or washes of fuzzy sound seem appropriate so I’m trying to dial things more in that direction.

Feeling better about the idea of playing trombone, too, since this is so far from being a performance per se, and it’s basically an opportunity for me to play long tones freely without a practice mute – something I don’t often do at home in the interest of not annoying the heck out of my neighbors. ;-)

Wednesday June 11th

I generally like where the sounds happening are going now, but the pacing and density needs a lot of work. There’s way too much happening! Transient moments of this kind of density would be nice, but this is going to become a bit exhausting if it just hangs out in this zone for the whole multi-hour block at pride.

This is a mono mix of what would usually be spread across 10 channels, through the exciters, resonators and booster speakers.

I’m on vacation this week… planning originally to attend ICMC but I couldn’t hack it in the end, so I’ve got some extra time this week to tune the full system. If I can finish the amplifiers this week I might take a smaller version of the system out to the park a bit early. I’m very curious how it’ll all sound outside, it fills up my apartment pretty well.