Cabbage Logo
Back to Cabbage Site

[CsoundUnity] Dynamically create Csound audio channels

Hello!
I’m trying to achieve the same behaviour of a CsoundUnityChild, but in a dynamic way, meaning that I don’t know the Csound audio channel name in advance. So I want to create multiple instances of the same instrument in different AudioSources, so that I can spatialize them separately but still update their content from the same CsoundUnity instance.

I’m creating each instrument with this C# code:

    private void CreateInstrument(int instrNumber)
    {
        // this creates the GameObject in the 3D world
        var instr = Instantiate(_instrumentPrefab, this.transform);
       
        // here I'm saving how many instances have been created, to correctly handle the channel name
        if (_instrumentInstanceCounter.ContainsKey(instrNumber))
        {
            _instrumentInstanceCounter[instrNumber]++;
        }
        else
        {
            _instrumentInstanceCounter.Add(instrNumber, 1); // lets start from 1 here, so the channel will be 1.1 aka instr 1 instance 1
        }
        var key = instrNumber;
        var value = _instrumentInstanceCounter[key];
        // a single instance will then update two audio channels, left and right
        // key is increased by one since the instruments start from 1 in Csound
        // we could have used instruments[instrNumber].number
        instr.Init(this._csound, $"chan{key + 1}.{value}L", $"chan{key + 1}.{value}R", instruments[instrNumber]);
    }

where _instrumentInstanceCounter is a Dictionary<int, int>, and instruments is a List <InstrumentData>, where I store all the parameters I want to pass to the instruments on creation.
This is the InstrumentData struct, and its parameters. I will want to randomize the values of the parameters between min and max.

[Serializable]
public struct InstrumentData
{
    public string name;
    public int number;
    public List<Parameter> parameters;
}

[Serializable]
public struct Parameter
{
    public string name;
    public float min;
    public float max;
}

The important bit is in the init method, where I am passing the strings of the names of the audio channels I want this instrument to write to.

So in my csd each instrument is not outputting directly to the output, but on an audio channel, for example like this:

   SchanL = p10
   SchanR = p11
   
   chnset a1, SchanL
   chnset a2, SchanR

This is the Init method of the class that will act like a CsoundUnityChild:

    public void Init(CsoundUnity csound, string chanLeft, string chanRight, InstrumentData data)
    {
        _chanL = chanLeft;
        _chanR = chanRight;
        _zerodbfs = csound.Get0dbfs();
        _csound = csound;

        // creating the buffers that will hold the data from Csound.GetAudioChannel
        _bufferL = new double[_csound.GetKsmps()];
        _bufferR = new double[_csound.GetKsmps()];

        var parameters = new List<string>();
        foreach (var p in data.parameters)
        {
            var value = Random.Range(p.min, p.max);
            parameters.Add($"{value}");
        }

        var score = $"i{data.number} {0} " + string.Join(" ", parameters) + $" \"{chanLeft}\" \"{chanRight}\"";
        Debug.Log($"score: {score}");
        _csound.SendScoreEvent(score);

        _initialized = true;
    }

This generates scores like this:

score: i1 0 5 0 11.97545 497.9954 0.415536 8.615604 0.6595364 "chan1.1L" "chan1.1R"

And this is the OnAudioFilterRead method that will write the samples on the newly created Unity AudioSource:

    private void OnAudioFilterRead(float[] samples, int numChannels)
    {
        if (!_initialized) return;
        for (int i = 0; i < samples.Length; i += numChannels, _ksmpsIndex++)
        {
            if ((_ksmpsIndex >= _csound.GetKsmps()) && (_csound.GetKsmps() > 0))
            {
                _ksmpsIndex = 0;
                _bufferL = _csound.GetAudioChannel(_chanL);
                _bufferR = _csound.GetAudioChannel(_chanR);
            }

            for (uint channel = 0; channel < numChannels; channel++)
            {
                samples[i + channel] = channel == 0 ? (float)_bufferL[_ksmpsIndex] : (float)_bufferR[_ksmpsIndex];
            }
        }
    }

Everything looks correct to me, but it sounds really bad, like this:


This should be 5 seconds of a stereo sine at 440 Hz.
Notice the final part where there shouldn’t be sound since the instrument has stopped playing, but there’s an ugly digital noise instead. So I will have to handle this also, when the instrument stops updating the channel there will be noise/garbage data in the channel, and I shouldn’t update the AudioSource anymore.

I really hope this setup makes sense and it’s not a thread desync issue :confused:
Any idea?

P.S. I had to modify the last lines of CsoundUnity.ParseCsdFileForAudioChannels like this:

// discard channels that are not plain strings, since they cannot be interpreted afterwards
// "validChan" vs SinvalidChan
if (!split[1].StartsWith("\"") || !split[1].EndsWith("\"")) continue;

var ach = split[1].Replace('\\', ' ').Replace('\"', ' ').Trim();
            
if (!locaAudioChannels.Contains(ach))
   locaAudioChannels.Add(ach);

because I don’t want those SchanL and SchanR to be recognized as channels, since they are Csound string variables instead :sweat_smile:

I can’t see anything obvious here. The routine looks good to me. Have you tried writing just _bufferL[] to samples[] to see if that is Ok? If so, could there be something up with the GetAudioChannel() routine?

The pitch sounds an octave above 440?. So it would appear that samples are being skipped, but I don’t see how tbh.

I tried with both channels, same result.
Btw this is the frequency analysis of the recording above:


maybe it can suggest something :thinking:
but not sure where 694 Hz and all those harmonics could come from :sweat_smile:

I think the harmonics are a good indicator that the waveform is mangled some way. I assume all tests of the audio channels in Csound produce consistent results when output to disk?

I did some investigation. No findings except that the issue is strictly related with the ksmps I set in the csd.
Have a look at this video where I go from ksmps = 1 to 256. The number of harmonics increments accordingly.

So my assumption is that I’m calling GetAudioChannel at the wrong time. Is there a Csound API method (maybe some callback) to sync with the ksmps calculation of the master Csound instance?

@giovannibedetti Are you calling GetAudioChannel() on each k-boundary?

Yes, but from a second script, that is not running in the same OnAudioFilterRead of the main CsoundUnity instance.
So I have no way of knowing when the ksmps block is computed. I’m just calling it every ksmps samples, but that could happen anywhere in between of the computation.
There is no way of synchronizing two separate OnAudioFilterRead methods in Unity (not that I know of), so I guess the Yield callback could help here probably.

For context:
For CsoundUnityChild we were smart and we are updating them from the main CsoundUnity instance, so each child is just reading from a precomputed buffer (with the Unity buffer length, not ksmps length). In that way they are synced.
But here I don’t know the name of the audio channel in advance, so I cannot do it from the main CsoundUnity instance

Yeah, I see the issue. But I’m not sure the yield callback will be of any help here because of the syncing issues you describe?

If that callback is executed straight after the ksmps computation then maybe it could work.
Otherwise my next attempts are:

Well maybe there’s an easier solution, which is adding a callback myself when my CsoundUnity instance executes PerformKsmps :man_facepalming:

Edit:
Just tried, this, no luck. So I guess that having a Csound callback for this purpose will have the same effect. :thinking:

A less elegant solution would be to define a fixed number of audio channels on init time and tap into them as needed?

Like a pool, yes, that’s an interesting option.
But I will have to edit the csd to hardcode the channels in advance, and those will be always updated, even if currently empty. But it’s definitely an option if everything else fails :cold_sweat:
As I am starting to believe that the issue lies within Unity…
I tried with a ksmps of 1024, that’s the buffer length in Unity, and things start to play as expected (but still with some noise), look what happens with ksmps 1024, 2048 and 4096 (it should be a 440hz sinusoid):

I quickly tried another approach (I am supposed to work atm but I’m too curious)
Using the callback and filling a bigger buffer with the same length of the Unity buffer.
Something like this:

        void OnCsoundPerformKsmps()
        {
            //Debug.Log($"Perform");
            _bufferL = _csound.GetAudioChannel(_chanL);
            _bufferR = _csound.GetAudioChannel(_chanR);
            var start = _ksmpsIndex;
            _ksmpsIndex += (int)_ksmps;

            if (_ksmpsIndex >= _UnityBufferSize - _ksmps)
            {
                start = 0;
                _ksmpsIndex = (int)_ksmps;
            }

            for(var i = 0; i < (_ksmpsIndex - start); i++)
            {
                _fullBuffer[i + start] = i % 2 == 0 ? 
                    _bufferL[i] / _zerodbfs : 
                    _bufferR[i] / _zerodbfs;
            }
        }

This is invoked everytime Csound performs a ksmps block.
Still some noise but it could be caused by the bad wrapping I made.
will try again later with better wrapping :wink:

Progress :smiley: regarding the pool idea, you could compile an instrument a i-time that will insert the channels into any exciting orc. Regarding the 1024 buffer size in unity, are your sure the buffer size is always the same? Many systems end up with variable sized processing blocks, which might explain the noise?

It depends on the number of channels, but usually yes it’s consistent at 1024 (at least at 44100).
so with two channels I have to multiply the buffer length by 2.
Though I will do some investigations on this

1 Like

I’m very close but still not there. This is the current status:

Apart from the clicks I have when I stop updating the samples and listening for the callback (and then destroying the instrument), there is still a noise I can’t remove.

Now the C# code looks like this:

        void OnCsoundPerformKsmps()
        {
            if (!_readingFromCsound)
            {
                _csound.OnCsoundPerformKsmps -= OnCsoundPerformKsmps;
                _canBeDestroyed = true;
                return;
            }

            _bufferL = _csound.GetAudioChannel(_chanL);
            _bufferR = _csound.GetAudioChannel(_chanR);

            var start = _ksmpsIndex;
            _ksmpsIndex += (int)_ksmps;

            if (start > (_UnityBufferSize * 2) - _ksmps)
            {
                start = 0;
                _ksmpsIndex = (int)_ksmps;
            }

            //Debug.Log($"[{AudioSettings.dspTime}]: Perform {start} {_ksmpsIndex}");

            for (var i = 0; i < _ksmps; i++)
            {
                _fullBuffer[i + start] = i % 2 == 0 ? 
                    _bufferL[i] / _zerodbfs : 
                    _bufferR[i] / _zerodbfs;
            }
        }

        private void OnAudioFilterRead(float[] samples, int numChannels)
        {
            if (!_initialized) return;
            if (AudioSettings.dspTime - _startTime >= (_totalDuration - _safeInterruptionTime))
            {
                _readingFromCsound = false;
                return;
            }
            for (int i = 0; i < samples.Length; i += numChannels)
            {
                for (int channel = 0; channel < numChannels; channel++)
                {
                    samples[i + channel] *= (float)_fullBuffer[i + channel];
                }
            }
        }

        private void Update()
        {
            if (_canBeDestroyed)
            {
                Destroy(this.gameObject);
            }
        }

:thinking: :face_with_monocle:

Further thoughts on this problem.
With the above code, I’m basically skipping every ksmps block (let’s assume of length 32) 16 samples from the left channel and 16 from the right.
So I tried filling 64 samples but the result is definitely worse, so that is not the way to go.
Probably I could try mixing the stereo channels in a single audio channel in Csound, and then write data with chnset that would be already interleaved.
But I’m not sure if an opcode like “chnmix” would do the trick, it looks like it will produce mono signals…
Will try later after work :wink:

It still sounds as if there are samples being dropped :thinking:

I found a way to achieve this behaviour, the solution will be implemented in the next version (3.5) of CsoundUnity.
I went for this: instead of using a callback or GetAudioChannel, I’m giving the user the possibility of adding/removing audio channels at runtime (on the C# side of things), so that they are all updated together (we already had those buffers for the CsoundUnityChildren, so I’m just adding / removing new buffers on the fly).
This seems to work great, I get no clicks and the sound is exactly as it should be.

So the user can use those calls to add / remove / read from an audio channel:

CsoundUnity.AddAudioChannel(channelName)
CsoundUnity.RemoveAudioChannel(channelName)
CsoundUnity.namedAudioChannelDataDict[channelName][sampleIndex]

I am also thinking of wrapping the third line in a more readable:

GetAudioChannelSample(channelName, sampleIndex)

I am still not sure if those methods will be confused with the native Csound call (GetAudioChannel), when they are not: all the buffers here (and the addition/removal operations on them) live on the C# side, even though they are updated from Csound.

I will add some detailed summary maybe :wink:

So party time! :partying_face:
Lots of new goodies are coming (I don’t have an ETA for v3.5 though :sweat_smile:)

great work @giovannibedetti! these new methods look really great. i must take a look over the source code to see how you managed this :+1: