Cabbage Logo
Back to Cabbage Site

Csound and Unity on Android

I’ll wait to hear back from you and Hector, happy to send on your ok. My understanding it that it’s working code, but not a formal release.

Also, I will have some clarifications on my recent question about tuning, relative intonation, sample rate and ksmps after more testing

Ok, I’ll ask and wait :smiley:
Thanks!

Hi, wanted to get back with the results of my testing, (and sorry for the delay)

  1. Overall tuning: Yes, in order to play at correct a=440 pitch on any script in CsoundUnity Android I needed to switch to sr=9600 (or else it plays an octave above). (Agreeing with Bill’s post above) tt appears to be a general artifact of CsoundUnity that the overall pitch/tuning is dependent on the Csound sample rate (on Windows it needs to be 48000), although the ksmps parameter seems to affect the sound quality but not the tuning (I stuck with ksmps=32). To be clear I know Android is not really running at 96000, but perhaps something about the way the sample data is being exchanged with the Unity Update() function is having this effect on tuning and playback.
  1. Intonation: this turned out to be a more narrow issue with the soundfont function sfinstr. It turns out that many (but not all) patches play back out of tune, especially the farther away they get from middle C (midi note 60). Timbres that contain odd harmonics (fifths) are generally unusable. (please note that this did not happen when using direct waveform Csound functions)

Using the widely available soundfont file sf_GMbank.sf2 some of the patches that play very out of tune include

  1. Piano 1 (and most of the pianos)
  2. Nylon Guitar
  3. Fretless Bass
  4. Fantasia String

A few that DO play in tune

  1. EP 1 Layer 2
  2. Organ 1
  3. Pan Flute

Hope this helps, and thanks again for the effort on this

Hi all,
I’m resuming this thread because I finally had time to do some tests with Unity on Android.
I can confirm that it works, my android device produces sound!
That’s great news :smiley:

First of all I am sorry for this long post, but I wanted to be clear explaining the problem I have.
One of my first experiments was trying to read an audio file.
Since on Android files gets compressed inside an apk, reading an audio file from the streaming assets like I was doing on the desktop version doesn’t work anymore.
This is the method I am using to load files:

//send score works also with instruments with string names, 
//but sending stop with i-instrName doesn't work
//so use ints for now
//also, to stop the instrument, this should be started with infinite duration
void SendScoreEvent(int instrNum, bool isStart)
{
    var sign = "";
    var score = "i ";
    if (!isStart)
    {
        sign = "-";
        score += sign + "" + instrNum + " 0 0";
    }
    else
    {
        var path = Path.Combine(Application.streamingAssetsPath, AudioFilesPaths[0]);
        Debug.Log($"Checking path: {path} \nFile Exists? " + (File.Exists(path) ? "true" : "false"));
        //sending score with full path enclosed in escape chars
        score += (instrNum + " 0 -1" + " \"" + path + "\"");
    }
    Debug.Log("sending score: " + score);
    csoundUnity.sendScoreEvent(score);
}

This gave me the ability to start/stop an audiofile playing.
When running on android, File.Exists(path) returns false. This is why I need all of this stuff, here is some Debug.Log of the error:

jar:file:///data/app/com.CSound.CsoundAndroidTest-2/base.apk!/assets/14 - Pray For Me.mp3 
sending score: i 1 0 -1 "jar:file:///data/app/com.CSound.CsoundAndroidTest-2/base.apk!/assets/14 - Pray For Me.mp3" 
INIT ERROR in instr 1 line 30:  
mp3in: jar:file:///data/app/com.CSound.CsoundAndroidTest-2/base.apk!/assets/14 - Pray For Me.mp3: 
failed to open file 

The counterpart in csound is this simple csd:

<Cabbage>
form caption("Read soundfile"), size(300, 200)
</Cabbage>
<CsoundSynthesizer>
<CsOptions>
-n -d -m0d
</CsOptions>
<CsInstruments>

sr = 48000
ksmps = 32
nchnls = 2
0dbfs = 1.0

instr 1

;prints p4 
ibufsize = 2048
ar1, ar2 mp3in p4, 0, 0, 0, ibufsize
;ar1, ar2 soundin p4, 0, 0, 0, ibufsize
outs ar1, ar2

endin

</CsInstruments>
<CsScore>

</CsScore>
</CsoundSynthesizer>

Now since I have to first ‘open’ the apk looking for the files, I was trying to preload all the audiofiles I need in a dictionary with a UnityWebRequest, something like this:

public string[] AudioFilesPaths;
private Dictionary<int, byte[]> _audioFileBytes;

// Use this for initialization
void Start()
{
    _audioFileBytes = new Dictionary<int, byte[]>();
    var count = 0;
    foreach (var fileName in AudioFilesPaths)
    {
        var path = Path.Combine(Application.streamingAssetsPath, fileName);
        Debug.Log($"Checking path: {path} \nFile Exists? " + (File.Exists(path) ? "true" : "false"));

#if UNITY_EDITOR_OSX
        path = "file://" + path;
#endif
        LoadBytesFromPath(path, (bytes) =>
        {
            Debug.Log($"bytes loaded: {bytes.Length}");
            _audioFileBytes.Add(count, bytes);
        });
        count++;
    }
}

void LoadBytesFromPath(string path, Action<byte[]> onBytesLoaded)
{
    StartCoroutine(LoadingBytes(path, onBytesLoaded));
}

IEnumerator LoadingBytes(string path, Action<byte[]> onBytesLoaded)
{
    Debug.Log($"LoadingBytes from path: {path}");
    using (var request = UnityWebRequest.Get(path))
    {        
        request.downloadHandler = new DownloadHandlerBuffer();
        yield return request.SendWebRequest();
        if (request.isNetworkError || request.isHttpError)
        {
            Debug.Log(request.error);
            yield break;
        }
        var bytes = request.downloadHandler.data;
        onBytesLoaded?.Invoke(bytes);
    }
}

And now the real question, since maybe I missed something from the docs:
How to send bytes data from Unity to CSound, and fill some tables to load from them later?
We have a setSample method that doesn’t seem viable, and what I need is something like setTableSample(s), but we only have a getTableSample method.
What am I missing?
Thank you for reading guys!

First of all, thank you for posting this most helpful information! I’m very happy you got it to work. It has been on my list of things to do forever :laughing:

I think a better idea would be to expose the csoundCopyTableIn() method. That would allow us to move blocks of samples rather than setting each one individually. Feel free to have a go, it should just be a matter of adding the function signature to CsoundCsharp.cs and then adding a method to CsoundUnityBridge.cs to access it.

For what it’s worth, I’m working on a game in Unity at the moment where I also have to update the contents of a table. It’s a sequencer component and whenever a user presses a button I need to update that index of a corresponding table. I do it by sending the current index to a named channel. I then trigger an instrument that will read this index and value before calling ftgen to recreate the table with the updates values. It works very well for this kind of simple updating. But would be very slow if I needed to update say 1024 values at at time.

I will have some time on Monday to look into the table stuff if you can’t work it out by then.

Thanx Rory!
I’ll give it a try this afternoon!

Just a quick update!
There was an easier way, copy the audio files in persistentDataPath (where files are not compressed as in the rest of the apk) like you did with csds.
mp3in can find the file and play it! wonderful! :joy:
I also managed to add the functions about tables, I am going to test them but I think that first I should convert the bytes in wav samples.
I’ll continue with my tests and keep this thread updated.
One thing I discovered is that Unity on Android cannot work with bluetooth speakers or headphones… :face_with_symbols_over_mouth:
https://forum.unity.com/threads/bluetooth-headphones-sound-problems.376842/#post-3636838

Do you want to make a pull request for the new methods you added? It would be nice to have it in the main repo.

btw, I was interested to hear your solution to the sound files problem. That’s good to know.

I’m working on it!
I’m adding all the methods about tables for completeness.
I’ll make a pull request as soon as it’s done.

pull request done!
tomorrow I can do some tests :wink:

Nice. I just merged. I’ll try them out when I get a chance.

I’m testing copyTableIn, Unity crashes :pensive:

Sorry, I haven’t got around to testing yet, but I will do tomorrow morning. It’s not so easy to debug these things as a problem in CsoundUnity brings down Unity. Are you declaring your tables in the core section of in the orchestra? I’ve found using table declared in the score section to cause problems before. That’s why I always use ftgen now. Perhaps it’s worth a shot.

Yes I think that is the problem.
getTableLength DOESN’T work and seems it can’t find the table, it always returns -1.
Trying now with GEN10 with 8 samples:

<Cabbage>
form caption("Test"), size(300, 200)
</Cabbage>
<CsoundSynthesizer>
<CsOptions>
-n -d -m0d
</CsOptions>
<CsInstruments>
sr 	= 	48000 
ksmps 	= 	32
nchnls 	= 	2
0dbfs	=	1 

</CsInstruments>
<CsScore>
f1 0 8 10 1
</CsScore>
</CsoundSynthesizer>

If it doesn’t work I’ll try with ftgen.
I’ll keep you posted.

Ok the problem was a very stupid thing I was doing:
I was calling the functions too early, before csound was initialized.
So by now getTableLength, copyTableIn and copyTableOut seem to work correctly!

Sadly I have a Unity crash with getTable and getTableArgs:

public int getTable(out double[] tableValues, int numTable)
{
    int len = Csound6.NativeMethods.csoundTableLength(csound, numTable);
    IntPtr tablePtr = new IntPtr();//Marshal.AllocHGlobal(sizeof(double) * len);
    tableValues = new double[len];
    int res = Csound6.NativeMethods.csoundGetTable(csound, out tablePtr, numTable);
    if (res != -1)
        Marshal.Copy(tablePtr, tableValues, 0, len);
    else tableValues = null;
    Marshal.FreeHGlobal(tablePtr);
    return res;
}

Also tried allocating the pointer (it shouldn’t be necessary), no change.
This is the native method:

 ///Stores pointer to function table 'tableNum' in *tablePtr, and returns the table length (not including the guard point). 
 ///If the table does not exist, *tablePtr is set to NULL and -1 is returned.
[DllImport(_dllVersion, CallingConvention = CallingConvention.Cdecl, CharSet = CharSet.Ansi)]
internal static extern Int32 csoundGetTable([In] IntPtr csound, out IntPtr tablePtr, [In] Int32 tableNum);

csound6Net does it in the same way:

public Double[] Copy()
    {
        Double[] copy = new Double[Length];
        IntPtr unmanagedAdr = new IntPtr();
        int result = NativeMethods.csoundGetTable(m_csound64.Engine, out unmanagedAdr, Fnumber);
        Marshal.Copy(unmanagedAdr, copy, 0, Length);
        return copy;
    }

maybe tomorrow I’ll find what’s wrong in my code :sweat_smile:

Here’s the code I am using to do the tests:

using System.Collections;
using UnityEngine;

public class TestCsound : MonoBehaviour
{
    private CsoundUnity csoundUnity;

    void Awake()
    {
        //assign member variable
        csoundUnity = GetComponent<CsoundUnity>();
    }

    // Start is called before the first frame update
    IEnumerator Start()
    {
        yield return new WaitForSeconds(4);

        var len = csoundUnity.getTableLength(1);

        Debug.Log($"table 1 length: {len}");

        for (var i = 0; i < len; i++)
            Debug.Log($"getTableSample(1, {i}) = {csoundUnity.getTableSample(1, i)}");

        var testValues = new double[8];
        for (var i = 0; i < testValues.Length; i++)
        {
            testValues[i] = i * 1.0d;
            Debug.Log($"Setting test values: {i}: {testValues[i]}");
        }

        csoundUnity.copyTableIn(1, testValues);
        Debug.Log($"copyTableIn: copied {testValues.Length} values in table 1");

        double[] outValues;

        csoundUnity.copyTableOut(1, out outValues);
        Debug.Log($"copyTableOut: retrieved {outValues.Length} values from table 1");

        for (var i = 0; i < outValues.Length; i++)
            Debug.Log($"outValues {i}: " + outValues[i]);

        //csoundUnity.getTable(out outValues, 1);
        //Debug.Log($"getTable: retrieved {outValues.Length} values from table 1");

        //for (var i = 0; i < outValues.Length; i++)
            //Debug.Log($"outValues {i}: " + outValues[i]);

        //double[] args;

         //csoundUnity.getTableArgs(out args, 1);
         //Debug.Log($"getTableArgs: retrieved {args.Length} args from table 1");

         //for (var i = 0; i < args.Length; i++)
             //Debug.Log($"args {i}: " + args[i]);

        Debug.Log("TESTS DONE!");
    }
}

Thanks, I’ll take a look tomorrow. Hopefully between the two of us we’ll figure it out. :wink:

This works fine for me here IF I used ftgen instead of f-statements in the score. Note I was getting some errors with MYFLT so I simply changed them to doubles. Looks like I have been doing this since I started this, so I guess it’s not an issue.

Great!
Indeed I didn’t understand fully the need of MYFLT, maybe is needed for the Android version?
I saw this in CsoundUnityBridge.cs in the Android unitypackage @ceberman sent us:

#if UNITY_EDITOR || UNITY_STANDALONE
using MYFLT = System.Double;
#elif UNITY_ANDROID
using MYFLT = System.Single;

Android doesn’t support double precision?

Ah, right, I will add that back in then! In fact, I better change all instances of double to MYFLT then.

Btw, do you know any other changes @ceberman made for Android? It would be nice to bring them into the main repo…

I just pushed through those updates, I also added you as a contributor on the main github page :wink: