Adding a set of standard Csound-based DSP Components to the next version of Csound Unitynts to

[Dr_Richard_Boulanger]

Dear Rory and Giovanni,

I have a group of incredible students here at Berklee who are working with Csound Unity.
Later this summer, we will have much to share on this forum and especially to show, and perform at the International Csound Conference in November! Thank you for all your work to make this research and development possible. We are pushing hard now to get our installation proposals ready and papers drafted for the upcoming submission deadlines.

Still, as we are going along a recent thought occurred in our discussions and developments.

As you know, Unity has some great components that one can attach to an object to deal with reverb, space, and 3D aspects of an object. As we were developing our unique collection of CsoundObjects for Unity (and adding to them and modifying them and exploring them) is that there are certain behaviors or processes that one might like to be able to attach to any or all CsoundObjects along the way rather than rewriting the Csound instruments to ‘include’ these DSP features for each one.

Maybe as a part of the Csound Unity 4.0 package…

For instance, it would be great to have a Csound-based REVERB (freeVerb, screverb) ‘component’ to add to this our that Csound Sound Object (from the Csound Catalog Collection for instance)

Or, it would be great to have a Csound-based HRTF component to add to this or that Csound Sound Object

Or, it would be great to have a Csound-based DOPPLER component to add to a CsoundObject

Or, a SPAT3d
Or, a VBAP
Or, a Spatial Volume & EQ - with distance calculations

Or, to make things really simple for beginners, a Csound-based Sample Player (mincer) component

Currently, I add the features to individual instruments. And, I am getting ready to make a set of Global instruments (or UDOs) that I can drop into my instrument to add these features to the soundObjects, and I can standardize my instrument models to use them, control them, modify them with gestures, etc. (and that is what I am doing now).

But, I am thinking of UNITY developers who are not Csound experts and just want to add this sonic ‘intelligence’ to their sounds (from a Csound Unity Collection of Instruments, NoiseMakers, and Sample Players perhaps)

For expediency, I could have my Csound Instruments use some of the UNITY components that do this already, and I am exploring that.
But it would be cool to have everything in the Unity Sound World be Csound.

What do you think about adding spatial, reverb, etc components to the next release of Csound Unity so that you show a general and standardized way to support adding Csound DSP capabilities to UNITY for all Immersive Music Experience developers.

Dr.B.

[giovannibedetti]

Hello Dr.B!

I’m glad that you and your students are experimenting with CsoundUnity, I can’t wait to try out all the amazing experiences you are working on!
I’ll surely attend the Csound Conference in November, I’d be more than happy to discuss further developments of the package.
We could hold a workshop there maybe? So that it’s official and also other people can attend, and we can collect suggestions from a diversified audience.
It would be nice creating a document together to plan the development milestones of the CsoundUnity project.

About this, I think that an improved implementation of Csound on Unity is possible, by using the Unity Native Audio Plugin SDK.
I haven’t explored this SDK yet, but it should be the way to go to create Csound based DSP effects that can be used out of the box in the Unity Audio Mixer.
That will open lots of possibilities, chaining of audio channels will be way easier, but most of all the performances will be great.
At the moment the current implementation of CsoundUnity consumes some cpu passing audio data from Csound to the C# Unity side (to fill the samples of the AudioSource), which we can probably skip (mostly) entirely.

While waiting for the Csound Conference I’ll try to dig a bit into the SDK: I will start with AudioMixer effects (they could coexist with the current version) and then I’ll try to do some prototypes for the next implementation of CsoundUnity, to see how much work it involves.

Looking forward to hearing from you and your students soon!
Take care

[rorywalsh]

FWIW, I did build a simple see sound need of audio plug-in in unity some years back. I decided not to follow through with it because it couldn’t deliver the type of score based event driven modularity that Csound can. But if we stick with DSP effects then it would be perfect.

[giovannibedetti]

Good to know!
Is the code accessible somewhere? It would be nice to have a look!
I’m wondering if DSP effects could receive score events and in general how much of the Csound API we can use

[rorywalsh]

They can work just like an affect on Cabbage. I don’t think I even have the code for this experiment anymore but it shouldn’t be that much work to do… when one has time to do such work!

[giovannibedetti]

Hehe that’s exactly the point!
But I’ll try to find some time because I’m curious!

[mateolarreaferro]

Hello Giovanni and Rory!

I’m Mateo, one of Dr. B’s students working on CsoundUnity. For my thesis, a group and I are creating a series of ‘examples’ and ‘utility’ classes that showcase the power of the the wrapper in the simplest possible way. It’s a truly inspiring area - your work has opened so many possibilities!

My team and I will be more than happy to collaborate with you in any way we can… Let’s get in touch soon! My email is mlarrea@berklee.edu - I would love to discuss what we’re doing with you and hear more about your plans.

Take care and thanks!

Mateo Larrea

[giovannibedetti]

Hi Mateo!

I’m so glad if CsoundUnity is being useful, but most of all I’m happy you’re creating examples and utilities for it! That’s what we need
It would be awesome if you could also create some videos: I wanted to add a “Made with CsoundUnity” section on the github page / website, but I have just a few videos to put there, so I’m waiting to collect some more.

About the development plans:

• I’d like to add Presets (a minor update): it has to be easy to save the Csound Channel settings, and also to save/load them at runtime, see this thread on the Csound Forum

• I’ll start investigating how we can use the Unity Native Audio Plugin SDK to create Csound based audio effects. This could open new development plans into a more organic setup. I’m not sure how, but in my thoughts I imagine a single Csound instance where we can add instruments on the fly, connect their output (audio chains), and choose the AudioSource that will play each audio channel.
  As Rory said, we should try and maintain the score based event driven modularity, so this requires some research/discussion on the architecture to design and implement.

• Investigate how we could make it easier for Unity devs to use CsoundUnity without knowing the Csound language.

• It would be nice to distribute the CsoundUnity package on the Unity Asset Store, and possibly create a showcase app to upload to the Google PlayStore, the AppleStore, and the Oculus store (all for free of course). We need to show how powerful Csound is to gain a stronger user base.

• Investigate if we can support the WebGL platform: we have Csound running on the web but on Unity we cannot use the same approach of the other platforms, because Unity audio support on WebGL is very basic: it doesn’t support the OnAudioFilterRead callback, that is where the passing of data from Csound and Unity happens. But it shouldn’t be too difficult to setup a messaging system to talk to CsoundWeb. We would lose the spatialization because audio won’t come from the AudioSources, but probably there are workarounds to obtain the same behaviour using Csound opcodes.

• Rewrite the CsoundUnity editor to use the UIElements, instead of IMGUI. UIElements are more performant, based on web languages and easier to maintain. You could have notice a high cpu usage on the editor when having lots of Csound channels drawn in the inspector. Implementing Cabbage GroupBoxes with the existing code could be a quick workaround to reduce cpu consumption (so that you can show/hide a subset of channels).

• Another area I’d like to explore is having Csound in Unreal MetaSound. I think Hector is working on it (https://forum.csound.com/t/csound-and-metasounds-in-unreal-engine/394), it would be nice giving him a hand

• Feel free to add any thoughts / ideas I have not covered in this list, these are the first things that came to my mind, but surely there could be more.

Also feel free to do pull requests on the CsoundUnity repo with your examples, I’ll review them!
Or if you prefer you can send me some .unitypackages and I can add them myself to the repo.

Only consider that I have a fulltime job during the week so sometimes it could take a while before I jump into it

I’d be glad to hear your thoughts and discuss what you’re doing, and, if you will attend, it would be nice if you could do a presentation at the Csound conference in November. I’ll try to help the more I can.
Of course feel free to ask if you need some Unity/C# support.

I’m very curious to see what you guys are working on!
Let me know!

[mateolarreaferro]

@giovannibedetti

I’d be glad to hear your thoughts and discuss what you’re doing, and, if you will attend, it would be nice if you could do a presentation at the Csound conference in November. I’ll try to help the more I can.
Of course feel free to ask if you need some Unity/C# support.

Dear Giovanni,

Thank you for your thoughtful response! A lot of very interesting ideas.

• Videos: Our intention is to create a series of video tutorials and demos once we have a decent amount of examples (probably between now and October). It would be fantastic to share these with you for the “Made with CsoundUnity” section. Before that, I’ll definitely share the examples and scripts as we create them. Ultimately, it would be fantastic to include them in the CsoundUnity repository.

• Presets: Caio, from the team, created a preset system for channels using scriptable objects - maybe this is something we can discuss when we meet. It really makes the workflow a lot friendlier.

• Putting CsoundUnity in the asset store: this would be a beautiful thing - Chuck (Chunity) does the onboarding like this and it is very smooth → Chunity. Let me know if we can help with that in any way.

• We will think more about some of the other points that you mentioned!

I agree it would be really nice to make a presentation for the conference. We’re writing the paper now, would you be interested in making some space in the workshop so that we can show some of the examples?

Again, thank you so much for your response and for your work

Mateo

[giovannibedetti]

Hi Mateo,

It would be great if you could send me the preset implementation, so that I can understand what is the best way to go (I have a partial working implementation without scriptable objects but I’m open to changes).

I’d like to give the final user the possibility to create presets at runtime. So yes, we can use SOs as the data container (and add a custom inspector so that they will show sliders and other widgets like in the CsoundUnity inspector), but we will have to save them as JSONs at runtime in the persistent data path. Is this already supported in Caio’s implementation?

About the Stores upload: I will ask on the Csound discord chan how to handle this, I’d like to use an official Csound account. Then we can make plans

About the workshop:

I can make the proposal if there’s interest.
I’ll try to draft a list of the proposed activities in the next days.

Keep in touch!

[caiojmini]

@giovannibedetti

on

Hi Giovanni!

So, my presets implementation is very crude (I’m not touching the Cabbage preset system at all). Basically we have a scriptable object with a struct that holds a channel name as a string and values as floats - the SO defines a public array of that struct so channels and values can be added as necessary through the inspector and accessed by other scripts (the fixedValue variable are being used for presets here)

[CreateAssetMenu(fileName = “CsoundChannelData”, menuName = “Csound/ChannelData”)]
public class CsoundChannelDataSO : ScriptableObject
{
[System.Serializable]
public struct CsoundChannelData
{
public string name;
public float fixedValue, minValue, maxValue;
}

public CsoundChannelData[] channelData;

}

Then we have another class (which we are calling CsoundSender for now) can hold an array of the Csound Channel Data assets and access them with functions like this:

public void SetPreset(int index)
{
    foreach (CsoundChannelDataSO.CsoundChannelData channelData in presetList[index].channelData)
    {
        //Passes values to Csound.
        csoundUnity.SetChannel(channelData.name, channelData.fixedValue);
    }

    //Set current preset index to the passed in index.
    currentPresetIndex = index;

}

I’m also a Dr. B adjacent undergrad at Berklee along with Mateo and second his feelings about your and Rory’s work on CsoundUnity, it’s really amazing!

[giovannibedetti]

Hi Caio!

Ok this is a good starting point, but to my understanding you have to create each value by hand, and it could take some time for a csd with many channels.

What I’m doing is gathering the serialized channels from a specific CsoundUnity component.

I have to use editor scripts for that (so I can do it out of play mode) but it’s not that difficult.

There will have to be some sort of checking when applying saved presets, I will add some warning message if some of the channels are not found.

I’ll try to find some time this week to bring it to a decent state, and push it on a separate branch where it can be tested.

I’ll keep you updated!

[giovannibedetti]

Hi!
First draft of presets available on the branch:

GitHub - rorywalsh/CsoundUnity

Use this link in the package manager:

https://github.com/rorywalsh/CsoundUnity.git#1632e230d95228268226dedec28524384a15f018

or better:

https://github.com/rorywalsh/CsoundUnity.git#feature/presets

Still WIP, not really documented, but I wanted to have some feedback before going on!
Let me know your thoughts!

EDIT: I think there is some issue with combo boxes yet

EDIT of EDIT: combo boxes should be fixed!

[caiojmini]

Hi!

Just got a chance to play with it a bit this morning, this is amazing! Combo boxes do look fixed on my end.

One thing that would be a nice addition would be the option to set a custom path to save the scriptable objects to help keep everything organized (as opposed to just saving them in the root of the assets folder.

Another thing that would be nice to have (and I have no idea how reasonable/feasible this is) would be if CsoundUnity could read a .snaps file generated by Cabbage and create CsoundUnityPreset assets based off that. But even without this this implementation is so much help, thank you so much!

[caiojmini]

There seems to be an issue with building the Unity project when using the version of the package with the preset system - I’ve been getting these two errors when trying to build both for MacOS and Android

Edit: the image turned out too small but the errors say:

Library/PackageCache/com.csound.csoundunity@e484923571/Runtime/CsoundUnity.cs(1579,30): error CS0103: The name ‘UnityWebRequest’ does not exist in the current context

Library/PackageCache/com.csound.csoundunity@e484923571/Runtime/CsoundUnity.cs(1581,43): error CS0246: The type or namespace name ‘DownloadHandlerBuffer’ could not be found (are you missing a using directive or an assembly reference?)

[giovannibedetti]

Thanks for the tests and the suggestions, will fix the build issue soon (should be a missing reference in the asmdefs)

About the custom path, yes! I was thinking of adding it but I didn’t have enough time

About reading the snaps, it shouldn’t be too hard, they should be csv I think. The only thing I’m not sure is if they hold the channel name inside, if not it could be hard to “link” the channels.

I’ll have a look!

[rorywalsh]

@giovannibedetti

About reading the snaps, it shouldn’t be too hard, they should be csv I think.

The Cabbage snaps are in JSON format, and follow a simple channel:value structure. Should be easy to convert from one to the other. Great work @giovannibedetti

[giovannibedetti]

Perfect!
I will add that in, hopefully while at the airport waiting for my flight back to Italy this saturday

[giovannibedetti]

Fix for the build issue pushed to feature/presets!

[giovannibedetti]

ah yes it will be easy to import Cabbage snaps and also export them from Unity!

{
    "test": {
        "form": 0.0,
        "keyboard2": 60.0,
        "att": 0.009999999776482582,
        "dec": 0.4000000059604645,
        "sus": 0.699999988079071,
        "rel": 0.800000011920929,
        "filebutton8": 0.0,
        "filebutton9": 0.0
    }
}