Cabbage Logo
Back to Cabbage Site

Csound wrapper/plugin for Unreal Engine

#1

Hello,

I’ve been working with UE4 and Max/MSP for some time and now I’m trying to get Csound integrated with UE4 (instead of using OSC messaging). I’m aware that there’s a new audio engine that will be released in UE4 4.16 that might make it easier to get a Csound wrapper or plugin done. I’m also aware of Rory’s efforts but I was wondering what the state of this is and if there’s anyone else interested in getting this working. I’m willing to help with development efforts if needed. I’m guessing the best is to wait for 4.16 to be released but I wanted to know what Rory and/or anyone else might have in mind.

I would love to see this working. As much as I wish I could switch to Unity and use Csound there, I just can’t get in Unity the same level of visual quality and performance that I get with UE4.

Cheers!

Hector

#2

I did create a wrapper before for UE4 which was Ok for music and generative scores, but the latency was pretty terrible. Creating procedural audio effects with it was not great. Unreal has a fixed buffer size that is hard coded into the engine, and it’s not small. I think it’s 4096 samples. One can get around this by using some black magic, but I never figured it out. I know it’s possible because Wwise and FMOD do it somehow. But I can’t study their implementations because they are closed source. In the end I just wrote a Csound FMOD wrapper so that users can at least access Csound in Unreal that way. I still have all my Csound Unreal code lying around however. Creating a wrapper won’t take long. Let’s hope that they provide some better tools in 4.16. I would really love to see Csound in Unreal.

#3

I’ve just been reading up on the new audio system in UE4. It looks like CsoundUnreal will be no trouble at all :wink:

1 Like
#4

Hi Rory,

Thank you for your reply. They are adding a whole new audio DSP chain so it’s possible to program your own plugins in C++ by just adding your own DSP algorithms so I’m guessing this should make Csound integration much easier. Have you watched the latest GDC presentation on the new audio engine? Very likely you had but just in case: https://www.youtube.com/watch?v=ErejaBCicds

In the case of using FMOD with UE4, I’m not experienced with it but from what I see it seems that each time an FMOD event triggers it reinitializes the Csound CSD, right? Does this mean that if I wanted to create a Csound/FMOD plugin that is always on, processing audio spatialization (ambisonic decoded to binaural), do I have to create an extremely long FMOD event? (not sure if this is even possible). Is there a way of having an FMOD event that is infinite and gets triggered once when UE4 starts playing?

Thanks!

#5

Now yet, but I’ve been reading the threads. Looks like a huge improvement. [quote=“virtualHC, post:4, topic:496”]
In the case of using FMOD with UE4, I’m not experienced with it but from what I see it seems that each time an FMOD event triggers it reinitializes the Csound CSD, right? Does this mean that if I wanted to create a Csound/FMOD plugin that is always on, processing audio spatialization (ambisonic decoded to binaural), do I have to create an extremely long FMOD event? (not sure if this is even possible). Is there a way of having an FMOD event that is infinite and gets triggered once when UE4 starts playing?
[/quote]

This is true, and one of the issue I have with the FMOD implementation. To be honest I don’t know if you can create an infinite event in FMOD. I guess there may be some way around this. Btw, do you know when 4.16 is planned to be released?

#6

Epic Games has been pushing UE4 development really hard and releasing massive updates very frequently so at the rate they’ve been going I’m guessing they will start releasing 4.16 preview versions around the end of this month or early next month. Although I think probably most of the new audio engine is already in their 4.16 branch in GitHub so we could test it now if we build it (I might try that soon).

#7

I might try that myself too but it won’t be for a week or so. From what I see in that video it shouldn’t take long at all to implement a Csound wrapper. This is great news.

1 Like
#8

Well, my prediction was right. They just released 4.16 preview 1 so no need to build from source. The new audio engine is still not on by default so a flag needs to be set in a ini file. It’s explained in this forum thread: https://forums.unrealengine.com/showthread.php?142071-Unreal-Engine-4-16-Preview

#9

Typical. I just built it 2 days ago! I won’t have time to get going on this till next week. Let me know if you make any headway.

#10

So I got it to work! It’s indeed quite easy. You just need to create a new blank plugin and then add a SynthComponent to it. It creates a “hello world” synth that produces a sine wave but from there you just have to replace the code with Csound’s initialization (with csoundSetHostImplementedAudioIO) and then add the calls to csoundGetOutputBuffer and buffer copy to the UE4 output buffer. The only buffer management difference between UE4’s and Csound is that Csound doubles the buffer when doing stereo whereas UE4 uses half the buffer per channel so it remains at 1024 regardless of using mono or stereo channels. I’ll post the code somewhere tomorrow but really is nothing special (although it did took me longer that needed due to the obscure UE4 documentation and my lack of experience with UE4’s C++ environment).

The synth output can then be spatialized using Steam Audio, which includes ray traced reflection computation using the scene geometry and HRTF encoding. Very exciting! Now I just need to implement messaging between Blueprints and Csoound and then do headtracked ambisonic-binaural playback for VR!

Cheers!

#11

Has it been 26 days since I last posted that! So much for having the time to take a look. I’m glad you got it going. Your processing function should be buffer-size agnostic if that makes sense. You shouldn’t make any assumptions on what size the Csound buffer will be as user can always change it. But you’ve probably already figured this out. The blueprints thing shouldn’t be that tricky either. When i did it before the only method I exposed were the basic chnget/chnset methods, along with a sendScoreEvent method. These are the bread and butter of any Csound API stuff. All the rest are candy.

I look forward to seeing this in action. Note I still plan to look into this myself. I would like to create an identical interface for Unreal as I have for Unity. But I’d be happy to work together on it either. I just think it would be nice if users could simply move between the two.

#12

Yes, it would be great to have a unified interface between the two engines. I haven’t tried Csound for Unity yet but I will try to do it soon so I can get an idea of your design model. For now I need to get it working in UE4 quickly and just bespoke for an upcoming personal project that will be exhibited soon, but I’m also keen on getting something done that can be distributed widely as a plugin. I’m definitively open to helping you achieve this, following your guidelines and as part of your project.

#13

Great. Once things quieten down a little here in work I should be able to get cracking. I’m thinking however that it won’t be till after June the 10th. Good luck with your project, are you working on a game or an installation?

#14

Thanks! It’s an installation. I’m trying to create VR experiences related to the ideas of acoustic ecology and soundscape composition that I’ve been exploring for some time, using spatial audio, realtime audio synthesis, photogrammetry, 3D video. There’s a sample of my VR work here: http://sitting.hcenteno.net This was my masters thesis work but I’m trying to expand it now. The idea is to create surreal sculptures using photorealistic elements and spatial audio captured from specific geographical locations… I’m still exploring the concept.

#15

Your work looks really interesting. Hmm, I got to get me some VR gear!

#16

Hey Rory.
I’m back since a long time, I was working on our last video game.
I’m very interested by a UE4 Csound plugin.
I’m currently working with UE4 blueprint to create generative music. But the internal UE4 new synth seems a bit limited. CSound could provide a lot more possibilities. Is Csound heavy for memory and performances? Or Csound is very light?
I do not want to use CSound with Fmod (I finally never found a way to use it on Mac).
What are your intentions concerning a UE4 plugin?
Thanks!

#17

I would love to develop an UE4 version of the Csound Unity wrapper. With the new audio stuff in UE it should be relatively easy to do, but time is of the essence. I have been promised a new PC at work that should run UE4 a lot better than my current PC, so perhaps that may encourage me to get this done sooner rather than later.

With regards to performance, Csound is highly optimized with no GUI overhead. It’s ideal for embedding into existing frameworks.

#18

Thank you for the answer Rory!
I’m trying to get my hands in C++ as a beginner so I have time :slight_smile:
Don’t hesitate to give us some news here. Meanwhile, good luck for everything!

#19

Good news Rory! I got it to work on Android (thanks go to you and Steven Yi). So far the basics are implemented for Windows and Android. Hopefully we can get it in shape so it can be released.

#20

I saw your post to the Csound list. That’s great. Steven is a genius, I don’t know how he finds time to learn about all these things. It would be really great for users if the UE4 wrapper had the same functionality as the CsoundUnity one so that users don’t feel like they have to learn an entirely new interface if there are moving back and forth. Maybe even called it CsoundUE4? We could move on to doing CsoundCryEngine next :wink: I hope to get time to work on the Android stuff in Unity this week. I would really love to get that sorted.