CsoundUnity for beat matching

Tao_Yao · 27 May 2017 14:33

I’m currently working on a DJ game and trying to use CsoundUnity in my Unity project for beat matching, however got these following issues:

I. I used the “temposcal” opcode in Csound (coz I think beat matching is all about time scaling), the default temposcal first argument value (time stretch) is 1.0 in Csound (means the original time scale of the audio file). But when I pass it to Unity with hslider, the 1.0 in Unity become twice as fast than the original speed (the normal time scale become around 0.45 in CsoundUnity.cs slider), I tried to set different range() for the hslider in Csound but seems that’s not the problem, when I set a different range and adjust to 1.0 in Unity will still make it twice as fast. But open and play it in Csound with 1.0 value will be normal speed. (See the .csd file I uploaded if necessary)

II. When I keep the Time Scale slider in 0.5, and play my track through CsoundUnity in game, the time scale of the song is really unstable (sometimes fast and sometimes slow), but when I play it in Csound alone the time scale is stable, what should I do?

III. Are there any other opcode that better than temposcal for beat matching?

IV. I have to specify the very specific path of my audio files (songs) as the 5th argument of temposcal, but what is the best way to pass multiple audio files and let the user can choose these songs in game? (by passing the a string path from Unity to Csound? if so, what’s the function name?) And as you can see the 5th argument of temposcal, I even have to specify “/user/…/Desktop/…”, but if player download the game it would be a different path in his computer.

V. Can I feed audio signal from Unity to Csound instead? I found this link says there are some functions in CsoundUnity.cs that can do this, but when I opened my CsoundUnity.cs I didn’t found some functions showing in the link. If those functions are removed, can I still feed my songs to Csound?
http://rorywalsh.github.io/CsoundUnity/html/class_csound_unity.html

I would really appreciate if someone can help me with it, thanks!

Csound_Time_Stretch.csd (1.3 KB)

rorywalsh · 27 May 2017 14:51

First things first, you should download the following files and put them into your CsoundUnityScripts folder. Then we know we are on the same page. Secondly, are you using the same sampling rate in both Unity and Csound?

I probably won’t be able to do much more than offer some suggestions for the rest of the weekend, but once Monday comes around I’ll be able to try out some examples myself and see what’s what.

You could use a combo of other PVS opcodes, but I think you’re on the right track using temposcal…

When CsoundUnity loads, it passes a path to a channel called “AudioPath”. If you put all your audio files in an Assets/Audio folder then you can do something like this to get them:

SPath chnget "AudioPath"
SAudioFile sprintf "%s/MySOundfiler.wav", SPath

Basically you just append your sound file names to the audio path.

Yes, in theory although I haven’t tested it in a while. With teh latest source I linked above, there should be a “processAudio” toggle button in the CsoundUnity inspector. If you check this, audio from the currentl audio clip will be piped to Csound. You’ll need to use the inch opcode to access the audio from the clip. As in the following if the clip is stereo:

aL inch 1
aR inch 2

To be honest, I added this feature thinking it would be really useful but ended up never needing to use it as it’s just as easy to load and manage the sound files directly from Csound

Tao_Yao · 28 May 2017 14:08

Thanks for taking your time to help me, Rory! I tried to change the sample rate but seems it has nothing to do with it, and I think I have already containing all files in my CsoundUnity folder since I imported the package in your github website.

It would be great if you can try out some examples after Monday, I can send you my Unity project file if you want.

And the processing audio by sending it from Unity to Csound using inch opcode is such a great idea and useful tool for all developers!

Tao_Yao · 28 May 2017 15:14

I “fixed” the issue by changing the pitch of Untiy AudioSource to 0.46 (when I set the timescale and pitch arguments as default 1.0 in Csound). But I have no idea why I have to set Unity AudioSource pitch to 0.46 to play it with a normal timescale and pitch while processed by my .csd …

rorywalsh · 28 May 2017 18:26

What is the sample rate of the sound file? I will try tomorrow with some of my own files and see what’s up.

Tao_Yao · 28 May 2017 20:18

All file are 44.1kHz sample rate and both settings of Unity and Csound are 44.1kHz sample rate too, thanks!

rorywalsh · 29 May 2017 09:22

I found a bug, and pushed a fix to github repo. You can update by simply replacing your existing CsoundUnity.cs file with the one linked below. Thanks for bringing this to my attention! Let me know if you have any further problems.

github.com

rorywalsh/CsoundUnity/blob/master/CsoundUnityScripts/CsoundUnity.cs

/*
Copyright (C) 2015 Rory Walsh. 

This interface would not have been possible without Richard Henninger's .NET interface to the Csound API. 
 
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), 
to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense,
and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF 
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR 
ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH 
THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
*/

using UnityEngine;
using System.IO;
using System.Collections.Generic;

This file has been truncated. show original

Tao_Yao · 31 May 2017 14:38

Hi Rory,

Thank you so much! This now works perfectly when I use the default my Macbook build-in speaker as output device, but there are couple other questions or problems might need your assistance.

I have a Focusrite Scarlett 2i2 audio interface (Connect to my monitor speaker), I have to use this audio interface in order to output audio with my wonderful monitor speaker. But when I switch the output device to Scarlett 2i2 instead of Macbook build-in speaker, this tempo/pitch goes faster and higher than the original, I have to set the pitch of CsoundUnity audio source to around ‘0.945’ in order to play with the original speed and pitch of the track.
.wav file is too big for building a game, is that possible to use temposcal opcode in CsoundUnity to process .ogg audio file? Which is way more smaller than the original .wav files.

Thanks!

rorywalsh · 31 May 2017 16:17

I’m not sure what could be causing this. Can you try a plain old AudioSource and clip and see if there are any pitch differences when you change sound card? [quote=“Tao_Yao, post:8, topic:553”]
.wav file is too big for building a game, is that possible to use temposcal opcode in CsoundUnity to process .ogg audio file? Which is way more smaller than the original .wav files.
[/quote]

I think it should be possible. Csound uses libsndfile to load sound files from disk, if it’s supported by libsndfile, then it’s supported in Csound.

[edit] just checked, .ogg files should be fine.

Tao_Yao · 31 May 2017 20:22

Hi Rory,

Thanks, I tried .ogg file and it works. But after I put the first .ogg file in the folder and play it, the pitches and speed of the AudioSource again goes higher even I switched to .wav file later (I’m using the Macbook speaker output now and before I put .ogg file in my project and script the pitch to play the normal speed and pitch was ‘1’). So I believe it’s the same bug occurs and haven’t been fixed completely yet.

You can also briefly tell me a little bit about the bug so that I can also try to modify CsoundUnity.cs and fix it? If I fix it I’ll send you the fixed version.

But if you can help me try it out with .ogg file in the same folder with .wav it would great.

rorywalsh · 1 June 2017 09:27

The issue was that I was incrementing the ksmps index by the number of channels, rather than one. It’s fixed now:

github.com

rorywalsh/CsoundUnity/blob/master/CsoundUnityScripts/CsoundUnity.cs#L166


{
    return csound.getKr();
}




/**
 * this gets called for every block of samples
 */
void OnAudioFilterRead(float[] data, int channels)
{
    if (csound != null)
    {
        processBlock(data, channels);
    }
}


/**
* Processes a block of samples
*/
public void processBlock(float[] samples, int numChannels)
{

I can’t see why using a .ogg would be any difference to a wave file. I’ll take a look.

rorywalsh · 1 June 2017 09:36

I’m not seeing any problems here, .ogg and .wav files play back at the correct speed for me. I’m testing with this little instrument:

<Cabbage>
form caption("Test"), size(300, 200)
</Cabbage>
<CsoundSynthesizer>
<CsOptions>
-n -d -m0d
</CsOptions>
<CsInstruments>
sr 	= 	48000 
ksmps 	= 	32
nchnls 	= 	2
0dbfs	=	1 

instr TEMPOSCALE
SPath chnget "AudioPath"
SFilename sprintf "%s/pianoSample.wav", SPath
a1, a2 diskin2 SFilename, 1, 0, 1
outs a1, a2*0
endin  

</CsInstruments>
<CsScore>
i"TEMPOSCALE" 0 [3600*12]
</CsScore>
</CsoundSynthesizer>

I can swap the .wav file for an .ogg file and hear no difference in playback pitch.

Tao_Yao · 1 June 2017 15:50

Hi Rory,

Thanks for telling me the issue, I will do some research on that.

I can swap between .wav and .ogg file and they are in the same speed and pitch too, but the problem now is that both .wav and .ogg files have to be adjusted the pitch of AudioSource component to ‘0.92’ in order to playback the original pitch and speed (like play directly through the quicktime player or a plain AudioSource with the default pitch ‘1.0’ ). Is that also happens to you? Or you mean you can play the original pitch and speed with CsoundUnity AudioSource pitch ‘1.0’?

Good news is my Scarlett Audio Interface have no problem anymore!

Thanks again for your time and patience:)

Tao_Yao · 1 June 2017 15:55

Csound_Time_Stretch.csd (1.8 KB)
Here’s the updated version of my .csd file used for my Unity project, I update it a little bit, if you have no problem with it, probably it’s because there’s something wrong with my Csound file?

You can change the path and put an audio file in your computer here, I used SetChannel() function to trigger the Csound from Untiy script instead of using area.

rorywalsh · 1 June 2017 16:02

I’m afraid I won’t have time to look into this right now, but I can confirm that a CsoundUnity streaming sound file plays back the same pitch for me as an AudioSource/Clip combination reading the same file. I might get a chance to look into this again tomorrow. FWIW, try the simple example I posted and see if things work as you would expect when you are not using the temposcal opcode. If they do, then we can assume that there is a) an issue with your code, or b) an issue with temposcal.

Tao_Yao · 1 June 2017 23:30

One interesting happens, I believe the main reason is sample rate in Csound.

When I use the example you gave me with ‘diskin2’ opcode, your original sample rate was set to 48000 and it works perfect with AudioSource pitch to 1, but when I set that to 44100, it behave just like my .csd file that you have to adjust that pitch to ‘0.92’ in Unity to play the normal pitch.

But when I changed the sample rate in my Csound file (with ‘tempscal’ opcode) to 48000 (originally was 44100), nothing happens and you still need to set pitch to ‘0.92’, and I even tried to change the my Csound sample rate to ‘32Hz’ and it’s still the same, when I set sample rate below 32 Hz in my Csound file it stopped playing. Which is really weird.

I believe either tempscal, ftgentmp or GEN01 (coz I used GEN01 to pass my audio file to tempscal) ignore or have an error with the sample rate we set in the header of Csound file.

But anyway, I would really appreciate if you can help me try this out tomorrow. I’ll also keep trying this until the problem has been resolved.

I think there’s nothing to do with your CsoundUnity API but Csound itself. Maybe I should try some other opcode and stop using GEN01 and see if that will work.

Tao_Yao · 2 June 2017 04:29

Hi Rory,

Don’t worry I’ve fixed the issue, there’s a severe problem with Unity compatibility in Unix operating system, I think that’s one of the reason why so many people hate Mac, haha

I think apple still haven’t realize this bug yet and I hope they can fix it in the next update, I’ll report this bug to one of my friend who works in apple as an engineer.

Let’s keep in touch and I’ll let you know if I found some good ideas or bugs about CsoundUnity!

-Tao

rorywalsh · 2 June 2017 08:41

I’m glad you got it sorted, but I’m none the wiser as to what was causing the issue? Can you give any more details in case it comes up again for others?

Tao_Yao · 2 June 2017 19:17

I was wrong, it’s caused by the sample rate, if the sample rate of the sound file is 44.1kHz, then the sample rate setting in csound must be 48kHz, and the output device (of computer or audio interface) have to be 44.1kHz in order to play the original pitch.

Tao_Yao · 2 June 2017 19:22

btw, I found another really interesting thing, you can try to use my Csound file with that tempscal opcode in Unity.

If you use purely an Audio Source in Unity without Csound processing, the negative frequency will play the audio clip backwards as we all know, and pitch higher than +1 will play the audio faster.

But when I use CsoundUntiy Audio Source, the negative frequency is no longer playing the audio back wards, instead it will just pause the audio just like you set the pitch to ‘0’, but pitch higher than +1 will still make it play faster, can you try that sometime when you are free? I’m curious what is the main reason.