[Clam-devel] Re: gsoc project - real time synthesizer for a woodwind instrument
Han, Yushen
yushen.han at gmail.com
Mon Jun 2 14:48:21 PDT 2008
hi, Greg and Pau
Thank you for your updates.
Greg's code was being extremely valuable to me.
I did more tests and just made an oboe version for
"ContinuousExcitationSynthesizer" by modifying Greg's code.
However, my program crashes unpredictably (for some loops) and the sound is
jitterring with my oboe sound samples in SDIF.
Here I have a number of questions about Greg's
"ContinuousExcitationSynthesizer" to ask you(mostly for Greg).
(1) The format of the score
I believed that the score for test is given in the format of
"frequency amplitude" frame by frame.
The test scores given by Greg work fine with his TestScore.
e.g.
TestScore::LoadScore. line: 452.596 0.333
TestScore::LoadScore. number of tokens: 2 // only 2 tokens - freq, amp
which is correct
scoreVetor [0][0] = floatToken =452.596
scoreVetor [1][0] = floatToken =0.333
But anything I edited by hand results in segmentation fault
because the program mistook multiple lines for one frame.
e.g.
TestScore::LoadScore. line:
456.313 0.444
433.111 0.431
413.131 0.421
436.121 0.421
TestScore::LoadScore. number of tokens: 5 // this should not be 5 !
scoreVetor [0][4] = floatToken =456.313
scoreVetor [1][4] = floatToken =0.444
scoreVetor [2][4] = floatToken =0.431
Segmentation fault
This problem is really odd to me since both the test score was essentially
the same
except for that I typed in the second file.
(2) the synthesized frames
In the output of the synthesis, I can see *every other frame* in the test
score was present in the synthesized audio:
e.g.
counter: 0, pitch: 452.596, amplitude: 0.333 // this is the 1st line of
longnote.txt
counter: 1, pitch: 452.783, amplitude: 0.342 // this is the 3rd line of
longnote.txt
Oraison: creating new voiceId 1212439648
counter: 2, pitch: 432.353, amplitude: 0.39 // this is the 5th line of
longnote.txt
Oraison: creating new voiceId 1212439648
counter: 3, pitch: 468.269, amplitude: 0.379 // this is the 7th line of
longnote.txt
......
Why is the even lines in the text score are skipped in the synthesis?
The assumption for "one frame" in the score is SR = 44100Hz and HOPSIZE =
256 Samples.
Is that correct?
Also, I noticed there was about 944 frames of non-zero frames ( about
5.5sec).
And there are 945 frames of "pitch 0 amplitue 0" appended to the synthesized
audio.
The score actually contains 1888 lines (frames).
Can you explain why I got only half of the frames synthesized?
(3) ( to both Pau and Greg )
Besides the problems above, the sound is not good, which perhaps is because
the loop points are not that good.
I don't know whether the jitterring effect was due to the zero-cross points
in the loop or the crossfading frame by frame.
Now I need to figure out this by finding some "perfect" loops.
Also, I did not use all 3 dynamics in my samples ( only mf now).
I am hoping to be able to use the loop in a note of different dynamics, when
the amplitude in the score is changed while the frequency does not jump for
more than a semitone. (Of course there should not be any new attack.)
Do you think this is the right way to go in order to incorporate the change
in dynamics? I am open to your suggestions.
I don't know if Greg has any official affiliation/obligation in this CLAM
project this year,
But I appreciate his help a lot!
Best regards,
Han, Yushen
On Fri, May 30, 2008 at 4:47 AM, Greg Kellum <greg.kellum at gmail.com> wrote:
> Hi Yushen,
> I was looking at the problem the wrong way. I was looking at it as a
> choice between looping at the zero crossings or cross fading. But of course
> you can do both, and I guess it will sound better if you do do both...
>
> Best,
> Greg
>
>
>
> On May 29, 2008, at 7:04 AM, Han, Yushen wrote:
>
> Greg,
>
> I see. That makes sense.
> I would not expect to have good loops without crossfading.
> It seems that one can't say much about the quality of the loop until he
> could hear the loop with crossfading.
> I need to look into LoopingSDIFFileReader before I can tell more about the
> loops that the loopMaker found.
>
> I did some test with my oboe samples and the loop (precise to sample) sound
> very good with zero crossings/no crossfading.
> Thanks for your reminder.
>
> Best regards,
> Han, Yushen
>
> On Thu, May 29, 2008 at 12:41 AM, Greg Kellum <greg.kellum at gmail.com>
> wrote:
>
>> Hi Yushen,
>>
>> The loop making algorithm is not trying to match up the start and end
>> points exactly so that you could just loop there; it's intended to be
>> used together with crossfading. In my own experience trying to match
>> up the zero crossings rather than cross fading doesn't work well;
>> Trying that in Kontakt always sounded like crap, but I suppose if
>> you're trying to loop very harmonic material with a stable period it
>> could work... In any case the looping algorithm is looking for
>> spectrally similar frames, and then in my synth, they are crossfaded.
>> This happens in the class LoopingSDIFFileReader... Before you
>> re-implement the algorithm you might want to try looping your material
>> in Kontakt or some other sampler to see if you can get a good loop
>> using only zero crossings and no crossfading...
>>
>> Best,
>> Greg
>>
>>
>> On Wed, May 28, 2008 at 10:10 PM, Han, Yushen <yushen.han at gmail.com>
>> wrote:
>> > Hi,
>> >
>> > I forget to mention that the most of my samples are 1 - 1.5 sec long.
>> (oboe)
>> > The stable state is as long as 1 sec usually, which is way too short for
>> > Greg's algorithm.
>> >
>> > I could get some good loops if I specify the startSample and endSample.
>> > But startTime endTime (in ms) did not generate good loops.
>> >
>> > Best regards,
>> > Han, Yushen
>> >
>> > On Wed, May 28, 2008 at 4:07 PM, Han, Yushen <yushen.han at gmail.com>
>> wrote:
>> >>
>> >> Hi, Pau and Greg
>> >>
>> >> Thanks to Greg's reply, I am able to covert my audio samples to SDIF
>> and
>> >> identify loops ( with startTime/endTime in ms specified in a .XML
>> file).
>> >> But If I extract the loop with startTime/endTime from the SDIF file,
>> the
>> >> loops contain "clicks" and hence sound bad.
>> >>
>> >> Here I suspect hat a ms may not be the best unit to specify a loop.
>> >> e.g. suppose SR = 44100 Hz, HOP = 256 samples (as used in Greg's code),
>> >> for a note C4 (f = 261.6Hz), we will have 3.8ms per period i.e 168.8
>> >> samples per period.
>> >> The fraction will lead to round error if we specify the loop by ms.
>> >>
>> >> In the hope that a sample-by-sample approach will avoid the click,
>> >> I am trying to develop my algorithm to identify an optimal loop based
>> on
>> >> the best f0 given by
>> >> segment.GetFrame(counter).GetFundamental();
>> >>
>> >> However, I check out the reference for segment, frame, and audio but
>> did
>> >> not know how to specify the unit by sample.
>> >>
>> >>
>> >>
>> http://www.clam.iua.upf.edu/doc/CLAM-devel-doxygen/classCLAM_1_1Segment.html
>> >>
>> http://www.clam.iua.upf.edu/doc/CLAM-devel-doxygen/classCLAM_1_1Audio.html
>> >>
>> >> Do you agree that it is the reason why the loop sounds like "clicking"?
>> >> Do you know if there is a way to refer to a chunk of audio by sample?
>> >> Thanks!
>> >>
>> >> Best regards,
>> >> Han, Yushen
>> >>
>> >>
>> >>
>> >>
>> >>
>> >>
>> >>
>> >>
>> >> On Thu, May 22, 2008 at 6:51 AM, Pau Arumí <parumi at iua.upf.edu> wrote:
>> >>>
>> >>> On dj, 2008-05-22 at 03:11 -0400, Han, Yushen wrote:
>> >>> > Hi, Greg and Pau
>> >>> >
>> >>> > I did some tests on SDIF files out of wave files and so far it works
>> >>> > fine.
>> >>> > Now I am trying to find out the loop of a single note and store the
>> >>> > loop in .xml file.
>> >>> > Here I have questions for each of you separately:
>> >>> >
>> >>> > (1) for Greg
>> >>> > I compiled your makeLoop (from the examples folder) successfully.
>> >>> > But if I specify the start time and the end time of the steady
>> state,
>> >>> > makeLoopExe always "calculate 0 EarthMover's distance".
>> >>> > If I do not specify the time, makeLoopExe would never finish. ( it
>> >>> > gives dots ".................." )
>> >>> >
>> >>> > Did you notice this problem before? Maybe it only happened to me.
>> >>> >
>> >>> > Also, in your previous message you said you "turned off the database
>> >>> > lookup function".
>> >>> > Can you please explain more about what "the lookup function" is
>> >>> > supposed to do and why you turned it off?
>> >>> >
>> >>> > ( I could see that you disabled the residual part in the
>> SDIFInConfig.
>> >>> > But I am not sure if this is what you were talking about... )
>> >>> >
>> >>> > (2) for Pau:
>> >>> > So far we have not decide which woodwind instrument we should work
>> >>> > with.
>> >>> > Should I work on Saxophone or Oboe?
>> >>> > Do we have annotated saxophone notes from SALTO with all loop
>> >>> > identified?
>> >>> >
>> >>> > I cut and labeled 72 oboe notes recorded in an anechoic chamber by
>> >>> > hand (24 pitches C4-B5 * 3 different dynamics).
>> >>> > It will take a while to find the loop for those notes. Before I
>> >>> > proceed, I would like to make sure I am working on the right
>> >>> > instrument :-)
>> >>>
>> >>> Hi Yushen,
>> >>>
>> >>> Yes please, proceed with your oboe notes.
>> >>> I'll also search for the old-salto saxophoe database, but there are
>> >>> chances they are not available.
>> >>> It would be interesting to work with a couple (or more) instruments,
>> >>> though we can concentrate on the sound quality for only one of them.
>> >>> I think that working with multiple instruments will force us to
>> improve
>> >>> the tools for creating instruments, which should be one of the goals.
>> >>> Another goal i'd like to propose is offering transformations not
>> typical
>> >>> in time-domain based samples, taking advantage of the underlying
>> >>> spectral model. Timber morphing from one instrument to another is the
>> >>> first that comes to my mind.
>> >>> Of course, the first milestone is having a usable instrument.
>> >>>
>> >>> Pau
>> >>>
>> >>>
>> >>> >
>> >>> > Best regards,
>> >>> > Han, Yushen
>> >>> >
>> >>> > On Sun, May 18, 2008 at 10:35 AM, Greg Kellum <
>> greg.kellum at gmail.com>
>> >>> > wrote:
>> >>> > Hi,
>> >>> >
>> >>> > In response to point 3, I turned off the database lookup
>> >>> > function at
>> >>> > some point, because of an exception that kept occurring...
>> I
>> >>> > can't
>> >>> > remember whether I eventually fixed this or not. I'll have
>> to
>> >>> > check...
>> >>> >
>> >>> > Greg
>> >>> >
>> >>> >
>> >>> > On Sat, May 17, 2008 at 6:04 PM, Han, Yushen
>> >>> > <yushen.han at gmail.com> wrote:
>> >>> > > Hello, Pau
>> >>> > >
>> >>> > > Thanks for your concerns. Here is some update from me.
>> >>> > >
>> >>> > > 1. Based on Greg's last update, I am able to compile the
>> >>> > command-line
>> >>> > > version of continuousExcitationSynthesizer.
>> >>> > > Also, I can read the XML file where the looping
>> information
>> >>> > is stored with
>> >>> > > his metadata reader.
>> >>> > > As for SDIF, I am able to use CLAM/examples/SDIF2Wav and
>> >>> > Wav2SDIF to convert
>> >>> > > between SDIF and WAV.
>> >>> > >
>> >>> > > 2. In the NetworkEditor, I still could not have
>> >>> > "SDIFDatabaseProcessing".
>> >>> > > (Greg had reported that he fixed this problem.)
>> >>> > > Right now I don't really use the network editor but the
>> >>> > command-line
>> >>> > > executable version.
>> >>> > >
>> >>> > > To resume our remote debugging, last time you concluded
>> >>> > > "It seems your OS is not loading some of the processings.
>> To
>> >>> > confirm
>> >>> > > this: how many processings do you have in the 3D Audio
>> >>> > category?
>> >>> > > You should see 14."
>> >>> > > However, I only had 1 processing - VectorBasedArrayPanning
>> >>> > available after
>> >>> > > compiling that 3D plugin.
>> >>> > >
>> >>> > > 3. I did some experiment with 12 oboe notes in one octave
>> >>> > and concluded that
>> >>> > > if there is only one source sounding,
>> >>> > > the first 4 harmonics are quite in phase ( I only observe
>> >>> > the first 4
>> >>> > > harmonics). Therefore I was thinking that we may disregard
>> >>> > the instantaneous
>> >>> > > phase in the SMS. However, this is not true for an
>> >>> > orchestra.
>> >>> > >
>> >>> > > "- [2+ days] To investigate the effect of slowly-changing
>> >>> > phase changes
>> >>> > > between different partials
>> >>> > > on the perception of the harmonic sound. If the
>> >>> > slowly-change phase did
>> >>> > > not affect the sound
>> >>> > > very much ( suppose all harmonics are all in phase ),
>> >>> > further
>> >>> > > simplification in synthesis
>> >>> > > could be done. This experiment is open to suggestion
>> >>> > from the mentor."
>> >>> > > I will write to you separately about that and more raw
>> >>> > ideas.
>> >>> > >
>> >>> > > 4. Right now I am trying to modify Greg's code and
>> Wav2SDIF
>> >>> > example for:
>> >>> > > "- [1-2 days] To synthesize with SMS with the analysis
>> >>> > result (in a simple
>> >>> > > setting).
>> >>> > > To produce some audible result to evaluate the
>> performance
>> >>> > of SMS
>> >>> > > analysis/transformation."
>> >>> > >
>> >>> > > It seemed that the Wav2SDIF example had done the majority
>> of
>> >>> > the work.
>> >>> > > I am studying the code but I don't know if I could improve
>> >>> > this part.
>> >>> > > Do you recommend me to read some updated documents on
>> SDIF?
>> >>> > > http://www.cnmat.berkeley.edu/SDIF/ does not seem to be
>> very
>> >>> > recent.
>> >>> > >
>> >>> > > 5. I apologize for my late updating.
>> >>> > > I am thinking of keeping a blog on this project...
>> >>> > >
>> >>> > > Best regards,
>> >>> > > Han, Yushen
>> >>> > >
>> >>> > >
>> >>> > >
>> >>> > >
>> >>> > > On Sat, May 17, 2008 at 7:40 AM, Pau Arumí
>> >>> > <parumi at iua.upf.edu> wrote:
>> >>> > >>
>> >>> > >> Hello Yushen,
>> >>> > >>
>> >>> > >> how is all doing? I'd like to hear in what you are up
>> >>> > regarding your
>> >>> > >> project.
>> >>> > >>
>> >>> > >> are you still blocked with the plugin installation? in
>> that
>> >>> > case is it
>> >>> > >> possible for you have also a linux box?
>> >>> > >>
>> >>> > >> i'd like you to write some raw ideas about what to be
>> >>> > implemented and
>> >>> > >> some planning, and then i'll comment and keep refining
>> the
>> >>> > planning.
>> >>> > >> is that ok?
>> >>> > >>
>> >>> > >> cheers!
>> >>> > >> pau
>> >>> > >>
>> >>> > >>
>> >>> > >
>> >>> > >
>> >>> >
>> >>> >
>> >>>
>> >>
>> >
>> >
>>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.clam-project.org/pipermail/clam-devel-clam-project.org/attachments/20080602/b3b88e06/attachment-0004.htm>
More information about the clam-devel
mailing list