1. A System for Hybridizing Vocal Performance
By Kim Hang Lau

2. Parameters of the singing voice
Parameters of the singing voice can be loosely classified as:
Timbre
Pitch contour
Time contour (rhythm)
Amplitude envelope (projections)

3. Vocal Modification
Vocal modification refers to the signal processing of live or recorded singing to achieve a different inflection and/or timbre
Commercially available units include
Intonation corrector
Pitch/formant processor
Harmonizer
Vocoder
Of particular interest is the Auto-Tune intonation correction from Antares system, which will be used to benchmark some of our tests.

4. Objectives Prototype a system for vocal modification
Modify a source vocal sample to match the time evolution, pitch contour and amplitude envelope of a similarly sung, target vocal sample
Simulates a transfer of singing techniques from a target vocalist to a source vocalist – thus a hybridizing vocal performance
Note that timbre is not included in the objectives to limit the scope of this thesis

5. Order of Presentation System Overview Individual components
System evaluation
System limitations
Conclusions and recommendations
We’ll first present an overview of the prototype system, followed by the individual components. The overall system will then be evaluated, and its limitations will be accessed. And finally, conclusions and recommendations.

6. System Overview Three components
Pitch-marking
Time-alignment
Time/pitch/amplitude modification engine
Inspired by Verhelst’s prototype system for the post-synchronization of speech utterances
The system consists of three components:
pitch marking applied to both the source and target vocal sample, which generate pitch information and amplitude envelope information.
The time-alignment unit generates the time-warping information that synchronized the source to the target.
Together with pitch and amplitude information, modification parameters are generated, which will be applied to the modification engine to modify the source vocal sample
According to my knowledge, this prototype system is an original contribution that suggest a new form of vocal modification. The individual components are however, implementation and adaptations of the existing techniques.
This system is implemented in software using Matlab.

7. Targeted System Specifications
Vocal performance
Commercial singing
Vocal pitch range Hz
Detection accuracy/resolution
10 cents
Detection dynamic range
40dB
Sampling rate
44.1kHz and 48kHz
Time-scale modification
±20%
Pitch-scale modification
±600 cents
The requirements of detection accuracy/resolution is stringent, because the system has to handle minute pitch inflections like pitch jitter and vibrato. Singing vibrato generally occurs between Hz at cents. The system must be able to detect and modify to produce smooth quasi-sine pitch contour at this frequency and depth.
Detection dynamic range is with reference to normalized power. For good singers, their dynamic range can be higher, but 40dB is the average.
The moderate time/pitch modification requirements are result of the assumption that two similarly sung vocal sample are compared
Without further ado, I’ll start presenting the individual components, starting with the pitch-marking system, followed by the modification engine, and then the time-alignment system

8. Component No.1 Pitch-marking

9. Pitch-marking and Glottal Closure Instants (GCIs)
Pitch-marks
5ms P P’
Information generated from pitch-marking
Pitch period
Amplitude envelope
Voiced/unvoiced segment boundaries
Pitch-marking is the process of placing markers in the signal waveform at a pitch synchronous rate for voiced sounds, and at constant rate for unvoiced sounds
For applications to the modification engine, these markers should ideally correspond to the time-instant when the vocal tract is most excited during a cycle of vocal fold vibration
This time instant is commonly accepted to be the instant when the glottis closes, hence Glottal Closure Instants
It is clear that pitch period and amplitude envelope can be derived from pitch-marking

10. Pitch-marking applying Dyadic Wavelet Transform (DyWT)
Kadambe adapted Mallat’s algorithm for edge detection in image signal to the detection of GCIs in speech signal
He assumed the correlation between edges in image signal and GCIs in speech signal
DyWT computation for dyadic scales 2^3 to 2^5 was sufficient for pitch-marking
If a particular peak detected in DyWT matches for two consecutive scales, starting from a lower scale, that time-instant is taken as a GCI
, which are both considered abrupt transition points in image and speech signals respectively

11. Mallat Kadambe Base-band Original Signal 2^1 2^2 2^3 2^4 2^5
In the left hand plot is an illustration from Mallat. On the top level is the original signal and subsequent plots are the wavelet coefficients of dyadic scale of power 1 to 5. It is clear that for every abrupt transition in the signal, the wavelet coeffients display a peak.
On the right-hand side is an illustration from Kadambe. The original signal is a synthesized vowel, and the ‘true’ GCIs are marked at the top of every graph.
Comparing the original signals, it will be quick to notice every oscillation in the speech signal can be considered as a abrupt transition point in Mallat’s context. This is because a GCI is embedded in the speech by way of convolution, and was never explicitly manifested in the signal waveform.
However, illustrated in the bottom right of Kadambe’s illustration, the time response of the wavelet transform is still very desirable for pitch-marking when higher harmonics are filtered, and the wavelet filter band that contain the fundamental frequency can accurately define GCIs. I’ll refer this band as the base-band.
2^4
2^5
Base-band

12. The proposed pitch-marking scheme
Detection principle
Detection of the scale that contains the fundamental period
Starting from a higher scale (of lower frequency), there is a considerable jump in frame power when this scale is encountered
Features
4X decimation to support high sampling rates
Frame based processing and error correction for possible quasi-real-time detection

13. The proposed pitch-marking system
The purpose of showing this system is to illustrate the immensity of the system that controls the behavior of the pitch-marking

14. Comparisons of results with Auto-Tune
Proposed system
Auto-Tune

15. Component No.2 The Modification Engine

16. Time/pitch/amplitude modification engine
D(n)
(n): time-modification factor (n): pitch-modification factor
(n): amplitude modification factor D(n): time-warping function

17. TD-PSOLA (Time-domain Pitch Synchronous Overlap-Add)
Time-domain splicing overlap-add method
Used in prosodic modification of speech
TD-PSOLA is implemented for the modification engine.
Analysis stage: short-time analysis are extracted from the signal via windows centered at a pitch-marks, as illustrated in the diagram
Pitch-scale modifications are attained by narrowing the distance between pitch-marks before overlap-add.
Time-scale modification is performed by adding or discarding short-time analysis signals in the synthesis stage.
The size and type of window are important analysis parameters. In general, windows with reasonable spectral behavior can be used, and the size of the window should be approximately 2 times the local pitch-period. The commonly used Hanning window was chosen.

18. Evaluation of the modification engine
Original
TD-PSOLA
In this test example, a female vocal sample was pitch-shifted to sustain a constant pitch.
The original
Our modification engine
Auto-Tune
Auto-tune has advance methods for the handling of pitch-transition, which is manifested by the non-instantaneous modification. The emphasis is however to show that Auto-Tune does not preserve the formants well.
Auto-Tune

19. Component No.3 Time-alignment

20. Time-alignment
Based on Verhelst’s prototye system that applies Dynamic Time Warping (DTW)
He claimed that the basic local constrain produces the most accurate time-warping path
Exponential increase in computation as length of comparison increases
Accuracy deteriorates as length of comparison increases
In order to find the modification parameters, the source and target time events has to be time-aligned.
It compares speaker independent parameters i.e. the LPC Cepstral parameter, and make use of dynamic programming to search for the best match between a target spoken word and a series of arbitrary spoken word.
In two similarly spoken word are compared, a time-warping path that synchronizes the source speech to the target speech is formulated
These constrains limit the search path

21. Adaptations from Verhelst’s method
Proposed to perform time-alignment on a voiced/unvoiced segmental basis
DTW for voiced segments
Linear Time Warping (LTW) for unvoiced segments
Global constraints are introduced to further reduce computations
Synchronization of voiced/unvoiced segments are required, which is manually edited in current implementation
LTW was chosen for unvoiced segment to confine our modification to voiced sound only. This is because the manipulation of unvoiced sounds can be more complicated than voiced sounds
Voiced/unvoiced segments boundaries are generated by the pitch-marking stage. But dissimilarity between source/target and limitations of pitch-marking will never yield the exact number of segments between the source and the target. A easy method was used to manually edit these segment boundaries.

22. Manipulation of modification parameters
Simple smoothing of (n), (n) using linear phase FIR low-pass filters are performed before feeding them to the modification engine

23. The Prototype System
With this, I’ll shall present the final

24. System Evaluation: case 1

25. System Evaluation: case 2

26. System Limitations Segmentation Modification engine
Lack of a reliable technique for voiced/unvoiced segmentation
Segmentation and classification of different vocal sounds is the key to devise rules for modification
Modification engine
Lack capabilities to handle pitch transition, total dependence to the pitch-marking stage

27. System Limitations Pitch-marking Time-alignment
Proposed system lacks robustness
Despite desirable time-response of the wavelet filter bank, its frequency response is not capable of isolating harmonics effectively and efficiently
Time-alignment
The DTW basic local constraint allows infinite time expansion and compression.
This factor often causes distortions in the synthesized vocal sample

28. Conclusions and Recommendations
Current systems works well for slow and continuous singing
Further improvements on the individual components are recommended to handle greater dynamic changes of the vocal signal, thereby extending the current good results to a wider range of singing styles

29. Questions
&
Answers

30. Wavelet filter bank

31. Dyadic Spline Wavelet

32. Wide-band analysis

33. DTW local constraints

34. Calculation of pitch-marks

35. DyWT