1. Developing a DSP Algorithm using a TTD process Application FIR filter

2. TTD and XPI and TTD XPI – eXtreme Programming inspired life cycle TTD – test driven development Main tool Automated testing framework Reference:- A More Agile Approach to Embedded System Development, Smith et al., IEEE Software magazine, March, 2009. 6/5/2016 Testing FIR filters Copyright smithmr@ucalgary.ca 2 / 33

3. XP inspired development process Stage 1: Envisioning embedded product This stage produces requirements. The customers or end users (domain-knowledgeable personnel) should be able to specify their ideas as a set of acceptance tests (in conjunction with standard user stories) in a straightforward manner. User stories – Wish list of what you would like the product to do Now prioritize the wish list for the first part that customer identifies as most important. Plan to develop, deliver and demonstrate a prototype. 6/5/2016 Testing FIR filters Copyright smithmr@ucalgary.ca 3 / 33

4. XPI: Stage 2: Prototyping Domain-knowledgeable technical personnel produce an initial prototype to analyze the proposed embedded solution. The solution would revolve around numerical algorithms implemented in a mathematically oriented scripting language such as Matlab. Developing additional acceptance test cases. Using TDD supported by a testing framework designed for Matlab development. Stage 1’s acceptance tests provide the initial skeleton for the prototype, Stage 2 adds “flesh” onto this outline. 6/5/2016 Testing FIR filters Copyright smithmr@ucalgary.ca 4 / 33

5. Stage 3: Initial XPI Production System We must rewrite the system in a high-level compiled language such as C or C++. The biggest issues here surround the refactoring necessary to support moving Matlab floating-point calculations onto high speed, low-cost embedded processors that can handle integer operations. Stage 3 basically repeats the pattern from Stage 2, but in a new programming language using a new unit-testing framework. However, because of the existence of the previous stages, we’ve already constructed numerous tests. 6/5/2016 Testing FIR filters Copyright smithmr@ucalgary.ca 5 / 33

6. Stage 4: Full Production System By the fourth stage, we’re fully involved with the target embedded environment itself. The system, although repeating earlier patterns, now needs to adapt to its new hardware environment with limited memory resources and must support interactions with device-specific peripherals on the target system. The system undergoes further modification to tune for the required performance by rewriting inefficient programming constructs, using signal-processing-specific #pragma state- to direct architecturally aware compilers, and rewriting critical components in low-level, highly customized machine code. 6/5/2016 Testing FIR filters Copyright smithmr@ucalgary.ca 6 / 33

7. Assign. 1 From individual assignment 1 you have C++ un-optimized averaging filter Compiler optimized C++ averaging filter ASM version – software loop – averaging filter Tests for averaging filter (function, timing and theory comparison) 6/5/2016 Testing FIR filters Copyright smithmr@ucalgary.ca 7 / 33

8. Steps in optimizing y[m] = Sum x[m – n] where n = 0 to N-1 Data stored oldest to newest x[m-n+1], x[m-n+2] … x[1], x[0] FIFO[0] ……………………………...FIFO[n-1] so that possible use of hardware circular buffers which go forward We code as sum = 0; for count = 0 to N-1 sum = sum + FIFO[n-1-count]; This “array” code requires specific DAG memory adds – slow because difficult to combine with other instructions (not enough instruction bits) 6/5/2016 Testing FIR filters Copyright smithmr@ucalgary.ca 8 / 40

9. Steps in optimizing – Step 2 We code as sum = 0; for count = 0 to N-1 sum = sum + FIFO[n-1-count]; This “array” code requires specific DAG memory adds – slow because difficult to combine with other instructions (not enough instruction bits) Change to pointer operation using post decrement FIFOpt = & FIFO[n-1-count]; sum = 0; for count = 0 to N-1 sum = sum + * FIFOpt--; Change to averaging FIR filter which access coefficients FIRcoeffs[N ] = {1, 1, 1, … 1, 1}; FIRcoeffpt = &FIRco eff[0] FIFOpt = & FIFO[n-1-count]; sum = 0; for count = 0 to N-1 sum = sum + (* FIFOpt--) * (* FIRcoeffpt++); Note the two meaning of the C++ operator * Multiplication of two variables (integer, float (single and extended), complex int are possible) Contents of memory location pointed to by a pointer 6/5/2016 Testing FIR filters Copyright smithmr@ucalgary.ca 9 / 40

10. Steps in optimizing – Step 3 Change to averaging FIR filter which access coefficients FIRcoeffs[N ] = {1, 1, 1, … 1, 1}; FIRcoeffpt = &FIRco eff[0] FIFOpt = & FIFO[n-1-count]; sum = 0; for count = 0 to N-1 sum = sum + (* FIFOpt--) * (* FIRcoeffpt++); Refactor to take advantage of processor DAG characteristics Most processors are faster at incrementing forward – and the instruction works in parallel with more instructions FIRcoeffs[N ] = {1, 1, 1, … 1, 1}; FIRcoeffpt = &FIRco eff[0] FIFOpt = & FIFO[0]; sum = 0; for count = 0 to N-1 sum = sum + (* FIFOpt++) * (* FIRcoeffpt++); Change away from {1 ….. 1} and you have normal FIR filter code 6/5/2016 Testing FIR filters Copyright smithmr@ucalgary.ca 10 / 40

11. Lab. 1 Code conversion: Averaging to FIR Average Filter -- Do Buffer update loop then do fetch a new data memory value with add loop FIR -- Modify second loop to bring in second data value (FIR coeffs) and perform multiply and add C++ un-optimized FIR -- data and coeffs from dm Compiler optimized C++ FIR -- data and coeffs from dm ASM version – software loop – FIR filter -- data and coeffs from dm Test -- (function, timing and theory comparison) Averaging filter has same functional results as FIR if you test with coefficients coefficients (1/N, 1/N, 1/N …. 6/5/2016 Testing FIR filters Copyright smithmr@ucalgary.ca 11 / 33

12. Lab. 1 Code conversion: New FIR Add new FIR specific tests – discussed today Modify one loop to bring in second data value (FIR coeffs) and perform multiply and add Compiler optimized C++ FIR – data from dm and coeffs from pm ASM version – software loop – FIR filter Other optimizations will become part of Lab. 2 – data from dm and coeffs from pm 6/5/2016 Testing FIR filters Copyright smithmr@ucalgary.ca 12 / 33

13. Lab. 1 – How to test a totally new DSP algorithm For any new algorithm We could write the code in Matlab Then write tests in Matlab to show works We then translate the MatLab code into C Then translate the MatLab tests into C and use them. In Lab. 1, we are going to take a short cut as we know what should be happening with FIR – straight to C and Testing 6/5/2016 Testing FIR filters Copyright smithmr@ucalgary.ca 13 / 33

14. Rough idea of function Insert a new value into the “new” location of a FIFO array Update FIFO filter to throw away oldest value Perform FIR filter operation For (I = 0; I < lengthFIR) sum = sum + FIFO(lengthFIR – I) * FIRcoeffs(I) Return sum Write code in a way that the code is testable 6/5/2016 Testing FIR filters Copyright smithmr@ucalgary.ca 14 / 33

15. Write for Testable means what? Ability to easily test with different length of FIR filters Ability to easily test with different FIR filter coefficients Ability to test with a variety of input values non-Functional test – ability to time execution time Ability to be able to test Outside uTTCOS – automated tests on ‘mock’ data Inside uTTCOS – human acceptance test using real audio data – Should be smooth audio performance – not hick-cups caused by lost signals 6/5/2016 Testing FIR filters Copyright smithmr@ucalgary.ca 15 / 33

16. What to worry about FIR filter of length ? Actual values of FIR coefficient are ???? Unimportant when not inside uTTCOS as we are not expecting ‘acceptable’ audio signals, just correct results Very important when used in human testing – inside uTTCOS Must be stable – otherwise will blow your ears out Must be known so you can use your ears to validate that the correct sort of filtering is occurring 6/5/2016 Testing FIR filters Copyright smithmr@ucalgary.ca 16 / 33

17. Story 1 – Basic ‘does anything work’ test FIR filter of length 4 “Fake” known FIR coefficients – A0, A1, A2 and A3 A0 = 2.0 (never use 1.0 since 1.0 * 1.0 is the same as 1.0 * 1.0 * 1.0 and 1.0 – meaning if there is a mistake in coding you would never know A1 = A2 = A3 = 0.0 (at the moment) Process the “fake” audio array Input[ ] = 0, 0, 0, 3, 0, 0, 0, 0, 0, 3, 0, 0, 0, 0, 0, 0 Note that the impulses of amplitude “3” are further apart than FIRlength 6/5/2016 Testing FIR filters Copyright smithmr@ucalgary.ca 17 / 33

18. Story 1 – Basic ‘does anything work’ test Input[ ] = 0, 0, 0, 3, 0, 0, 0, 0, 0, 3, 0, 0, 0, 0, 0, 0 Note that the impulses of amplitude “3” are further apart than FIRlength Values read out of the array and placed in Leftchannel 1 each time we apply the filter Used as In1 = 0; In2 = 0; In3 = 0; In4 = 0; In5 = 0 etc 6/5/2016 Testing FIR filters Copyright smithmr@ucalgary.ca 18 / 33

19. What is expected result? Write a test, Watch it Fail when only a stub CPP or ASM function is present the test of the test. Must fail when no code is present otherwise is useless test Write the code to satisfy the test The test of the code Remember – the test writing is expected to be easy for the C++ code So generate sensible written tests for the C++ code and modify them for use when we optimize the assembly code 6/5/2016 Testing FIR filters Copyright smithmr@ucalgary.ca 19 / 33

20. What is the expected result? Fake FIR coefficients 2.0, 0, 0, 0 Therefore result (two sets of impulse responses) would be filteredInput[ ] = 0, 0, 0, 6, 0, 0, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0 6/5/2016 Testing FIR filters Copyright smithmr@ucalgary.ca 20 / 33 DATA FILTER COEFFS ARRANGED IN REVERSED ORDER as A0 is multiplied by newest value

21. Go to.xls file 6/5/2016 Testing FIR filters Copyright smithmr@ucalgary.ca 21 / 40

22. FIR in imaging a row FIR filter of size 40 on image of size 256 Get initial and final transients Image might look like this B1, B1, B1, B1, B1, I, I, I, I, I, I, I, I, B2, B2, B2, B2 Where B is background (or more realistically – valid image but not of great interest So we prime the filter to avoid transients by assuming image repeats in space B2, B2, B2, B2, B1, B1, B1, B1, B1, I, I, I, I, I, I, I, I, B2, B2, B2, B2, B1, B1, B1, B1, B1 6/5/2016 Testing FIR filters Copyright smithmr@ucalgary.ca 22 / 33

23. What is the expected result? FIR coefficients 5.0, 0, 0, 0, 0, 0, input[ ] = 0, 0, 0, 3, 0, 0, 0, 0, 0, 4, 0, 0, 0, 0, 0, 0 Therefore result (two sets of scaled impulse response) would be filteredInput[ ] = 0, 0, 0, 15, 0, 0, 0, 0, 0, 20, 0, 0, 0, 0, 0, 0 6/5/2016 Testing FIR filters Copyright smithmr@ucalgary.ca 23 / 33

24. Story Want to be able to evaluate what happens (speed issues with compiler) when you write the code with fixed-length external arrays FIR[32] rather than FIR[N] where N can be any number, and array outside function (global) Want to be able to evaluate what happens (speed issues with compiler) when you write the code with passing array parameters into the filter 6/5/2016 Testing FIR filters Copyright smithmr@ucalgary.ca 24 / 33

25. Write the Test – FIR with parameter passing TEST(Story1_A1) { float input[ ] = 0, 0, 0, 3.3, 0, 0, 0, 0, 0, 4.4, 0, 0, 0, 0, 0, 0; float FIRcoeffs[ ] = 2.0, 0, 0, 0; (4 FIR coeffs) float expectedResult[ ] = 0, 0, 0, 6.6, 0, 0, 0, 0, 0, 8.8, 0, 0, 0, 0, 0, 0; float actualResult[ ] = 0; for (I = 0; I < length(input)) actualResult[I] = FIRparamPassing(input[I], FIRcoeffs, length(FIRcoeffs)); CHECK_ARRAY_CLOSE(expectedResult, actualResult, length(expectedResult), threshold); } 6/5/2016 Testing FIR filters Copyright smithmr@ucalgary.ca 25 / 33

26. Write the Test – FIR with parameter passing V2 TEST(Story1_A2) { float input[ ] = 0, 0, 0, 3.3, 0, 0, 0, 0, 0, 4.4, 0, 0, 0, 0, 0, 0; float FIRcoeffs[ ] = 2.0, 0, 0, 0, 0, 0; (6 coefficients) float expectedResult[ ] = 0, 0, 0, 6.6, 0, 0, 0, 0, 0, 8.8, 0, 0, 0, 0, 0, 0; float actualResult[ ] = 0; for (I = 0; I < length(input)) actualResult[I] = FIRparamPassing(input[I], FIRcoeffs, length(FIRcoeffs)); CHECK_ARRAY_CLOSE(expectedResult, actualResult, length(expectedResult), threshold); } 6/5/2016 Testing FIR filters Copyright smithmr@ucalgary.ca 26 / 33

27. Write the Test – FIR with static array float staticArray[ ] = { 0.} // The C++ optimizer may find FIR filters accessing external arrays might be easier to optimize (fixed location) FIRTEST(Story1_B) { float input[ ] = 0, 0, 0, 3, 0, 0, 0, 0, 0, 3, 0, 0, 0, 0, 0, 0; float FIRcoeffs[ ] = 2.0, 0, 0, 0; float expectedResult[ ] = 0, 0, 0, 6, 0, 0, 0, 0, 0, 6, 0, 0, 0, 0, 0, 0; float actualResult[ ] = 0; Copy(FIRcoffs, staticFIRcoeffs, length(FIRcoeffs)); for (I = 0; I < length(input)) inValue = input[I]; // Global variable FIRstaticNum4( ); // Read global input, coefficients and produces global output actualResult[I] = outValue; } CHECK_ARRAY_CLOSE(expectedResult, actualResult, length(expectedResult), threshold); } 6/5/2016 Testing FIR filters Copyright smithmr@ucalgary.ca 27 / 40

28. Write the Test – FIR with static array V2 TEST(Story1_B) { float input[ ] = 0, 0, 0, 3, 0, 0, 0, 0, 0, 3, 0, 0, 0, 0, 0, 0; float FIRcoeffs[ ] = 2.0, 0, 0, 0, 0, 0; float expectedResult[ ] = 0, 0, 0, 6, 0, 0, 0, 0, 0, 6, 0, 0, 0, 0, 0, 0; float actualResult[ ] = 0; Copy(FIRcoffs, staticFIRcoeffs, length(FIRcoeffs)); for (I = 0; I < length(input)) inValue = input[I]; FIRstaticNum6( ); // Using different FIR filter routine for actualResult[I] = outValue; // Each filter length might be faster } CHECK_ARRAY_CLOSE(expectedResult, actualResult, length(expectedResult), threshold); } 6/5/2016 Testing FIR filters Copyright smithmr@ucalgary.ca 28 / 33

29. Write the stubs – expect to fail File 1 float FIRparamPassingIndex(float inValue, float FIRcoeffs[ ], int numCoeffs( ) { return -3.1459; } File 2 float FIRparamPassingPointer(float inValue, float *FIRcoeffs, int numCoeffs( ) { return -5.1459; } 6/5/2016 Testing FIR filters Copyright smithmr@ucalgary.ca 29 / 33

30. Write the stubs – expect to fail File 3 float outvalueNum4Index = 5.6789; float FIRcoeffsNum4[ ] = { 0 }; void FIRstaticNum4Index(void) { // Implement via Index outvalueNum4Index = -8.1459; } File 4 float outvalueNum6Index = 7.6789; float FIRcoeffsNum6[ ] = { 0 }; void FIRstaticNum6Index(void) {// Implement via Index outvalueNum6Index = -9.1459; } 6/5/2016 Testing FIR filters Copyright smithmr@ucalgary.ca 30 / 33

31. Write the Test – FIR with static array – V3 TEST(Story1_C) { float input[ ] = 0, 0, 0, 3, 0, 0, 0, 0, 0, 3, 0, 0, 0, 0, 0, 0; float FIRcoeffs[ ] = 2.0, 3.0, 4.0, 5.0; float expectedResult[ ] = 0, 0, 0, 6, 9.0, 12.0, 15.0, 0, 0, 6, 9.0, 12.0, 15.0, 0, 0, 0; float actualResult[ ] = 0; Copy(FIRcoffs, staticFIRcoeffs, length(FIRcoeffs)); for (I = 0; I < length(input)) inValue = input[I]; FIRstaticNum4( ); actualResult[I] = outValue; } CHECK_ARRAY_CLOSE(expectedResult, actualResult, length(expectedResult), threshold); } 6/5/2016 Testing FIR filters Copyright smithmr@ucalgary.ca 31 / 33

32. Write the Test – FIR with static array – V4 TEST(Story1_C) { Each input produces an “impulse response” float input[ ] = 0, 0, 0, 3, 0, 0, 3, 0, 3, 0, 0, 0, 0; float FIRcoeffs[ ] = 2.0, 3.0, 4.0, 5.0; float expectedResult[ ] = 0, 0, 0, 6, 9, 12, 15 + 6, 0 + 9, 12 + 6, 15+ 9, 12, 15, 0; float actualResult[ ] = 0; Copy(FIRcoffs, staticFIRcoeffs, length(FIRcoeffs)); for (I = 0; I < length(input)) inValue = input[I]; FIRstaticNum4( ); actualResult[I] = outValue; } CHECK_ARRAY_CLOSE(expectedResult, actualResult, length(expectedResult), threshold); } 6/5/2016 Testing FIR filters Copyright smithmr@ucalgary.ca 32 / 33

33. When do you have enough tests? It is a standard question Salary tax function If salary < 10000 then no tax If 10000 <= salary < 20000 then 10% tax If 20000 <= salary < 40000 then 20% tax If 40000 <= salary then 30% tax Partition of space Salary = 5000, 15000, 25000, 40000 – typical Salary = -10000, 0 -- special cases Salary = 10000, 9999, 10001 – boundary test Etc. 6/5/2016 Testing FIR filters Copyright smithmr@ucalgary.ca 33 / 33

34. How do you test FIR algorithm? void FIRNum4(void) { FIFO[3] = invalue; float sum = 0; for (int I = 0; I < 3; i++) { sum = sum + FIFO[I] * FIRcoeffs[I]; } outValue = sum; for (int I = 0; I < 2; i++) { FIFO[I] = FIFO[I + 1]; } 6/5/2016 Testing FIR filters Copyright smithmr@ucalgary.ca 34 / 33

35. Introduce deliberate mistakes You have a Personal Software Process You know what sort of errors you typically make E.g. Too many times around the loop Forgetting important constants Go in a deliberately make the mistake you commonly make See what sort of tests spot that mistake Always include those sort of tests 6/5/2016 Testing FIR filters Copyright smithmr@ucalgary.ca 35 / 33

36. Test should spot the following error void FIRNum4(void) { FIFO[3] = invalue; float sum; // float sum = 0; for (int I = 0; I < 3; i++) { sum = sum + FIFO[I] * FIRcoeffs[I]; } outValue = sum; for (int I = 0; I < 2; i++) { FIFO[I] = FIFO[I + 1]; } 6/5/2016 Testing FIR filters Copyright smithmr@ucalgary.ca 36 / 33

37. Exam question Why might this incorrect algorithm pass the first test? Give me example code that would probably cause this incorrect algorithm always pass tests void FIRNum4(void) { FIFO[3] = invalue; float sum; // float sum = 0; for (int I = 0; I < 3; i++) { sum = sum + FIFO[I] * FIRcoeffs[I]; } etc 6/5/2016 Testing FIR filters Copyright smithmr@ucalgary.ca 37 / 33

38. Code causing test to pass TEST(Story1_C) { float input[ ] = 0, 0, 0, 3, 0, 0, 3, 0, 3, 0, 0, 0, 0; float FIRcoeffs[ ] = 2.0, 3.0, 4.0, 5.0; float expectedResult[ ] = 0, 0, 0, 6, 9, 12, 15 + 6, 0 + 9, 12 + 6, 15+ 9, 12, 15, 0; float actualResult[ ] = 0; Copy(FIRcoffs, staticFIRcoeffs, length(FIRcoeffs)); for (I = 0; I < length(input)) inValue = input[I]; DoGarbage Function( ); FIRstaticNum4( ); actualResult[I] = outValue; } CHECK_ARRAY_CLOSE(expectedResult, actualResult, length(expectedResult), threshold); } 6/5/2016 Testing FIR filters Copyright smithmr@ucalgary.ca 38 / 33

39. Example – why does DoGarbage( ) make FIRNum4( ) work? void FIRNum4(void) { FIFO[3] = invalue; float sum; // float sum = 0; for (int I = 0; I < 3; i++) { sum = sum + FIFO[I] * FIRcoeffs[I]; } etc void DoGarbage(void) { int garbage = 0; } 6/5/2016 Testing FIR filters Copyright smithmr@ucalgary.ca 39 / 33

40. POSSIBLE solution to incorrect Test results Have a couple of tests that are similar TEST(Story1_C) { float input[ ] = 0, 0, 0, 3, 0, 0, 0, 0, 0, 3, 0, 0, 0, 0, 0, 0; float FIRcoeffs[ ] = 2.0, 3.0, 4.0, 5.0; float expectedResult[ ] = ….. } TEST(Story1_D) { float input[ ] = 3, 0, 0, 0, 0, 0, 3, 0, 0, 0, 0, 0, 0; float FIRcoeffs[ ] = 2.0, 3.0, 9.0, -5.0; float expectedResult[ ] = ….. } Use the automated framework to run the list of tests both forward and backwards 6/5/2016 Testing FIR filters Copyright smithmr@ucalgary.ca 40 / 33

41. Mututation testing Have a tool that deliberately generates a new version of the file with an inserted mistake E.g. change Change + to – Change * to + (note 2 * 2 = 2 + 2) 6/5/2016 Testing FIR filters Copyright smithmr@ucalgary.ca 41 / 33

42. How do you test FIR algorithm? void FIRNum4(void) { FIFO[3] = invalue; float sum = 0; for (int I = 0; I <= 3; i++) { sum = sum + FIFO[I] * FIRcoeffs[I]; } outValue = sum; for (int I = 0; I < 2; i++) { FIFO[I] = FIFO[I + 1]; } 6/5/2016 Testing FIR filters Copyright smithmr@ucalgary.ca 42 / 33

43. How do you test FIR algorithm? void FIRNum4(void) { FIFO[3] = invalue; float sum = 0; for (int I = 0; I < 3; i++) { sum = sum - FIFO[I] * FIRcoeffs[I]; } outValue = sum; for (int I = 0; I < 2; i++) { FIFO[I] = FIFO[I + 1]; } 6/5/2016 Testing FIR filters Copyright smithmr@ucalgary.ca 43 / 33

44. Test should spot the following error void FIRNum4(void) { FIFO[3] = invalue; float sum; // float sum = 0; for (int I = 0; I < 3; i++) { sum = sum + FIFO[I] * FIRcoeffs[I]; } outValue = sum; for (int I = 0; I < 2; i++) { FIFO[I] = FIFO[I + 1]; } 6/5/2016 Testing FIR filters Copyright smithmr@ucalgary.ca 44 / 33