1. Computational Rocket Science How we built the PSAS LV2 avionics How you can build your own
Bart Massey Computer Science Department Portland State University

2. About This Course
PSAS
Rocketry
Avionics
Embedded system view
Technology view
Autonomous agent view
Project view
Subsystem project

3. About Me
6th year Asst. Professor of Computer Science at Portland State University
Interests in open source, software engineering, AI, embedded systems
Degrees:
B.A. Physics, Reed College
Thesis M.Sc. CS, University of Oregon
Ph.D. CS, U. Oregon CIRL

4. Why Rockets? Get to play with smart people Get to look cool
Get to build software and hardware
Lots of hard problems

5. About PSAS ~ 8 years old, ~ 15 active members, ~ 40 members total
World's best amateur rocket avionics (as far as I know)
Me: “faculty advisor” = take credit for student genius work

6. How A Rocket Works Newton's 2nd Law Fuel out bottom
What about mass loss, drag?
Key idea: state vars at instant + simulate to predict
1 2 𝑚 𝑓 𝑣 2 = 𝑚 𝑟 𝑔ℎ

7. State Variables x,y,z & rho, phi, theta
& their 1st and 2nd derivatives
Relation between complicated, so ignore it when possible

8. Split The Problem Treat instances separately = 4-way split of problem
Navigation: Where is rocket now?
Control: Where should rocket go?
Communication: Human on ground finds out state, changes state
Simulation: What will rocket do?

9. The Navigation Problem
Given measurements of current rocket state variables...
...and estimates of current/future rocket state:
Update current/future estimates

10. Why Navigation Is Hard
Sensors give what they give: must guess state indirectly
these inference algorithms are hard
at least the air & space are empty!
Compute power is limited
little embedded computers
doing many other things than nav

11. The Control Problem Given estimates of current/future rocket state...
...and wanted future state:
Make the rocket do what is wanted
e.g. open a parachute
e.g. follow orbital path

12. Why Control Is Hard
Must use estimates from nav: these are likely wrong
“I canna change the laws of physics, Captain!” Control can only get certain small changes from rocket
Math is hard, and must be fast
(Pencil-balancing exercise)

13. The Communications Problem
Send current measurements, estimates, plan to the ground
for human checks of
safety
goodness
to record for later analysis
Maybe send commands to rocket
usually in an emergency

14. Why Comms Is Hard Long way away, but power goes like inverse square
Antennas want to stick out
Never enough bandwidth
send safety info
send full state estimates
send full measurements
send SW state
...

15. The Simulation Problem
Given models of ...
Sensors (instruments)
Effectors (controls)
Environment (physics, comm)
Onboard computers
... Determine long-term behavior
under normal conditions
during failure

16. Why Simulation Is Hard Want accurate models—impossible
Want fastest possible computation
Want to be able to “mix & match”
e.g. LV2 FC SW (real!)
e.g. LV2 CAN node (real!?)
Programmer time is for “real work”

17. More Problems: Reliability
Rockets are complex!
This means many chances for things to break
Two ways to fight it:
Simplify (not always possible)
Backup systems (more complex!)
(c.f. on-board video downlink)

18. Real Problems: Safety Not the same as reliability!
We want to allow system to fail when it might hurt someone
Again, can simplify or backup
Safety analysis is hard: simplify
Get big margin of error from
distance
over-engineering
not software

19. Parts Of A Rocket Project
Airframe
Motor
Avionics & Software
Recovery System
Payload (optional)
Ground Systems

20. Airframe Mostly a boring metal tube Stuff sticks out of it
makes drag hard to compute
tears off
Use Finite-Element Analysis to design (I guess this is hard)

21. Rocket Motors Solid (simple), Hybrid (throttle)
Deliver total energy (“impulse”, N∙s)
Solids have fixed burn time
Want high exhaust v (“ISP”)

22. Avionics: Rocket Electronics
All the stuff we've talked about
Lots and lots in our rocket
Every (modern) rocket has some
Several kinds
power
computers
radios
instruments & controls

23. Avionics: Rocket Software
Every CPU needs its software
Hard to write
distributed, real-time, concurrent
often small CPU, memory, etc
low-level, tightly tied to HW
mission, safety critical
C.f. medical devices

24. Recovery System Rockets fall down (fins) Use parachute(s)
Must be reliable
Triggered by FC

25. Ground Systems Launch Control Telemetry downlink
Should be (are) combined
Lots of radios (lots of trouble)
Need proper testing / V&V, but rarely get it
LC fail safe: don't launch

26. Onboard System Architectures
Monolith
Star
Ad hoc

27. Buses Lots of choices: ethernet, USB, serial Want power with comm
Of course, ad hoc interconnect also popular
We chose CAN
Real systems always have ad hoc

28. CAN “Controller Area Network” ~1 Mb/s true bus (for cars)
Prioritized, failure-resistant
Tiny 16B packets & funny addresses
Many embedded devices, ugly

29. FC SW Architectures State machine polling Threaded dataflow
Threaded messaging
Blackboard

30. Embedded Systems Common features: Single function Small, cheap
Low-power
Real-time control
Big business

31. Linux and Rockets Scalability: onboard, ground, LC, sim, etc.
Free and open source
Nice application space
Awkward low-level, real-time

32. “Real-Time”
“If it's late, it's wrong”
“hard” vs “soft” real-time
End-to-end: no “partly”
Linux is “soft” real-time
Rockets need milliseconds

33. Finite State Machines
Standard embedded trick
Edges for state transitions

34. State Change Triggers Sensor change Time change Complex heuristic
External control

35. Rocket States Ready Launch Boost Coast Descent Drogue Main Landing
+ Error states!

36. Microcontrollers Many are around. We use...
8-bit Microchip PIC 16/17/18x
32-bit ARM
We want QFP, not BGA
Keep C programs simple!

37. FC Reboot! At least two kinds: cold boot: at start test things well
warm boot: on error
must happen fast
must be able to tell
how much time was lost?
state machine makes it easier

38. Override! Must listen to ground commands
Nice: can shut off FC and take over manual ground control, but....
May not be able to ground control
Remember: the rocket moves quite fast; debug+fix cycle during flight??
Safety overrides are critical!

39. Testing
Really “V&V”: Validation (does system meet requirements?) & Verification (do subsystems meet specification?)
Testing + inspection + formal methods
We should do structured inspections!

40. Requirements??
Traditional SE: build careful requirements spec document
In open source, specs are often handled loosely
OK for mission/safety critical systems? (maybe not)
But critical reqs should be enforced outside SW anyhow...

41. Code Reading Code walkthroughs “Pair programming”
Part of open source philosophy: “Many eyes make shallow bugs” --Eric Raymond
C is hard to read: use wisely

42. Unit Test Pick single function or block: Does “unit” work? (framework)
Test case = input + correct output(s)
Tests before code? (XP)
Assertions: not unit tests, but a good thing
We should do more unit tests

43. System Test Can test subsystems as with units
Test case = input + correct output(s) is complex, hard
CAN nodes are nice test unit: test HW + SW, simple function
System test FC by simulation

44. Simulation Testing If you simulate... physics etc.
instruments / control
...you can use to “flight-test” SW
Best: test real SW, pref unchanged
OK: hack SW a bit to fit in sim
Not so good: test model of SW
100s of sim flights killed many bugs

45. Physics Simulator
Remember: evolve state of rocket over small time steps
Compute next state from current state & accelerations / forces
Can be very crude, or very accurate
For predicting flight, want accurate
For testing, crude & fast is OK

46. Exercise: Build A Physics Sim
Parameters:
motor = 10s 10g
chute = 10 m/s, ff = 100 m/s
Hints:
g = 10 m/s2, drag acc = -kv2
dv = a dt, dx = v dt

47. Exercise: Physics Sim Questions
What if time step is too small? Too large?
How long is the coast phase compared to boost phase? Without air resistance?
How would you hook in instruments? FC?

48. Discrete Event Simulator
Slow to take tiny time steps
Discrete Event Simulator (class you missed, sorry) uses a priority queue to jump to next time something interesting will happen
But physics? Big-step physics? Hybrid sim (what we do now)?

49. Instrument Simulator
CAN node architecture means nice unit for instrument simulation
But what to do with node SW?
Model it (what we do now)?
Simulate C & real node SW?
How to test node HW & SW?
Mix & match real / sim instruments?

50. V&V Is Hard Testing etc is a lot of work!
Would rather work on building, but
We find that test time more than pays for lost build time
In any case, mission / safety critical stuff must get thorough V&V
We need to V&V non-avionics!

51. Technologies So far, “common-sense” systems design and SE for avionics
But making some things work needs clever tech tricks
navigation
DSP
sensor fusion
control theory

52. Inertial Navigation
Recall: nav error adds up badly (double integral / sum)
Soln: fuse IMU (good source of short- term data) with GPS (poorer absolute data), other?
GAINS: GPS-Aided Inertial Nav Sys

53. GPS
satellite
satellite
satellite
rocket

54. Tricks With GPS
Differential GPS: calculate from nearby fixed GPS to cancel error
Doppler GPS: GPS velocity measurements way better than position measurements
Phase Differential GPS: One in the nose, one in the tail, ...!

55. Other Navigation Sources
Video (very hard)
Solar cell
3D magnetometer (in progress)
Ground radar uplink
???

56. Integrating Navigation Data
We have: a bunch of measurements of different state variables, most of which disagree at least some
We want: accurate estimate of all state variables now
Sensor fusion: combine measurements in best possible way (MLE)

57. Bayes' Rule pr(H|E)∙pr(E) = pr(HE) = pr(E|H)∙pr(H)
So pr(H|E) = pr(E|H)∙pr(H) / pr(E)
We want LHS, usually have RHS
Let E = measurement
Let H = state var (e.g. position)
Bayes' Rule tells us how to combine nav estimate with measurement to get new nav estimate

58. Naïve Bayesian Reasoning
Assume (wrongly) independence
Then pr(E1E2...) = pr(E1)∙pr(E2)∙...
Now Bayes' Rule tells us how to combine many measurements with current estimate
This works OK in practice!

59. Bayesian Particle Filtering
1) Generate point estimates from probability density
2) Evolve each point using measurements, Bayes' Rule
3) Resample to get new probability density
Open source online for this

60. Bayesian Particle Filtering Illustrated

61. Digital Signal Processing
Ref: “Discrete Time Signal Processing” Oppenheim, Schafer, and Buck
Given samples of signal(s):
Produce samples of new signal(s)
Analyze signal(s)
Major field, ~4 slides: get help

62. FIR Filters
Average signal over last 5 samples = “low pass filter”, but...
Can do better with weighted avg
frequency

63. IIR Filters Feedback filter output into input
more filter per unit work
harder to make work right
Classic example: EWMA filter
𝑥 𝑖1 = 𝑗=0 𝑘 2 −𝑗 𝑥 𝑖−𝑗

64. FFT
Any repetitive signal can be viewed as amplitudes & phases of frequencies
FFT converts between digital time domain & frequency domain
Requires n log n time
Easy to overuse / misuse

65. Implementing DSP Need fast MAC (multiply & add) Want floating point
Overflows / underflows are a disaster
Most FPUs have fast MAC
Most embedded CPUs have no floating point---limits DSP

66. Control Some kinds are boring (chutes) But continuous is hard
Another two slides on a complex field that can be a whole career: get help

67. Damping
Under, over, critically damped

68. Hybrid Control & Domains
Sometimes when things change, need to change whole control plan
Idea: cut parameter space up into regions, with different control for each
Result: “Hybrid Control”---standard way of dealing with the problem

69. Autonomous Agent View Agent: has state sensors effectors
Rocket = Autonomous Agent
Intelligent Agent?

70. Artificial Intelligence
Field of study since 1960s
These days, less about thinking like a person, more about making good decisions in tough spots
Thus, nice basis for rockets
Today, just one example: learning

71. Naïve Bayesian Machine Learning
Recall Naïve Bayesian
Now, imagine simulating with control settings, launch outcome
Use this data to get Bayesian estimate of control settings that give best outcome
Better yet, hybrid control: do under many situations

72. Magic Bullet / Smarter
Tim Menzies and I got NASA money (SBIR, US$68,000) to do this for LV2
Not just simulation (offline), but maybe during flight (online)---recover from unexpected faults!
buzzword = Intelligent Vehicle Health Maintenance Systems

73. Project and Process
OK, most of what has been said so far is good for NASA/ESA, not just PSAS
What makes us unique?
low-cost
volunteer
open hardware / open source
open process

74. Open Source Come to my talk last year
Idea: open development of code made freely available in source form.
Usually way different process than closed source development
emphasis on tools
worldwide, asynchronous dev
community

75. Open Hardware
Idea: Try to achieve benefits of open source for HW projects
PSAS (& other Portland State projects) lead in open HW dev
Barriers: fewer folks can play, hard to get tools, harder to play

76. Hardware / Software Co-Development
Really important for PSAS: Develop HW with SW
Normal industry practice
But not so much in open source
Allows way better tradeoffs
Requires skilled team

77. Design Agreement
One of the hardest things is to get agreement on design
PSAS process is pretty standard: controlled consensus (strong leader(s)) & smart discussion
Architecture is group, details are implementer's choice

78. Wiki Getting volunteers to document is very hard
Wiki provides low-cost way to capture non-code
Also serves as community bulletin board
Also public face for PSAS

79. Source Code Management
Critical for open source
We moved from CVS to Subversion
Critical functions:
capture history
allow working together
part of backup plan

80. Kinds of Software Have mixture of bog-standard GUI apps
Linux device drivers
embedded C code
microcontroller assembly
etc
Tough to manage so many things
Tough to integrate them

81. HW Engineers As Coders
SEs have a lot to learn from HW engineers about coding
One key problem: HW modularity
Make your HW engineers
get code review
pseudocode
take your architecture
make good HW/SW tradeoffs

82. Debugging Device Drivers
Debugging is hard
Debugging code buried in a device driver is very hard
PSAS case study: Intel CAN controller

83. Debugging C Firmware Different kind of hard buggy tools tiny memory
poor I/O
PSAS case study: recovery node

84. System Debugging Very complex system, with lots of emergent behavior
So, need to instrument very well
Sometimes better to redesign and reimplement than debug
But this can be carried way too far

85. Scenarios and Analysis
It is easy to lose your way in trees
Scenarios let you see the forest
Build a library of scenarios, and use them in design, implementation, V&V
(We need to do this)

86. Lessons From The PSAS School Of Hard Knocks
I've mentioned a number of places where we need to improve at PSAS
Also some places where we did improve
Let's sum up some of these lessons

87. KISS: Keep It Simple, Stupid
Most of our troubles are from very complex systems
If you can make a system simpler without major loss, just do it
Must be able to tell things that are just very wanted from things that are really needed

88. Take Small Steps
Every one of our new avionics systems has taken 2-3 times longer to complete than we planned
Mainly because we keep starting over rather than small steps
Working on a mostly-working system is way easier

89. Check Everything
So far, our major problems have come from not enough V&V
failure to launch due to many bugs
loss of airframes w/o mechanical V&V
Better simulation has really helped the avionics

90. Freeze Early
Our biggest source of failures to launch has been not getting things done in time
Working on site does have advantages: pressure + everyone there + esprit de corps
But it can also be a colossal waste of time

91. Get Architecture Right
Our system architecture we got right early on (almost made a bad wrong turn, though)
Our FC SW architecture has changed 4 times in 5 years
This is painful for everyone, and probably unnecessary

92. Track Team Status
SW & HW engineers are really bad at people / team management
Worse, each team tends to work by itself
Result: missed deadlines, bad coordination, loss of people
Wiki helps, list helps, but need a person in charge

93. Leading Is Hard So make sure your leaders are up to the job
Also, spread the job among many leaders
Responsibility is leader job #1
leading by doing
leading by managing
leading by communicating

94. Balance Cost and Work This is a real PSAS sore spot!
our goals are to have open HW / SW systems anyone can afford to build
yet we need to get things done
Must sometimes seek and spend money to make progress
hard to find folks to make effort
need to find best leverage for money

95. Concluding Comments Rocketry is huge fun The way PSAS does it is great
Try it!
get involved with PSAS
build your own club—start small and get some help: you can do it
Keep in touch. I'm not hard to find

96. Project Options (Pick One)
Select a subsystem (small or large) of PSAS rocket. Do some analysis of it (e.g. read code, run on simulator, examine history). Write a report.
Design a PSAS experiment payload. Report on how you would tie it into the PSAS avionics.
Extend your simulator. Write a report on the simulator.