Computational Rocket Science How we built the PSAS LV2 avionics How you can build your own Bart Massey bart@cs.pdx.edu Computer Science Department Portland State University

About This Course PSAS Rocketry Avionics Embedded system view Technology view Autonomous agent view Project view Subsystem project

About Me 6th year Asst. Professor of Computer Science at Portland State University Interests in open source, software engineering, AI, embedded systems Degrees: B.A. Physics, Reed College Thesis M.Sc. CS, University of Oregon Ph.D. CS, U. Oregon CIRL

Why Rockets? Get to play with smart people Get to look cool Get to build software and hardware Lots of hard problems

About PSAS ~ 8 years old, ~ 15 active members, ~ 40 members total World's best amateur rocket avionics (as far as I know) Me: “faculty advisor” = take credit for student genius work

How A Rocket Works Newton's 2nd Law Fuel out bottom What about mass loss, drag? Key idea: state vars at instant + simulate to predict 1 2 𝑚 𝑓 𝑣 2 = 𝑚 𝑟 𝑔ℎ

State Variables x,y,z & rho, phi, theta & their 1st and 2nd derivatives Relation between complicated, so ignore it when possible

Split The Problem Treat instances separately = 4-way split of problem Navigation: Where is rocket now? Control: Where should rocket go? Communication: Human on ground finds out state, changes state Simulation: What will rocket do?

The Navigation Problem Given measurements of current rocket state variables... ...and estimates of current/future rocket state: Update current/future estimates

Why Navigation Is Hard Sensors give what they give: must guess state indirectly these inference algorithms are hard at least the air & space are empty! Compute power is limited little embedded computers doing many other things than nav

The Control Problem Given estimates of current/future rocket state... ...and wanted future state: Make the rocket do what is wanted e.g. open a parachute e.g. follow orbital path

Why Control Is Hard Must use estimates from nav: these are likely wrong “I canna change the laws of physics, Captain!” Control can only get certain small changes from rocket Math is hard, and must be fast (Pencil-balancing exercise)

The Communications Problem Send current measurements, estimates, plan to the ground for human checks of safety goodness to record for later analysis Maybe send commands to rocket usually in an emergency

Why Comms Is Hard Long way away, but power goes like inverse square Antennas want to stick out Never enough bandwidth send safety info send full state estimates send full measurements send SW state ...

The Simulation Problem Given models of ... Sensors (instruments) Effectors (controls) Environment (physics, comm) Onboard computers ... Determine long-term behavior under normal conditions during failure

Why Simulation Is Hard Want accurate models—impossible Want fastest possible computation Want to be able to “mix & match” e.g. LV2 FC SW (real!) e.g. LV2 CAN node (real!?) Programmer time is for “real work”

More Problems: Reliability Rockets are complex! This means many chances for things to break Two ways to fight it: Simplify (not always possible) Backup systems (more complex!) (c.f. on-board video downlink)

Real Problems: Safety Not the same as reliability! We want to allow system to fail when it might hurt someone Again, can simplify or backup Safety analysis is hard: simplify Get big margin of error from distance over-engineering not software

Parts Of A Rocket Project Airframe Motor Avionics & Software Recovery System Payload (optional) Ground Systems

Airframe Mostly a boring metal tube Stuff sticks out of it makes drag hard to compute tears off Use Finite-Element Analysis to design (I guess this is hard)

Rocket Motors Solid (simple), Hybrid (throttle) Deliver total energy (“impulse”, N∙s) Solids have fixed burn time Want high exhaust v (“ISP”)

Avionics: Rocket Electronics All the stuff we've talked about Lots and lots in our rocket Every (modern) rocket has some Several kinds power computers radios instruments & controls

Avionics: Rocket Software Every CPU needs its software Hard to write distributed, real-time, concurrent often small CPU, memory, etc low-level, tightly tied to HW mission, safety critical C.f. medical devices

Recovery System Rockets fall down (fins) Use parachute(s) Must be reliable Triggered by FC

Ground Systems Launch Control Telemetry downlink Should be (are) combined Lots of radios (lots of trouble) Need proper testing / V&V, but rarely get it LC fail safe: don't launch

Onboard System Architectures Monolith Star Ad hoc

Buses Lots of choices: ethernet, USB, serial Want power with comm Of course, ad hoc interconnect also popular We chose CAN Real systems always have ad hoc

CAN “Controller Area Network” ~1 Mb/s true bus (for cars) Prioritized, failure-resistant Tiny 16B packets & funny addresses Many embedded devices, ugly

FC SW Architectures State machine polling Threaded dataflow Threaded messaging Blackboard

Embedded Systems Common features: Single function Small, cheap Low-power Real-time control Big business

Linux and Rockets Scalability: onboard, ground, LC, sim, etc. Free and open source Nice application space Awkward low-level, real-time

“Real-Time” “If it's late, it's wrong” “hard” vs “soft” real-time End-to-end: no “partly” Linux is “soft” real-time Rockets need milliseconds

Finite State Machines Standard embedded trick Edges for state transitions

State Change Triggers Sensor change Time change Complex heuristic External control

Rocket States Ready Launch Boost Coast Descent Drogue Main Landing + Error states!

Microcontrollers Many are around. We use... 8-bit Microchip PIC 16/17/18x 32-bit ARM We want QFP, not BGA Keep C programs simple!

FC Reboot! At least two kinds: cold boot: at start test things well warm boot: on error must happen fast must be able to tell how much time was lost? state machine makes it easier

Override! Must listen to ground commands Nice: can shut off FC and take over manual ground control, but.... May not be able to ground control Remember: the rocket moves quite fast; debug+fix cycle during flight?? Safety overrides are critical!

Testing Really “V&V”: Validation (does system meet requirements?) & Verification (do subsystems meet specification?) Testing + inspection + formal methods We should do structured inspections!

Requirements?? Traditional SE: build careful requirements spec document In open source, specs are often handled loosely OK for mission/safety critical systems? (maybe not) But critical reqs should be enforced outside SW anyhow...

Code Reading Code walkthroughs “Pair programming” Part of open source philosophy: “Many eyes make shallow bugs” --Eric Raymond C is hard to read: use wisely

Unit Test Pick single function or block: Does “unit” work? (framework) Test case = input + correct output(s) Tests before code? (XP) Assertions: not unit tests, but a good thing We should do more unit tests

System Test Can test subsystems as with units Test case = input + correct output(s) is complex, hard CAN nodes are nice test unit: test HW + SW, simple function System test FC by simulation

Simulation Testing If you simulate... physics etc. instruments / control ...you can use to “flight-test” SW Best: test real SW, pref unchanged OK: hack SW a bit to fit in sim Not so good: test model of SW 100s of sim flights killed many bugs

Physics Simulator Remember: evolve state of rocket over small time steps Compute next state from current state & accelerations / forces Can be very crude, or very accurate For predicting flight, want accurate For testing, crude & fast is OK

Exercise: Build A Physics Sim Parameters: motor = 10s burn @ 10g chute = 10 m/s, ff = 100 m/s Hints: g = 10 m/s2, drag acc = -kv2 dv = a dt, dx = v dt

Exercise: Physics Sim Questions What if time step is too small? Too large? How long is the coast phase compared to boost phase? Without air resistance? How would you hook in instruments? FC?

Discrete Event Simulator Slow to take tiny time steps Discrete Event Simulator (class you missed, sorry) uses a priority queue to jump to next time something interesting will happen But physics? Big-step physics? Hybrid sim (what we do now)?

Instrument Simulator CAN node architecture means nice unit for instrument simulation But what to do with node SW? Model it (what we do now)? Simulate C & real node SW? How to test node HW & SW? Mix & match real / sim instruments?

V&V Is Hard Testing etc is a lot of work! Would rather work on building, but We find that test time more than pays for lost build time In any case, mission / safety critical stuff must get thorough V&V We need to V&V non-avionics!

Technologies So far, “common-sense” systems design and SE for avionics But making some things work needs clever tech tricks navigation DSP sensor fusion control theory

Inertial Navigation Recall: nav error adds up badly (double integral / sum) Soln: fuse IMU (good source of short- term data) with GPS (poorer absolute data), other? GAINS: GPS-Aided Inertial Nav Sys

GPS satellite satellite satellite rocket

Tricks With GPS Differential GPS: calculate from nearby fixed GPS to cancel error Doppler GPS: GPS velocity measurements way better than position measurements Phase Differential GPS: One in the nose, one in the tail, ...!

Other Navigation Sources Video (very hard) Solar cell 3D magnetometer (in progress) Ground radar uplink ???

Integrating Navigation Data We have: a bunch of measurements of different state variables, most of which disagree at least some We want: accurate estimate of all state variables now Sensor fusion: combine measurements in best possible way (MLE)

Bayes' Rule pr(H|E)∙pr(E) = pr(HE) = pr(E|H)∙pr(H) So pr(H|E) = pr(E|H)∙pr(H) / pr(E) We want LHS, usually have RHS Let E = measurement Let H = state var (e.g. position) Bayes' Rule tells us how to combine nav estimate with measurement to get new nav estimate

Naïve Bayesian Reasoning Assume (wrongly) independence Then pr(E1E2...) = pr(E1)∙pr(E2)∙... Now Bayes' Rule tells us how to combine many measurements with current estimate This works OK in practice!

Bayesian Particle Filtering 1) Generate point estimates from probability density 2) Evolve each point using measurements, Bayes' Rule 3) Resample to get new probability density Open source online for this

Bayesian Particle Filtering Illustrated

Digital Signal Processing Ref: “Discrete Time Signal Processing” Oppenheim, Schafer, and Buck Given samples of signal(s): Produce samples of new signal(s) Analyze signal(s) Major field, ~4 slides: get help

FIR Filters Average signal over last 5 samples = “low pass filter”, but... Can do better with weighted avg frequency

IIR Filters Feedback filter output into input more filter per unit work harder to make work right Classic example: EWMA filter 𝑥 𝑖1 = 𝑗=0 𝑘 2 −𝑗 𝑥 𝑖−𝑗

FFT Any repetitive signal can be viewed as amplitudes & phases of frequencies FFT converts between digital time domain & frequency domain Requires n log n time Easy to overuse / misuse

Implementing DSP Need fast MAC (multiply & add) Want floating point Overflows / underflows are a disaster Most FPUs have fast MAC Most embedded CPUs have no floating point---limits DSP

Control Some kinds are boring (chutes) But continuous is hard Another two slides on a complex field that can be a whole career: get help

Damping Under, over, critically damped

Hybrid Control & Domains Sometimes when things change, need to change whole control plan Idea: cut parameter space up into regions, with different control for each Result: “Hybrid Control”---standard way of dealing with the problem

Autonomous Agent View Agent: has state sensors effectors Rocket = Autonomous Agent Intelligent Agent?

Artificial Intelligence Field of study since 1960s These days, less about thinking like a person, more about making good decisions in tough spots Thus, nice basis for rockets Today, just one example: learning

Naïve Bayesian Machine Learning Recall Naïve Bayesian Now, imagine simulating with control settings, launch outcome Use this data to get Bayesian estimate of control settings that give best outcome Better yet, hybrid control: do under many situations

Magic Bullet / Smarter Tim Menzies and I got NASA money (SBIR, US$68,000) to do this for LV2 Not just simulation (offline), but maybe during flight (online)---recover from unexpected faults! buzzword = Intelligent Vehicle Health Maintenance Systems

Project and Process OK, most of what has been said so far is good for NASA/ESA, not just PSAS What makes us unique? low-cost volunteer open hardware / open source open process

Open Source Come to my talk last year Idea: open development of code made freely available in source form. Usually way different process than closed source development emphasis on tools worldwide, asynchronous dev community

Open Hardware Idea: Try to achieve benefits of open source for HW projects PSAS (& other Portland State projects) lead in open HW dev Barriers: fewer folks can play, hard to get tools, harder to play

Hardware / Software Co-Development Really important for PSAS: Develop HW with SW Normal industry practice But not so much in open source Allows way better tradeoffs Requires skilled team

Design Agreement One of the hardest things is to get agreement on design PSAS process is pretty standard: controlled consensus (strong leader(s)) & smart discussion Architecture is group, details are implementer's choice

Wiki Getting volunteers to document is very hard Wiki provides low-cost way to capture non-code Also serves as community bulletin board Also public face for PSAS

Source Code Management Critical for open source We moved from CVS to Subversion Critical functions: capture history allow working together part of backup plan

Kinds of Software Have mixture of bog-standard GUI apps Linux device drivers embedded C code microcontroller assembly etc Tough to manage so many things Tough to integrate them

HW Engineers As Coders SEs have a lot to learn from HW engineers about coding One key problem: HW modularity Make your HW engineers get code review pseudocode take your architecture make good HW/SW tradeoffs

Debugging Device Drivers Debugging is hard Debugging code buried in a device driver is very hard PSAS case study: Intel CAN controller

Debugging C Firmware Different kind of hard buggy tools tiny memory poor I/O PSAS case study: recovery node

System Debugging Very complex system, with lots of emergent behavior So, need to instrument very well Sometimes better to redesign and reimplement than debug But this can be carried way too far

Scenarios and Analysis It is easy to lose your way in trees Scenarios let you see the forest Build a library of scenarios, and use them in design, implementation, V&V (We need to do this)

Lessons From The PSAS School Of Hard Knocks I've mentioned a number of places where we need to improve at PSAS Also some places where we did improve Let's sum up some of these lessons

KISS: Keep It Simple, Stupid Most of our troubles are from very complex systems If you can make a system simpler without major loss, just do it Must be able to tell things that are just very wanted from things that are really needed

Take Small Steps Every one of our new avionics systems has taken 2-3 times longer to complete than we planned Mainly because we keep starting over rather than small steps Working on a mostly-working system is way easier

Check Everything So far, our major problems have come from not enough V&V failure to launch due to many bugs loss of airframes w/o mechanical V&V Better simulation has really helped the avionics

Freeze Early Our biggest source of failures to launch has been not getting things done in time Working on site does have advantages: pressure + everyone there + esprit de corps But it can also be a colossal waste of time

Get Architecture Right Our system architecture we got right early on (almost made a bad wrong turn, though) Our FC SW architecture has changed 4 times in 5 years This is painful for everyone, and probably unnecessary

Track Team Status SW & HW engineers are really bad at people / team management Worse, each team tends to work by itself Result: missed deadlines, bad coordination, loss of people Wiki helps, email list helps, but need a person in charge

Leading Is Hard So make sure your leaders are up to the job Also, spread the job among many leaders Responsibility is leader job #1 leading by doing leading by managing leading by communicating

Balance Cost and Work This is a real PSAS sore spot! our goals are to have open HW / SW systems anyone can afford to build yet we need to get things done Must sometimes seek and spend money to make progress hard to find folks to make effort need to find best leverage for money

Concluding Comments Rocketry is huge fun The way PSAS does it is great Try it! get involved with PSAS build your own club—start small and get some help: you can do it Keep in touch. I'm not hard to find

Project Options (Pick One) Select a subsystem (small or large) of PSAS rocket. Do some analysis of it (e.g. read code, run on simulator, examine history). Write a report. Design a PSAS experiment payload. Report on how you would tie it into the PSAS avionics. Extend your simulator. Write a report on the simulator.