1. Emir Zeylan - 6036791 Stylianos Filippou - 6224598
DeepLoco: Dynamic Locomotion Skills Using
Hierarchical Deep Reinforcement Learning
Emir Zeylan
Stylianos Filippou

2. Learning Physics-based locomotion skills
Learning Physics-based locomotion is difficult problem
Technique for achieving locomotion skills with limited prior knowledge
Technique that uses Reinforcement learning

3. Related work Physics-based Character Control
Controllers that are developed around a FSM structure and make use optimization methods to improve the results, like policy search and trajectory optimization.
RL learning for Simulated Locomotion
Agents in 2D and 3D physics-based simulations that learn specific tasks with minimum prior knowledge.
Motion Planning
Path planner that can be used to compute steering and forward speed commands to the locomotion controller to navigate in the environment

4. Two-level hierarchical control framework
LLC (Low Level Controller):
Responsible for coordinating joint torques to achieve the goal given from HLC
HLC (High Level Controller):
Responsible for high-level task-specific objectives
SIM (Simulation):
Simulation of Bipedal Locomotion
Overview
Two-level hierarchical control framework

6. Policy representation and learning
Both controllers are trained with common policy learning algorithm
During training an action is selected based on a stochastic or a deterministic policy. This can be done using the equation:
The objective is to get an optimal policy, which maximize the rewards Rt in the long run.
Policy representation and learning

7. Low-level controller LLC State
Consists mainly of features describing the character’s configuration
LLC Goal
Consists a footstep plan including the position for the character swing foot, the target location and root heading for the next step.
LLC Action
The action aL indicates for each joint the target position for PD controllers.
Low-level controller

8. Reference Motion
Joint torques coordination has as purpose to mimic reference motion
Helps to achieve a desired walking style
At each timestep a reference motion provides a reference pose and a reference velocity.
Multiple reference motion clips are used to achieve better result
A kinematic controllers is constructed to make use of multiple motion clips.
The selection of the most suitable motion is achieved with the use of the extracted clip features

9. Low Level Reward
Reward rL helps the user to guide the behaviour of the agent by changing the reward function provided as input to the system
The LLC reward rL is defined as a weighted sum of objectives that encourage the character to imitate the style of the reference motion while following the footstep plan.

10. Bilinear Phase Transform
Helps the LLC to synchronize with reference motion and better distinguish between different phases
Inspiration of bilinear pooling models
The equation indicates at each phase φ the current state and the current goal that must be achieved

11. Low Level Network & Training
LLC Network
The LLC is represented by a 4-layered neural network that receives as input sL and gL, and outputs the aL.
LLC Training
LLC training proceeds episodically where the character
is initialized to a default pose at the beginning of each episode.

13. High-level controller
HLC State
The state is consist of information regarding the character and the environment
HLC Goal
A specified High Level task
HLC Action
The process of the high level task goal(gH) produce an action (aH) which is the low level goals
High-level controller

14. High Level Network & Training
HLC Network
Three convolutional layers processing the terrain map. The result is merged with the character features and the goal gH, which will be processed by two convolution layers. Finally, the last layer produces the action aH.
HLC Training
During training the character is initialize at a default pose and each episode end at the 200s timeout or when the character falls

16. High Level Tasks Path Following
Given a rocky terrain needs the character to navigate through paths carved in this terrain
Soccer Dribbling
A task which as the name indicating it requires the character to move a ball through several random locations
Pillar Obstacles
Requires the character to travel across a dense area, Similar to the path following task

17. High Level Tasks Block Obstacles
Another variant of pillar obstacles, with larger blocks
Dynamic Obstacles
A dynamic changing environment to reach target the location

18. Results LLC Performance LLC performance depence on motion clips
HLC Performance
Indicates that HLC is able to learn high level tasks

20. Contributions of the method
With limited amount of prior knowledge the method:
Achieves a significant robust bipedal locomotion.
Achieves a significantly more natural locomotion.
Achieves the ability to walk with multiple styles that can be interpolated.
Achieves challenging tasks as soccer dribbling.

21. Conclusion
Hierarchical learning-based framework for 3D bipedal walking skills with minimal prior knowledge
Easy-directable control over motion style to produce highly robust controllers
Reuse of controllers are allowed by hierarchical decomposition

22. Any questions?

23. Discussion

24. What are the limitations of this method?
It proved difficult for the HLC to achieve good performance when face with a more difficult dynamic obstacles environment.
Without a reference motion, the LLC fails to learn a successful walk, so it depends on them.
Another limitation of LLC is that the training does not consist of stopping but only constant forward walks and turns
Without the hierarchical decomposition, LLC failed to perform their respective tasks.
It proved difficult for the HLC to achieve good performance when face with a more difficult dynamic obstacles environment.
Without a reference motion, the LLC fails to learn a successful walk, so it depends on them.
Without normalization LLC’s robustness is not as they expected(After normalizing for character weight and size dierences) //removed.
TA limitation of LLC is that the training does not consist of stopping but only constant forward walks and turns
Without the hierarchical decomposition, LLC failed to perform their respective tasks.
To train the policies without the control hierarchy, the LLC’s inputs are augmented with gH and for the path following task, the terrain map T is also included as part of the input. Convolutional layers are added to the path following LLC. The augmented LLC’s are then trained to imitate the reference motions and perform the high-level tasks. Without the hierarchical decomposition, both LLC’s failed to perform their respective tasks
The more dicult dynamic obstacles environment, proved challenging for the HLC, reaching a competent level of performance, but still prone to occasional missteps, particularly when navigating around faster moving obstacles. We note that the default LLC training consists of constant speed forward walks and turns but no stopping, which limits the options available to the HLC when avoiding obstacles.

25. Could we apply this method to another high level tasks?
Yes if we provide the appropriate motion clips for the LLC to learn a different style if necessary and if needed some modification on the HLC (as they did for dribbling task) so that the character is encourage to achieve the specific goal.
If the high level task requires the character to stop, then not because of the limitation of this method.

26. Can this technique be used in industry ?
No, but is possible to use in medical area for patients which are struggling with walking. This technique could be applied for simulations to help the patient.
Given that the character simulation consist only of moving motion with no stopping, then for games this will be inappropriate.
But further improvements of this method could seen this technique to be successful in industry as already achieves really complicated tasks.