1. An Interactive System for Hiring & Managing Graduate Teaching Assistants Ryan Lim Venkata Praveen Guddeti Berthe Y. Choueiry Constraint Systems Laboratory University of Nebraska-Lincoln

2. Outline Task & Motivation System Architecture & Interfaces Scientific aspects  Problem Modeling  Problem Solving  Comparing & Characterizing Solvers Motivation revisited & Conclusions

3. Task Hiring & managing GTAs as instructors + graders  Given A set of courses A set of graduate teaching assistants A set of constraints that specify allowable assignments  Find a consistent & satisfactory assignment Consistent: assignment breaks no (hard) constraints Satisfactory: assignment maximizes 1.number of courses covered 2.happiness of the GTAs Often, number of hired GTAs is insufficient

4. Motivation Context  “Most difficult duty of a department chair” [Reichenbach, 2000]  Assignments done manually, countless reviews, persistent inconsistencies  Unhappy instructors, unhappy GTAs, unhappy students Observation  Computers are good at maintaining consistency  Humans are good at balancing tradeoffs Our solution  An online, constraint-based system  With interactive & automated search mechanisms

5. Outline Task & Motivation System Architecture & Interfaces Scientific aspects  Problem Modeling  Problem Solving  Comparing & Characterizing Solvers Motivation revisited & Conclusions

6. System Architecture 1.Web-interface for applicants Password Protected Access for GTAs http://cse.unl.edu/~gta Cooperative, hybrid Search Strategies Other structured, semi-structured, or unstructured DBs In progress Visualization widgets Password Protected Access for Manager http://cse.unl.edu/~gta 2.Web-interface for manager  View / edit GTA records  Setup classes  Specify constraints  Enforce pre-assignments Local DB 3.A local relational database  Graphical selective queries Interactive Search Automated Search Heuristic BT Stochastic LS Multi-agent Search Randomized BT 4.Drivers for  Interactive assignments  Automated search algorithms

7. GTA interface: Preference Specification

8. Manager interface: TA Hiring & Load

9. Outline Task & Motivation System Architecture & Interfaces Scientific aspects  Problem Modeling  Problem Solving  Comparing & Characterizing Solvers Motivation revisited & Conclusions

10. Constraint-based Model Variables  Grading, conducting lectures, labs & recitations Values  Hired GTAs (+ preference for each value in domain) Constraints  Unary: ITA certification, enrollment, time conflict, non-zero preferences, etc.  Binary (Mutex): overlapping courses  Non-binary: same-TA, capacity, confinement Objective  longest partial and consistent solution (primary criterion)  while maximizing GTAs’ preferences (secondary criterion)

11. Outline Task & Motivation System Architecture & Interfaces Scientific aspects  Problem Modeling  Problem Solving  Comparing & Characterizing Solvers Motivation revisited & Conclusions

12. Problem Solving Interactive decision making  Seamlessly switching between perspectives  Propagates decisions (MAC) Automated search algorithms  Heuristic backtrack search (BT)  Stochastic local search (LS)  Multi-agent search (ERA)  Randomized backtrack search (RDGR)  Future: Auction-based, GA, MIP, LD-search, etc. On-going: Cooperative/hybrid strategies

13. Manager interface: Interactive Selection

14. Dual perspective Task-centered viewResource-centered view

15. Heuristic BT Search Since we don’t know, a priori, whether instance is solvable, tight, or over-constrained  Modified basic backtrack mechanism to deal with this situation We designed & tested various ordering heuristics:  Dynamic LD was consistently best Branching factor relatively huge (30)  Causes thrashing, backtrack never reaches early variables 24 hr: 51 (26%) 1 min: 55 (20%) Max depth: 57 Shallowest level reached by BT after … Number of variables: 69 Depth of the tree: 69

16. Stochastic Local Search Hill-climbing with min-conflict heuristic Constraint propagation:  To handle non-binary constraints (e.g., high- arity capacity constraints) Greedy:  Consistent assignments are not undone Random walk to avoid local maxima Random restarts to recover from local maxima

17. Multi-Agent Search (ERA) [Liu et al. 02] “Extremely” decentralized local search  Agents (variables) seek to occupy best positions (values)  Environment records constraint violation in each position of an agent given positions of other agents  Agents move, egoistically, between positions according to reactive Rules Decisions are local  An agent can always kick other agents from a favorite position even when value of ‘global objective function’ is not improved ERA appears immune to local optima Lack of centralized control  Agents continue to kick each other  Deadlock appears in over-constrained problems

18. Randomized BT Search Random variable/value selection allows BT to visit a wider area of the search space [Gomes et al. 98] Restarts to overcome thrashing Walsh proposed RGR [Walsh 99] Our strategy, RDGR, improves RGR with dynamic choice of cutoff values for the restart strategy [Guddeti & Choueiry 04]

19. Optimizing solutions Primary criterion: solution length  BT, LS, ERA, RGR, RDGR Secondary criterion: preference values  BT, LS, RGR, RDGR  Criterion: Average preference Geometric mean Maximum minimal preference

20. More Solvers… Interactive decision making Automated search algorithms  BT, LS, ERA, RGR, RDGR.  Future: Auction-based, GA, MIP, LD- search, etc. On-going: Cooperative / hybrid strategies

21. Outline Task & Motivation System Architecture & Interfaces Scientific aspects  Problem Modeling  Problem Solving  Comparing & Characterizing Solvers Motivation revisited & Conclusions

22. Comparing Solvers Using the same CSP encoding, students implements solvers separately and competed for best results Experience lead to the identification of behavioral criteria and regimes that characterize the performance of the various solvers in the context of GTAP

23. Characterizing Solvers General criteria  Stability, solution length, vulnerability to local optima, deadlock, thrashing, etc. Tight but solvable instances  ERA  RDGR  RGR  BT  LS Over-constrained instances  RDGR  RGR  BT  ERA  LS

24. Outline Task & Motivation System Architecture & Interfaces Scientific aspects  Problem Modeling  Problem Solving  Comparing & Characterization Solvers Motivation revisited & Conclusions

25. Motivation (revisited) “Most difficult duty of a department chair”  Keeps the manager in the decision loop while removing the need for tedious and error-prone manual assignments  Helps producing quick (3 weeks down to 2 days) and satisfactory (stable) assignments Initially, assignments were manually done on paper  Now, on-line data acquisition process  Enabled department to streamline & standardize GTA selection, hiring, and assignment Overworked staff, unhappy GTAs  Overjoyed staff (relieved from handling application forms and massive paperwork)  Enthusiastic anonymous online reviews from applicants

26. History & Evaluation System entirely built by students Modeling started in January 2001 Prototype system used since August 2001  Features improved and added as needs arised No formal longitudinal study Since August 2003: 109 GTA users, 23 feedback responses Since April 2004, CSE implemented on-line GTA evaluation by faculty on top of GTAAP

27. GTA Online Feedback 23 responses Navigation Data entry

28. Conclusions Integrated interactive & automated problem- solving strategies  Reduced the burden of the manager  Lead to quick development of ‘stable’ solutions Our efforts  Helped the department  Trained students in CP techniques  Paved new avenues for research Cooperative, hybrid search Visualization of solution space

29. <<< end of presentation I welcome your questions Please contact me for a live demo

31. Manager interface: Course Load Specification

32. Manager interface: Preassignment

33. Manager interface: Constraint Specification