Information Elicitation in Scheduling Problems

Information Elicitation in Scheduling Problems
Ulaş Bardak Ph.D. Thesis Defense Committee Jaime Carbonell (chair), Eugene Fink, Stephen Smith, and Sven Koenig (University of Southern California)

Ulas Bardak - Thesis Defense
Outline – Introduction – Related Work – Domain – Optimization – Elicitation - Evaluation - Conclusion Outline Introduction Related work Domain Optimization Elicitation Evaluation Conclusions Let’s look at what I will talk about today. First I will briefly introduce what we are trying to do and give a simple example problem. Then I will talk about the related work and state how this work differs from them. I will then introduce the domain we used for implementing our work I will then go over various parts of the implemented system including the optimization and the heart of this work, our elicitation algorithm. I will go over the evaluation process and the results of the evaluation before the concluding. 12/3/2018 Ulas Bardak - Thesis Defense

What is information elicitation?
Outline – Introduction – Related Work – Domain – Optimization – Elicitation - Evaluation - Conclusion What is information elicitation? For example… Mention optimizer in the demo. 12/3/2018 Ulas Bardak - Thesis Defense

Outline – Introduction – Related Work – Domain – Optimization – Elicitation - Evaluation - Conclusion Why elicitation? Scheduling problems include information about resources, constraints, and preferences Uncertain information can lower the quality of schedules We need to select and ask questions that help to reduce uncertainty What do we mean by information elicitation in scheduling problems? Well, a scheduling problem basically consists of taking users and a set of resources and assigning those resources to the users based on their preferences. The thing is, if there is uncertainty in the either the user preferences or properties of resources, the quality of the produced schedule may be lower than the quality of the schedule that would’ve been produced with full certainty. That’s where the elicitation part comes in – we ask questions to the user to resolve uncertainty. 12/3/2018 Ulas Bardak - Thesis Defense

Outline – Introduction – Related Work – Domain – Optimization – Elicitation - Evaluation - Conclusion Example problem We are organizing a small conference, using three available rooms We have incomplete information about speaker needs Let me demonstrate how uncertainty can cause problems and how elicitation can help make the schedules better. Let’s say we are organizing a small academic conference with 2 sessions and 3 rooms. Furthermore, I suppose just like in a real conference setting, we don’t have complete information about the speakers’ needs. 12/3/2018 Ulas Bardak - Thesis Defense

Outline – Introduction – Related Work – Domain – Optimization – Elicitation - Evaluation - Conclusion Initial schedule Available rooms: Initial schedule: Talk Posters Room num. Area Projector Large Med. Small Yes No Yes 1 2 3 Requests (by importance): Invited talk, 9–10am: Needs a large room Poster session, 9-11am: Needs a room Missing info: Invited talk: – Projector need Poster session: – Room size – Projector need Assumptions: Invited talk: – Needs a projector Poster session: – Smaller room is OK – Needs no projector In this particular problem we are assigning rooms to requests and we have three rooms. Room 1 – which is a big room with a projector, room 2 the smaller sized room without a projector and room 3 the small room which also has a projector. Our two requests are the invited talk which, we are told, needs a large room and the poster session about which all we know is that it needs a room. Given this information we realize that certain pieces are missing – we don’t know if either session needs a projector and we don’t know whether or not the poster session needs a large room. Therefore, before we can produce a schedule we have to make some assumptions – we assume that the invited talk probably needs a projector whereas the poster session is probably okay with a smaller room and no projector. Once we make these assumptions, we can build a schedule and assign the talk to the large room and posters to the medium sized room. 12/3/2018 Ulas Bardak - Thesis Defense

Outline – Introduction – Related Work – Domain – Optimization – Elicitation - Evaluation - Conclusion Choice of questions Initial schedule: 1 2 Posters 3 Talk Candidate questions: Invited talk: Needs a projector? Poster session: How big a room? Needs a projector? Requests: Invited talk, 9–10am: Needs a large room Poster session, 9–11am: Needs a room Useless info: There are no large rooms w/o a projector × If we want to make sure we have a good schedule and possibly improve what we have, we can ask questions to the user to resolve uncertainty. In this case there are three potential questions we can ask. If we look closer though we can see that only one of the three actually would be useful in this case – Asking if the invited talk needs a projector would not be very useful since there are no other big rooms with a projector. We can also ask if posters session would need a bigger room because that may determine if the session would prefer to stay in room 2 or be okay in room 3. We can also ask if the posters session would need a projector. Potentially useful info √ Potentially useful info √ 12/3/2018 Ulas Bardak - Thesis Defense

Outline – Introduction – Related Work – Domain – Optimization – Elicitation - Evaluation - Conclusion Improved schedule Requests: Invited talk, 9–10am: Needs a large room Poster session, 9–11am: Needs a room Initial schedule: 1 2 Posters 3 Talk Info elicitation: 1 New schedule: Talk 2 3 System: Does the poster session need a projector? How big a room does it need? So, we go ahead and pose these questions to the user – in this case we see that a projector would be potentially useful in a poster session and that the poster session would be OK in a small room. So we can improve the quality of the schedule by moving the poster session to the room with a projector. Posters User: A projector may be useful. A small room is OK. 12/3/2018 Ulas Bardak - Thesis Defense

Outline – Introduction – Related Work – Domain – Optimization – Elicitation - Evaluation - Conclusion Motivation Improve optimization results by reducing uncertainty of the available knowledge. This brings us to the main motivation – what we are trying to do is to improve the results of optimization, that is the quality of the schedule we produce, by reducing the uncertainty in the knowledge we start out with. 12/3/2018 Ulas Bardak - Thesis Defense

Outline – Introduction – Related Work – Domain – Optimization – Elicitation - Evaluation - Conclusion Related work Example critiquing (Burke) Have users tweak result set Collaborative filtering (Resnick and Hill) Have the user rank related items Similarity-based heuristics (Burke) Look at past similar user ratings Focusing on targeted use (Stolze) Information elicitation has its uses in many different domains and therefore there are many different approaches which has been taken in the past. Some approaches require the users to give a lot of feedback to the system in terms of tweaks on the recommended allocations, rankings of different possible allocations or other users in terms of similarity. Another approach tries to elicit what the targeted use of resources is instead of preferences. 12/3/2018 Ulas Bardak - Thesis Defense

Outline – Introduction – Related Work – Domain – Optimization – Elicitation - Evaluation - Conclusion Related work Clustering utility functions (Chajewska) Decision tree (Stolze and Ströbel) Min-max regret (Boutilier) Choose question that reduces max regret Auctions (Smith, Boutilier, and Sandholm) Some approaches rely on having only a few varying preference functions. If the variance in the possible utility functions is small, they can be clustered together or a decision tree can be built and only a few questions need to be asked by the system. Another big category of approaches rely on asking questions that minimize maximum regret in all possible assignment scenarios. There are also approaches which are used for elicitation in auctions for eliciting things like non-price preferences of bidders. 12/3/2018 Ulas Bardak - Thesis Defense

Outline – Introduction – Related Work – Domain – Optimization – Elicitation - Evaluation - Conclusion What is different? No bootstrapping Both continuous and discrete variables Large number of uncertain variables Tight integration with the optimizer Synergy of multiple approaches Our approach sets itself apart from each of these other works, with at least one of these properties. The system does not need initialization data. We don’t make an assumption that the uncertain variables are discrete or are only few in number. We have a tight integration with the optimizer and we don’t rely on the users to make a lot of tweaks We integrate multiple elicitation approaches together. 12/3/2018 Ulas Bardak - Thesis Defense

Outline – Introduction – Related Work – Domain – Optimization – Elicitation - Evaluation - Conclusion Explored domains An academic conference Assigning rooms to sessions Placing vendor orders Assigning orders to sessions Social networking Matching users to other users The domain we chose for applying our approach is planning an academic conference – like a much more scaled up version of the example. The resources we are trying to assign are rooms and our users are sessions. We are trying to assign the rooms to the sessions and form the conference schedule. We will look at how these are represented, next. 12/3/2018 Ulas Bardak - Thesis Defense

Outline – Introduction – Related Work – Domain – Optimization – Elicitation - Evaluation - Conclusion Selected domain Scheduling a conference Rooms are our resources We need to assign rooms to sessions The domain we chose for applying our approach is planning an academic conference – like a much more scaled up version of the example. The resources we are trying to assign are rooms and our users are sessions. We are trying to assign the rooms to the sessions and form the conference schedule. We will look at: Interaction overview - Where elicitation fits in the implemented system Representation - Domain independent with examples in domain Optimization Elicitation - Fuses domain independent and dependent methods 12/3/2018 Ulas Bardak - Thesis Defense

Collaborative scheduling
Outline – Introduction – Related Work – Domain – Optimization – Elicitation - Evaluation - Conclusion Collaborative scheduling Automatic operations invoke the auto scheduling return the control to the user Manual operations • Edit resources and constraints • Modify the schedule • Provide advice to the system So if we look at a picture of the overall interaction – we have the user who starts out the system to do automatic scheduling after which the system passes the control back to the user. The user is responsible for possibly editing resources and constraints. The user may want to manually modify the schedule and the user can provide advice to the system – this can be due to a question the system generates for example to resolve uncertainty. 12/3/2018 Ulas Bardak - Thesis Defense

Collaborative scheduling
Outline – Introduction – Related Work – Domain – Optimization – Elicitation - Evaluation - Conclusion Collaborative scheduling Automatic operations Automatic operations Process new data and advice Optimize schedule Generate and send questions to the user Manual operations • Edit resources and constraints • Modify the schedule • Provide advice to the system invoke the auto scheduling return the control to the user The automatic operations that take place inside the system are processing the new data coming from the user, updating the schedule accordingly and when there is uncertainty generating questions for the user to answer. 12/3/2018 Ulas Bardak - Thesis Defense

Architecture Administrator Top-level control and learning
Outline – Introduction – Related Work – Domain – Optimization – Elicitation - Evaluation - Conclusion Architecture Top-level control and learning Graphical user interface Administrator Info elicitor Representation Optimizer Process new info Optimize the schedule Choose questions If we look at in particular the interaction that takes place in the implemented version of our approach, we can see this interaction and the modules that are responsible for each piece. We have a parser which parses the input, the optimizer updates the schedule and the elicitor chooses the questions to generate. The user sits in front of a GUI and works with the system. 12/3/2018 Ulas Bardak - Thesis Defense

Outline – Introduction – Related Work – Domain – Optimization – Elicitation - Evaluation - Conclusion Rooms 2000 ft2 1 2 3 Rooms have a set of properties Size, seating capacity,... Microphones, projectors,... We also know distances between rooms Room 1 is 2000 square feet and has one projector. We represent each room as a set of properties like the size of a room, how many microphones there are in the room, etc. For example, in our simple problem, room 1 was 2000 square feet and it had one projector. We also represent distances between rooms. For example it may be the case that room 1 is 400 feet away from room 3. Room 1 is 400 feet away from Room 3. 12/3/2018 Ulas Bardak - Thesis Defense

Outline – Introduction – Related Work – Domain – Optimization – Elicitation - Evaluation - Conclusion Sessions Session description includes Importance Hard constraints, such as the minimal acceptable room size Soft preferences, such as the desired room size The system knows about the relative importances of the sessions and each session also has a set of preferences like how big a room it would like to have and a set of constraints like how small a room can be until it can no longer be used to host the session. An example of this would be saying the invited talk is more important than the poster session and it needs a room that is at least 500 square feet but if possible it would prefer a room double that size. The invited talk is more important than the poster session. The assigned room has to be at least 500 square feet, and preferably 1000 square feet. 12/3/2018 Ulas Bardak - Thesis Defense

Outline – Introduction – Related Work – Domain – Optimization – Elicitation - Evaluation - Conclusion Sessions We represent preferences by piecewise-linear functions. 1.0 Quality 0.5 Room size 250 500 750 1000 Unacceptable Each preference of each session is basically a piecewise-linear function that maps an attribute to schedule quality. This is the preference function for room size we talked about in the previous slide. The quality is unacceptable until we hit 500 square feet, then it gets better and better until we reach 1000 square feet. The invited talk is more important than the poster session. The assigned room has to be at least 500 square feet, and preferably 1000 square feet. 12/3/2018 Ulas Bardak - Thesis Defense

Outline – Introduction – Related Work – Domain – Optimization – Elicitation - Evaluation - Conclusion Uncertainty We usually have incomplete knowledge of room properties, session importances, and constraints and preferences. Up to now we talked about certain properties and certain preferences. As we have seen in the example however, we are very likely to get incomplete information about users preferences and we might even have uncertain information relative session importances and about room properties. For example, we may not know exactly how many microphones are in a room because no one went and counted them in the last year. 12/3/2018 Ulas Bardak - Thesis Defense

Outline – Introduction – Related Work – Domain – Optimization – Elicitation - Evaluation - Conclusion Uncertain properties We represent an uncertain value as either a completely unknown value, or a probability density function, approximated by a set of uniform distributions. Uncertain room properties can be represented either as complete unknown or as a set of uniform distributions. A concrete example would make this clearer. 12/3/2018 Ulas Bardak - Thesis Defense

Outline – Introduction – Related Work – Domain – Optimization – Elicitation - Evaluation - Conclusion Uncertain properties Example: An auditorium has about 600 seats. 0.2 chance: [ ] 0.6 chance: [ ] 0.2 chance: [ ] Probability 0.006 .6 0.004 Let’s say that we know an auditorium has about 600 seats. The way we represent this in the system is by a set of three uniform distributions. We say that there is a 20% chance the auditorium has between 450 and 549 seats , 60% chance it has between 550 and 650 seats and 20% chance between 651 and 750 seats. Uncertain importances would be represented very similarly to this but uncertain preferences are a little more complicated. 0.002 .2 .2 200 400 600 800 Capacity 12/3/2018 Ulas Bardak - Thesis Defense

Uncertain preferences
Outline – Introduction – Related Work – Domain – Optimization – Elicitation - Evaluation - Conclusion Uncertain preferences We represent an uncertain preference as completely unknown function, piecewise-linear function with uncertain y-coordinates of endpoints, or set of possible piecewise-linear functions with related probabilities. An uncertain preference, just like an uncertain property, can be completely unknown. It can also be a piecewise linear function where a given point may have uncertain utility or it can be a set of preference functions each with a probability. 12/3/2018 Ulas Bardak - Thesis Defense

Uncertain preferences
Outline – Introduction – Related Work – Domain – Optimization – Elicitation - Evaluation - Conclusion Uncertain preferences The description of a demo session does not include a room-size preference. 1.0 Quality .95 chance .05 chance 0.5 Room size 250 500 750 1000 Unacceptable This is an example uncertain preference function. Lets say that we don’t know the room size preference for the demo session but we know that a typical preference would be to have a minimum room size of at least 250 square feet and preferably 750 square feet. But in this case we also want to represent the fact that there is a very small chance a big sponsor may decide to show up in which case we would need a bigger room. Demo sessions usually require at least 250 square feet, and preferably 750 square feet; however, there is a 5% chance that a big sponsor shows up unexpectedly and asks for additional 250 square feet. 12/3/2018 Ulas Bardak - Thesis Defense

Outline – Introduction – Related Work – Domain – Optimization – Elicitation - Evaluation - Conclusion Optimization The optimizer assigns rooms to sessions. Input: Rooms and sessions Output: Room and time for each session Now that we know about representation, let’s take a quick look at the optimization process which is responsible for assigning rooms to sessions and producing a schedule. 12/3/2018 Ulas Bardak - Thesis Defense

Outline – Introduction – Related Work – Domain – Optimization – Elicitation - Evaluation - Conclusion Session quality Quality value of a session is based on how much each preference is satisfied Uncertainty is taken into account when calculating quality Optimizer calculates a quality for each session when it considers placing that session in a given room. The quality is the sum of how much each of the sessions’ preferences is satisfied and it takes uncertainty into account when calculating quality. 12/3/2018 Ulas Bardak - Thesis Defense

Outline – Introduction – Related Work – Domain – Optimization – Elicitation - Evaluation - Conclusion Schedule quality Overall schedule quality value is a weighted sum of session quality values If any session violates hard constraints, the whole schedule is unacceptable What the optimizer tries to maximize is the overall schedule quality which it calculates as a weighted sum of individual session qualities. If a session has a violated constraint in the schedule, the whole schedule is unacceptable. 12/3/2018 Ulas Bardak - Thesis Defense

Outline – Introduction – Related Work – Domain – Optimization – Elicitation - Evaluation - Conclusion Optimizer Simple version is based on hill-climbing Advanced version uses randomized hill-climbing, similar to simulated annealing The optimization algorithm we use is randomized hill climbing. The optimizer also lets us limit search time which lets us use the optimizer not just for full optimization but quick bursts of improvements to a given schedule if we need to. 12/3/2018 Ulas Bardak - Thesis Defense

Outline – Introduction – Related Work – Domain – Optimization – Elicitation - Evaluation - Conclusion Elicitation We use elicitation to reduce uncertainty User can selectively answer any questions Now that I talked about representation and optimization we can talk about the elicitation. Elicitation is what we use to generate questions and resolve uncertainty. When we generate a list of questions for the user to answer, he or she can choose to answer a subset and does not have to answer everything. 12/3/2018 Ulas Bardak - Thesis Defense

Outline – Introduction – Related Work – Domain – Optimization – Elicitation - Evaluation - Conclusion Elicitation Synergetic Elicitor Heuristic Elicitor Rule-based Elicitor Search Elicitor Heuristic Elicitor All Potential Questions There are three individual elicitors that make up the unified elicitor. The heuristic elicitor generates an initial list of questions which are then passed to the auxiliary elicitor which may tag on more questions and the search elicitor takes the final list and can modify the ordering of questions. We will talk about each of these elicitors in more detail. Merged List Re-ranked List Rule-based Elicitor 12/3/2018 Ulas Bardak - Thesis Defense

Outline – Introduction – Related Work – Domain – Optimization – Elicitation - Evaluation - Conclusion Heuristic elicitor Synergetic Elicitor Heuristic Elicitor Rule-based Elicitor Search Elicitor Selection of questions based on the standard deviation of schedule quality Fast calculation, once per variable Domain-independent 12/3/2018 Ulas Bardak - Thesis Defense

Heuristic elicitor Get list of questions
Outline – Introduction – Related Work – Domain – Optimization – Elicitation - Evaluation - Conclusion Heuristic elicitor Get list of questions Each uncertain variable is a potential question For each question, determine impact on schedule quality of possible answers Plug in possible answers to the quality function to get change in schedule quality For each question, calc. question score Let’s start with the Heuristic elicitor. Heuristic elicitor has four main steps. First, we get the list of possible questions. Each uncertain variable is a potential question. Then for each question, we look at what utility values we get for different possible answers. Using these different utility values we can get the standard deviation in utility due to each question. Score of a question is then the difference between this standard deviation and the cost of asking that question. We add the question to the list of questions if the score is above a certain threshold. Finally, we sort the list of questions and return questions with a score higher than a treshold. Return top questions 12/3/2018 Ulas Bardak - Thesis Defense

Outline – Introduction – Related Work – Domain – Optimization – Elicitation - Evaluation - Conclusion Rule-based elicitor Synergetic Elicitor Heuristic Elicitor Rule-based Elicitor Search Elicitor Selection of additional questions, based on domain-specific heuristics, such as “Room capacity is more important than ceiling height.” The auxiliary elicitor is the simplest of the three elicitors. It relies on simple heuristics to rank all the possible questions. One such heuristic is the fact that a question about room capacity would be ranked higher than a question about room heights. As the heuristic elicitor is likely to generate questions on only a subset of all the possible uncertain variables, auxiliary elicitor is very useful for augmenting that list. 12/3/2018 Ulas Bardak - Thesis Defense

Outline – Introduction – Related Work – Domain – Optimization – Elicitation - Evaluation - Conclusion Search elicitor Synergetic Elicitor Heuristic Elicitor Rule-based Elicitor Search Elicitor Ranks selected questions using B* search Relies on the optimizer for evaluating nodes in the search space Domain-independent and optimizer-independent B* by Berliner, here at CMU in This is not related to the B* trees and we use a variety of it for non-adversarial search. 12/3/2018 Ulas Bardak - Thesis Defense

Outline – Introduction – Related Work – Domain – Optimization – Elicitation - Evaluation - Conclusion Example Uncertain room size versus uncertain projector number 0.5 0.1 0.2 0.3 0.4 :40% :60% Min qual.:0.1 Max qual.:0.5 0-1:50% 2-3:50% Min qual.:0 Max qual.:0.4 :50% :50% Min qual.:0.15 Max qual.:0.35 0-1:50% 2-3:50% Min qual.:0.1 Max qual.:0.25 :25% :25% Min qual.:0.28 Max qual.:0.33 Search elicitor improves question weights by using the optimizer. What happens is basically a best first search using the optimizer as a heuristic function. To get the weight for a variable we start with the complete range of values for that variable and see how much of a better job the optimizer can do at the extreme values. If this value is smaller than a certain threshold, we split the range into two equally likely regions and repeat the process. Once the optimizer can not improve the schedule or can improve beyond the threshold we stop. The minimal possible utility of asking about the room size is greater than the maximal possible utility of asking about the number of projectors. 12/3/2018 Ulas Bardak - Thesis Defense

Outline – Introduction – Related Work – Domain – Optimization – Elicitation - Evaluation - Conclusion Evaluation The synergetic elicitor is far more effective than each of its individual components, simple heuristics, and random selection of questions. Before we move on to the actual evaluation details I want to take a second to go over the initial research hypothesis which is also what we found out after evaluation. Our approach produces a ranking of questions that is significantly better at improving the quality of optimized schedules than the ranking produced by individual components, simple heuristics or random picking of questions. 12/3/2018 Ulas Bardak - Thesis Defense

Outline – Introduction – Related Work – Domain – Optimization – Elicitation - Evaluation - Conclusion Evaluation Four scenarios with 88 sessions 10 rooms, 100 uncertain values 20 rooms, 500 uncertain values 50 rooms, 1000 uncertain values 84 rooms, 3300 uncertain values More complex Less We do evaluation using 4 different scenarios of different sizes going from 3400 potential questions to only 100 potential questions. 12/3/2018 Ulas Bardak - Thesis Defense

Outline – Introduction – Related Work – Domain – Optimization – Elicitation - Evaluation - Conclusion Evaluation For each setting, we use five different elicitation systems Synergetic Elicitor Heuristic & rule-based Search & rule-based Rule-based Random Synergetic Elicitor Heuristic Elicitor Rule-based Elicitor Search Elicitor We use five different elicitation system on each problem. The heuristic elicitor with the auxiliary elicitor augmenting the list, the search elicitor with the auxiliary elicitor providing a starting list of questions, the complete system, auxiliary elicitor by itself and ordering questions randomly. 12/3/2018 Ulas Bardak - Thesis Defense

Outline – Introduction – Related Work – Domain – Optimization – Elicitation - Evaluation - Conclusion Evaluation We plot: Change in the schedule quality Change in the quality loss due to uncertainty (100%  0%) For each problem we answer questions in order and as we answer we plot the schedule quality and the percentage remaining loss in quality. The remaining loss in quality is a percentage that shows how far we are from the quality of the schedule we produce when all the possible questions are answered. This number is assumed to be 100% at the starting point , and 0% after we answer all the questions. 12/3/2018 Ulas Bardak - Thesis Defense

Evaluation 100 variables % of questions needed for 85% of full quality
Outline – Introduction – Related Work – Domain – Optimization – Elicitation - Evaluation - Conclusion Evaluation 80% 12.5% 15% 70% 33% % of questions needed for 85% of full quality 100 variables y In the smallest problem, with only 100 questions, the auxiliary and random picking perform about the same. The other three systems however, have very similar behavior. We believe that this happens because the total change in the schedule quality between the starting point with 100 unknowns and the ending point where everything is certain is very small. (2.87 x 4.01) 12/3/2018 Ulas Bardak - Thesis Defense

Outline – Introduction – Related Work – Domain – Optimization – Elicitation - Evaluation - Conclusion Evaluation 45% 17.5% 33% 44% 26% % of questions needed for 85% of full quality 3300 variables y We start out with the random picking of questions. We can see that there is a very gradual but slow increase in the schedule quality as more questions are answered. The auxiliary elicitor works about the same. Heuristic and search elicitors improve on that and we can see that the best case is with the full system. (2.87 x 4.01) 12/3/2018 Ulas Bardak - Thesis Defense

Evaluation Problem size % of questions needed for 95% of full quality
Outline – Introduction – Related Work – Domain – Optimization – Elicitation - Evaluation - Conclusion Evaluation 98% 18% 73% 42% % of questions needed for 95% of full quality 100 q. 500 q. 1000 q. Problem size 3400 q. One interesting observation we can make here is that as the problem size gets smaller and smaller it takes a higher percentage of the questions to get close to the quality level of a fully certain schedule. We believe that this is related to what we mentioned in the last slide, that is as the difference between fully certain and the starting point of a problem gets smaller, the full system finds it harder to get to the full quality. 0% 25% 50% 75% 100% 12/3/2018 Ulas Bardak - Thesis Defense

Outline – Introduction – Related Work – Domain – Optimization – Elicitation - Evaluation - Conclusion Summary We have applied the elicitor to conference scheduling Synergetic elicitor outperforms its components and simple heuristics Improvement is more prominent for larger problems Synergetic elicitor does better than simple heuristics including random picking of questions. 12/3/2018 Ulas Bardak - Thesis Defense

Outline – Introduction – Related Work – Domain – Optimization – Elicitation - Evaluation - Conclusion Contributions We have investigated a novel approach to information elicitation, which has led to three main contributions. Fast heuristic computation of the expected utility of potential questions Use of B* search for determining more accurate question utilities Synergy of domain-independent and domain-specific elicitation techniques This work includes three main contributions: An elicitation method which uses standard deviation as expected impact has been implemented. We use non-adversarial B* search for refining question weights We unify different elicitation strategies under one elicitation system. We have investigated a novel approach to information elicitation, which has led to three main contributions. 12/3/2018 Ulas Bardak - Thesis Defense

Outline – Introduction – Related Work – Domain – Optimization – Elicitation - Evaluation - Conclusion Future work Learning question costs Learning elicitation strategies 12/3/2018 Ulas Bardak - Thesis Defense

12/3/2018 Ulas Bardak - Thesis Defense

Additional Slides 12/3/2018 Ulas Bardak - Thesis Defense

Vendor elicitation Domain Sessions can require services that external vendors provide e.g. mobile equipment, food deliveries Each item can satisfy multiple services e.g. Laptop  Computer, Portable computer Penalty for spending money A vendor optimizer finds a near optimal placement of vendor orders Uncertainty can exist in prices, availability of items 12/3/2018 Ulas Bardak - Thesis Defense

Vendor elicitation Elicitation Algorithm Enumerate all of the services Order based on affecting the overall cost penalty 12/3/2018 Ulas Bardak - Thesis Defense

Vendor elicitation Evaluation 12/3/2018 Ulas Bardak - Thesis Defense

Information Elicitation in Scheduling Problems

Similar presentations

Presentation on theme: "Information Elicitation in Scheduling Problems"— Presentation transcript:

Similar presentations

About project

Feedback

Log in

Auth with social network:

Information Elicitation in Scheduling Problems

Similar presentations

Presentation on theme: "Information Elicitation in Scheduling Problems"— Presentation transcript:

Similar presentations

About project

Feedback