2. Choose n out of these k objects For example:  Choose your three favorites out of these ten photographs  Of these fifty apps, which ten would you download to your phone?  Which two of these seven movies would you want to watch?

3. Could we prove that that there is dependence within each person’s choices? For example, do people have a certain “taste” in sushi rolls? Objectives:  We wanted to prove that each person does not choose randomly. Some items are chosen together more often than they would be otherwise.  In particular, we wanted to find which items are similar to one another. If a person chooses a given object, which other objects is he also more likely to choose?

4.  SUSHI Preference Data Set -- survey taken by 5000 people in which they were asked to rank ten different types of rolls from best to worst (http://www.kamishima.net/sushi/ )http://www.kamishima.net/sushi/  The ten rolls: shrimp (0), sea eel (1), tuna (2), squid (3), sea urchin (4), salmon (5), egg (6), fatty tuna (7), tuna roll (8), cucumber (9)  We just looked at each respondent’s first three choices and ignored the order in which they listed them. (This way, the data fit our “choose n out of k” format.)

5. 0123456789 0x 1394x 2451411x 3369223299x 4398519373193x 5395387412191806x 622723814411462140x 7774925141242112961148253x 81291564041228813474468x 9834342522147666135x The following is a matrix of how often each pair of sushis appeared together in someone’s top three: Most popular pairs Least popular pair

6. Doesn’t this answer our questions?  The most popular pairings were (2,7) and (4,7). So those who like roll #7 were more likely to choose roll #2 or #7.  The least popular pairing was (5,9) – only 21 respondents listed them as two of their top three! They must be very dissimilar.

7. That ignores the fact that some rolls were just more popular overall. It makes sense that (2,7) and (4,7) were chosen together so often since 2, 4, and 7 were popular overall. The reverse is true for 5 and 9. There’s no clear proof that these pairings tell us anything about people’s taste – they may just reflect each roll’s popularity.

8. We needed to generate a matrix of how often each pair of rolls would be expected to appear together. We could then compare the actual results to the expected results. To generate this matrix, we decided to run a simulation.

9.  Each respondent needs to randomly choose three rolls  The rolls must be chosen without replacement – each respondent needs to choose three different rolls  Each roll’s overall popularity must be held fixed

10.  Simply choose three rolls out of ten without replacement, using sample(0:9,3,replace=FALSE,prob=P 1,P 2,…) in R  Imagine that a number line between 0 and 3 is split up into 10 parts where the size of each part is proportional to the frequency of each subsequent roll.  A random number between 0 and 3 is then generated, corresponding to one of the rolls. For example, if 1.4 was generated, then roll #4 would be chosen.

11.  A new number line is then drawn, leaving out whichever roll was chosen the first time, while proportionally increasing the size of each remaining part. For example, this would be the new number line if #4 were chosen:  Once again, a number between 0 and 3 would be chosen, corresponding to the second roll chosen.  This same process would be repeated to choose the third roll.

12.  We have to redraw the number line after the first choice. As a result, the probabilities for the second and third choices are not the same as the overall probabilities.  The overall distribution of choices from the simulation is not equal to the overall distribution of choices from the actual survey: How can we fix this? We somehow need to keep the overall probabilities constant for each choice, while still not allowing for repeats. 0123456789 Actual Frequency0.1070.1100.1320.0660.1250.1220.0440.2250.0540.015 Simulated0.1110.1130.1320.0720.1260.1240.0490.1950.0600.017

13. Hartley and Rao (1962) describe an approach to solve this problem: 1. Randomize the order of the rolls. This was accomplished by calling sample(0:9) in R. 2. Split up the number line between 0 and 3 into 10 parts where the size of each part was still proportional to the frequency of each subsequent roll, but using the new order. For example, when the new order of the roll is [3,7,5,9,1,2,4,0,8,6] we use the following number line:

14. 3. A random number between 0 and 1, d, is chosen. 4. The three rolls selected are the ones corresponding to d, d+1, and d+2. In the following example d =.95, meaning that rolls 5, 2, and 6 – the rolls corresponding to.95, 1.95, and 2.95 – are chosen.

15. Our simulation shows that each roll is chosen with the same frequency using this technique as in the actual survey. 0123456789 Actual Frequency0.1070.1100.1320.0660.1250.1220.0440.2250.0540.015 Technique #20.1070.1100.1320.0660.1250.1220.0440.2250.0540.015 Technique #10.1110.1130.1320.0720.1260.1240.0490.1950.0600.017

16. Using this second method, we found our matrix of expected results. The fact that our expectations were so different from the actual data implies that people don’t make their choices independently. 0123456789 0x 1387.57x 2470.63481.2x 3221.9227.63262.73x 4441.5455.07552.37249.87x 5432.27441.6529.13245.13507.77x 6145.63144.43176.5381.5167.7163.43x 7911.9935.731196.3550.131128.41080.9367x 8177.03180.2214.43116.4208.6200.9748.367456.23x 949.150.23351.73323.76752.33351.218.667124.4720.3x

18. 0123456789 0x 10.327x 2-0.905-3.2x 39.875-0.3072.237x 4-2.072.997-7.632-3.598X 5-1.792-2.598-5.092-3.45813.235x 66.7427.786-2.4493.6-8.162-1.833x 7-4.567-0.3516.236-5.5064.9892.041-5.951x 8-3.61-1.80312.950.519-8.35-4.7243.6860.551x 94.838-1.021-1.3535.791-4.331-0.58710.96-5.6893.26x *Remember how 2 and 7 initially seemed to be the most similar pair? It still looks like they are similar, but there are many other pairings which are much more similar. For example, 6 and 9 were chosen together only 66 times yet has a larger residual!

20. 0 - shrimp 1 - sea eel 2 - tuna 3 - squid 4 - sea urchin 5 - salmon 6 - egg 7 - fatty tuna 8 - tuna roll 9 - cucumber

21. To further support these results, we re-ran the analysis by looking at each respondent’s top five choices. These were the results of the new multidimensional scaling: The fact that this plot is so similar to our prior one (see previous slide) proves that our results were not merely a result of the fact that we arbitrarily chose to look at the top three choices and that any value of k and n (where k<n) should work.

22. The groupings made by the MDS make sense when we look back at what each type of roll was.

23. Look at the clusters it formed:  6 and 9  Egg and Cucumber, the two non- fish choices  2, 7, and 8  All three are different types of tuna rolls Since those clusters make sense on their own, and were confirmed by our statistical analysis, we could also trust the other clusters we formed:  4 and 5  Sea Urchin and Salmon  0, 1, and 3  Shrimp, Sea Eel, Squid

24.  In our study, we looked at associations in choice data using simulations.  The simulation was done by sampling without replacement yet still proportional to size.  We showed that people did not make their choices randomly.  MDS and clustering based on the identified associations revealed the specifics of people’s taste.  This general approach can be readily applied to other choice data.