1 Some Guidelines for Good Research Dr Leow Wee Kheng Dept. of Computer Science

2 Research is...  extending human understanding?  publishing papers?  solving a challenging puzzle?  making things work?  having fun?

3 Research involves...  selecting a topic  studying existing work  solving problems  publishing results

4 Problem Solving  define problem  specify requirements/quality of solution  formulate solution method  evaluate performance of method

5 Problem Definition  Purpose  clarify problem  focus attention  Define problem precisely  It's usually difficult to define problem. So, just skip it?  The more difficult it is, the stronger is the need, the harder one should try.

6 Problem Definition Example: image retrieval 1st trial: Retrieve images that a user wants. Problem: too vague  What does user want?

7 Problem Definition 2nd trial: Retrieve images that are similar to a specific example called the query. Better, but still vague:  What does it mean by “similar”?

8 Problem Definition 3nd trial: Given a query Q, retrieve images I i, i = 1,...,n, such that a similarity measure s(Q, I i ) is large. Better, but  What is this s(Q, I i )? Notice  There’s an input: Q  There’re expected outputs: I i  There’s requirement specification: large s(Q, I i )

9 Problem Definition 4th trial: Given a query Q, retrieve images I i, i = 1,...,n, such that a similarity measure s(Q, I i ) is large, and s(Q, I i ) is consistent with human’s perception. Good try  but difficult to measure human’s perception; still a difficult research topic. Notice  We are talking more about quality of solution.

10 Problem Definition 5th trial: Given a query Q, retrieve images I i, i = 1,...,n, such that precision p and recall rate r are maximized.  each Q and I i contains one or more regions  p and r are performance indices  p and r are defined in terms of similarity s(Q, I i ) Now, the problem definition is more specific:  requirement of solution also given  still haven’t said what are s(Q, I i ), p, r

11 Problem Definition Good problem definition should include  inputs  expected outputs  relationships between inputs and outputs  requirements about outputs  performance measures

12 Formulate Solution Method What it is not...  not writing a program  not thinking of an algorithm  not thinking of what tools to use So what is it?  give more details to problem definition  use mathematics  divide the problem into sub-problems  map sub-problem into known problem with known solution methods

13 Formulate Solution Method Example: image retrieval First sub-problem: define similarity s(Q, I i )  Q has regions R k  I i has regions R ij  some R k are identical/similar to some R ij  some R k are not identical/similar to some R ij  need to find best matching pairs How to say all these in math?

14 Formulate Solution Method  One possibility: use a mapping function:  f: Q  I i  R ij = f(R k ) means R k corresponds to R ij  define region similarity s(R k, R ij )  s(R k, R ij ) is large if R k is similar to R ij  this is another sub-problem  s(R k, f(R k )) = match between R k  Q and corresponding f(R k )  I i

15 Formulate Solution Method First, define s f (Q, I i ): Given a query Q, an image I i, and a mapping function f, define s f (Q, I i ) as Then, define s(Q, I i ): Given a query Q and an image I i,

16 Formulate Solution Method Now, perform a “magic” which I call problem transformation.  Given a query Q and an image I i, find the mapping f such that s(Q, I i ) is maximized.  this is an optimization problem  Given a query Q and a set of images I i,  compute s(Q, I i ) by solving optimization problem  return images I i with large s(Q, I i ). Now, we have an algorithm!

17 Formulate Solution Method Sounds complicated. Do we really do all these?  Yes. The more complicated, the stronger the need. Further reading: my IJCAI 2001 paper on conceptual graph  map query and images to conceptual graphs; define problem as subgraph matching problem (problem transformation)  map subgraph matching to search problem (transform once more)  implement search algorithm

18 Formulate Solution Method Some challenges:  Define the problem that you’re working on using math.  Try to find an interesting CS research problem that cannot be defined using math.

19 Performance Evaluation Purpose  evaluate solution method  understand its strengths and weaknesses (with the aim of improving it)  determine values of system parameters

20 Performance Evaluation  What’s your impression if you see this: The accuracy of the method is 90%.  The statement is not very meaningful.  Is the method really good?  may be the problem is simple  may be the data is not representative  easy to differentiate elephant from apple; difficult to differentiate between African and Asian elephant

21 Performance Evaluation So what makes a good evaluation?  good test data  test various aspects of method  good test cases  expose strengths and weaknesses of method  good comparison  set baselines for assessing performance  compare with state-of-the-art

22 Performance Evaluation Example 1: Color histograms: finding good operating condition (i.e., parameter values)  100 colorful images  2 parameters:  radius R  separation ratio   2 performance measures:  mean number of color bins  mean error of color

23 Performance Evaluation

24 Performance Evaluation

25 Performance Evaluation good operating condition

26 Performance Evaluation Example 2: Estimation of surface normals of 3D points.  3 sets of data points  2 methods of getting good neighboring points: point-based (P), mesh-based (M)  2 methods of computing surface normals: PCA, linear extrapolation (LE)  baseline: PCA on raw 3D points

27 Performance Evaluation What can you conclude from the result?  M is better than P  LE is better than PCA  LE/M is the best

28 Performance Evaluation Example 3: Classification of images by color distributions.  100 classes, 20 samples each  2 methods of computing color histograms: clustered (c), adaptive (a)  4 dissimilarity measures:  L2, JD: only for c  EMD: too slow for c, so only for a  WC: ok for both c and a  baseline: c + L2

29 Performance Evaluation What can you conclude from the result?

30 final points...  Always relate to the big picture.  good research is never performed in isolation  Know the strengths & weaknesses of your tools.  nothing is perfect for everything  Understand the problem first.  choose the solution method last  Learn various methods and tools.  you never know when you need one

31 good readings...  H. S. Fogler & S. E. LeBlanc, Strategies for Creative Problem Solving, Prentice-Hall,  general ideas, no math  D. Huff, How To Lie With Statistics, W. W. Norton,  know how not to get cheated