1. Notes on Research Proposals

2. Components of the Research Proposal
Problem Description/Statement
Research Objectives
Importance/Benefits of the Study
Literature Review
Research Design / Data Analysis
Deliverables
Schedule
[Facilities and Special Resources]
References
Budget (Appendix)

3. Problem Statement
Review the discussion from Week 2 on problem statements.

4. Purpose of the Problem Statement
Represents the reasons/motivation behind your proposal (based on the specific domain of study).
It specifies the conditions you want to change or the gaps in existing knowledge you intend to fill (this is the specification of the research problem).
Should be supported by evidence.
Specifies your hypothesis that suggests a solution to the problem.
Shows your familiarity with prior research on the topic and why it needs to be extended.
Even if the problem is obvious, your reviewers want to know how clearly you can state it.

5. Guidelines for writing a good abstract/problem statement
All should have the following elements in this order:
State the general case / problem
Describe what others have done
What’s missing / where is the gap in knowledge?
Describe the proposed solution or research objectives/questions
Specify one or more specific hypotheses
Should include specific metrics/measurements
Discuss how their validation addresses the research questions
Specific results (or research design, if it is a proposal)

6. Purpose of the Research Objectives Section
Specify the outcome of your project, the end product(s)
Keep you objectives
Specific: indicate precisely what you intend to change through your project
Measurable –what you accept as proof of project success
Logical – how each objective contributes to systematically to achieving your overall goal

7. Research Objectives Flows naturally from the problem statement
state your hypotheses clearly
give the reader a concrete, achievable goal
Verify the consistency of the proposal
check to see that each objective is discussed in the research design, data analysis and results sections

8. Literature Review Recent or historically significant research studies
Always refer to the original source
Discuss how the literature applies, show the weaknesses in the design, discuss how you would avoid similar problems
How is your idea different/better?

9. Importance/Benefits of the Study
Importance of the doing the study now
What are the potential impacts on
Research in the area
Applications of the research if successful
Broader impact (in other areas, on the society, in education, etc.)
If you find this difficult to write, then most likely you have not understood the problem

10. Research Design What you are going to do in technical terms.
May contain many subsections
Be specific about what research methodology you will use and why
Provide details of your proposed solutions to the problem and sub-problems
Provide information for tasks such as sample selection, data collection, instrumentation, validation, procedures, ethical requirements

11. Schedule & Deliverables
Include the major phases of the project
exploratory studies, data analysis, report generation
Critical Path Method (CPM) of scheduling may help
Deliverables:
Measurement instruments
Algorithms
Computer programs / prototypes
Comparative evaluation
Other technical reports

12. Budget and Resources Itemized Budget Provide a Budget Narrative
Access to special systems or computers
Infrastructure needs
Costs of surveys, user studies, etc.
Cost of travel if related to research design
Provide a Budget Narrative
This part is usually an appendix.

13. Proposal Characteristics
Straightforward document
No extraneous or irreverent material
Don’t tell us why you became interested in the topic
The first words you write are the most important ones
Not a literary production
Clear, sharp and precise
economy of words; no rambling sentences
Clearly organized
Outlined with proper use of headings and subheadings

14. Suggested Organization
Title, Abstract, Keywords (problem statement)
Introduction and Overview
Background information; problem description in context
Hypotheses and objectives
Assumptions and delimitations
Importance and benefits
Related Work/Literature Review
Research Design and Methodology
Plan of Work and Outcomes (deliverables, schedule)
Conclusions and Future Work
References
Budget (appendix)

15. Weaknesses in Research Proposals
Research Problem
unfocused
unimportant (done before!)
more complex
limited relevance

16. Weaknesses in Research Proposals
Research Design
so vague it prevents evaluation
inappropriate or impossible data
procedures inappropriate for problem
Threats to validity
Lack of reliable measures
lacking controls

17. A Sample Research Proposal
Read (and study) the sample proposal in Chapter 5 of in Practical Research
Fill in the critique in Chapter 12 for this proposal.
Since the critique is designed for a REPORT, simply change the tense for most questions.
Is the sample size adequate? -> Will the sample size be adequate

18. Guide to Writing the Research Proposal

19. 5 Key Questions to Answer in Your Problem Statement
Does your problem statement:
Demonstrate a precise understanding of the problem you are attempting to solve?
Clearly convey the focus of your project early in the narrative?
Indicate the relationship of your project to a larger set of problems and justify why your particular focus has been chosen?
Demonstrate that your problem is feasible to solve?
Make others what to read it further?

20. 5 Key Questions to Answer for Purpose and Objectives
Does this section
Clearly describe your project’s objective, hypotheses and/or research question?
Bury them in a morass of narrative?
Demonstrate that your objectives are important, significant and timely?
Include objectives that comprehensively describe the intended outcomes of the project?
State objectives, hypothesis or questions in a way they can be evaluated or tested later

21. Writing Tips for Objectives Section
Don’t confuse your objectives (ends) with you methods (means).
A good objective emphasizes what will be done, whereas a method will explain why or how it will be done.
Include goals (ultimate) and objectives (immediate)

22. Purpose of the Research Design
Describes your project activities in detail
Indicates how your objective will be accomplished
Description should include the sequence, flow, and interrelationship of activities
It should discuss the risks of your method, and indicate why your success is probable
Relate what is unique about your approach.

23. Data Analysis Data Analysis is essentially a four step process
Identify precisely what will be evaluated. If you wrote measurable objectives, you already know.
Determine the methods used to evaluate each objective. More precisely, you will need to describe the information you will need and how you propose to collect it.
3. Specify the analyses you plan to make and the data you need to collect. Your design may be simply to observe behavior of a particular population or something more complex like a rigorous experimental and multiple control group design.
4. Summarize the resulting data analyses and indicate its use. Consider mock data tables that show what your resulting data might look like.

24. Key Questions to Answer for Research Design/Data Analysis
Does the research design and data analysis section
Describe why analysis is needed in the project?
Clearly identify the purpose of your analysis?
Demonstrate that an appropriate analysis procedure is included for each project objective
Provide a general organizational plan or model?
Demonstrate what information will be needed to complete the analysis, the potential sources and the instruments that will be used to collect it.

25. Writing Tips for Research Design
Begin with your objectives
Describe the precise steps you will follow to carry out each objective, including what will be done, and who will do it.
Keep asking and answering the “What’s next?” question.
Once you have determined the sequence of events, cast the major milestones into a time-and-task chart

26. Scientific Writing Prosaic Clear, accurate, but not dull
Economy – every sentence necessary but not to the point of over condensing
Ego less – you are writing for the readers not yourself

27. Scientific Tone Objective and accurate To inform not entertain
Do not over qualify – modify every claim with caveats and cautions
Never use idioms like “crop up”, “loose track”, “it turned out that”, etc.
Use examples if they aid in clarification

28. Scientific Motivation
Brief summaries at the beginning and end of each section
The connection between one paragraph and the next should be obvious
Make sure your reader has sufficient knowledge to understand what follows

29. Other Writing Issues
The upper hand – inclusion of offhanded remarks like “ …this is a straightforward application …”
Obfuscation – aim is to give an impression of having done something without actually claiming to have done it
Analogies – only worthwhile if it significantly reduces the work of understanding, most of the time bad analogies lead the reader astray

30. Writing Issues
Straw men – indefensible hypothesis posed for the sole purpose of being demolished
“it can be argued that databases do not require indexes”
Also use to contrast a new idea with some impossibly bad alternative, to put the new idea in a favorable light

31. Unsubstantiated Claims
Example:
Most user prefer the graphical style of interface.
We believe that ….
Example
Another possibility would be a disk-based method, but this approach is unlikely to be successful.
Another …, but our experience suggests that …

32. References and Citation
Up-to-date
Relevant (no padding)
Original source
First order: books and journal articles
Second order: conference article
Third order: technical report
No private communications or forums ( material cannot be accessed or verified) if you must leave as a footnote not in the bibliography
Do not cite support for common knowledge

33. Reference and Citation
Carefully relate your new work to existing work, show how your work builds on previous knowledge, and how it differs from other relevant results.
References – demonstrate the claims of new, knowledge of the research area, pointers to background reading

34. Citation Style References should not be anonymous
Other work [6] -> Marsden [6] has …
In self-references, readers should know that you are using yourself to support your argument not independent authorities
Avoid unnecessary discussion of references, Several authors …., we cite …

35. Citation style
Ordinal-number style, name-and-date style, superscripted ordinal numbers, and strings.
Use anyone, but use one!
Entries ordered
By appearance of citation
alphabetically

36. Quotation Text from another source
If short – enclosed in double quotes
If long – set aside in an indented block
Long quotations, full material, algorithms, figures may require permission from the publisher and from the author of the original
Use of quotes for other reasons is not recommended

37. Acknowledgements Anyone who made a contribution
Advice, proofreading, technical support, funding resources
Don’t list your family, unless they really contributed to the scientific contents

38. Ethics Don’t Present opinions as fact Distort truths Plagiarize
Imply that previously published results are original
Papers available on the internet – authors put out an informal publication and becomes accepted as a formal. It is expected that the informal version will be removed

39. Notes on Research Design
You have decided
What the problem is
What the study goals are
Why it is important for you to do the study
Now you will construct the research design which describes what you are going to do in technical terms.

40. General Structure of Research Proposals

41. Research Design
Is a plan for selecting the sources and types of information used to answer the research question.
Is a framework for specifying the relationships among the study’s variables
Is a blueprint that outlines each procedure from the hypothesis to the analysis of data.

42. Research Design
The research design will provide information for tasks such as
Sample selection and size
Data collection method
Instrumentation
Procedures
Ethical requirements
Rejected alternative designs

43. Classification of Research Designs
Exploratory or formal
Observational or communication based
Experimental or ex post facto
Descriptive or causal
Cross-sectional or longitudinal
Case or statistical study
Field, laboratory or simulation

44. Exploratory or Formal
Exploratory studies tend toward loose structures with the objective of discovering future research tasks
Goal - to develop hypotheses or questions for further research
Formal study begins where the exploration leaves off and begins with the hypothesis or research question
Goal – test the hypothesis or answer the research question posed

45. Observational or Communication Based
Observational studies – the researcher inspects the activities of a subject or the nature of some material without attempting elicit responses from anyone.
Communicational – the researcher questions the subjects and collects response by personal or impersonal means.

46. Experimental or Ex Post Facto
In an experiment the researcher attempts to control and/or manipulate the variables in the study. Experimentation provides the most powerful support possible for a hypothesis of causation
With an ex post facto design, investigators have no control over the variables in the sense of being able to manipulate them. Report only what has happened or what is happening. Important that researches do not influence variables

47. Descriptive or Causal
If the research is concerned with finding out who, what, where, when or how much then the study is descriptive.
If is concerned with finding out why then it is causal. How one variable produces changes in another.

48. Cross-sectional or Longitudinal
Cross-sectional are carried out once and represent a snapshot of one point in time.
Longitudinal are repeated over an extended period

49. Case or Statistical Study
Statistical studies are designed for breath rather than depth. They attempt to capture a population’s characteristics by making inference from a sample’s characteristics.
Case studies – full contextual analysis of fewer events or conditions and their interrelations. (Remember that a universal can be falsified by a single counter-instance)

50. Field, Laboratory or Simulation
Designs differ in the actual environmental conditions

51. Quantitative v. Qualitative Approaches
Categorize research studies into two broad categories
Quantitative – relationships among measured variable for the purpose of explaining, predicting and controlling phenomena
Qualitative – answer question about the complex nature of phenomena with the purpose of describing and understanding from the participant’s point of view

52. The Validity of Your Method
Accuracy, meaningfulness, an credibility
Most important questions:
Does the study have sufficient controls to ensure that the conclusions we draw are truly warranted by the data? (internal validity)
Can we use what we have observed in the research situation to make generalizations about the world beyond that specific situation? (external validity)

53. Strategies to reduce internal validity problems
Controlled laboratory study
A double-blind experiment
Unobtrusive measures ( to see where people use the library look at worn flooring)
Triangulation – multiple sources

54. Strategies to enhance external validity
A real-life setting – artificial settings may be quite dissimilar from real-life circumstances
Representative sample
Replication in a different context

55. Formal Notion of Validity
“The best available approximation to the truth of a given proposition, inference, or conclusion”
Source: Research Methods Knowledgebase

56. Types of Validity Conclusion Validity: Construct Validity:
Is there a relationship between the two variables?
Internal Validity:
Assuming that there is a relationship, is it a causal one?
Construct Validity:
Assuming that there is a causal relationship, can we claim that the program reflected our construct of the program and that our measure reflected well our idea of the construct of the measure?
External Validity:
Can we generalize the (causal) effect to other settings, domains, persons, places or times?

57. Types of Validity
Source: Research Methods Knowledgebase

58. Hypothesis &
Experiments

59. Setting of the Problem Statement of the Problem Goal Establishes
Additional information to comprehend fully the meaning of the problem
hypothesis
scope
definitions
assumptions

60. Hypotheses Tentative proposition formulated for empirical testing
Means for guiding and directing
kinds of data to be collected
analysis and interpretation
have nothing to do with proof
acceptance or rejection is dependant on “data”

61. Examples of Hypotheses
Error-based pruning reduces the size of decision trees (as measured in the number of nodes) without decreasing accuracy (as measured by error rate)
The use of relevance feedback in an information retrieval system, results in more effective information discovery by users (as measured in terms of time to task completion)
The proposed approach for generating item recommendations based on association rule discovery on purchase histories results in more accurate predictions of future purchases when compared to the baseline approach.
[From a recent Google experiment] Longer documents tend to be ranked more accurately than shorter documents because their topics can be estimated with lower variance.

62. Rejecting the Hypothesis
Often researchers set out to disprove an opposite/competing hypothesis
Example: We believe that test strategy A uncovers more faults than test strategy B. So our hypothesis will be that
Programmers using test strategy A will uncover more faults than programmers using test strategy B for the same program.

63. Rejecting the Hypothesis
However, we cannot actually prove this hypothesis, we instead will try to disprove an opposite hypothesis
There will be no difference in the fault detection rate of programmers using test strategy A and those using test strategy B for the same program.

64. Types of Hypotheses Existential Compositional
An entity or phenomenon exists (perhaps with a specified frequency)
“Atoms contain uncharged subatomic particles (neutrons)”
Compositional
An entity or phenomenon consists of a number of related parts or components (perhaps with a specified frequency)
“Atoms consist of proton, electrons, and neutrons.”
“All decision tree algorithms can be divided into a growing phase and a pruning phase.”

65. Types of Hypotheses Correlational Casual
Two measurable quantities have a specified association
“An element’s atomic weight and its properties are correlated.”
“The size of a decision tree constructed using error-based pruning grows linearly with the size of training set.”
Casual
A given behavior has a specified causal mechanism
“The low reactivity of noble gases is caused by their full outer shell of valence electrons.”
“The use of relevance feedback results in more effective information discovery by users”

66. Rejecting the Hypothesis
If there is a significant difference in the fault detection rate we can reject the “no difference” and by default, support our research hypothesis
the “no difference” = null hypothesis

67. Recall: Falsifiability
Falsifiability is the logical possibility that an assertion can be shown to be false by evidence
Does not mean “false.” Instead, if a falsifiable proposition is false, its falsehood can be shown by experimentation, proof, or simulation.
There are different degrees of falsifiability
What make a hypothesis unfalsifiable?
Vagueness – theory does not predict any particular experimental outcome
Complexity/Generality – theory “explains” any experimental result
Special pleading – traditional experimental methods are claimed not to apply

68. Delimiting the Research
This is what the researcher does not want to do in the project
Should be stated clearly and explicitly.
What will be done is part of the problem statement.

69. Experiments
Studies involving the intervention by the researcher beyond that required for measurement
usually, manipulate some variable in a setting and observe how it affects the subject (cause and effect)
there is at least one independent variable and one dependent variable

70. Independent Variable Variable the researcher manipulates
For our hypothesis concerning test strategies, we may take a sample of software engineers and randomly assign each to one of two groups: one using test strategy A and the other test strategy B. Later we compare the fault detection rate in the two groups.
We are manipulating the test strategy, thus it is the independent variable

71. Dependent Variable
Variable that is potentially influenced by the independent variable
in our last example, the dependent variable is fault detection rate
Presumably the fault detection rate is influenced by test strategy applied
there can be more than one dependent variable

72. Conducting an Experiment
Seven activities
select relevant variables
specify the level(s) of treatment
control the experimental environment
choose the experimental design
select and assign the subjects
pilot-test, revise, and test
analyze the data

73. Select the Relevant Variables
Translate our problem into the hypothesis that best states the objectives of the research
how concepts are transformed into variables to make them measurable and subject to testing
research question:
Does a product presentation that describes product benefits in the introduction lead to improved retention of the product knowledge?

74. The Speculation
Product presentations in which the benefits module is placed in the introduction of a 12 minute message produce better retention of product knowledge that those where the benefits module is placed in the conclusion.

75. Researcher’s Challenge
Select variables that are the best operational representations of the original concepts.
Sales presentation, product benefits retention, product knowledge
Determine how many variables to test
constrained by budget, the time allocated, the availability of appropriate controls, and the number of subjects

76. Researcher’s Challenge
Select or design appropriate measures/metrics for them
thorough review of the available literature and instruments.
Adapted to unique needs of the research situation

77. Choosing an Experimental Design
Experimental designs are unique to the experimental method
statistical plans to designate relationships between experimental treatments and the experimenter’s observations
improve the probability that the observed change in the dependent variable was caused by the manipulation of the independent variable

78. Validity in Measurements
Validity: the extend to which instrument measures what is supposed to be measured
E.g., thermometer  temperature
E.g., IQ Test  Intelligence?
E.g., CPU time  algorithm complexity or efficiency

79. Reliability of Measurement
Reliability: accuracy and consistency by which the instrument can perform measurement
Accuracy exists only if there is consistency (not necessarily the other way around)
Need to measure more than once
Reliability is a necessary but not sufficient condition for validity