Synopsis Six: operationalizing, reliability, validity.

We've noted that operationalizing a definition helps set up a way to measure something that you otherwise can't measure directly. We used the example of self-esteem: you can't measure that directly, but you can measure traits of it, such as confident walk, tone of voice, etc.

Taking this a step further, to re-state your research question or hypothesis in measurable terms, you need to operationalize variables. For instance, a question using "intercultural communication competence" as a variable might involve operationalizing by restating: "Scores on a set of scales used by observers to rate behaviors of workshop participants as they describe their reaction to potential situations involving people of different cultures." Herbert Feigl (1945) set up several criteria used by researchers to evaluate operationalized definitions, including:

1. Is it empirically (observedly) based and definite?

2. Is it logically consistent with conceptual definition?

3. Is it intersubjective--that is, do other researchers agree with the definition?

4. Is it technologically possible to implement?

5. Is it repeatable by others?

6. Is it suggestive of constructs--that is, does it help advance theories?

This brings us to the idea of measurement, actually examining a set of variables. Different levels of measurement are available to researchers, including nominal (categorizing) and ordinal (ranking). You can rank ordinally on an interval level scale (intervals equal in size) or add the idea of zero (no detectable level of the variable) using a ratio level scale. Most qualitative research falls into the category of nominal measurement; quantitative researchers can use ordinal measurement. A common measurement communication researchers use is named after its inventor, the Likert scale. (five equivalent categories).

Reliability is simply defined as "the internal consistency of a measure." That is, if someone else tried to duplicate your research, would results be consistent? Researchers have to deal with a number of threats to reliability, including intercoder problems, test fatigue, vague measure items, etc. Reliability can be determined using a formula which results in a decimal number between 0 and 1, 1 being perfectly reliable. "Good" reliability is about .80 or above. This is called the reliability coefficient.

Validity tells us if we can believe the results of the study; that is, did we measure what we set out to measure? Threats to validity include confounding variables and the "Hawthorne effect," meaning changes in the results due to the effect of the measurement. We can try to establish validity in a study by relying on common sense arguments ("face validity"), a panel of experts, or building on other researchers' methods known to be valid.