ATPS Tribal Lecture Notes: Weekend II
Friday
1:00pm
• Talk through assignments #3 & #4
There are two umbrella terms for our roles as researchers: participant observer and non-participant observer. A non-participant observer is an expert on the outskirts, one degree removed from the situation and people. A participant observer seeks community buy-in and takes part in research activities with community members.
pros and cons of each=
• involvement of the research has an impact on the study's participants, people react to seeing you, hearing you, and your actions. You may “mess with the field” and not even realize it. By changing the field of study you may change the results you observe. Participant observation requires that the researcher be physically present. This can lead the respondents to alter their behavior. An interview is an interruption in the natural stream of behavior. Respondents can get tired of responding to questions or resentful of the questions asked.
• The act of research may influence the subject of study itself. By asking people about their voting intentions, we probably influence those intentions to some degree: maybe respondents become more definite in their choices by committing to them out loud.
• By being a non-participant researcher you lack the ability to ask for clarification if the subject does or says something you don't understand.
• insider vs. outsider, access, authentic voice/authority to speak for the group of people being researched, concept that you just might not have a right to know something--- if you are not invited to be a participant there is probably a reason for that---
• “going native” meaning of this deragatory term and why this used to be thought of as a negative and now it seems to be revered (ex. “Living with the Mek”). Affect on the observer . You may become very empathetic with who you are observing and want to change what you are “testing.”
• At a point, the researcher may become the one being observed.
• Participant observation can be very unsafe for the researcher--- prison research---- it may also be unethical if the researcher does not reveal why he or she is there----
Within these two umbrella terms of participant observer and non-participant observer for our “roles” as researchers, there are another two big ways to classify how we actually “do” research: obtrusive and unobtrusive.
Classifications of doing research: Obtrusive research and unobtrusive research.
Unobtrusive Measures :
Observation without participant knowledge (trash in trashcans outside a business or agency, social movement--WTO witnesses, erosion of wear and tear in stairwell, accretion of flyers in stairwell) Ex. Historical/document/archival data- content analysis
Data already collected for another purpose: secondary sources, secondary data analysis
Unobtrusive measures are measures that don't require the researcher to intrude in the research context. Unobtrusive measurement presumably reduces the biases that result from the intrusion of the researcher or measurement instrument .
However, unobtrusive measures reduce the degree the researcher has control over the type of data collected.
* A research technique is considered unobtrusive when they have no impact on what/who is being studied. An indirect measure is an unobtrusive measure that occurs naturally in a research context.
Examples: The types of indirect measures that may be available are limited only by the researcher's imagination and inventiveness. For instance, let's say you would like to measure the popularity of various exhibits in a museum .
• It may be possible to set up some type of mechanical measurement system that is invisible to the museum patrons. We could, for instance, construct an electrical device that senses movement in front of an exhibit. Or we could place hidden cameras and code patron interest based on videotaped evidence.
• In one study, the system was simple. The museum installed new floor tiles in front of each exhibit they wanted a measurement on and, after a period of time, measured the wear-and-tear of the tiles as an indirect measure of patron traffic and interest.
• In a similar manner, if you want to know magazine preferences at TESC, you might rummage through the trash on campus.
These examples illustrate one of the most important points about indirect measures -- you have to be very careful about the ethics of this type of measurement . In an indirect measure you are, by definition, collecting information without the respondent's knowledge . In doing so, you may be violating their right to privacy and you are certainly not using informed consent. Of course, some types of information may be public and therefore not involve an invasion of privacy.
Obtrusive Measures :
Observations of behavior with participant knowledge
Perceptions, opinions and attitudes gathered through interviews, surveys, focus groups For some issues there may simply not be any available unobtrusive measures.
2:00pm
Craig Bill guest speaker (Governor's Office of Indian Affairs)
3:00pm-5:00pm
Don't be unreasonable in the questions you ask of your data- don't try to coax information out of it that is not here. Rabbit hole of interpretation- sometimes we can will ourselves into transforming data into information that doesn't exist.
Sampling--- whole point of statistics is to reduce the possibility that relationships between variables occurred by chance alone.
Probability (random) and Non-probability (non-random)--- explain that they will probably use non-random, targeted sampling. This means that they cannot generalize their results beyond the people who responded. So! In the spring, their final report will read “Based upon the responses…..” or “According to the people who participated……” You cannot say “All Tribes believe….” or “All tribal programs are…..”
Usually, sampling is the process by which we check out a small portion of a thing and then generalize to the entire thing. However, we may also use sampling to get at really specific information from a small group--- we may not have any desire to generalize. We all use sampling in everyday life. When you had that first sip of coffee this morning, was it representative of every sip after that?
The preference for studying samples instead of entire populations mostly comes from practical concerns. Usually, lack of money and time dictate why we use samples of populations instead of entire populations.
Why do we sample? - could be to get estimates and predictability and generalization, or it could be to get very specific information about a particular group.
How pick Sample approach? Depends on what you want to get out of the sample (measurably) Methods of measuring samples can be very simple or very complex.
Three steps in sampling : identify the target population, put together a population list, select the sample (using random or non-random, cluster, convenience, quota, etc.).
First, define some terms:
Population: the entire set of people, things, events or groups that is of interest to the researcher. Residents on the Nisqually reservation.
Sampling frame : the list of all elements in the population. A list of names of those residents.
Sample : a subset of the population.
Parameter : a number that describes some attribute of the population. Average income. Usually, this is the number we do not know and it is why we conduct a sample in order to estimate that value.
Statistic : a number that describes some attribute of the sample. The average income of the residents.
What are the two basic types of sampling?
Probability sampling : researcher decides which segment of the population will be used in order to accurately portray the larger population. Every subject in the sample has the same chance of getting selected. Therefore, the sample group possesses the same characteristics of the larger population. The assumption is that this random selection will result in a normal distribution and we can then generalize to the larger population. From a quantitative perspective: Social scientists like to measure things. By measuring things, we are hoping to describe, explain and/or predict our social world. Probability sampling allows us to estimate how closely the sample statistics are clustered around the true value. It enables us to estimate the sampling error (the degree of error expected for a given sample design) because of the square root, the standard error is reduced by half if the sample size is quadrupled. Why? Do we use a true experimental design in social science? No. we typically use a post-test only design, we react to events/stimulus which makes pre-testing difficult. examples …cluster (naturally occurring elements of population: city blocks and then subgroups are sampled within) Simple random sample : procedure that generates numbers or cases strictly on the basis of chance. ex. random digit dialing. Stratified random sampling : uses information already known about the population before sampling. First breaks the population down into strata (male students, female students) and then randomly selects sample from stratum. Allows for oversampling and weighting: proportionate stratified sample - (equal unit) each stratum is represented exactly in proportion to the population (50 men, 50 women), disproportionate stratified sample - (unequal unit) varies intentionally in proportion to the population (20 men, 80 women).
Nonprobability sampling : used when probability sampling is too expensive or when exact representation of population is not important to study or when population cannot be defined. What purpose does nonprobability sampling serve if we cannot generalize? We cannot measure the sampling error with nonprobability samples because there is not a normal distribution. examples … quota (weighted to end up with same percent as larger population), judgment/purposive (group of women w/ HIV- they can tell their story best), convenience (results from hanging out, use whoever is around)
Nonprobability sampling=
• Snowball sample (telephone game): one member of the sample is identified and then they identify another person who could take part in the study and so on. (gang members, wine drinkers)
• Systematic sampling with a random start: your sample is a city's phone book, you randomly select what letter to start with, you then pick every tenth name on a list
How pick Sample size? Depends on what you are doing. If you want to interview female nuclear physicists who grew up in San Diego , your sample is likely going to be very small. Further, it depends on how complex your ultimate measurement is and if you want to generalize to the larger pop. According to the laws of numbers, the more we sample, the more accurate we can be. The idea being that the larger the sample the more we can capture the diversity of the population. (homogenous or heterogenous) The rule most researchers follow is the more heterogenous the population, the bigger the sample should be. By increasing the sample size, we increase the range of estimates available. This assumes a normal distribution. * common problem with a lot of newspaper reports.
Is this true? What are some of the problems we have had with sample size and Native Americans (census)? I have been taught that size is not as important as randomization when you want to decrease your sampling error. Why? Because randomization results in a natural distribution of variance. It tells me that things did not just happen by chance. I may keep increasing my numbers, but be less and less accurate dependent upon the split in the sample….I may not have a normal distribution.
This is a major debate between social scientists, size vs. substance.
Saturday
9:00am
Surveys -----lecture/workshop
Show sample surveys of how not to design surveys. Discuss: cover letter/informed consent, incentive to participate?, show survey monkey, show the output, show how they will have to code every response.
Refer to Fowler and talk about different kinds of questions and different response options. scale (1-5) make them symmetrical vs. index (yes/no)
Quick note about surveys - don't use “check all that apply” or “rank” type of questions- they are a nightmare to code.
HANDOUT SURVEY DESIGN---limit to 25 questions, put demographic questions at end, be sure to have some open ended questions.
Interviews & Focus Groups : lecture/workshop
HANDOUT INTERVIEW TIPS---review
HANDOUT FOCUS GROUP PROTOCOL---have a facilitator, note taker, observer---incentive for participants?---informed consent, access, timing/schedule, on bus route, child care, facilitator should establish ground rules, try to get everyone to participate----when the session is done IMMEDIATELY write down your thoughts/observations and go through your notes. Do not wait. You will forget.---always send a thank you note to each person---
Show what output can look like/how to code? Find patterns/themes.---this is where making sure you are focused in on answering your research question is crucial----
Sticky video http://www.youtube.com/watch?v=7U6eHgesUtA&feature=related
11:00am
Review why HSR was created in the first place. Show HSR history on our course website. http://academic.evergreen.edu/curricular/atpsmpa/humansubjetsreview.htm
Walk through every piece of the HSR pdf. http://www.evergreen.edu/deans/docs/Official%20HSR%20Form%20Update%201-2-09.pdf
12:00---LUNCH
1:00pm
Move to computer lab, fill out as much of HSR application as possible while there, work on assignments #3 & #4.
Cover concept of risk & note that there is always something to put in this category, cover the differences between confidentiality and anonymity. Note that they have to physically sign the document and so do faculty.
TAKE TIME TO PILOT THEIR INSTRUMENTS ON A FELLOW CLASSMATE WHO IS NOT FAMILIAR WITH THEIR PROJECT!!!!!
Sunday
9:00am
Milgram Experiment Revised: video/discussion about the ethics of this new present day "obedience" experiment
Freakonomics: video/discussion about causation vs. correlation and what we cannot account for in research
Causation vs. Correlation
Causality: 1) cause must precede effect, 2) two variables must be empirically associated, 3) effect cannot be explained by a third variable (it is non-spurious).
Correlation: variables are observed to be related= when one occurs or changes, so does the other.
11:00---LUNCH
12:00
Videos
http://www.youtube.com/watch?v=DVlddHOx5JM
Tony Garcia Interview with Lummi member (example of the good and the not so good when it comes to interview style)
http://www.youtube.com/watch?v=RzaqK3EaIG8
National Urban Indian Family Institute (example of a case study impacting policy)
Reliability, Validity, & Triangulation
When operationalizing concepts, we want to create measures that have reliability and validity .
* Reliability is repeatability…when a measure will consistently produce correct measures. E.g.,most (but not all) thermometers are reliable. Intelligence tests are problematic. * Checking for reliability can involve examining if: (1) the same measure measures the same thing for different subpopulations, e.g., whether a SAT question is interpreted the same way by different ethnic groups (group bias effect); (2) a different version of the same measure works the same way for same population (parallel forms reliability); (3) the same measure is repeated on the same population at two different points in time to see how stable it is over time (test-retest reliability ); or (4) Different people do the measuring to see if that affects the results (interobserver reliability) .
* Internal Validity is when a measure measures what it is supposed to measure. The research literature identifies many different types of internal validity.
* External Validity is the generalizability of a finding or conclusion. There are limits to the extent to which we can generalize from any particular study, since the research setting is likely to have unique features.
One way to start to compensate for the inevitability of some level of error in our measures is to engage in triangulation . Triangulation is where we rely on more than one measure or more than one method of data collection to study the same thing. If multiple measures or multiple data sources reinforce each other, then we can be more confident about our findings. Example= Case Studies.
Case Studies Handout ---review
Go to computer lab and work on assignments #3 & #4---- pulling it all together---
GO OVER EACH PART OF THE RESEARCH PLAN FROM ASSIGNMENT #4---limitations of the study always include the things you cannot account for that may impact your results, assumptions include the biases you are bringing to the research, a plan of how you will specifically actually physically collect the data means just that----make a outline for yourself---
Assignment #4: Draft Research Proposal
Due: Monday November 9th
Page Length: Depends on Project
Revised draft of assignment #3. Define key terms, specify measures, and provide research plan (limitations, assumptions, how you will physically collect the data, access, contact lists, schedule, location, delivery method, feasibility: who, when, what, how). Attach a draft instrument (survey, interview, or focus group questions plus protocols). Attach draft HSR application with 6 questions answered and cover letter/consent form.
• Discuss our process for evals= we write evals of them together, they submit their self eval to us and they write evals of both faculty and submit to us, in person/phone eval conferences are optional.
VLL: Valuable Lesson of Leverage
One of the most valuable lessons I have learned in life came from working around the house and on cars. You move a lot of heavy objects in home improvement and car maintenance.
If you get in too close to the object you are trying to move, what are you relying on? Yourself- your own brut strength. However, if you move back away from the object, you can rely on yourself AND the object through counter balance. This is an important lesson to use in research when we are transforming data into usable information- back away from the data.
Don't force it. In research it is crucial to be open to the unintended .
End by 2:45pm.
Move back to classroom-----------
3:00pm
Larry present winter quarter syllabus
3:30pm
Alan talks about Tribal Economies course