help with research design

[achandler1]

I work at a Zoo and early last year, I did a survey with visitors regarding their perceptions and knowledge about some specific animals. Over the course of the past year, we've done a series of events and exhibits about these animals and now I am wanting to test for any change in visitor knowledge/perceptions. I'm trying to decide if I go back to exact same sample (I have most of their emails) and ask the same questions, or should I conduct the survey with a different set of visitors? Or should I do both? Any thoughts as to the most appropriate design?
Thanks so much.

[Ian Straus]

Two important questions: Do you know the rate at which visitors return? What was your response rate?

Without knowing your answers, I suggest you use a new sample of visitors.

Why? with last year's sample you have no assurance that they have visited the zoo again since your earlier survey. I imagine many do not, and many more won't respond for the usual reasons people don't respond.
I don't know the size of your old sample, but most people invest the resources for a sample that is adequate to reach their chosen level of confidence, and not much more.
So I suspect your old sample is not in the thousands.
So a decision to re-sample the old respondents means you will have a considerably smaller sample the second time around, and you may not be able to announce any result with confidence.

Of course you could do both: You could survey not only new visitors but also former visitors and compare the answers of those who did visit again with those who did not. then you'll see if I'm right about the disadvantages of revisiting your old sample. As a by-product you would get an estimate of the proportion of visitors who return again in x period of time.

[achandler1]

Thanks Ian.

The visitors surveyed are locals and typically come annually, actually 3-5x a year, on average. The survey was conducted online and the response rate was approx. 20%. I have just under 600 completes.

My concern with surveying a new sample was about the appropriateness of conducting stat testing to look for differences between independent samples. I was told recently by a 'statistician' that you could only do pre-post stat testing if it is the same sample. I found that difficult to believe and haven't been able to confirm that in anything that I've read.
What do you think?

Thanks again, Amy

[Bill Golden]

How about trying several ways - and have some fun while doing it -

Use survey types #1 and #2 as you have planned, and see how they compare or contrast -

#3 is the in-person surveys you will do, at the zoo.
I did these at a zoo some years ago (8 years ago to be exact)
It ended as a real good, productive experience -
We picked a good location, at the confluence of three walking paths near the animals and seals, where people naturally approached us -
Almost everybody said yes and agreed to do the survey -
An item of interest which was to a degree a surprise, was a very high
"pay attention" rate on the part of most of our respondents, in this zoo environment; a zoo is a GOOD place to do market research surveys -

For #4, realize that most people at the zoo will be in groups, meaning groups of people walking (few people go to the zoo alone)
So survey type #4 will be for KIDS - there are usually lots of them -get their input, see what they think - "good cat/gato bueno", etc.
And because of the survey respondents coming in groups, you should have at least 2 interviewers; there is room for 3, and we operated with 4 when we ran the project, and stayed busy all afternoon - (somebody has to be good and talented with the quota sheet, if you are running quotas...)
We found virtually everybody who did our surveys that day to be really interested in what we were doing and asking -

Let me know how it goes, our team would really be interested in your zoo experience -
goldenwilliam@ymail.com

[Ian Straus]

Thanks Ian.

My concern with surveying a new sample was about the appropriateness of conducting stat testing to look for differences between independent samples. I was told recently by a 'statistician' that you could only do pre-post stat testing if it is the same sample. ...What do you think?

Amy,
People do tracking studies all the time, and find them useful.
Also remember statistical process control, which manufacturers have found very useful all my life. You don't do that by taking repeated looks at the same sample of items; you do it by taking new samples periodically.

The commment may have been about this:
There are paired-sample situations and independent-sample situations. Paired-samples allows separating the differneces you find into within-subject and between-subject effects.
There is a different T-test for paired sample situations.
But that doesn't mean you can't apply any statistical test to an independent-sample situation.

[Ian Straus]

Thanks Ian.

The visitors surveyed are locals and typically come annually, actually 3-5x a year, on average. The survey was conducted online and the response rate was approx. 20%. I have just under 600 completes. ...

Amy,
Is this the membership of a frequency program or season pass program? It doesn't sound as if that includes out of town tourists or infrequent visitors. And ordinarily I'd expect the zoo, or at least a big zoo, to have a lot of infrequent and one time visitors.

If so, and if your entrance system logs visits by member, you may be able to cross the survey results with your database and produce interesting stuff.
- You might identify people who visited during a particular exhibit or program, and estimate the effect on knowledge & beliefs about the subject of the program.
- You might begin to model for what precedes ceasing to visit / dropping out of the program.
- You might just identify differences in demographics or in-zoo spending by frequency of visit.
Any of those might be useful.

[Seán]

The ideal design would a paired test. Test people before programme exposure and then test after would, with the testing time before and after exposure consistent between participants to control for memory loss. Was your initial sample of people who were later exposed to your events?

To get a paired test you will need to clean-out any one who would not have had the exposure. Then you are just left with the problem of time since exposure and people who have had multiple exposure (this could be checked in the survey).

Testing a separate set of people would need the underlying assumption that they were also exposed to the programme/ stimulus/ manipulation. This assumption generally works for tracking studies because one of our core questions is the coverage of the manipulation, with an underlying assumption the manipulation was effective (we pre-test the programme – TVC, price change, etc).

Without prior knowledge of the programmes effectiveness, any separate sample result will be ambiguous. For example, a ‘no change’ result could mean the programme was not effective and/ or not enough people were exposed to the programme. If you just compare those in the survey who claim they were exposed to the programme then you are just doing a paired test with the added expense of over sampling and not knowing if they actually were really exposed to the programme (memory is a fickle thing).

Cross-sectional surveys are more something we in market/ social research are forced to use because we can not go back to our original samples. In manufacturing the original sample is destroyed by the testing process. However, in manufacturing we often do go back to the larger sample we take from the original batch. Incidentally this is how we can get ‘use by date information’ plus check consumer complaints or is sued.