Response Bias in China

[Louiseb]

Does anyone have any experience with response bias in China? In my experience in Japan, I have found that people tend to use lower points on the scale more often than other countries. Does anyone know whether there is a bias in Chinese respondents I should be aware of?

[dotdash]

I've found the same kind of bias in Taiwan -- I'd expect it in China, too, and generally in Asia/Pacific Rim.

[Allanon]

I can't compare with other countries because my research experience is mainly in China, but our experience for China is as follows:

* Chinese respondents rarely choose negative responses. Many use the neutral response to express negative views. On the other hand, some neutral responses actually do represent a neutral attitude.

* Moderate positive responses similarly may indicate either a neutral attitude or a positive attitude. Consumers may say something is "somewhat good" even if their real feeling is only neutral, in order to "give face". Only the top responses can be considered unambiguously positive.

* An example from our internal database of taste testing data - we find the average distribution of a 5-point likeability scale is:

5 (like very much) - 19%
4 (like somewhat) - 45%
3 (so-so) - 11%
2 (not like very much) - 19%
1 (not like at all) - 6%

I'm curious to know how this compares with other countries!

David

[Philip Moore]

Hi David,

The typical american responses to a 5-point satisfaction score are so skewed toward the positive that researchers have created new indices to try and find some value. I'm used to seeing numbers like...

5 (very positive) - 74%
4 (somewhat positive) - 20%
3 (neutral) - 3%
2 (somewhat negative) - 1%
1 (very negative) - 2%

So the clients built "exception" policies to focus on the 3% of customers who were somewhat or very dissatisfied.

Many researchers, both commercial and academic, moved away from the lone overall satisfaction score and added a likelihood to recommend and a likelihood to repurchase/revisit to an index. This combination of three questions received many names: secured customer index, customer loyalty index, delighted customer index, etc. The basic approach is to measure and track the intersect set of the top-box responses.

Reicheld suggested dropping the overall satisfaction question all together and using only the likelihood to recommend question. But in order to put a name on his approach he also subtracted the neutral and negative responses from the very positive and coined the phrase "net promoter."

Either of these approaches is probably superior to simply tracking and reporting the top-box percentage of an overall satisfaction score.

[etnlessard]

I guess the Canadian perspective is somewhat difference from what you have observed. The bias here is often towards the center of the scale. Here are the percentages for a 7 point scale on the evaluation of different aspects of a mall.


1: 7%
2: 8%
3: 10%
4: 25%
5: 18%
6: 17%
7: 15%

We must prefer not to give to strong an opinion (???).

[Philip Moore]

Yes.

Actually, I think this topic may have some articles in the archive. I know Harris Interactive did a nice cross-cultural scale usage ROR project in the EU a few years back and discovered some meaningful differences between the Scandivian, Anglo, and Mediterranean respondents.

In Canada, I would guess you even see differences between scale use in Montreal and Vancouver.

[Louiseb]

Thanks all. Would you say that this positive response bias extends to yes/no questions? For instance, would they tend to overstate a "yes" response in order to "give face"?

[Allanon]

Um ... yes?

I'm a little uncomfortable with the term "bias" in this context. Bias: "an inclination of temperament or outlook; especially : a personal and sometimes unreasoned judgment" (Webster's online). The word has a negative connotation, but when we ask respondents (for example) to evaluate how much they like a product on a five-point scale, what indeed can we expect other than a personal and unreasoned judgment? There usually is no actual fact with which to compare the results. This is equally true in all countries, not particularly so in China.

Philosophical rants aside - for yes / no questions, whenever I am concerned about the possibility of bias (due to a desire either to "give face" or to "save face"), I would reformulate the question as a multiple choice. For example, rather than asking "Are you planning to buy a car within 1 year?" ("yes" means "yes, I have enough money to afford a car"), I would ask "Which of the following items, if any, are you planning to buy within 1 year?" Still not perfect, but better.

[Seán]

Completely agree with Allon on this in terms of the connotative meaning of bias. From whose perspective do we judge ‘bias’? In a technical sense it has good meaning but in a social sense it is a loaded word. If results are different, maybe that’s because the experience is as well! When looking at others we often forget the actor-observer attribution biase of attributing our experiences based on context (e.g. good service) and others behaviour on their personality/ culture (Chinese give more positive answers because of some cultural reason).

To add more complexity to the issue. In countries that have multiple languages you may get strong differences depending on the language you used. If your survey is in the native language (e.g. Cantonese) response can be very different to using an official language (e.g. Mandarin) or foreign (e.g English).

A colleague doing is doing doctorial work on this topic in South East Asian country on culture and found the language used determined the culture expressed. In a mono-language culture a parallel would be in how we answer an ‘officially’ question and one that is more ‘colloquial’.

Assuming you can’t change to a more concrete measuring system, maybe indexing everyone to 100 as a start point and then look at the trend over time rather that get hung-up on the current score.

[etnlessard]

With regards to the semantics issue that was broached above, I was wondering if perhaps "bias" might not have several meanings and when it is used in statistics it has a particular meaning, whether it has a negative connotation in a different circumstance or not. Here is a definition from a technical dictionary used in Quebec (www.granddictionnaire.com).

"The difference between the expectation of the sample estimator and the true value, depriving a statistical result of representativeness by systematically distorting it."

In other words, wouldn't bias here mean that it is the difference between the response given by the respondant and the response the researcher meant to receive for that respondant's situation. For example, if you ask the question "Do you intend to buy this TV", and as the researcher, you are expecting "yes" if there is intention and "no" if there isn't but the respondent says "yes" to make you feel better, or for whatever other reason, then the bias is the difference between what the researcher expected and what the respondent answered.

My personal opinion is that the best way to go about it is to approach the score obtained simply as a point of reference. I would therefore agree with Seán that comparing over time might be best. In the case of comparing two populations, you could pretest on other issues and weight the obtained scores having measured the possible "bias".

[Allanon]

Originally Posted by etnlessard

"The difference between the expectation of the sample estimator and the true value, depriving a statistical result of representativeness by systematically distorting it."

In other words, wouldn't bias here mean that it is the difference between the response given by the respondant and the response the researcher meant to receive for that respondant's situation.

I agree with this way of looking at it, but still feel there's a problem with the word bias in this case, because usually the questions we are asking don't refer to any objective fact. For example, "how likely are you to buy this product on a scale of 1 to 5?" There is not actually a scale of 1 to 5 somewhere in the respondent's brain. There is not even such a thing as "intention" which is either present or not present. Both the scale and the term "intention" are semantic conveniences we use because of the complexity of the real situation. Unlike a thermometer or a ruler, the scale is not a precise measuring schemes associated with the objective facts in a systematic way. Since there is no "true value", I don't see the relevance of discussing the difference between the estimator and the true value.

In my opinion the only case where the term "bias" would become relevant is if we are actually asking the respondent about a fact, such as "Do you own a car?" In this case there is a systematic correspondence between the measuring instrument and the facts, so we can meaningfully question whether respondents' answers match that correspondence. But I strongly feel the above is not true for scales of likeability and purchase intention.

But we seem to reach the same conclusion - I totally agree that the best way to analyze this kind of data is by using historical data as a reference point.

[etnlessard]

Originally Posted by Allanon

There is not actually a scale of 1 to 5 somewhere in the respondent's brain. There is not even such a thing as "intention" which is either present or not present. Both the scale and the term "intention" are semantic conveniences we use because of the complexity of the real situation. Unlike a thermometer or a ruler, the scale is not a precise measuring schemes associated with the objective facts in a systematic way.

I completely agree with this and would add that this lends even more support for ethnographic research (see "gorillas in the midst" thread).

[Louiseb]

Thanks to all for your input. In this case, I'm asking whether respondents have seen a particular advertisement and showing stimuli. So, there isn't an opportunity for scaling the question or making it multiple response - you say you have seen it or you say you haven't. I'm concerned that respondents may feel obligated to say they have seen it even when they haven't for any number of reasons - to establish that they are responsible workers or for any number of cultural reasons. Additionally, I inserted some ads that were never run and their ghost awareness was also very high. Does anyone have a point of view as to why this has ocurred?

[Philip Moore]

I'm not sure you can do anything about this at this point in your project, but if you get another job like this you could try projective questioning either in place of or in addition to your current survey questions.

Ex.

How likely is it that your neighbors have seen this ad before?

How effective do you think this ad would be in convincing your neighbors to buy this product?

etc.

Good luck

[mzhummer]

hi everyone.. i needed to revive this thread as we are in need of an article of some sort that helps 'validate' the appropriateness of using 5-point rating scale vs. the 7-point rating scale when conducting surveys in asia (vs. US for example). it has been more of a general knowledge that 5-point scale does work better among asians but we seem to lack some written proof to support this.

would very much appreciate if you can direct me to any relevant/related document. thanks in advance!