Standards of the MR Industry

[Raincatcher]

Hi All,

Could anyone tell me if what I'm experiencing now is a common phenomenon in the market research industry, particularly in Hong Kong? Or is it mainly a problem of the research firm I'm working in? Thanks!

But before I tell you the story, I'd like to briefly introduce myself so that you have an idea of why I feel this way. I'm a junior, at least compared to most of you here, market researcher in Hong Kong with 2 years of market research experience. But before this, I have other research experience (both qualitative and quantitative) that is mainly academic-based. I've helped out a professor with a quantitative research study funded by the government and have also conducted an indepedent qualitative research for my Masters dissertation. In addition to actual research experience, I've also taken compulsory research-related courses, such as (social) research methodology and social statistics (beginner and advanced for both), for my Bachelor degree as well as for my Masters degree. Therefore, I am confident to say that I have a good foundation that will give me the knowledge that I need for further learning and for developing my career in research.

I have encountered two particular incidents in my company that have given me doubts to the standard of the research industry which I hope people here can help dismiss. I was recently involved in a quantitative tracking study which my company has done for the second year - second and third waves of the study (I was not involved in this project last year). No change has been made to the questionnaire and the report format for this year. I found that my manager (5 years of research experience) has not only misunderstood what multiple regression is, but she has also misunderstood what "statistical significance" is!! Indeed, she has been using terms of standard deviation and confidence interval (always 95%) on all quantitative reports only out of habit and as a "standard procedure".

1. For the multiple regression analysis:
The client requested it last year when we did the study for them for the
first time. The initial research design (i.e., the first wave of this tracking
study) was done by another research company. Our company have taken
over the study from its second waves onwards. Although I was not
involved in the study last year (second wave), I did give my inputs and
explained to my manager what "multiple regression" is.

To my surprise, I found that she has gone back to her old/own
interpretation of multiple regression this year when we were discussing the
findings!! What I can't understand is how can someone present findings to
a client without even understanding what she's presenting?

2. For the significance test that our client requested us to do for this study:
We split our total sample into two: Samples A & B. We did a significance
test (at 95% confidence interval), using the proportion difference between
the two (independent) samples , for every attribute that we measured.

Again, to my surprise, I found that my manager has misunderstood what
significance test is when she did a mock presentation in front of me. She
stated that Sample A is significantly lower than Sample B on Attribute X,
explaining that the difference between A & B is "very big". That is, she
has misunderstood "significant" here in terms of the "magnitude" of the
difference! I almost died from shock when I heard this.

I cannot believe how someone with 5 years of MR experience and at a managerial grade can allow herself to do something without fully understanding what it is and the theory behind it, especially for something that is as basic as "significance test". In my opinion, "significance test" is equivalent to "ABC" in the English language.

Please tell me that this is her own problem and not instrinct to the market research industry in Hong Kong. I'm beginning to doubt whether my enthuasiasm for research is really in academic research, not commercial one. Indeed, I'm beginning to have doubts about my career choice and am wondering if I should switch. However, I find that I have a hard time letting go of market research and am certain that I'll miss it if I do give up.

Any advice would be greatly appreciated. Thanks!

Best regards,

Raincatcer

I was involved in academic research 10 years ago as well before going to the "darker commercial side" I have encountered many experienced and good researchers and many good managers. Finding both qualities in the same person is more rare but don't give up ( if you have both qualities you are heading for a bright future)

If you are a stastical purifist, change again. You will be frustrated otherwise.

not so much by certain people who do the talk but don't know what they are talking about , but just by the fact that you have to do water in your wine given timings, budgets, methodologies etc...

You will find sins against the most basic stuff hapens more often than more sophisticated stuff = a lot of people in research (commercial) do not understand the true notion of significance but it was never explained to them and they don't dare to ask because like you said it should be abc.

I have not the intention of defending some hong kong manager but I would say this in his/her favour although you are absolutely right.

In many cases the sample sizes are limited and to get a significant result the difference might be of the order that everybody feels is big. A big difference is therefore not significant and significant does't have to be big but in commercial research it is often so.

Further clients want to hear if it is big or small ( they are accounted for that not for signifcance) Try to explain to them that size doesn't matter...

Letting you career path depend on what other people are doing is not the right attitude. Take the situation into account, try to grasp why the situation is like that. if you feel uncomfortable with it, try to change it and do it differently if you like what you are doing. If you have some commercial responsibility there is a saying in th industry: you get the clients and (managers) you deserve

Not one resarch is flawless and certainly the industry and the people who are working in it are either but is the "academic aura" so pure ( we could start a section about that as well).

[Evgeny Belov]

By my opinion, that happens due to bad knowledge of statistics of the client side. By my experience there aren't so many people at client side who really understand what regression, discriminant etc. even siginificance is. Moreover, they desire for simple reports, because many of them do not understand if you go further crosstabs. This is especially true for local companies, but frequently happens in large multinational companies.

Another thing with significance testing is that if sample size is large enough (for instance 1,500 U&A) a difference of 2 percent can be significant. But just overall sense says that in question for prompted awareness it is almost nothing (however it can worthwile in question of favorite brand). So clients really do not look for significance but for large difference between them and competitors. So not all significant differences means something worthwile by the client point of you. I even use two words "significant" ant "noticeable" difference in reports.

Nevertheless, as Cherchill said: "pessimist see troubles in every oportunity, and optimist see opprtunity in every truble". So it is good chance for you to get to the top. Because as I believe the more proffesional you are the more market value you have. Start to propagandize scientific approach to the problem, because unproper use of sofisticated analysis and methodologies can lead to confusion, and can kill any considered market plan.

[Raincatcher]

Hi Johan and Evgeny Belov!

Thank you both for your helpful advice!

I guess my biggest concern is what I should do when such incidents happen again, but this time concerning things that I have no prior knowledge of and thus depend, to a significant extent, on my manager to teach me. If this ever happens, I will "absorb" everything she says without any other channels to verify since my company is a very small research firm with only 4 people in the research department, one of whom being the owner of the company who's thus too busy to teach. Moreover, he expects my manager, rather than himself, to play the training role.

Indeed, I've encountered another incident before the ones I mentioned earlier. I wrote a report for a qualitative project - the first qualitative report that I wrote all by myself. My manager thought it's quite good and handed it to my boss for his comments without much changes/corrections made to it, with the intention that this piece of work will be THE FINAL deliverable to the client.

My boss's view was that the report was only good for the detailed findings section which the client can refer to if they want more information. It's also good for a person, like me, who's still learning how to write a MR report. He said that what we need, in addition to what I've written, is a "Management Overview" and that this is indeed what the client is paying for.

After hearing my boss's comment, my manager changed her verdict and explained that the reason why she said my version was good was because she was too busy with other projects at that time and did not have much time to think about the structure of the report carefully! Thus, she sort of put herself in my shoes and read the report from my point of view. Of course, my boss gave her a good scolding for this.

I don't think I can live with another of such incident again.

Originally Posted by Raincatcher

I don't think I can live with another of such incident again.

I can say that I have gone through a very similar situation - the same busy boss, the same not-too-educated manager etc. Of course, it is up to you to decide whether to stay in the company or not. But if you concentrate on research by itself and try to do your best, you will find your own experience rising. And then some day you will be a better analyst than your manager is by now... It is the essence of capitalism, isn't it?

[Evgeny Belov]

The thing I can suggest in this situation is to read smart books about research, statistics etc. Try to get reports from another research companies (if it is possible) and learn reading them.

Also try to explain your company owner what good research is from your point of you refering to admitted statisticans and well-known books.

And nevertheless try to maintain good attitude with your manager.

[Philip Moore]

Not sure what opportunities you have in Hong Kong, but in the US we have dozens of very good conferences every year where you could learn most of what you need to get to the next level. I too came from an academic research background. Although my first commercial research mentors were forthcoming with whatever information I needed and tried to teach through example, I still learned most of the skills that continue to serve me in my current role by attending conferences and workshops.

Good luck

Your manager should have been honnest, telling you upfront look i did not look at it properly but it seems ok lets see what the big guy thinks of it and work from their on.
If he/she honestly said to his/her boss that she didn't have the time, you got in the end some honnest feedback I think, whether from your direct line or the line above.

I don't know whether the hong kong culture allows it but you could ask indicate that you would appreciate honest feedback.
"Sorry paul, I want to do a great job and what to grow that is why your opinion is important to me. I understand your time is limited but I would like to ask you whether we can proceed in the future by giving me some honnest feedback on the work I delivered. In the end it will allow me to take some work burden from you faster than planned"

It is further not uncommon that certain people disagree on what a good presentation/report should look like. It all depends on the context of project, type of client ad trend in companies. 10 years ago we had to deliver 100 pages with a management summary to be stamped OK, now less is more rules in general (my perception)but again it remains personal.

If you are in a small company it is unlikely that they will send you to many conferences given they are rather expensive and they cannot lack the extra hand for certain number of days so mostly it is on the job training. This doesn't mean that it cannot be structured somewhat.

You had some negative feedback. Ask both your manager to indicate 3 to 5 reports from your company they felt were great and for what reason and which were well received by the client. Go over them and check how they differentiate from yours. It will learn you the house style (it can be different from yours). If the company exist already for some time they must be doing something great that clients appreciate.

Being exhaustive is not what it about. It is a cliché but I use it everytime. From each research try to come to 5 learnings from your research. What would you like your client to remember from this research (given the client's question and decision to take) when the report has been red or presented.

If the research is really huge chunk the information in layers never more than 5. ( the average memory span of a normal person is 5 to 7)

You would be stunned if you would ask people in a room what they have learned and taken up after a presentation of 2 hours. You will get more info across with a 30-45min presentation: That is one slide with 5 bullet point conclusions. For each bullet use 1 to 3 extra pages with graphs and extra information to back up the conclusions, you add 1 introduction slide, 1 objective slide, 2 methodology slides and you have a 10-20page report they will love. All further details etc... put them behind this report in separate chapters or annex for consultation

In most cases the 10pp-20pp memo will circulate thru the company. The 100 page report will end up in the draw of the mr-responsible and in time it will come back if they need some additional info etc.

It doesn't mean that your 20page report shouldn't be backed up with the necessary statistics but like selling cars sometimes it is necessay to tell what is under the hood often this is not the case. This doesn't mean the motor is irrelevant. All those little details are nicely written in the manual but not in the nice brochure they used to convince people to buy the car and that is exactly what your report must do. Convince client's that they can make the right decision based on your conclusions. You might think you need all the statistic gizmo's to be reported to achieve this but that is often not the case.

If you want to change given the relationship with your boss runs sour try to end it in beauty. Always think plenty of oppurtunities elsewhere, just don't tell that you want to change given this quarrel with your boss. It doesn't look good. Just tell you want new horizons, a bigger firm to develop your skills further....usual blabla and you will turn out.

After a couple of years you can decide yourself how to write your reports, style and people you want to work with... It takes some time to become the master of your own faith. If you like research don't let one misfortunate career step turn you away from it and wait until you can become a knight yourself

[G----]

You just need to face the fact that your manager is not good. She is faking her way through things. It happens all the time! It's in no way isolated to MR or Hong Kong. There are people everywhere in every business who make a living, get promoted, etc. without really trying to develop the real skills that should be needed for the job.

I would bet that your manager really thought it was a good report because she doesn't have the insight and understanding of what a client wants/needs like the owner does. (When caught, she tried to backpeddle her way out, luckily the owner held her responsible.) With your manager always try to get a second opinion, maybe not directly, but try to find some backup for those things she is "teaching" your. Take all the opportunities you can to learn from the owner or from other experienced researchers outside the company (marketing or MR association meetings are a place to start).

If you get frustrated every time you meet someone who is not top-quality, you won't be satisfied anywhere. I've seen plenty of people faking it in acadenia as well. Make the most of your situation, learn what you can and when a better opportunity comes along, jump on it.

[Raincatcher]

Thank you all!

I have been feeling frustrated about my situation and indeed have thought about switching career. But after finishing another project (and report), I find that I really love this job, although maybe not the company. I think I will persevere no matter what. I believe the best way to overcome this situation is to think critically and use my judgement rather than "absorbing" everything that my manager (or anyone else) says, which is what I should be doing anyway even if my manager never gets confused about her work.

But thank you all for your helpful advice and kind words!

Raincatcher