By Ron Kurtz There are many ways that weak research, combined with poor analysis, can lead to bad conclusions. Research among “luxury” and “affluent” consumers is vulnerable to errors in various areas. For example, the lack of an objective or universal definition of “luxury” creates uncertainty as to what respondents understand the word to mean. In addition, the level of “affluence” associated with true luxury consumers is subject to various differences of opinion among researchers. Some use a level of income, a criteria which seems to be much less stable and reliable than net worth, according to a recent study by the Federal Reserve Board. The specific level of income or net worth can also vary substantially among researchers. Rich definition? Most consumer research, including that among luxury and affluent consumers, is based on online surveys among panels of people who have “volunteered” to respond to frequent and often lengthy questionnaires. The validity and reliability of such research should be carefully considered given a recent study that showed how panel members are often treated poorly. The study showed that the poor treatment of panel members included “paying them pennies, giving them boring, lengthy, or irrelevant surveys, frustrating them with multiple closed studies, and bombarding them with opportunity after opportunity.This is most definitely not how you want to treat people upon whom you are depending for your success. And if you or your research vendors are not paying attention, this is exactly what may be happening in your research.” While the report provides a review of general consumer panels, these same panels are the sources of respondents for surveys of affluent and luxury consumers. It is hard to imagine how such people can be representative of the true luxury and affluent consumers. And, as a result, it is doubtful that the research results are projectable. The new report is a sequel to a 2009 report by a prominent market research firm that ran an internal test on a few panels it had used or was considering. It arranged for a selection of mystery shoppers to sign up for each panel and be typical respondents for a month. The new test used the same methodology to evaluate 12 major panels, including Toluna, e-Rewards, Clear Voice, Surveyhead, Opinion Outpost, MySurvey.and six more. All were evaluated from the perspective of the typical panel member. Panel, not pummel Some of the problem scenarios identified in this study of research panels were: “Yours is the tenth questionnaire in a row that the respondent has completed that morning, and many of the others were long, boring, and irrelevant. The respondent is tired and inattentive. “Or let us say that the respondent has attempted 12 different questionnaires this morning before trying yours. One of them asked 10 minutes’ worth of questions before telling the respondent they were not qualified – and tossed them out with no reward. “One of them froze when the respondent was mostly done. “Another one told the respondent they were not qualified and kicked them out before they could answer a single question. “Two more were actually called “surveys” but were trying to get the respondent to compare car insurance rates. “Five of them were already closed by the time the respondent tried to respond, even though the invitations were all sent yesterday or today. The respondent was disqualified for two more because they do not own a pet, even though they stated in their panel profile that they have no pets. “The respondent is tired, frustrated, and annoyed, and now they are evaluating a new product concept that you really hope they will like. Now just how reliable is your data?” BOTTOM LINE, if you buy syndicated research, you should ask if it is based on a panel survey and if so what company provided the panel. If you are doing your own survey, you should be careful in how you select a panel provider. Or perhaps you should consider a different survey methodology. Good research can be a valuable contributor to good planning and decision making. Good research requires careful planning and analysis. While this is not always easy, it is well worth the effort. Ron Kurtz is founder/president of The American Affluence Research Center, Atlanta. Reach him at kurtzgroup@comcast.net.