If media headlines are to be believed, 2014 was the year in which Catholics, having been “consulted” about their adherence to Catholic teaching, firmly rejected traditional doctrine on sexual morality. “Poll: Catholic Beliefs at Odds With Vatican Doctrine” ran one headline. “German, Swiss Catholics Reject Many Church Teachings on Family,” claimed another. The supposedly official “survey” in question was described by some bishops as an “unprecedented questionnaire” and even as an “opinion poll,” creating the unfortunate perception that the Vatican was “polling” members of the Church on future pastoral priorities and perhaps even on doctrine itself.

The remarkable thing about all of this is that there never was an “official Vatican survey” to begin with.

The first discussion document for the Extraordinary Synod of Bishops on the Family this past October contained a number of technical discussion questions that were really not designed for laypeople. Consider the following typical question: “How is the theory and practice of natural law in the union between man and woman challenged in light of the formation of a family? How is it proposed and developed in civil and Church institutions?” It strains definitions to the breaking point to describe a list of three dozen questions of this nature as a “poll” of laypeople.

The idea of surveying laypeople seems to have arisen from a letter by Cardinal Lorenzo Baldisseri, general secretary of the synod, asking that the entire document be distributed to deaneries and parishes so that feedback could be gathered from local sources. Somehow or other, this normal consultation process transformed into a rather haphazard series of diocesan “opinion polls” accompanied by potentially unreliable statistical analysis.

Given the confusion that surrounded the synod on the family last October, and the likelihood of ongoing controversy in the months preceding next October’s synod, it is worth examining how this “survey” was conducted in many dioceses in order to avoid making the same mistakes again.

Survey design involves more than just listing questions on paper or on a website and hoping for the best. There are well-established principles that should be followed. A very quick review of the way in which a number of dioceses approached this process reveals some major flaws.

The first flaw has already been noted — the questions themselves were not designed for ordinary laypeople. This problem was recognized by some dioceses. For example, the Archdiocese of Louisville, Ky., adapted their online survey, presenting laypeople only with questions that were within their competence to answer. But how many dioceses or parishes around the world actually took this sensible step? The answer, of course, is that we don’t know, and it is almost impossible to find out. So when we talk about a “Vatican survey,” we are potentially talking about hundreds of different consultation processes that may or may not be comparable to each other.

Connected to this is the fact that, even if the questions were relevant and comprehensible, they were not presented in the form of a survey designed according to generally accepted scientific methods. The quality of results one gets from a questionnaire is directly proportionate to the quality of the research instrument itself. The way in which questions are designed and phrased matters a great deal. In the synod’s preparatory document, we are dealing with a set of questions designed for discussion by bishops, not a professionally designed questionnaire.

Some dioceses, like St. Petersburg in Florida and Arundel and Brighton in England, actually designed simplified online surveys based on the discussion questions in the document. From a technical point of view, this is a step forward, but is still insufficient. The questionnaires show basic technical flaws in the way questions are worded. The result is a deficient survey instrument. Bishops and priests are not professional market researchers and should not be placed in a position where they have to take on this task.

But this is not all. A more serious error relates to the sampling strategy employed throughout this consultation. The bottom line: There was no sampling strategy. We can only generalize the results from a survey sample to the population as a whole when a random sample has been used. Did any diocese think about an appropriate random sample? What was to stop a person from filling in the survey multiple times? Furthermore, not everyone may have had an equal chance of completing the “survey” — the likelihood of even seeing a questionnaire may have depended on the attitude that the bishop or pastor took towards the consultation process in the first instance.

While the consultation process was undoubtedly well intentioned, these basic missteps in questionnaire design and sampling strategy cast a major doubt over the reliability of the entire exercise.

This critique is not merely an exercise in pedantry. These flaws need to be identified and corrected precisely because the Church should conduct its business in a professionally competent manner. But they also matter because the preparatory document for next October’s Ordinary Synod of Bishops on the Family has now been published and contains a further 46 discussion questions. If anything, the questions are even more inaccessible than those posed last year. But, already, the document is being misconstrued as another official “survey” of Catholics.

Bishops must avoid another consultation approach that yields unreliable results and also contributes to the mistaken perception that the Church, like a political party, is now polling its members on matters of policy. Conducting a quantitative survey can capture information from large numbers of people, and this has a certain understandable attraction. But without proper design and sampling, any information generated is of very doubtful value.

Bishops could facilitate better consultation on pastoral practice by facilitating parish assemblies or inviting written submissions from interested laypeople, recognizing, of course, that this qualitative approach is not amenable to statistical analysis and generalization.

An even better approach would be to actually meet with families across dioceses to discuss pastoral priorities face to face. While not lending itself to dramatic newspaper headlines, pastoral outreach of this nature also has the advantage of not being easily exploited by others as a political weapon in an already divided Church.

Patrick Kenny writes from Ireland, where he is a lecturer in marketing strategy.