Right Data, Wrong Decision: How Organizations Misread Their Own Surveys

2026-11-01

Suppose an agency wants to know what children think about organ donation, so it can prepare policy. It surveys 10,000 children. The response rate is 95%. The analysis uses standard statistical methods and the report is full of charts. The result: most are in favor.

That report may still not be enough to base any decision on.

The reason is not the software or the sample size. It is that the organization does not know whom those numbers came from, under what conditions, or how each respondent understood the question. (This is an illustrative scenario, not a description of any real survey.)

If you run a company, the mechanism is familiar. It appears whenever you survey employee satisfaction, assess ERP readiness, ask which teams “already use AI,” or check compliance with SOPs. The questions sound far more ordinary. The errors belong to the same family.

Executive summary

• Having a lot of data is not the same as having knowledge. Knowledge also needs provenance, context, an understanding of how well respondents grasped the question, and a clear limit on what can be inferred.
• The five classic survey errors can each be read as a failure point in creating knowledge: listening to the wrong group, listening through pressure, listening to a misunderstood question, not recording the conditions of collection, and not counting the cost of collecting.
• Data can be accurate as a record and still wrong as an interpretation. The biggest risk is a conclusion that goes further than the data proves.
• An organization that already has a preferred direction can design a survey that confirms it without anyone intending to.

Why start with an extreme example

Asking children about organ donation is a case where every methodological flaw shows up clearly. The topic is sensitive. Adults heavily influence the answers. The concepts are abstract. And the act of asking can cause harm.

Inside companies, the same errors exist. They are just harder to see because the questions look harmless.

It helps to separate two layers. The five errors below are well established in survey methodology. Reading them as five failure points in a chain that creates knowledge is OKELAS’s analysis, not a conclusion from the academic literature.

Figure 1. Five survey errors and matching company examples (illustrative).

Five errors, read as knowledge failures

1. You only hear from people who are easy to reach

In survey methodology this is coverage error: the list used to draw the sample does not cover the group you need to hear from, so results are systematically skewed. A larger sample does not fix it, because more responses still come from the same list.

For a children’s survey, that list is often urban schools, children of medical staff, or patients at large hospitals. Children in remote areas or outside the school system are absent.

Read as a knowledge problem, what is lost is more than statistical representativeness. Each group holds a different kind of experience. A child who has seen a relative die reads a question about death differently from one who has not. The groups hardest to reach often hold exactly the part the organization is missing, and the organization does not know it is missing.

In a company: assessing ERP readiness by surveying only the IT department. Accounting, the warehouse and production supervisors, the people who will work in the system every day, are never asked.

2. The answers were shaped by authority

Respondents tend to choose answers that look socially acceptable, especially on sensitive topics. This is social desirability bias, and it has been reviewed extensively in the survey literature.

Picture a teacher standing nearby, introducing organ donation as a noble act that saves lives. A student’s answer then blends several things: what they actually think, what their parents taught them, what the teacher just said, and what they guess adults want to hear. The chart may look excellent and still fail to reflect independent opinion.

In knowledge terms, the organization receives a version already adjusted to fit the environment. The risk is mistaking agreement for compliance, and silence for acceptance.

In a company: a satisfaction questionnaire handed out and collected by the direct manager. “Satisfied” is one data point. Whether the employee would dare to say otherwise is another, and the form does not record it.

3. One answer, several states of understanding

This is the error we consider most important for knowledge management.

Suppose a child is asked: “Would you agree to donate your organs to save others after you die?” An answer of “yes” can come from four very different states:

• A. Does not understand the concept; thinks it is like giving blood.
• B. Misunderstands; does not distinguish brain death from a coma.
• C. Understands, but answers as adults have instructed.
• D. Understands fully and genuinely agrees.

The survey records the same value: “Yes.” But these are four entirely different states of knowledge.

Figure 2. One answer, several states of understanding (illustrative example).

The equivalent in a company is: “Does your unit use AI in its work?” “Yes” might mean a few people use a chatbot to translate emails, or it might mean AI sits inside a controlled process with someone accountable for it. The same word hides very different readiness. (See also AI Reasoning vs. Organizational Truth.)

If you do not know how respondents understood the question, an answer is still data, but it cannot yet be called knowledge.

4. You do not know the conditions under which the answer was produced

In pediatric research ethics, guidance distinguishes parental permission from the child’s own assent. These are two separate layers of protection: a parent’s agreement does not mean the child participates willingly, and it does not license ignoring the child’s objection.

That has a knowledge consequence. When parents coach a child on how to answer, or push the child to take part, the answer carries the family’s view. If a system stores only the final result, four situations look identical:

• The child agrees because they genuinely believe it.
• The child agrees because a parent said to.
• The child objects because the family objects.
• The child did not want to answer but was told to participate.

The organization has lost provenance: the ability to trace who supplied the information, how, and under what conditions.

In a company: a self-assessment form, or an SOP audit checklist, filled in by the very department being assessed during audit-preparation week and marked “Pass.” No one records who filled it in, when, or under what pressure.

5. You do not count the cost of collecting

When designing a survey, teams usually track the number of responses, the response rate and the margin of error. With children and a topic like death, those measures are not enough. A child who has just lost someone may be hurt by being asked. That is an ethical risk, and it also distorts the data, because defensive emotional reactions end up in the answers.

The management lesson: not all knowledge should be collected at any price. The value of knowledge has to be weighed against the cost and risk to the people providing it.

In a company: a post-incident interview run like an inquiry into who is to blame. Employees may learn that the safest thing next time is to say nothing. This is analysis, not a measured result, but the mechanism is easy to recognize in many organizations.

When one link breaks, the chain breaks

Figure 3. The chain that creates knowledge, and five points where it can break (OKELAS discussion framework).

The five errors correspond to five links in a chain that creates knowledge from a survey. The sixth link is where it is synthesized into a decision.

A survey can collect 10,000 responses, reach a 95% response rate and be processed with modern methods. But if the frame is skewed, respondents are under pressure and the question is misunderstood, all of that can still fall short of a sound basis for a decision. That is an organization with the capacity to process data but not yet the capacity to manage knowledge.

A bigger error: confusing “who decides” with “who knows”

One mistake remains that the five errors do not reach: treating different kinds of knowledge as interchangeable.

In the organ donation case, there are at least four bodies of knowledge, held by different parties:

Figure 4. Decision capacity is not the same as knowledge value.

None can replace another. A physician understands brain death but cannot stand in for a family living through it. A legislator does not automatically speak for a community’s ethical views. A child may lack the capacity to make a legal decision, but that does not mean the child’s voice has no knowledge value.

Decision capacity and knowledge value are different things. When an organization fails to separate them, it tends to make two opposite mistakes: giving one group authority beyond its role, or discounting a group simply because it holds no authority.

On a plant floor it looks like this: a shift supervisor has no authority to change an SOP but knows exactly which SOPs no longer match reality. If the decision process listens only to people with approval signatures, that knowledge never enters the system.

Right data, conclusion beyond the evidence

Consider an illustrative results table:

Illustrative figures, not real statistics.

Figure 5. The gap between what the data records and what the organization wants to prove (illustrative figures).

An organization might conclude “85% of children support organ donation.” That holds only if the sample is representative, age groups are stratified appropriately, children understood the question, adults did not sway them, participation was voluntary, and the question did not create moral pressure.

Even if every one of those conditions is met, the result describes only the children asked, under specific conditions. It does not prove that children have legal capacity to consent, and it does not replace medical, legal and ethical review of policy.

A more careful statement would be: “In the group surveyed, 850 of 1,000 chose to agree. There is not enough evidence to establish respondents’ independent understanding of brain death and donation, so this cannot be treated as the view of children nationwide.”

The point for knowledge management: control the gap between what the data actually proves and what the organization wants to use it to prove.

When an organization asks questions to hear what it already wants to hear

Many organizations are inside a loop they do not recognize:

1. A direction has already been set.
2. Supporting evidence is needed.
3. A survey is designed to measure support.
4. The data comes back matching the way the question was framed.
5. The data is used to reinforce the original direction.

Confirmation bias in individuals has long been described in psychology. Whether it operates at the organizational level is our inference, not a measured result. But the mechanism is plain enough: an individual bias can become an official report, and then the basis for the next decision. Once a flawed belief has been standardized into a procedure, a report or a policy, it is far harder to correct than when it was still an opinion.

An organization needs more than the ability to learn from data. It needs the ability to notice when the method that produced the data is wrong.

Self-check: six questions before you use a survey result to decide

Pick the survey or internal assessment that currently influences one of your decisions: employee satisfaction, ERP readiness, SOP compliance, or an AI usage survey. Ask:

1. Who was not asked? Which groups with different experience were missing from the list?
2. Could respondents tell the truth? Who stood next to them while they answered, and would they face consequences for an unwelcome answer?
3. How many ways could one answer be understood? Was there any step to check that respondents understood the question?
4. Who filled it in, under what conditions? Does the record keep that information?
5. Did collecting it put pressure or harm on participants?
6. Does the conclusion you want to draw go beyond what the data proves? Was this survey designed to find out, or to confirm a decision already made?

If for an important survey you cannot answer three or more of these, treat the result as a hint rather than a basis. (The three-question threshold is a discussion heuristic from us, not a measurement standard.)

The problem is not the data

Looking back, the five errors are not unique to research on organ donation. They are ways an organization can hold plenty of information and still not have knowledge it can trust.

For OKELAS, this is why a knowledge system should not simply store and retrieve answers. The value lies in preserving the provenance, context, evidence, contradictions and limits of a piece of information before it is used to decide. That is OKELAS’s view of the approach, not a description of a standardized verification step.

Bad knowledge does not always come from bad information.

Sometimes it comes from entirely accurate information that was collected the wrong way, placed in the wrong context, and interpreted beyond what it actually proves.

Next steps

• Evaluate your organization’s knowledge management maturity: Knowledge Management Maturity Assessment
• (pillar) From Individual Knowledge to Organizational Knowledge
• (related) Tacit vs. Explicit Knowledge · ERP Data Readiness: The Hidden Complexity of “Clean Data”

Sources

The children’s organ donation survey scenario and the 1,000/850/100/50 table are illustrative and are not based on any specific survey. Methodological concepts are drawn from:

• Groves, R. M., Fowler, F. J., Couper, M. P., Lepkowski, J. M., Singer, E., Tourangeau, R. (2009). Survey Methodology, 2nd ed. Wiley. On coverage error.
• Krumpal, I. (2013). Determinants of social desirability bias in sensitive surveys: a literature review. Quality & Quantity, 47(4), 2025–2047.
• American Academy of Pediatrics, Committee on Bioethics (1995). Informed consent, parental permission, and assent in pediatric practice. Pediatrics; and the update by Katz, A. L., Webb, S. A. (2016). Informed consent in decision-making in pediatric practice. Pediatrics, 138(2).
• Nickerson, R. S. (1998). Confirmation bias: a ubiquitous phenomenon in many guises. Review of General Psychology, 2(2), 175–220.
• Ackoff, R. L. (1989). From data to wisdom. Journal of Applied Systems Analysis, 16(1), 3–9. On the data, information, knowledge chain; the framework is debated among researchers.

The “six-link chain for creating knowledge” is a discussion framework proposed by OKELAS, not a certified methodology or an industry-standard term. This article is a knowledge-governance analysis, not guidance on research design, biomedical ethics or legal advice.