Rescue this survey: answer key
Introduction to Social Research Methodology
Task 1: Flaw-by-flaw diagnosis and model rewrites
Mark for three things per item: the correct name of the flaw, a mechanism (what the answers would fail to mean), and a rewrite that actually removes the flaw without introducing a new one. Several items admit secondary diagnoses; credit any defensible extra observation, but the primary flaw listed below is the one each item was built around. Rewrites other than the models are fine if they cure the disease.
Item 1 — Double-barrelled. “Pay” and “opportunities for promotion” are two objects; an employee satisfied with pay but blocked from promotion has no honest single answer, so responses mix two attitudes in unknown proportions. The tell-tale and. Model rewrite: split into two items with the same five-point satisfaction scale: “How satisfied are you with your pay at Meridian?” and “How satisfied are you with your opportunities for promotion at Meridian?”
Item 2 — Leading question. “Don’t you agree…” plus “one of the best things about working here” signals the expected answer and invites ratification; agreement will be inflated and the item measures acquiescence, not opinion. Model rewrite: a neutral statement with a balanced scale: “The new flexible scheduling policy works well for me — strongly agree / agree / neither / disagree / strongly disagree”, or a direct neutral question: “How well does the new flexible scheduling policy work for you?”
Item 3 — Loaded wording and prestige bias. Two biases stacked: the position is attributed to the CEO (“Our CEO has said…”), pulling answers toward — or, for some, against — the prestigious source; and “hard-working employees have nothing to fear” is loaded, implying that a “No” marks the respondent as not hard-working (a social-desirability trap). The forced Yes/No binary compounds it. Model rewrite: drop the attribution and the loading: “How concerned are you, if at all, about the new performance targets? — very concerned / fairly concerned / not very concerned / not at all concerned.”
Item 4 — Jargon. “Leverage cross-functional synergies”, “omnichannel customer journey”: shop-floor respondents cannot map this onto anything they do, so answers measure exposure to management vocabulary (or guessing) rather than any attitude. Credit students who also name a competence/relevance problem — many staff have no basis to rate their “store” on this. Model rewrite: ask about the concrete thing actually meant, e.g. “How well do the store and online teams work together when a customer order involves both? — very well / fairly well / not very well / not at all well / don’t know.”
Item 5 — Overlapping response categories (not mutually exclusive). “Up to 20 / 20–30 / 30–40 / 40 or more”: an employee working exactly 20, 30, or 40 hours fits two boxes, and respondents at the boundaries resolve the ambiguity inconsistently — pure noise at exactly the values where contracts cluster. Model rewrite: “19 or fewer / 20–29 / 30–39 / 40 or more”, or ask for the number of contracted hours directly.
Item 6 — Non-exhaustive categories. The bands stop at 10 years: anyone with more than 10 years’ service — precisely the long-tenured staff a turnover study needs — has no box. (Sharp pairs may also note that fractional tenures fall between bands, e.g. 3.5 years; the bands assume whole years. Credit the observation.) Model rewrite: add the top band: “Less than 1 year / 1–3 years / 4–6 years / 7–10 years / More than 10 years.”
Item 7 — Double negative. “I do not think that management is failing…” requires respondents to compute the negation of a negation, and then express it on an agree–disagree scale that itself carries direction; many will read past one negative and answer the opposite of what they mean, and their errors are undetectable. Model rewrite: state it positively and let the scale carry direction: “Management keeps staff informed about changes that affect them — strongly agree … strongly disagree.”
Item 8 — Unanswerable recall demand. Nobody can recall customers served over 12 months, but the write-in box demands a number, so every respondent fabricates one: precision without accuracy. Model rewrite: bound the recall period to something rememberable and use bands: “On a typical working day, roughly how many customers do you serve? — fewer than 20 / 20–49 / 50–99 / 100 or more / I do not serve customers directly.” (The last option also fixes the hidden relevance problem for non-customer-facing staff.)
Item 9 — Missing “don’t know” / “not applicable” where needed, on a required item. Not all staff use the computerised rota system (head-office staff on fixed hours; brand-new hires), yet the item is required with no opt-out. The genuinely unable must either fabricate a rating or abandon the form — and fabricated ratings are indistinguishable from real ones. Model rewrite: keep the item and its scale, add “I don’t use the rota system / don’t know”, and keep it required — the opt-out makes “required” legitimate. (Alternative full-credit fix: a contingency question — “Do you use the computerised rota system?” — routing non-users past the rating.)
Item 10 — Order/context effect between two adjacent items. The wording of item 10 is fine; its placement is not. A specific — and, for many, irritating — question about the rota system immediately precedes the general schedule-satisfaction item, so the rota system is at the top of every respondent’s mind when they answer item 10: the general rating is contaminated by the specific one. Today’s rule: general before specific. Model fix: move item 10 before item 9 (or to the start of the scheduling block), and keep that order identical for all respondents. Credit pairs who add that in CAWI one could randomise block order — but the general-before-specific fix is the expected answer.
Plenary tip: items 3 and 10 generate the best discussion — item 3 because students must untangle two named biases plus a response-format problem, item 10 because the flaw is invisible when items are read one at a time, which is exactly the point.
Task 2: Model questionnaire outline
Do not mark against this outline item-for-item; mark against the structural requirements (intro with consent line, 6–8 items, at least two labelled groups with one-line introductions, clean response formats, opt-outs where needed, one open question at the end) and against the wording rules from Task 1. A strong submission looks like this:
Introductory text. “This short survey about work schedules at Meridian is run by the research team on behalf of the board. It takes about 5 minutes. Taking part is voluntary and your answers are anonymous: results will be reported only in aggregate, and no one at Meridian will see individual responses. By continuing, you confirm that you agree to take part.”
Group A — Your current schedule. “The first questions are about the schedule you work now.”
- Overall, how satisfied are you with your current work schedule? (Very satisfied … Very dissatisfied, 5 points, all labelled) — the general item, placed before any specific ones.
- How well does your schedule fit your commitments outside work? (Very well / Fairly well / Not very well / Not at all well)
- How often do you work hours you would prefer not to work (e.g. late evenings, weekends)? (Never / Rarely / Some weeks / Most weeks / Every week)
Group B — Rota changes and communication. “The next questions are about how schedules are set and changed.”
- How much notice do you usually receive of your rota? (Less than a week / 1–2 weeks / 3–4 weeks / More than 4 weeks / Not applicable — my hours are fixed)
- In the past month, how often was your schedule changed after it was published? (Never / Once / 2–3 times / 4 or more times / Don’t know)
- If you ask for a schedule change, how easy is it to get one? (Very easy / Fairly easy / Fairly difficult / Very difficult / I have never asked)
- The people who set my rota take my preferences into account. (Strongly agree … Strongly disagree, plus “Don’t know”)
Open question (last). 8. “Is there anything else about scheduling at Meridian you would like to tell us? A sentence or two is plenty.” (Optional text box.)
Points of craft worth praising when you see them: bounded recall periods (item 5’s “in the past month”); behavioural opt-outs that are informative rather than evasive (“I have never asked”, “my hours are fixed”); the general satisfaction item placed first; no required opinion items without an opt-out; honest stated length.
The pilot record. Require two specific findings in “problem → change” form, e.g. “pilots read ‘schedule’ as including breaks → changed to ‘shift pattern’”; “no option for staff on fixed hours in item 4 → added Not applicable”; “form demanded an answer to the open question → made it optional”. Generic findings (“it was fine”, “one question was a bit unclear”) do not meet the bar. If a pair reports the pilot caught nothing, invoke the iron law from the lecture: the pilot always finds something — send them back to ask their pilots what they hesitated over.
Task 3: Model invitation and reminder plan
Model invitation e-mail
Subject: Your say on work schedules — 5 minutes, anonymous
Dear colleague,
Meridian’s board has asked an independent research team to find out how well work scheduling is working for staff — and what should change. We would like to hear from everyone, whatever your role or hours.
The survey takes about 5 minutes. Taking part is voluntary. Your answers are anonymous: results go to the research team only and will be reported in aggregate — no one at Meridian will see individual responses.
Please complete it by Friday 19 September: [survey link]
Thank you — your answers will directly shape the board’s decisions on scheduling.
[Name], on behalf of the research team
Required elements to check off: sponsor and purpose; honest length; voluntariness; anonymity and how it is protected; deadline; link. Sender: the strongest answers name a trusted, senior, but non-threatening sender — the CEO or the research team lead, explicitly not the respondent’s line manager and preferably not a bare HR system address — with a reason grounded in the willingness/trust argument from the lecture: answers about schedules are answers about one’s manager, so the sender must not be the person being rated.
Model reminder schedule (3-week window)
| Day | Contact | Audience | Message |
|---|---|---|---|
| Day 1 (Mon) | Invitation | Everyone with company e-mail | Full invitation as above |
| Day 8 (Mon) | Reminder 1 | Non-respondents only (or all, with “if you have already completed it, thank you — please ignore this message”) | Short: purpose, 5 minutes, deadline, link |
| Day 17 (Wed) | Final reminder | Non-respondents only | “Closing Friday” — urgency, one line, link |
| Day 19 (Fri) | Field closes | — | — |
Any schedule with an invitation, at least one mid-field reminder, and a “closing soon” contact inside a 2–3-week window earns full credit; bonus credit for restricting reminders to non-respondents where the platform allows, or for the anonymous-design courtesy wording, and for scheduling the PAPI store collection in parallel (distribution and a sealed collection box, with a mid-field chase of quiet stores).
Monitoring sentence — should name at least one concrete dial from the lecture: completes per day against reminder dates, sample composition (stores vs head office, regions), abandonment points, or item nonresponse. Stores-at-half-rate sentence — the expected answer is targeted follow-up: re-engage store managers as endorsers, chase specific quiet stores, extend the window if needed — and explicitly not blanket reminders to everyone (which deepen the skew) and not redefining the target population.
Timing and discussion guidance
Task 1 runs comfortably in 20 minutes if pairs are told to spend no more than 2 minutes per item; warn at 15 minutes and let them finish the rewrites for any items still bare-named. In the plenary, sample two diagnoses per pair rather than walking all ten items with everyone. Task 2’s build phase overruns if pairs polish wording — remind them the pilot exists to find flaws, so a rough complete form beats a perfect half form. If time is short, Task 3 can be assigned as a take-home; if so, collect the pilot records from Task 2 before the class disperses, since they are the evidence that the pilot actually happened.