The Hiring Committee That Killed the Unstoppable Candidate
It started like any other Tuesday at one of the big tech companies. I walked into the product hiring committee room, coffee in hand, expecting another run-of-the-mill slate of senior PM candidates. Then I saw her resume: Stanford CS, ex-Google AI team, launched two breakout features at a high-growth Series D startup, glowing references. On paper, she was flawless.
Two interviewers gave her top marks. Engineering lead said she “thought like a founder.” Design partner called her “the most user-obsessed PM I’ve worked with in five years.”
But then the third interviewer, a quietly intense data scientist who rarely spoke up, dropped a bomb.
“She couldn’t explain her North Star metric from her last role,” he said. “Asked her what it was. She said ‘engagement.’ I asked for the quantitative definition. She said ‘time in app.’ I asked how they validated that correlated with retention. She didn’t know.”
A pause. Then the hiring lead turned to me. “Johnny, you ran growth at a similar company—does that track?”
I shook my head. “No way. If she led that launch, she should know the causal model behind their KPIs. Either she didn’t run it, or she didn’t understand it. Both are disqualifiers at this level.”
We rejected her. Later, we found out her manager had inflated her role. The feature succeeded—but she wasn’t the driver. One data source—peer praise—was real. Another—her resume—was manipulated. The third—her operational grasp of metrics—was missing.
That’s when it hit me: the strongest validation comes not from consistency across data sources, but from their tension.
Because anyone can game one stream. Two is harder. All three? Nearly impossible.
The Three Sources No One Can Simultaneously Control
In Silicon Valley, we’re obsessed with data. A/B tests. NPS. Retention curves. But we misunderstand what data is for. Most teams use data to confirm decisions. The best use it to challenge them.
After two decades building products at scale, I’ve found that real insight emerges only when you triangulate across three independent data sources, each controlled by different stakeholders, with different incentives. I call this Triple-Source Validation (TSV).
They are:
- User Behavior (the clickstream)
- Stakeholder Narrative (what people say in meetings, 1:1s, emails)
- Operational Metrics (system-level KPIs, error logs, backend signals)
Most leaders rely on one, maybe two. But the moment you demand all three, fraud collapses, assumptions break, and truth surfaces.
Let me show you how.
Source 1: User Behavior — The Clickstream That Never Lies
Last year, I advised a fintech startup building a retirement planning tool. Their user research was glowing. “We love it,” users said. “Finally, something that makes saving simple.”
But the behavioral data told a different story. Only 12% completed the onboarding flow. Average session time: 90 seconds. And zero users returned the next day.
The product team insisted: “But the interviews were positive! We even recorded them.”
I replied: “People are polite. They don’t want to hurt your feelings. But their thumbs don’t lie.”
We dug into session recordings. Watched users tap through, eyes glazing over at the third form field. Saw them exit when asked for income details. One user literally sighed and said, “This feels like my taxes.”
We redesigned around micro-commitments: one field per screen, optional inputs, instant visual feedback. Completion jumped to 68%. Seven-day retention: 41%.
The insight? User behavior is the only data source where incentives are aligned with truth. No one benefits from clicking the wrong button. No one pretends to use a feature they hate. The clickstream doesn’t care about your roadmap or investor deck.
But behavior alone isn’t enough. Because it can’t tell you why.
Source 2: Stakeholder Narrative — The Unfiltered Truth in Meetings
At another company, our analytics showed a 22% increase in feature usage after a redesign. The product lead presented it as a win. Leadership nodded. Slides were green.
Then I sat in on a support team sync.
“We’re getting crushed,” the lead said. “Calls are up 40%. Users can’t find basic functions. The new nav buried everything.”
I pulled the ticket logs. “Feature confusion” tickets spiked from 15 to 89 per week. Average resolution time doubled.
We’d moved fast, shipped without observing real-world use. Success was measured by adoption (clicks), not usability (support load).
I went back to the product manager. “Your metric says success. Your support team says disaster. Which one do you trust?”
He paused. “I guess… the team closest to the pain?”
Exactly.
Stakeholder narrative—what engineers complain about in standup, what sales says clients beg for, what support hears in calls—is raw, emotional, and often ignored. But it’s also real-time qualitative signal from people incentivized to surface problems.
Executives dismiss it as “anecdotal.” But anecdotes are data points with context. And when multiple stakeholders—unconnected, unsynchronized—say the same thing, you have a pattern.
The trick? Create rituals where these voices are heard before decisions, not after failures.
Source 3: Operational Metrics — The Hidden System Truth
A marketplace client once told me their buyer conversion was “stable.” Their funnel dashboard showed 18% from browse to purchase.
But their backend logs told another story: 43% of search requests timed out or returned errors. Median latency: 2.8 seconds. And their A/B test framework wasn’t logging exposure correctly—20% of users in the “control” group were actually seeing the new UI.
We fixed the logging first. Overnight, the “18%” conversion dropped to 11%. The real number.
“Why didn’t anyone catch this?” I asked.
“Engineers were focused on uptime. Product was盯着 the frontend metrics. No one connected the dots.”
Operational metrics—latency, error rates, logging accuracy, infra load—are the nervous system of your product. But they’re often siloed in engineering, treated as “hygiene,” not strategy.
Yet they’re the most tamper-proof data source. You can fake a survey. You can misreport a KPI. But if your API is dropping requests, no amount of PowerPoint will fix it.
Operational truth doesn’t care about your org chart.
Why You Can’t Hold All Three — And Why That’s Good
Here’s the uncomfortable truth: no individual or team can credibly control all three sources at once. And that’s by design.
- User behavior is decentralized. You can’t dictate how millions act.
- Stakeholder narrative is politically risky. Speaking up can cost jobs.
- Operational metrics require technical access and honesty—rare in deadline-driven cultures.
When someone does claim alignment across all three, alarm bells should ring.
I once reviewed a product lead’s promotion packet. Her project had:
- 30% increase in engagement (behavior)
- Rave reviews from peers (narrative)
- Clean system metrics (operations)
Perfect? Too perfect.
We dug deeper. Found that the engagement metric was redefined after launch—changed from “weekly active users” to “sessions with video play,” which inflated results. Peer reviews came mostly from her close collaborators. And the system logs? She wasn’t the one monitoring them—her engineering partner was, and he was up for promotion too.
They were gaming the system—cooperatively, quietly. Not maliciously, but to survive the promotion gauntlet.
We delayed both promotions. Re-ran the analysis with neutral parties. Real impact: flat engagement, mixed feedback, backend instability during peak load.
The triple-source framework exposed the fraud because the sources should contradict—gently. Perfect harmony is a red flag.
Tension between sources isn’t noise. It’s the signal.
How to Run a Triple-Source Validation Session
You don’t need new tools. You need new rituals.
Here’s how we run TSV at my current venture:
Step 1: Assign Source Owners (No Overlap)
Before any major decision—launch, kill, pivot—we assign one person per source:
- Behavior Owner: Typically growth or data science. Owns clickstream, funnel analysis, cohort retention.
- Narrative Owner: Rotating role—support lead, sales engineer, QA tester. Brings unfiltered frontline feedback.
- Operations Owner: Backend engineer or SRE. Reports on system performance, error rates, data integrity.
No person holds two hats. Period.
Step 2: Pre-Meeting Submission (Blind)
48 hours before the meeting, each owner submits a one-pager:
- What does your data say?
- What’s the biggest red flag?
- One recommendation.
No group edits. No alignment. Raw input only.
Step 3: The 90-Minute Validation Meeting
We start with silence. Everyone reads the submissions.
Then, in order: Behavior → Narrative → Operations.
Each owner presents. No interruptions.
Then, the hard part: we look for dissonance.
- Did behavior show growth but operations show instability?
- Did narrative flag usability issues but behavior show high retention?
- Did all three agree? If so, we question the methods.
Last quarter, Behavior reported 15% lift in checkout completions. Narrative said users were “frustrated by the new layout.” Operations revealed the tracking pixel fired on page load, not confirmation—meaning we were counting visits as purchases.
The discrepancy caught a $2M error in revenue reporting.
We call this the Dissonance Review. It’s not about consensus. It’s about conflict as a diagnostic tool.
Step 4: Action with Accountability
We end with one decision:
- Proceed, pause, pivot, or kill.
And one owner accountable for each data stream’s health going forward.
No vague “we’ll monitor.” Clear ownership, clear KPIs.
Three Counter-Intuitive Truths From the Trenches
After running 70+ TSV sessions across six companies, three insights keep surfacing—ones that defy conventional wisdom.
1. The Most Accurate Data Is Often the Ugliest
Leaders want clean dashboards. But clean data is often sanitized. The most reliable signals come from messy sources: support ticket verbatims, error logs with stack traces, raw session recordings with swear words.
At a health tech company, we discovered a critical UX flaw not from analytics, but from a single user muttering “Why won’t this f***ing button work?” in a recording. The button had a 0.3-second delay due to a race condition. Metrics showed 99% success. Behavior? Rage.
Truth often hides in the noise you’re trained to filter out.
2. “Vocal Minorities” Are Usually Silent Majorities
We dismiss angry users as outliers. But in one case, a single support agent flagged that “older users keep asking where the back button went.” We assumed it was one confused person.
But Narrative Owner dug into call logs. Found 217 similar complaints in two weeks—mostly from users aged 65+. They weren’t in our core segment, but they represented 38% of paid subscribers.
We added a persistent back button. Retention in that cohort jumped 52%. Lifetime value increased by $180 per user.
The “minority” was a buried revenue stream.
3. You Need Fewer Metrics, Not More
Teams drown in KPIs. But TSV forces you to pick one metric per source.
- Behavior: Pick one north star (e.g., 7-day retention).
- Narrative: One recurring theme (e.g., “users say checkout is confusing”).
- Operations: One system health indicator (e.g., API error rate < 0.5%).
When we reduced our dashboard from 47 metrics to three per initiative, decision speed increased 3x. Clarity went from “meh” to “obvious.”
Focus beats volume. Triangulation beats aggregation.
FAQ
Q: What if sources contradict but we need to move fast?
Move, but label it a bet. Say: “We’re proceeding despite narrative concerns because behavior and ops are strong.” Or: “Ops is shaky, but the behavior signal is too big to ignore.” Just make the tension visible.
Q: How do you prevent retaliation against narrative owners who speak up?
Anonymize input early. Rotate roles. Protect with leadership air cover. At one company, we had the CEO open every TSV with: “If you don’t bring bad news, you’re not doing your job.”
Q: Can TSV work for small teams?
Better. With fewer people, you must wear multiple hats—but not during validation. Founders can play all roles, but not in the same meeting. Separate the acts: collect data as a team, validate as independent sources.
Q: What about qualitative research—where does that fit?
It’s part of narrative. But distinguish between facilitated research (which can be biased) and organic feedback (calls, tickets, chats). The latter is higher-fidelity.
Q: How often should we run TSV?
Before any major decision. Quarterly for ongoing initiatives. After any system outage or user backlash.
Triple-Source Validation isn’t a process. It’s a mindset.
It’s the recognition that truth is messy, distributed, and often inconvenient. That the data you like is the one you should distrust most.
The best products aren’t built on consensus. They’re forged in the friction between what users do, what people say, and what the system knows.
Demand all three. Celebrate their tension. Let the contradictions guide you.
Because in the end, the data you can’t control is the only data you can trust.