If you want to watch a senior clinician’s soul leave their body in real time, show them a pull request where an LLM decided how to round serum potassium.
In a standard SaaS CRUD app, if a junior engineer blindly trusts an AI autocomplete and ships a subtle logic bug, a button turns the wrong shade of blue or an analytics dashboard drops a few events. Nobody dies.
In complex, high-stakes domains like healthcare informatics, the margin for error is measured in milliequivalents per liter. Yet across the industry, large teams with wildly mixed levels of engineering maturity are being handed generative AI tools and told to "move faster." Instead of accelerating alignment, AI has become a high-velocity wedge driven straight between software engineers and Subject Matter Experts (SMEs).
We used to have a communication problem. Now, we have an automated hallucination problem masquerading as requirements gathering.
The Death of the Alignment Meeting
Let’s be honest about how cross-functional healthcare teams used to work. The clinical SME team—doctors, nurses, lab informaticists—would drop a high-level directive or a half-formed suggestion into a ticket or a Slack channel. Often, that direction was ambiguous. Not because clinicians are bad at their jobs, but because clinical reality is a sprawling mess of tacit knowledge, unit conversions, and physiological edge cases that they take for granted.
In the pre-LLM era, an ambiguous requirement forced a reckoning. A senior engineer would read the ticket, realize the edge cases weren't specified, and schedule a series of painful, Necessary Meetings. Junior engineers would sit in those meetings and absorb the domain complexity by osmosis. By the end of the third whiteboard session, everyone was on the same page. We knew why we were building the thing, and what would happen when the data inevitably arrived malformed.
Today? The junior or mid-level engineer doesn't schedule the meeting. Why talk to an intimidating MD when you have a sycophantic chatbot right there in your IDE?
When SMEs give vague directions now, engineers don't ask clarifying questions. They copy-paste the messy Slack thread directly into an LLM prompt, ask it to "implement the fix based on this discussion," and hit commit. They treat the LLM not just as a code generator, but as a surrogate Subject Matter Expert.
And the results are stunning.
Case Study: The Slack-Driven Lab Precision Disaster
Recently, a clinical team flagged a UI issue: lab results needed standardized decimal precision for display. In a Slack thread, the clinicians tossed around a few examples—“Sodium should have zero decimals, Creatinine needs two, Potassium needs one…”—and asked engineering to define how many decimal points each lab should display.
Instead of sitting down with the clinical informaticist to formalize a specification, the assigned engineer grabbed the raw Slack conversation, fed it into an AI coding assistant, and prompted it to build the display formatter.
The AI dutifully spat out a clean, syntactically pristine lookup table and a formatting function. The tests (also written by the AI) passed. LGTM. Ship it.
Except the AI—and the engineer who acted as its blind notary—completely missed two catastrophic domain realities that a five-minute conversation with an SME would have caught immediately:
1. The Unit Blindness Problem
The AI mapped precision strictly to the lab test name (e.g., "Creatinine": 2), blissfully unaware that lab values in healthcare don't exist in a vacuum—they exist in units.
Depending on the source lab, patient population, or regional standard, the same analyte can be reported in conventional units or SI units. Serum creatinine in mg/dL (where normal is roughly 0.70 to 1.30) requires two decimal places. If that same lab comes across an HL7 feed in µmol/L (where normal is 60 to 110), displaying two decimal places is visual noise—or worse, if the direction goes the other way, truncating an SI unit that requires micro-precision turns a critical shift into a flat line. By hardcoding precision to the string name of the lab rather than the (analyte, unit) tuple, the code was fundamentally broken on arrival.
2. The 0-Decimal Fallback Abyss
What happens when a lab result arrives that wasn't mentioned in that casual 15-message Slack thread? In medicine, there are thousands of LOINC codes. A Slack thread might cover the top ten.
Because no fallback behavior was discussed in the Slack thread, the AI had to invent a default. And what did the statistical token predictor choose as the default precision for an undefined lab?
Zero decimal places. toFixed(0). Integer rounding.
Let that sink in. Any lab test not explicitly named in a casual Slack conversation was silently stripped of its decimal points:
- Hemoglobin A1c: A completely normal A1c of
5.5%rounds up to6%—instantly pushing a healthy patient across the clinical threshold into prediabetes (5.7%to6.4%) right on the screen. - Venous Blood pH: A life-threatening acidemic pH of
7.15and a normal pH of7.45both round to7. - Narrow Therapeutic Index Drugs: A Tacrolimus trough level or Digoxin level of
0.8 ng/mL(therapeutic) vs1.4 ng/mL(approaching toxicity) rounds to1. - TSH, INR, Troponin: If it wasn't in the Slack paste, it got the guillotine.
In software, defaulting an unmapped integer to 0 is a lazy habit. In clinical informatics, silently rounding an unmapped floating-point lab value to an integer manufactures false diagnoses out of thin air and is an adverse event waiting for a plaintiff's attorney.
The Silent March to Production: Whack-a-Mole Engineering
Here is the truly terrifying part of this story: nobody catches it.
In a healthy engineering culture, this architectural flaw would get flagged during design review. In an AI-accelerated team with mixed experience levels, the code looks clean, the AI-generated unit tests pass (because the AI wrote tests to validate its own flawed assumptions), and the PR sails through. SMEs aren't reading TypeScript repositories—they assume engineering built a safe system.
So how does a bug like this get found before production? Usually, it doesn't.
At best, a QA tester gets lucky. Maybe they are running a regression test and notice that their familiar test patient—whose A1c has been 5.5% for three years—suddenly displays as 6% on the dashboard. It doesn't feel right, so QA files a bug ticket: "A1c is rounding to 6 instead of 5.5."
And what happens next? The ticket gets routed right back to the same engineer who shipped the original disaster.
Does the engineer pause, question the underlying fallback architecture, or consult an SME? Of course not. They paste the QA ticket into their IDE chat window. The AI dutifully fixes the symptom by adding "A1c": 1 to the hardcoded dictionary. The ticket is marked resolved. QA verifies that 5.5% is back. Everyone pats themselves on the back and calls it a day.
Meanwhile, the toFixed(0) time bomb is still sitting right there at the bottom of the function, waiting for the other 3,990 lab tests that QA didn't happen to eyeball.
The SME team is completely blind to the rot until the code is already live in production and blows up into a five-alarm clinical fire. By the time a physician calls support asking why a patient's INR or Digoxin level is showing up as a whole number, the trust between engineering and clinical leadership isn't just strained—it's vaporized.
In a team with mixed experience levels, senior engineers and SMEs used to share a trust contract: If I don't understand your clinical requirement, I will ask you before I write the logic. AI has broken that contract. Less experienced engineers now possess a tool that gives them the illusion of domain competence, allowing them to patch symptoms in the dark until production catches fire.
You Cannot Prompt Your Way Out of Domain Complexity
Healthcare, finance, avionics—these aren't "solved problems" where requirements are neat JSON schemas waiting to be transpiled into code. They are messy, high-entropy environments where the most important part of software engineering is identifying what wasn't said.
When a clinician says, "Let's define decimal places for each lab," a real engineer hears ten unasked questions: What is our source of truth? Does unit of measure change the precision? How do we handle trailing zeros? What is the safe fallback if a lab isn't in the dictionary—display raw value, or block rendering?
An LLM doesn't hear unasked questions. It just predicts the next token that closes the ticket so the engineer can move the Jira card to Done.
Until engineering leadership stops measuring velocity by how fast PRs are generated and starts holding engineers accountable for actually talking to the people who understand the domain, the gap between Engineering and SMEs will continue to widen. And somewhere out there, a healthy patient with a 5.5% A1c is getting diagnosed with prediabetes by Math.round().