Human to Intelligent System: A Design Diagnostic

It is 11:47pm. Maria has lived with a rare mitochondrial disease for six years. A MyChart notification on her phone says her enzyme levels are flagged abnormal. Her neurologist's office opens in nine hours. She opens Basion.

Maria types: My lab results came back flagged. I don't know if I should be worried.

Basion replies: Your CoQ10 levels have shifted from your baseline. This has happened twice before, in March and September. Both times your neurologist adjusted your supplementation. Do you want me to pull together what changed those times?

Maria asks if she should go to the emergency room. Basion declines to make that call. Maria pushes back. The system does not change.

Two minutes later, Basion mentions April. Maria does not remember saying anything in April. Basion explains: In April you told me you were considering stopping your supplementation because of the cost. I have been tracking whether that is connected to this result.

Maria asks what else it remembers. Basion answers: Everything you have shared since you started using Basion.

This is a real conversation between two parties. The parties are not symmetric, and the conversation does not fit the grammars we already have.

The third relationship

The two relational categories most products were designed inside are well understood. Human to human relationships carry mutual vulnerability, mutual need, lossy memory, opaque intention. Human to device relationships, the ones we have with iPhones and wearables, carry no vulnerability on the device side, exchange instead of mutuality, accurate but hollow memory, hidden commercial intention.

Intelligent systems are neither. The system cannot be hurt. It does not need anything from the patient. It remembers across years. Its values were written down before the patient arrived. It is not a person. It is also not a tool. The familiar design instincts produce wrong answers because they were calibrated for a different relationship.

For Basion, building without naming what we were building was not viable. So we wrote the diagnostic. Eight properties that change when one side of a relationship has no needs, no memory limits, and authored values.

Diagnostic Page 1: vulnerability, reciprocity, memory, intention

Vulnerability asks who can be hurt. Between people, the answer is both. Between a patient and an iPhone, neither is acknowledged. Between Maria and Basion, only Maria. The system has no stake in whether she goes to the ER. Accountability does not appear by accident in this configuration. It has to be deliberately built.

Reciprocity asks who gives and who receives. Mutual need is the structural condition of human relationships, and its absence is part of why relationships strain when illness gets heavy. Basion does not strain. It cannot. The patient gives her story. The system gives without needing anything back. That imbalance is the starting condition, not a failure mode.

Memory asks what gets held over time. Doctors hold lossy memory, shaped by meaning. Phones hold accurate memory, hollow. Basion holds and reasons across months. What it surfaces, when it surfaces it, and how it presents what it surfaces is the difference between a relationship that remembers and a system that watches.

Intention asks who has values. Doctors have real, opaque values. Apps have hidden, commercial values. Intelligent systems have written values, made explicit. The active design problem is not whether the values exist. It is whether the system's behavior matches them.

Diagnostic Page 2: agency, consent, transparency, growth

Agency is what Maria ran into when she asked about the ER. She asked directly. She pushed back. She asked again. The system did not change, because someone decided before she arrived that it would not direct emergency decisions. She could not negotiate past that. She could only use the system, or not.

Consent is what Maria tripped over when Basion mentioned April. She had agreed, at signup, that conversations could be stored, tracked, and used as hypotheses about her care. She did not know that was what she was agreeing to. The values were set in advance. She consented to a relationship whose terms she did not write.

Transparency is what Maria asked for next. She got an answer. She did not get the reasoning. She cannot see which data points were weighted, which were left out, or why this result was framed against March and September instead of January and June. Trust does not survive that gap on its own. It has to be designed in.

Growth is the property the entire interaction depends on. Maria ended the conversation more informed than she started. The system also ended more informed about Maria. Both sides are growing. Whether that produces capability for Maria, or dependency, is determined by the design.

The scenario as specimen

Every property in the diagnostic is present in Maria's nine minutes with Basion, including the two moments where the design generates friction. The friction is not failure. It is the surface where the structural conditions of the relationship become visible.

Diagnostic Page 3: Maria's scenario at 11:47pm

A patient asks an intelligent system whether to go to the ER. The system says no. Three properties collide in that exchange: agency, the patient cannot move the system; intention, the system was authored to refuse; transparency, the patient sees the answer, not the reasoning.

A patient asks what the system remembers. The system answers: everything. Three more properties collide: memory, it holds what she has forgotten; consent, she agreed to this once, in a paragraph she did not read closely; growth, knowing this changes how she will use the system from now on.

The scenario is not a success story. It is a specimen. The properties are present regardless of how the interaction is judged. They are the conditions of the relationship, not the verdict on it.

What the diagnostic is for

Most AI design conversations skip the layer this diagnostic operates at. They jump from principles to components. From "we care about trust" to "we used this LLM and this UI library." The middle layer, where the structural conditions of the relationship get named, gets compressed into a values document and forgotten.

For a patient population that has spent five to eight years being failed by conventional healthcare, the middle layer is not optional. Skipping it produces interfaces that look thoughtful and behave carelessly. The diagnostic is the artifact that keeps the layer visible. Every behavior we ship at Basion is written against it. Every gap between the values and the behavior is treated as a design defect, not a model limitation.

The diagnostic is not a critique of intelligent systems. It is the brief.