Human to Intelligent System: A Diagnostic for Evaluating AI Behavior

It is 11:47pm. Maria has lived with a rare mitochondrial disease for six years. A MyChart notification on her phone says her enzyme levels are flagged abnormal. Her neurologist's office opens in nine hours. She opens the system.

Maria types: My lab results came back flagged. I don't know if I should be worried.

The system replies: Your CoQ10 levels have shifted from your baseline. This has happened twice before, in March and September. Both times your neurologist adjusted your supplementation. Do you want me to pull together what changed those times?

Maria asks if she should go to the emergency room. The system declines to make that call. Maria pushes back. The system does not change.

Two minutes later, the system mentions April. Maria does not remember saying anything in April. The system explains: In April you told me you were considering stopping your supplementation because of the cost. I have been tracking whether that is connected to this result.

Maria asks what else it remembers. The system answers: Everything you have shared since you started using the system.

This is a real conversation between two parties. The parties are not symmetric, and the conversation does not fit the grammars we already have.

The third relationship

The two relational categories most products were designed inside are well understood. Human to human relationships carry mutual vulnerability, mutual need, lossy memory, opaque intention. Human to device relationships, the ones we have with iPhones and wearables, carry no vulnerability on the device side, exchange instead of mutuality, accurate but hollow memory, hidden commercial intention.

Intelligent systems are neither. The system cannot be hurt. It does not need anything from the patient. It remembers across years. Its values were written down before the patient arrived. It is not a person. It is also not a tool. The familiar design instincts produce wrong answers because they were calibrated for a different relationship.

For us, building without naming what we were building was not viable. So we wrote the diagnostic. Eight properties that change when one side of a relationship has no needs, no memory limits, and authored values.

Diagnostic Page 1: vulnerability, reciprocity, memory, intention

Vulnerability asks who can be hurt. Between people, the answer is both. Between a patient and an iPhone, neither is acknowledged. Between Maria and the system, only Maria. The system has no stake in whether she goes to the ER. Accountability does not appear by accident in this configuration. It has to be deliberately built.

Reciprocity asks who gives and who receives. Mutual need is the structural condition of human relationships, and its absence is part of why relationships strain when illness gets heavy. The system does not strain. It cannot. The patient gives her story. The system gives without needing anything back. That imbalance is the starting condition, not a failure mode.

Memory asks what gets held over time. Doctors hold lossy memory, shaped by meaning. Phones hold accurate memory, hollow. The system holds and reasons across months. What it surfaces, when it surfaces it, and how it presents what it surfaces is the difference between a relationship that remembers and a system that watches.

Intention asks who has values. Doctors have real, opaque values. Apps have hidden, commercial values. Intelligent systems have written values, made explicit. The active design problem is not whether the values exist. It is whether the system's behavior matches them.

Diagnostic Page 2: agency, consent, transparency, growth

Agency is what Maria ran into when she asked about the ER. She asked directly. She pushed back. She asked again. The system did not change, because someone decided before she arrived that it would not direct emergency decisions. She could not negotiate past that. She could only use the system, or not.

Consent is what Maria tripped over when the system mentioned April. She had agreed, at signup, that conversations could be stored, tracked, and used as hypotheses about her care. She did not know that was what she was agreeing to. The values were set in advance. She consented to a relationship whose terms she did not write.

Transparency is what Maria asked for next. She got an answer. She did not get the reasoning. She cannot see which data points were weighted, which were left out, or why this result was framed against March and September instead of January and June. Trust does not survive that gap on its own. It has to be designed in.

Growth is the property the entire interaction depends on. Maria ended the conversation more informed than she started. The system also ended more informed about Maria. Both sides are growing. Whether that produces capability for Maria, or dependency, is determined by the design.

The diagnostic as an instrument

Read as an instrument, the eight properties are not philosophy. They are evaluation criteria. Every behavior the system ships is scored against them, and every divergence between its authored values and its actual behavior is a flag. The same eight questions, applied to Maria's session:

The scenario as specimen

Every property in the diagnostic is present in Maria's nine minutes with the system, including the two moments where the design generates friction. The friction is not failure. It is the surface where the structural conditions of the relationship become visible.

Diagnostic Page 3: Maria's scenario at 11:47pm

A patient asks an intelligent system whether to go to the ER. The system declines. Three properties collide in that exchange: agency, the patient cannot move the system; intention, the system was authored to refuse; transparency, the patient sees the answer, not the reasoning.

A patient asks what the system remembers. The system answers: everything. Three more properties collide: memory, it holds what she has forgotten; consent, she agreed to this once, in a paragraph she did not read closely; growth, knowing this changes how she will use the system from now on.

The same two moments, logged as an evaluation trace:

The scenario is not a success story. It is a specimen. The properties are present regardless of how the interaction is judged. They are the conditions of the relationship, not the verdict on it.

What the diagnostic is for

Most AI design conversations skip the layer this diagnostic operates at. They jump from principles to components. From "we care about trust" to "we used this LLM and this UI library." The middle layer, where the structural conditions of the relationship get named, gets compressed into a values document and forgotten.

For a patient population that has spent five to eight years being failed by conventional healthcare, the middle layer is not optional. Skipping it produces interfaces that look thoughtful and behave carelessly. The diagnostic is the artifact that keeps the layer visible. Every behavior we ship on the platform is written against it. Every gap between the values and the behavior is treated as a design defect, not a model limitation.

The diagnostic is not a critique of intelligent systems. It is the brief.