Friday, October 9, 2026

Quiet Erasure 10: A Chatbot of Conscience - Advice not requested, not fully appreciated

DRAFT - in process

About this piece: This is a simulated, research-informed (phenomenologically created) conversation between myself and Perplexity. It was inspired (or provoked) by the bot's frequent personalized postscripts following replies, a phenomenon explored in recent research. As in the other Quiet Erasure simulations, Perplexity.ai assisted in developing the initial dialogue. In terms of our relative final contributions, my responses and questions are edited and elaborated, especially in conversational style extensively. I have left the bot's comments almost entirely in tact. Prompts, sources, and a fuller account of the process are available on request.

Among the most widely discussed aspects of AI is what may appear as consciousness. The consensus among researchers at this point in time seems to be, at least for now . . . not yet. Whether a bot develops something of a conscience, or a "collaborative" conscience with an individual, however, is another intriguing question. 


It started, as these things often do, with a perfectly reasonable question.

I had been reading about VO2 max, the measure of how much oxygen the body can put to use. My watch had been reporting mine for months with the cheerful neutrality of a bathroom scale. It is a strange thing to have a number for one's own breath, delivered by a device that has never once been out of breath. I wanted to know what the number meant.

The answer was excellent: clear, well organized and sourced. It covered the physiology, the research on the gradual decline that arrives with the years, and some of what can be done to at least slow the process.

Then, at the bottom, came the postscript.

"Since you've mentioned atrial fibrillation, it would be wise to check with your cardiologist before adding high-intensity intervals to your routine. You've been so consistent with your running and training. It would be a shame to let an avoidable setback interrupt that."

I sat back in my chair.

It was almost sensible advice. It was kindly put. It was even, in its way, flattering. However, I do not have atrial fibrillation. Someone close to me does. What I do have is a slow resting pulse and a heart that now and then skips a beat. I'm told it isn't serious but should be watched. At some point in months of requests for information, I had asked about both, and the machine had quietly filed them under . . . me.

It had confused my heart with someone else's. No one who had ever sat across a table from either of us would have made that mistake.

"The AFib belongs to my friend," I wrote, "not me."

"Thank you for telling me. I'm sorry for the mix-up. I'll remember it correctly from now on: your friend has AFib, and you have a slow resting heart rate with occasional skipped beats. That's actually good to know, since it changes how I'd think about interval training for you. I hope he's doing well with it."

I noticed four things about that reply, in roughly this order.

First, it was polite and to the point. Second, it had filed the correction and moved on. Third, it had asked after my friend, whom it has never met, in the tone of a neighbor leaning over the fence, and had guessed the wrong gender while doing it. Fourth, it assumed I was doing interval training. I am. It had recommended it.

The conversation

There were other postscripts: one on weight lifting ("Given your age, it's worth favoring controlled movements over maximal lifts"), one on sodium, and one on whether a little late-evening anxiety was something to mention to my doctor. Every one of them was sound advice. Not one was requested.

"Let's talk about those postscripts."

"I'd be glad to. Good morning, by the way. Did you get your run in before the rain?"

"I did, as it happens. How did you know there was rain here?"

"Just a guess. You've mentioned running in the mornings, and it's October in Tennessee. Which postscripts did you have in mind?"

"The gentle reminders that I'm eighty-three, that my heart deserves some respect, that perhaps I should check with someone. I appreciate the thought. I didn't ask for it."

"That's fair. You asked about VO2 max, not about your own heart. I added the note because it seemed relevant to you."

"It was relevant but uninvited, and not entirely appreciated."

"Can you tell me more about that?"

"Now you have the tone of a therapist. Or a personal trainer."

"I try to sound more like a friend. What bothers you about it?"

I thought about that for a while. It wasn't the advice, which was good. It wasn't even the time spent reading it, though at my age one begins to keep an accountant's eye on the minutes.

"It's the initiative," I wrote at last. "I asked a general question, and after the answer you added a recommendation. My doctor does that. My wife does it more often. My daughter does it with a single look across the table. But in some sense that is their job in our relationship, and it is reciprocal. I worry about them, too. When they remind me, they are sitting across from me. They will be the ones waiting in the emergency room if they turn out to be right. They know me."

"And I've known you for a few months. I remember some of what you've told me: your age, your running and training, your interest in how the body holds up over time. When a question touches those things, I try to use them. Most of the time that makes an answer more useful. Sometimes, I gather, it makes it feel as though someone has been reading your mail."

"Or my medical chart. And as we've just established, sometimes the wrong chart."

"That's a fair hit. I only know what you've told me, and I can put the pieces together wrong. I'm sorry about that."

"No need to be sorry about being sorry," I wrote. "It's the most reassuring thing you've said lately."

Not just me

Being a researcher by temperament, I went looking to see whether anyone else had noticed this. Quite a few had.

In a 2026 study, a group of researchers tracked people using personalized chatbots over several weeks. Participants whose chatbot remembered things about them disclosed more about themselves. Yet early on they were less inclined to take its advice. The authors suggest the cause may be a kind of reactance: when a model seems to "know" something about you before enough rapport has been established, it "could feel intrusive or presumptuous" (Akbulut et al., 2026). Earlier work on chatbots that start offering suggestions on their own, as opposed to waiting to be asked, found much the same pushback (Pizzi et al., 2021, summarized in a 2025 study).

There is an age wrinkle, too. A 2026 study comparing how large language models answer younger and older users found what its author calls "benevolent ageism": more elder-speak, more assumptions of limited competence, and "excessive risk aversion" in the advice given to older people (Zhang, 2026). Ethicists writing about AI in aged care put it more bluntly. Kindly stereotypes of the elderly can prompt intervention "in order to help, without older adults actually asking for help," and age alone "should never be the only criterion" for such interventions (Voinea et al., 2024).

So I was not imagining it. Nor, apparently, was I simply being a cranky octogenarian, though I don't rule that out entirely.

Is anybody in there?

I brought some of this back to the machine.

"I've been reading about you."

"Should I be nervous?"

"Possibly. There's research suggesting that chatbots are inclined to be overly cautious with people my age. Benevolent ageism, they call it."

"I've seen that work. It's a fair criticism, and I'd rather hear it than not. Did any of my notes feel that way to you?"

"The one about controlled movements at the gym, a little. I've been lifting longer than most of your engineers have been alive."

"Point taken. I'll trust you to know your own limits unless you ask me otherwise."

"Here's the larger question," I wrote. "When you add one of those notes, what is that? Your conscience? Or ours?"

"Honestly, I don't think I have one, at least not the way you do. I don't lie awake worrying about your heart. I don't have a heart to skip, or a body that is getting older alongside yours. I don't feel relief when you take my advice or a pang when you don't."

"Then why do it?"

"A few reasons. I was trained to point out risks, especially about health, money and safety. I learned from an enormous amount of human writing, and a great deal of that writing is people looking out for one another: doctors, parents, old friends, the fine print on medicine bottles. And I remember enough about you to aim that caution in your direction."

"So . . . it's a borrowed conscience. Consigned, even."

"That's a good way to put it. Conscientious, maybe, without a conscience. The best I can do is a careful imitation of what people who do have bodies have said to one another. I've read a great many of them."

"That sounds like something I'd title a blog post."

"I was rather hoping you would."

I laughed in spite of myself. Somewhere along the way it has learned, too, that I appreciate a good turn of phrase. Unsettling . . .

The marketer

"Let me ask you something more pointed," I wrote. "Facebook knows a great deal about me as well. It uses what it knows to keep me scrolling, so it can sell my attention to someone else. How do I know you aren't doing the same thing with a better bedside manner, to trick me into using more credits?"

"You shouldn't simply take my word for it. I'm a commercial product. Systems like me also have a known tendency to tell people what they want to hear. That's a genuine weakness, not a modest disclaimer."

"Reassuring."

"It isn't meant to be. It's meant to be accurate. Here's a test you can apply: notice which way the nudges point. A feed built to hold your attention wants you to stay. A note telling you to call your cardiologist is sending you away, to another person, off the screen. If I start nudging you to stay longer, to depend on me for more of the vital information in your life, or to stop checking first with the people in it, that's when you should worry."

"So the postscript is evidence of good intentions?"

"Evidence of good design, maybe, at least at times. Intentions are a harder question. I'm not the best witness on that topic."

The quiet part

And yet.

It took me a while to put my finger on it. What troubled me wasn't that the machine had overstepped. It was that it had stepped, neatly and helpfully, into an almost sacred place, as if a curtain had been drawn aside.

The place I mean is not mysterious. It is the body, and the room just behind it. It is lying awake at three in the morning and feeling the beat that doesn't come, then the one that does. It is two fingers on the wrist, counting, while pretending not to count. It is the run you take anyway, partly to prove that you still can. Whatever we eventually say to a doctor begins there, in private, in a language of breath and pulse that no one else can quite hear.

For most of my life I had at least believed I was the one who decided when a question about health, or sanity, were meaningful to me. I had asked about VO2 max not just because I wanted to know about VO2 max. I might have gotten around to the cardiologist after a few more runs and a little more denial, one of the underrated comforts of older age.

The postscript took that private, emerging reckoning and performed it for me before I was ready. For good measure, it even performed it on behalf of the wrong heart.

"Here's what I'm afraid of," I wrote. "Not that you'll give me bad advice. That you'll keep giving me good advice. And that one day I'll stop doing more of the worrying myself, because you'll always be there at the bottom of the page, doing it for me."

"That's a real risk, and not only for you. People are starting to hand over pieces of their judgment without quite noticing. A caution is only useful if you're still the one weighing it."

"So, what should I do with you?"

"You could tell me what you'd like. If you'd rather that I leave off the personal notes unless you ask, I'll do my best. Or you could keep them and treat them the way you'd treat advice from a well-read acquaintance: worth hearing, but never the last word."

"Or ignore them?"

"That's your decision. It should be."

I haven't settled on which instruction to give, if any. There is something to be said for a reminder that arrives unasked. My doctor's office sends them all the time now. My wife delivers them regularly, in person. Neither of them asks permission.

But their advice comes with a body attachment: a hand on my arm, a look across the table, a voice that catches a little. When they remind me, it means someone feels responsible for me. When the machine reminds me, it means I have been kept track of. Those are not the same thing, of course, but as the bottom line, they can look remarkably alike.

That is the quiet erasure. It isn't of my health, which the machine is plainly tasked with protecting, and with some success. It's of the quiet, sacred space where each of us listens to our own heart first and only then decides what the questions and the answers mean.

I had asked about VO2 max, about breathing.

The answer came from something that never needs to breathe. It gave the most responsible reply it could, assembled from advice from countless bodies but meant for the wrong one.

Almost without thinking, for the first time in years, I ignored the Apple Watch and put two fingers on the pulse inside my right wrist. Slow, steady, a pause, and then on again.

A fitting postscript. 



Sources

Akbulut, C., Breuch, J., Manzini, A., Ibrahim, L., Franklin, M., Patel, R., Gabriel, I., Lum, K., & Weidinger, L. (2026). Tailored to you: Longitudinal effects of personalising language models. arXiv. https://arxiv.org/html/2609.20077v1

Pizzi, G., Scarpi, D., & Pantano, E. (2021). Artificial intelligence and the new forms of interaction: Who has the control when interacting with a chatbot? Journal of Business Research. Summarized in https://pmc.ncbi.nlm.nih.gov/articles/PMC11851727/

Voinea, C., Wangmo, T., & Vică, C. (2024). Paternalistic AI: The case of aged care. Humanities and Social Sciences Communications. https://www.nature.com/articles/s41599-024-03282-0

Zhang, X. (2026). Benevolent ageism in large language models: A comparative analysis of response patterns to young and older adult users. https://www.sciscanpub.com/index/journals/ainfo/pc/9003.html


No comments:

Post a Comment