Transformers figure out the deep structures of language, well above and below the level of anything people can understand about their own language. That is exactly what is so troubling. What will we find out about how we mean things? I remember a fact that I learned when I was forced to study Old English for my Ph.D.: in English, the terms for food eaten at the table derive from French—beef, mutton—while the terms for animals in the field derive from Anglo-Saxon—cow, sheep. That difference registers ethnicity and class: the Norman conquerors ate what the Saxon peons tended. So every time you use those most basic words—cow, beef—you express a fundamental caste structure that differentiates consumer from worker. Progressive elements in the United States have made extensive attempts to remove gender duality from pronouns. But it’s worth noting that, in French or in Spanish, all nouns are gendered. A desk, in French, is masculine, and a chair is feminine. The sky itself is gendered: the sun is male, the moon female. Ultimately, what we can fix in language is parochial. Caste and gender are baked into every word. Eloquence is always a form of dominance. Government is currently offering no solutions. Sam Altman, the C.E.O. of OpenAI, which created the deep-learning network GPT-3, has been very open about his pursuit of any kind of governance whatsoever. In Washington, he has found, discussing the long-term consequences of artificial intelligence leads to “a real eyes-glazed-over look.” The average age of a U.S. senator is sixty-three. They are missing in action.

Let’s imagine an A.I. engineer who wants to create a chatbot that aligns with human values. Where is she supposed to go to determine a reliable metric of “human values”? Humanities departments? Critical theory? Academic institutions change their values systems all the time, particularly around the use of language. Arguably, one of the most consistent, historically reliable, widely accepted system of ethics in existence belongs to the Catholic Church. You want to base a responsible A.I. on that? No doubt, in practice, the development of the ethics of natural language processing will be stumblebum. There will be the piecemeal work of the technologists, elaborate legal releases indemnifying the creators, P.R. responses to media outrage—and, of course, more chatbot fails.