Transformers shine at interface tasks, struggle with mechanistic understanding.
Transformers excel as interfaces to messy healthcare data because they’re great at sequence modeling like language and longitudinal notes. They summarize records, find context, and support decisions. They work well over EHR data or ICD-10 codes to predict likely codes or find patients for trials. However, for drug discovery, most transformers just do next-token prediction through autoregression. This creates impressive language competence but doesn’t automatically give you conceptual understanding or reliable causal reasoning. Drug development needs mechanism understanding and generalization under uncertainty, and in regulated settings, outputs need guardrails and real-world validation.
— Healthcare’s AI Lesson: Autocomplete Isn’t Understanding · Forbes