Higher-order representation in AI

Higher-order representations are those that are about other representations. Humans and some other animals form higher-order mental representations concerning representations in our own minds, through the operation of processes of metacognition and introspection. These have been linked with a wide range of mental capacities and attributes, including consciousness. Recent research on large language models (LLMs) has explored their knowledge of their own ‘minds’, sometimes suggesting that these models represent their own inner representational states in activations. This paper surveys this research, arguing that there is some evidence of higher-order representation in LLMs but that substantial empirical and philosophical questions remain unresolved.