Search for a command to run...
Abstract Despite significant progress made in drilling automation over the last decade, autonomous drilling of a hole section remains at an early stage. Autonomous drilling cannot happen without near 100% situational awareness (SA). Going by the adage "what cannot be measured cannot be improved", the first step in the process towards autonomous drilling is measuring the SA capability of drilling advisory models. The objective of this paper is to demonstrate SA measurement with example use cases. We utilize the Situation Present Assessment Method (SPAM) to evaluate the SA capability of a drilling advisory model on historical datasets from land operations in the Middle East. SPAM measures SA by querying the model outputs in real-time during the historical playback to assess awareness under operational conditions. Responses and response times are used to evaluate the three levels of SA (perception, comprehension and projection). We also apply a Large Language Model (LLM) to evaluate the SA of the model in a dynamic, interactive scenario, such as a simulated conversation with the end user. The questions for testing the model were carefully crafted to test all 3 levels of SA. An example of a level 1 SA question that aims to test perception is "What is the current status of the sensors used?" An example of a level 2 SA question which tests comprehension is "Why is the pipe stuck? A level 3 SA question on the other hand tests the projection capability. They are usually of the type "What should be done next"? On the datasets used in this study, the questions were posed roughly after every 30 minutes of historical playback in drawing statistics and test results. The use of LLM allowed for the questions to be refined. After each such query a subject matter expert evaluated the answers provided for all the questions. The model used in this test achieved varying SA accuracies for the three levels. As expected, when additional sensor data was made available in a particular test run, the SA accuracy increased. Also, level 3 SA questions were better answered when offset well data was available. This paper outlines an approach to measure the SA capability of event detection (drilling advisory) models, which is key to making progress towards autonomous drilling. The approach is integrated with an LLM to facilitate testing the SA capability in a conversational manner. This is the first paper that presents an approach to quantifying the SA capability of a drilling advisory model.