IceEdge: Thermal-Aware Machine Learning Inference Serving for Emerging Edge Applications

Smart city applications require machine learning (ML) capabilities at the edge, for which they require edge devices equipped with specialized processors (e.g., GPUs). However, deploying such high-end edge devices in large numbers is challenging, particularly in regions that face infrastructure and budget constraints (e.g., cities lacking ubiquitous broadband) or extreme weather conditions (e.g., hot temperatures over sustained periods of time). Hence, custom ML frameworks are necessary that can support efficient and safe consolidation of multiple applications with diverse workloads on fewer edge servers. We present IceEdge, an inference serving framework for CPU-GPU based edge computing servers, which isolates colocated applications using existing container mechanisms and provides a deadline- and thermal-aware scheduler for the ML workloads. We implement IceEdge for a Dockerized environment on NVIDIA Jetson Xavier AGX platform. IceEdge outperforms state-of-the-art real-time inference serving frameworks Triton and DeepRT, improving the frame drop rate by up to 15% and reducing the temperature rise by around 12°C compared to Triton inference serving framework.