ApproxIFER: A Model-Agnostic Approach to Resilient and Robust Prediction Serving Systems
ApproxIFER: A Model-Agnostic Approach to Resilient and Robust Prediction Serving Systems
Due to the surge of cloud-assisted AI services, the problem of designing resilient prediction serving systems that can effectively cope with stragglers and minimize response delays has attracted much interest. The common approach for tackling this problem is replication which assigns the same prediction task to multiple workers. This approach, …