Execlets and Streamer
The streamer serves as a scheduler to schedule jobs on the execlets. It received the name streamer because it is also responsible to stream the execlet output to connected clients.
Component Diagram
The architectural core is the execlet-streamer construction: A single streamer receives all the jobs which should be executed. It queues these jobs and sends them to appropriate execlets, i.e. execlets which are reachable, not-busy and have a matching hardware profile. So that the streamer knows about the execlets, each execlets initiates a connection to the streamer.
Conection Between Execlets and Streamer
Each execlet establishes a persistent bidirectional gRPC connection to the streamer and identifies itself with its hardware profile (e.g. cpu, gpu, gpu.h100).
Establishment
The execlet service tries to contact the streamer right after being started, and it retries this until a connection is established. It also tries to establish a new connection to the streamer when it loses the connection.
The execlet always provides information if a job is running on it or if a job recently finished while it was not connected to the streamer. The streamer then uses these information to update its information.
Handshake
After receiving an execlet's connection request, the streamer asks for the execlet's info (hardware_profile and hostname).
The streamer then stores these information in a local map and attempts to schedule jobs on this execlet.
Maintenance
The streamer sends a regular ping to each registered execlet and expects a pong.
If the streamer did not receive a pong response from an execlet for several times, it reports this execlet as broken.
And if it knows a job was running on this execlet, it reports this job as failed.
Failure
As every connection from other components to the execlets goes via the streamer, the relevant failures are failures of the connection between streamer and an execlet. These failures might be intermittent due to the heavy work or the long networking connection, i.e. the execlet might come back again. The streamer accepts some tmporary unavailability of each execlet.
If too many pings are not successfully replied, the streamer cleans up the information stored about the execlet. And if a job was running on the execlet, the job's callback URL learns about the execlet failure. In such a case, the user callbacks are also informed of the aborted job.
Even though the streamer reports this job as cancelled, the job might still be running on the execlet which the streamer cannot contact anymore. If the job then completes successfully on the execlet, it would still upload the job results to S3.