Skip to content

Hydra -- Autoscaler

For spinning up computing instances with different providers, new instances go through the following statuses, which involves going through our own provisioning. As the newly spun-up instances live at several sites, we put some emphasis on having a secure connection between our servers.

Status

Inside Hydra, instances created with the actual compute provider through a backend (e.g. Prime Intellect) have different statuses:

  • pending: Instance is created but not yet set up by the provider
  • running: Provider's instance setup is ready and we can access the instance
  • provisioning: Our provisioning is running on the instance (usually through Ansible)
  • ready: Instance is ready for customer usage
  • terminated: Instance is terminated, also on provider's side
  • error: Could be reported either by the provider (very unlikely) or Hydra itself, having issues with this instance, most likely a failed provisioning attempt

Our Provisioning of New Machines

After a provider is declared as running by the provider, Hydra rans an own provisioning flow in form of an Ansible playbook. The used playbook varies based on whether Hydra needs to prepare an execlet or a bare VM.

When a playbook execution does not succeed, we attempt rerunning it based on on our config. When a run succeeded, we persist these information to Hydra's own database so that these information survive Hydra service restarts. This means that restarting Hydra will not automatically rerun the Ansible provisioning if it already succeeded.

Instance Managers

Each provider gets one or more pools, usually one for bare VMs and one for execlets. As this means an InstanceManagerBackend can have several pools, each instance is assigned to a pool based on their name prefix.

Example pools

Examples pools are verda-vm-pool, prime-intellect-vm-pool and prime-intellect-execlet-pool.

On a slightly higher level than the backends, we have InstManagers, which are able to direct actions to specific pools. For each pool, we create one SinglePoolInstManager, which inherently is able tell its backend to perform actions for its pool. And for Hydra to interact with a set of SinglePoolInstManagers, we have the MultiPoolInstManager, which also implements the InstManager interface and routes actions for specific pools to the relevant SinglePoolInstManager.

uml diagram

Providers

Online services supplying GPU compute capacity are integrated as providers. Each provider interacts outside of Hydra through a custom client. To abstract the client's functionalities, each provider gets an InstanceManagerBackend and Hydra knows every provider inform of this backend.

uml diagram

Tip

Create an unnamed variable for letting the compiler check that a struct implements an interface. 

var _ as.InstanceManagerBackend = (*InstanceManagerBackend)(nil)

Supported Providers

The following already integrated providers have each some relevant unique details. Hydra's interaction with these providers happens via the account <provider>@lyceum.technology. The relevant password can be found in the SOPS inventory files, but you can/should also use an own account to interact with the compute instances (ask to be added to each team).

Whenever we want to spin up an instance with any of our providers, Hydra looks through the different providers and starts the cheapest option.

Atlas

Atlas is our internal provider and its instances will always be preferred because of an artifially cheap price.

Mithrill

  • Quite modern provider which is not very stable, offers only US compute instances in spot and reserved modes (no on-demand)
  • One project with own SSH key per env. However, API secret is able to access all projects. Acceptable because we also specify to which project to connect to.
  • Interaction through REST API

Currently disabled

The providers code is already present in our production code but its usage is disabled, because it needs more testing and adjustment to the providers model. See the related Linear ticket ⧉ to see more details about the paused integration.

Mithril ⧉ is used as a provider for spot instances only (no on-demand). The provider itself functions through a Vickrey auction, i.e. a sealed 2nd price auction. This means that we have to bid to then potentially receive an instance; bid and instance would therefore have to be considered together to handle the instance further. We currently only create bids for single instances, and whenever an instance is preempted, we automatically cancel the bid because we do not want to receive a new instance again (our current sales model demands this).

We determine the price we want to bid by first querying the spot instance prices which we would have to pay with other providers. Afterwards, we bid a cent less with Mithril. However, we currently only place a Mithril bid if we think we can immediately win the auction to get an instance. To decide this, we have a look at the highest price which recently won a bid and only place a bid if our price is a certain percentage higher.

The current problem seems to be that our availability prediction is not good enough and we try to place a bid which then does not win, meaning lots of customer wait on our side to then tell the customer that it did not work.

Prime Intellect

  • app.primeintellect.ai ⧉, quite flaky provider wrapping other providers (also Verda)
  • Offers access to providers at many different global location (also a lot in the US)
  • Uses same way of accessing for every env (also same SSH key)
  • Interaction happens through REST API ⧉ (might changes fast as company seems to be young)

UpCloud

  • upcloud.com ⧉, European provider offering fixed GPU server plans (H100, B200, L40S, L4)
  • Servers are created from fixed plans (e.g. GPU-12xCPU-240GB-1xH100) instead of arbitrary instance types; the api_names in supported_gpus.yml match the plans' gpu_model attribute (e.g. NVIDIA H100 80GB HBM3, NVIDIA B200)
  • Spot capacity exists as separate GPU-SPOT- plans and the backend supports it, but we currently only enable on-demand in supported_gpus.yml
  • Authentication via an API subaccount (username + password); the OS disk is cloned from UpCloud's AI/ML-ready GPU Ubuntu template which ships with NVIDIA drivers preinstalled
  • Interaction with provider happens through UpCloud's own Go SDK ⧉

Verda

  • verda.com ⧉, very reliable provider in Finland offering currently EU-only hardware
  • Different cloud credentials and SSH keys used for the different environments
  • Each environment could easily modify instances in other environments
  • Interaction with provider happens through Verda's own Go SDK ⧉

Supported GPUs

To decouple the configuration and the actual code implementation of Hydra, we use an inventory file called supported_gpus.yml. In there, we define as a type how we call and address this GPU hardware profile internally and how many GPUs of this type we allow per instance.

Relevant for the providers is how they address these hardware profiles on their end. And this is what is listed under api_names of each provider, how a provider calls the hardware profile which we call as mentioned under type.

Example

If we want to create an l40s on our end, we would have to tell Prime Intellect to create an L40S_48GB or Atlas to create an NVIDIA L40S.

See the supported_gpus.yml reference for the full schema of this file.