Autoscaling not retiring nodes

#1

Four node cluster started with zero nodes, set Desired number to one to get autoscaling going, rans jobs, nodes scaled up to four (max). Queue is empty now but nodes will not scale down.

autoscaler.log (shown below) indicates nodes are found empty. Log entries shows queues disabled, presumably prepping for termination, but then are immediately turned back on. Nodes never get terminated.

Any idea why this is happening?

Dec 30 01:10:03 [autoscaler:aws-scale-out] Retrieving job state data for scheduler slurm in queues for t2.small
Dec 30 01:10:03 [autoscaler:aws-scale-out] Autoscaling group t2.small (t2TestCluster-FlightComputeGroup-1BEJ4MH578B83) has demand for 0 nodes
Dec 30 01:10:03 [autoscaler] Performing scale-in check
Dec 30 01:10:03 [autoscaler:scale-in] Found empty nodes: flight-017 flight-018 flight-209 flight-229
Dec 30 01:10:03 [autoscaler:scale-in] Disabling queues on flight-017 (in group: t2TestCluster-FlightComputeGroup-1BEJ4MH578B83)
Dec 30 01:10:03 [autoscaler:scale-in] Disabling queues on flight-018 (in group: t2TestCluster-FlightComputeGroup-1BEJ4MH578B83)
Dec 30 01:10:03 [autoscaler:scale-in] Disabling queues on flight-209 (in group: t2TestCluster-FlightComputeGroup-1BEJ4MH578B83)
Dec 30 01:10:03 [autoscaler:scale-in] Disabling queues on flight-229 (in group: t2TestCluster-FlightComputeGroup-1BEJ4MH578B83)
Dec 30 01:10:03 [autoscaler:scale-in] Enabling queues on flight-017 (not exhausted)
Dec 30 01:10:03 [autoscaler:scale-in] Enabling queues on flight-018 (not exhausted)
Dec 30 01:10:03 [autoscaler:scale-in] Enabling queues on flight-209 (not exhausted)
Dec 30 01:10:03 [autoscaler:scale-in] Enabling queues on flight-229 (not exhausted)

#2

The autoscaling cycle runs every 5 minutes though nodes are only scaled down once they are “exhausted” (reaching the end of the current billing hour).

The process is roughly:

  • Detect which nodes are idle (i.e. have no jobs running on them)
  • Temporarily disable idle nodes in the scheduler so further jobs can’t get scheduled during the shutdown process
  • Determine whether the node is “exhausted” by which we mean that it is within the last 7 minutes of its current billing hour. This number is chosen so a node has at most 7 minutes run time left and at least 2 minutes, which gives it a chance to be terminated before slipping into a further billing hour.
  • If it is exhausted, shut it down, otherwise leave it alone until the next autoscaling cycle in 5 minutes.