Four node cluster started with zero nodes, set Desired number to one to get autoscaling going, rans jobs, nodes scaled up to four (max). Queue is empty now but nodes will not scale down.
autoscaler.log (shown below) indicates nodes are found empty. Log entries shows queues disabled, presumably prepping for termination, but then are immediately turned back on. Nodes never get terminated.
Any idea why this is happening?
Dec 30 01:10:03 [autoscaler:aws-scale-out] Retrieving job state data for scheduler slurm in queues for t2.small
Dec 30 01:10:03 [autoscaler:aws-scale-out] Autoscaling group t2.small (t2TestCluster-FlightComputeGroup-1BEJ4MH578B83) has demand for 0 nodes
Dec 30 01:10:03 [autoscaler] Performing scale-in check
Dec 30 01:10:03 [autoscaler:scale-in] Found empty nodes: flight-017 flight-018 flight-209 flight-229
Dec 30 01:10:03 [autoscaler:scale-in] Disabling queues on flight-017 (in group: t2TestCluster-FlightComputeGroup-1BEJ4MH578B83)
Dec 30 01:10:03 [autoscaler:scale-in] Disabling queues on flight-018 (in group: t2TestCluster-FlightComputeGroup-1BEJ4MH578B83)
Dec 30 01:10:03 [autoscaler:scale-in] Disabling queues on flight-209 (in group: t2TestCluster-FlightComputeGroup-1BEJ4MH578B83)
Dec 30 01:10:03 [autoscaler:scale-in] Disabling queues on flight-229 (in group: t2TestCluster-FlightComputeGroup-1BEJ4MH578B83)
Dec 30 01:10:03 [autoscaler:scale-in] Enabling queues on flight-017 (not exhausted)
Dec 30 01:10:03 [autoscaler:scale-in] Enabling queues on flight-018 (not exhausted)
Dec 30 01:10:03 [autoscaler:scale-in] Enabling queues on flight-209 (not exhausted)
Dec 30 01:10:03 [autoscaler:scale-in] Enabling queues on flight-229 (not exhausted)