Autoscaling appears not to be working


#1

I have flight running on EC2. I asked for one head node with a minimum of one worker and up to 4 workers to be autoscaled.
I have a bunch of 2 core c4.large instances.

I’ve submitted dozens of jobs that take a couple of minutes each and some are queued for well over 10 minutes. No other nodes are being spun up though. How do I start to debug this please?

Cheers,
Mike


#2

Hi Mike,

You can start by looking at /var/log/clusterware/autoscaler.log to see if Flight is attempting autoscaling. Also the AWS Ec2 console autoscaling tab will show you the groups created, requests and any errors with failed instance launches.

Hope that helps!

Steve


#3

Thank you. Will do.

Is there also a way to kick it? i.e. say ‘Just fire up all the nodes up to the maximum I specified in the CF template?’

Cheers,
Mike


#4

The CF template for flight just sets the min/max values for a scaling group. If your scaling group exists for the cluster in the AWS console and is not reporting any errors, you could try modifying it manually.

Thanks

Steve