Configure-users feature not working with encrypted bucket


#1

Problem Description

When using an encrypted S3 bucket to store Alces cluster config files, like features and customizer scripts, the configure-users feature does not work. All customizers run successfully out of the same bucket.

Cluster Deployment Setup

The cluster is being deployed in AWS and access to the encrypted bucket and its corresponding KMS key is granted using EC2 roles. The role is applied to both, the login node and compute nodes. Access to the S3 bucket has been tested to work from login and compute nodes

Preliminary Findings

Encryption is not a problem for customizers as they are able to successfully run and pull updates from the encrypted bucket. The problem only applies to the configure-users feature. This feature does not run neither on cluster bootstrap nor manually ran with the command alces customize apply feature/configure-users

Question

Is there a way to make the configure-users feature work with an S3 bucket?


#2

Hi @jmenbo,
Are there any clues in the logs held in /var/log/clusterware, or output from alces customize apply feature/configure-users that you could share? With that feature, we’d usally expect to see something like this, even in a failure scenario:

[alces@login1(mycluster) ~]$ al customize apply feature/configure-users
Running event hooks for configure-users
Running configure hook: /opt/clusterware/var/lib/customizer/feature-configure-users/configure.d/configure-users
FATAL: Could not locate user list: ERROR: S3 error: 404 (Not Found)
S3 config file is: /tmp/cluster-customizer.s3cfg.z8ViV3tK
User file is: /tmp/cluster-customizer.user-list.8NtVZyor
Group file is: /tmp/cluster-customizer.group-list.5SAJvJXC
Running configure hook: /opt/clusterware/var/lib/customizer/feature-configure-users/configure.d/grouplist.cfg
Running configure hook: /opt/clusterware/var/lib/customizer/feature-configure-users/configure.d/userlist.cfg
No start hooks found in /opt/clusterware/var/lib/customizer/feature-configure-users
No node-started hooks found in /opt/clusterware/var/lib/customizer/feature-configure-users
No member-join hooks found in /opt/clusterware/var/lib/customizer/feature-configure-users

When the configure-users feature encounters a problem, it leaves the s3cmd config file, along with any retrieved user and group files in /tmp for further inspection/diagnosis.

Thanks,

Stu


#3

Hi @stuts,
Yes, I do get an error similar to what you posted when I run the command manually. The /tmp/cluster-customizer.{user,group}-list.xxxxxx files are both empty, which is consistent with the fact that accessing the S3 bucket is failing. The contents of the cluster-customizer.s3cfg.xxxxx file are:

[default]
access_key = ""
secret_key = ""
security_token = ""
use_https = True
check_ssl_certificate = True

/var/log/clusterware/instance.log on a cluster that is configured to use an encrypted S3 bucket shows the following error

[cluster-customizer:configure] FATAL: Could not locate user list: ERROR: S3 error: 400 (Bad Request)

Where as a cluster using an unencrypted S3 bucket does not have the above error.

I’m aware that the s3cfg file does not have any AWS keys, but it shouldn’t need them since we are using EC2 instance roles to give all cluster instances access to the S3 bucket with the config files.

It’s worth mentioning that the EC2 role that gives the cluster access to the S3 bucket works fine for all other customizers we are using. The problem seems to be specific to features customizers, like configure-users.

Thanks for taking the time to look into this. Let me know if there is any additional info I could provide to help narrowing down the source of the problem.

-j


#4

Hi @jmenbo,

Thank you for the further information.

KMS-encrypted files are not supported by the version of s3cmd in the current release of Flight however AES-256 encrypted buckets are fully supported for customization profiles.

Could you check the encryption used on some of your working customization scripts, it may be that they are set to AES-256.

Thanks,

Stu