How to create Gridware Depots


#1

Is there a way to create Gridware Depots?


#2

Hi @jmenbo!

Thanks for your interest in Alces Flight!

Have you got a bit more information on what you’re trying to achieve here? If you just jot down some information on what you’re aiming for, we’ll try and get back to you with some ideas on how is best to achieve this with Alces Flight!

Hope to you hear from you soon!

Cheers,
Ruan


#3

Hi Ruan, thanks for the reply. In short, I’m looking for a way to make a set of packages available to our cluster users as a “named” set that they can use to deploy. Basically the same way that one can use “Gridware Depots” to deploy, let’s say, the bio depot. I would like to be able to package a custom set of applications into our own depots.

I looked at the “gridware depot –help” output and there seems to be an option to create depots, i.e. “alces gridware depot init newdepotname”

This command creates a file structure under ~/gridware/depots/{random_8_digit_string} which is symlinked as ~/gridware/newdepotname

And that’s as far as I got. I don’t know how to:

  • add packages to this empty depot

  • export the depot to, let’s say an S3 bucket, to be reused on other clusters

  • reference the newly created depot when deploying new clusters

  • add/remove packages from a depot

I also took a look at the Github gridware-depots repo, but didn’t help me get the answers I was looking for.

https://github.com/alces-software/gridware-depots/blob/master/bioinformatics.yml

Any advice or links to docs would be greatly appreciated.

Thanks


#4

Hi @jmenbo,

You’re on the right track! Here’s an overview of the steps you need to perform:

  1. Create your new depot:
alces gridware depot init yourdepot
  1. Add packages to the depot: e.g.:
alces gridware install -d yourdepot libs/blas
  1. Export your depot to /tmp/yourdepot:
alces gridware depot export yourdepot
  1. Update the depot descriptor file, /tmp/yourdepot/yourdepot.yml, to update the title, summary, description and region_map fields.

    Pay particular attention to the S3 URLs in the region_map field - they should point to the S3 bucket(s) you’ll upload your depot to in the next step. It’s fine to make all of the regions point to a single bucket, but be aware that you may incur cross-region ingress charges, or if you’re only using a single region, it’s also fine to drop the other regions.

  2. Sync the exported depot to S3:

s3cmd --access_key <your access key> --secret_key <your secret key> \
  put -r -P /tmp/yourdepot/ s3://yourbucket/

Once you’ve done the above your depot is ready to be used on other clusters. You’ll need to arrange the following to happen on future clusters:

  1. Grab your depot descriptor from the S3 bucket and add it to the directory that contains local depots:
s3cmd get s3://yourbucket/yourdepot.yml /opt/gridware/etc/depots/local/data/
  1. Update /opt/gridware/etc/gridware.yml to enable the local depots directory by uncommenting the /opt/gridware/etc/depots/local line:
:depot_repo_paths:
 - /opt/clusterware/var/lib/gridware/depots/official
 - /opt/gridware/etc/depots/local

Alternatively you can simplify the above by making use of the site-depots customization feature, which you can specify when you launch your cluster in the “Additional features to enable” field. This feature will make all depots configured in your cluster customization bucket available when the cluster is launched.

To enable this, ensure your depot descriptor file(s) are in your S3 customization bucket under the /site-depots/data/ prefix, e.g. s3://yourcustomizationbucket/site-depots/data/yourdepot.yml.

If you’re using the default customization bucket, you can find out what its name is by using alces about customizer, e.g.:

[alces@login1(mycluster) ~]$ alces about customizer
 Customizer bucket prefix: s3://alces-flight-zzz0mtdmy6m3kmm8/customizer

[alces@login1(mycluster) ~]$ sudo -s
[root@login1(mycluster) alces]# s3cmd put /tmp/yourdepot/yourdepot.yml s3://alces-flight-zzz0mtdmy6m3kmm8/site-depots/data/

If you want some/all of the depots to be automatically installed when the cluster launches, add a config.rc file to the customization bucket at s3://yourcustomizationbucket/site-depots/config.rc`, e.g.:

cw_SITE_DEPOTS_default="yourdepot"

HTH!


Capturing customised gridware environments for later cloud use