I had a play with Cluster Flow on Alces Flight today and got it working in the end. It's not super simple unfortunately, but with a bit of magic I hope that it may be possible.
I started off with the intention of writing some documentation, but it ended up being just a list of commands that I used (with comments). You can see that here: https://github.com/ewels/clusterflow/blob/alces-flight/docs/alces-flight.md
The Cluster Flow config file I used is here: https://github.com/ewels/clusterflow/blob/alces-flight/clusterflow_aws.config
Note that a bunch of things in that config file could be set automatically on your end, which would be helpful. Mainly
@total_mem, so that modules don't request more resources than are available (I had this - jobs just sat in the queue for ever).
One thing I wasn't expecting is that you guys have your own version of the environment module system, so the built in Cluster Flow stuff doesn't work (it calls
modulecmd perl load XXX and evals this). Not sure if we can get around this without just installing and adding everything to the node! Maybe that's easiest.
I found a few things that needed fixing during this, it's really helpful to have a blank cluster to test on! That's always been very difficult when developing this tool.
Anyway, let me know what you think.