cloud particles

At finklabs we create our infrastructure via AWS cloudformation and in order to do that efficiently we wanted to use reusable building blocks. For that to achieve we where evaluating different solutions available to us via open source.

So this is basically conceptual paper-ware (structure and a few helpers). You still need to write the particles.

Please be aware not to let cloud particles stop you from anything. In case you do not have particles or you do not want to write any you can still build beautiful infrastructure from the raw services provided by AWS.

cloud particles are perfectly optional. There is no tight coupling! You can totally bring your own building-block-mechanism and still use cloud for deployment. You do not even have to use troposphere - as long as your mechanism can export a valid json cloudformation template we are fine. Actually we encourage you to do so. Please share with us what you come up with.

Goals

  • codify best practices for infrastructure
  • use cloudformation.py to assemble a stack from particles
  • complexity is handled in particle

Detailed requirements

  • particle has default parameters that can be overridden
  • particle provides default permission that can be overridden
  • we distribute particles as python packages (later move away from github subprojects)
  • we want to distribute generic std. particles company wide (e.g. finklabs-particles)
  • we want to distribute squad specific particles (e.g. mes-particles)

Status on cloud particles implementation

  • cloud particle implementation is based on MES template_generator.py
  • answered “what is the minimum information we need to provide to use a particle?”
  • restore troposphere character (talked about context => template is the context)
  • added SERVICE_NAME and DEFAULT_TAGS to template
  • I liked the “template_generator” implementation but class structure gets in the way when creating stacks from multiple particle sources
  • move cloudformation parameters and outputs into particle
  • move permissions profile to particle

TODOs

  • create particle profiles using awacs to further shrink the particles
  • look into naming conventions / tooling for autogenerated resource names here it is important that in case we generate random names we can regenerate the same names during testing (fink placebo tools)
  • share particles via package (need Github repo, Jenkins build, ...)

Usage

To build better infrastructure at finklabs we want to assemble infrastructure from reusable particles.

The fink.cloud_particle_helper module contains the functionality (initialize, get_particle_permissions, and Particle) to integrate the cloud particles into troposphere

Sample particles

instance

will create or update a CloudFormation stack

Quickstart example using particles

With cloud particles you can import particles from multiple sources:

from fink_cloud import cloud_particle_helper as ph
import eventbus_particle as eb
import reusable_particles as rp

We use cloudformation.py to assemble a stack from particles:

def assemble_particles(template):
    ################# parameters #############################################
    param_sns_alerts_topic = template.add_parameter(troposphere.Parameter(
        'SNSAlertsTopic',
        Description='Name for topic that receive notifications for validation.',
        Type='String'
    ))

    ################# particles ##############################################
    particles = []  # list of Particle()
    sg_frontend_web = Ref('%sFrontendWeb' % template.SERVICE_NAME)

    ################# s3 bucket ##############################################
    particles.append(rp.create_s3_bucket(template))
    ...

Under the hood we use troposphere to code cloudformation templates. The troposphere template instance is used as a common context to exchange information between cloud particles. With cloud each cloudformation.py needs to implement a generate_template function.

def generate_template():
    template = troposphere.Template()
    ph.initialize(template, 'miaImportProcessor')
    assemble_particles(template)
    return template.to_json()

Developing your own particles

We just started with cloud particles and plan to provide more help on particle development in the future.