Installing fink

This chapter covers the fink installation. fink’s behaviour can be customized using plugins. The fink plugin mechanism relies on standard Python package mechanisms. In order to get a good experience and get the most out of fink you need to know a few things about Python packaging.

This chapter aims to provide you with all the information you need to know on this topic.

What you need to know about python package management

There is now a lot of packaging infrastructure in the Python community, a lot of technology, and a lot of experience. We will try cover some basic things and give you best practice what to use for Python package management.

  1. Always use virtualenv. A virtualenv is effectively an overlay on top of your system Python install. Creating a virtualenv can be thought of as copying your system Python environment into a local location. When you modify virtualenvs, you are modifying an isolated container. Modifying virtualenvs has no impact on your system Python.
  2. Use pip for installing packages. Python packaging has historically been a mess. There are a handful of tools and APIs for installing Python packages. As a casual Python user, you only need to know of one of them: pip. If someone says install a package, you should be thinking create a virtualenv, activate a virtualenv, pip install <package>. You should never run pip install outside of a virtualenv. (The exception is to install virtualenv and pip itself, which you almost certainly want in your system/global Python.)
  3. Use requirements file for installing all project dependencies. Always strictly specify the package version. Bad one: somepackage=>2.0.3. Good one: somepackage==2.0.3

Here is some useful links if you want dive deeper into Python package management.

fink package structure

The following diagram gives an overview on the fink packages. Please note how we grouped the fink packages in the following categories:

  • fink - the fink core (livecycle mechanism, fink tools)
  • fink plugins - packages to customize how you use fink
  • fink generators and tools - scaffolding and tools to make your work even more efficient

fink package structure overview

At finklabs we have very few (currently one) fink packages we do not want to open-source. The finklabs-config-reader has very opinionated defaults on how we use fink on our AWS infrastructure that is very specific and optimized for our media usecase.

Maintaining dependencies for your project

It is a very common practice not to install Python packages by hand. Instead dependencies and version are managed in a documented and repeatable way. Basically you add the names and versions of your packages to a text file. Most projects also group their dependencies into direct dependencies of the service or application and packages they need to develop, build, test and document.

The grouping is not enforced by packaging but to have a std. within an organization is beneficial especially if your want to reuse CI/CD tools.

A little opinionated but pretty common:

  • requirements.txt tools and packages your service directly depends on
  • requirements_def.txt tools and packages you need to develop and test your service
  • requirements_fink.txt fink and fink plugins you use to deploy your service to AWS

The easiest way to install fink is via pip and virtualenv.

Defining which fink plugins to use

fink needs at least some fink-glugins so you should want to install these together. Add fink and the plugins you use to requirements_fink.txt

fink
fink.say-hello
fink.config-reader
fink.lookups
fink.bundler
fink.slack-integration
fink.gen-serverless
fink.cloud
fink.code
fink.lambda
fink.api

This is also a best practice to use the same requirements_fink.txt file on your build server, too.

Setup virtualenv

Using virtualenvs for Python is considered best practice. This is what you need to do:

  • create a virtualenv (‘$ virtualenv venv’)
  • install the packages you want to use (see above)
  • a virtualenv works basically like every other technical device, you need to switch it on before you can use it (‘$ source ./venv/bin/activate’)

Prepare the venv:

$ virtualenv venv

Activate the venv for use:

$ source ./venv/bin/activate

Installing all dev dependencies in one go

Install the dependencies into venv:

$ pip install -U -r requirements_fink.txt

Now you can start using fink:

$ fink version

BTW, fink version shows you all the versions of fink and installed plugins. So you can use this to quickly check which plugins are installed.

Deactivate a virtualenv

I do not throw away my lawn mower once I am done but with my terminals I do that. But you can deactivate a virtualenv:

$ deactivate

Updating fink

You should frequently update your fink installation to get access to new features and bugfixes. When updating your fink installation, please update fink and all the plugins. Just updating fink or a single plugin could easily break your fink installation.

$ pip install -U -r requirements_fink.txt

This will update fink and all fink plugins specified in requirements_fink.txt

General remarks on “breaking changes” and deprecated features

We have two conflicting goals when maintaining fink:

  • we want to introduce new features and replace older ones with newer implementations
  • we want to be compatible with existing infrastructure and configurations

Besides that this is a pretty standard situation for a deployment tool and many other software projects. Nevertheless we want to make it explicit and consequently document here how we handle this.

We make sure that fink does work with our existing configurations. Frequent releases are ok but our squads expect downward compatibility for new patch versions. We could accept minor releases with breaking changes but expect an “update-documentation” which documents all steps that are necessary to upgrade.

We recently discussed and confirmed this in the fink grooming on 05.07.17 with representatives of the CO and VE squads.

In case we need to deprecate something we announce that in the fink changelog respectively in the changelog of fink plugins and tools.

The following sections go into dependency specification and provide information for how to upgrade from one version to the next.

Dependency specification (pinning versions)

Like we said above we guarantee compatibility with your installation procedures and configurations for patch versions. If you need to enforce compatibility it is recommended to best pin fink packages to minor versions in requirements_fink.txt.

This is just a sample, you need to pin to the actual versions you want to use:

fink==1.0.*
fink.config-reader==1.0.*
fink.lookups==1.0.*
fink.bundler==1.0.*
fink.slack-integration==1.0.*
fink.cloud==1.0.*
fink.code==1.0.*
fink.lambda==1.0.*
fink.api==1.0.*

Detailed information on Version Identification and Dependency Specification.

Updating from gcdt to fink 1.0.0

changed tool names

To better match AWS service names we changed the tool names in fink 1.0.0. The old Japanese names used by gcdt have been nice but proved difficult to remember. To ease migration the gcdt commands still can be used for another say 3-4 month but after that we use the new command names only.

fink command gcdt command AWS service name
cloud kumo CloudFormation
code tenkai CodeDeploy
lambda ramuda AWS Lambda
api yugen API Gateway

removed long deprecated hook mechanism

Removed long deprecated hook support from cloud, code and lambda. No more ‘pre_bundle’, ‘pre_hook’, ‘pre_create_hook’, ‘pre_update_hook’, ‘post_create_hook’, ‘post_update_hook’, ‘post_hook’ any more.

Please use the fink lifecycle hook mechanism instead:

cloud lifecycle hooks using hooks in fink list of available hooks to use

you explicitly need to install all used fink tools

You need to install fink-tools (fink-cloud, fink-code, fink-lambda, fink-api). Please add the tools to your requirements_fink.txt like described in the installation section above.

remove the deprecated ami lookup

Newer finklabs base_ami uses a different naming scheme. Consequently the ami lookup implemented in fink provides you with old base_ami ids.

moved cloudformation helpers to fink.cloud

If you use iam and route53 helpers you need to change imports for these submodules from gcdt to fink.cloud.

we introduce config validation for all fink tools and plugins

The new config validation looks for required properties and does format checks in some cases. fink now also validates datatypes. We found some errors where string type was uses in existing configurations instead of the correct integer type and you need to fix this before your configuration can pass as valid:

A sample for a valid use of integer values:

"lambda": {
  ...
  "memorySize": 128,
  "timeout": 15,
}