Development of fink

Contributing

If you find any bugs or if you need new features please feel free to issue a pull request with your changes.

Issues and Feature Requests

Please open a GitHub issue for any bug reports and feature requests.

Common for all Tools

  • All tools imply that your working directory is the directory that contains the artifact you want to work with.
  • Furthermore you are responsible for supplying a valid set of AWS credentials. A good tool is aws-mfa
  • You you need to set an environment variable “ENV” which indicates the account/staging area you want to work with. This parameter tells the tools which config file to use. Basically something like fink_$(ENV).json is evaluated in the configuration component.

Installing the development version locally

To install your local development version (after checkout):

$ pip install -e .

use pip to install the dev requirements:

$ pip install -r requirements_dev.txt

Running Unit-Tests

Use the pytest test-runner to run the fink unit tests. A few tests (with ‘_aws’ in the file name) need AWS. Please turn on your VPN and set the AWS_DEFAULT_PROFILE, ENV, and ACCOUNT environment variables. Details here: https://confluence.finklabs.com/display/OPSSHARED/Deployment+on+AWS.

You need to install the development version of this package so you can run the tests:

$ pip install -e .
$ export AWS_DEFAULT_PROFILE=superuser-dp-dev
$ export ENV=DEV
$ export ACCOUNT=dp # => or your team account

Note: You need to enter an MFA code to run the tests.

$ python -m pytest tests/test_cloud*

Please make sure that you do not lower the fink test coverage. You can use the following command to make sure:

$ python -m pytest --cov fink tests/test_lambda*

This requires the coverage package (included in the requirements_dev.txt file):

$ pip install -r requirements_dev.txt

Mock calls to AWS services

For testing fink together with botocore and AWS services we use placebo_awsclient (a tool based on the boto maintainers placebo project). The way placebo_awsclient works is that it is attached to the botocore session and used to record and later playback the communication with AWS services.

The recorded json files for fink tests are stored in ‘tests/resources/placebo_awsclient’.

fink testing using placebo playback is transparent (if you know how to run fink tests nothing changes for you).

To record a test using placebo (first remove old recordings if any):

$ rm -rf tests/resources/placebo_awsclient/tests.test_code_aws.test_code_exit_codes/
$ export PLACEBO_MODE=record
$ python -m pytest -vv --cov-report term-missing --cov fink tests/test_code_aws.py::test_code_exit_codes

To switch off placebo record mode and use playback mode:

$ export PLACEBO_MODE=playback

To run the tests against AWS services (without recording) use normal mode:

$ export PLACEBO_MODE=normal

Please note:

  • prerequisite for placebo to work is that all fink tools support that the awsclient is handed in as parameter (by the test or main). If a module creates its own botocore session it breaks fink testability.
  • in order to avoid merging placebo json files please never record all tests (it would take to long anyway). only record aws tests which are impacted by your change.
  • fink testing using placebo works well together with aws-mfa.
  • if you record the tests twice the json files probably get messed up. Please do not do this.
  • Please commit the placebo files in a separate commit. This makes reviewing of pull requests easier.

documenting fink

For fink we need documentation and we publish it on Readthedocs. Consequently the tooling is already set like sphinx, latex, ... We would like to use markdown instead of restructured text so we choose recommonmark.

Detailed information on using markdown and sphinx

Installation of docu tools

$ pip install -r requirements_docs.txt

If you need to create the pdf docu install pdflatex (... you also need texlive!).

$ brew cask install mactex

build docu

In order to build the html and pdf version of the documentation

$ make html
$ make latexpdf

Release docu to Readthedocs

To release the documentation to Readthedocs most of the time there are no additional steps necessary. Just connect your rtfd account to your github repo.

Initialize api docu

We used the sphinx-apidoc tool to create the skeleton (80_fink_api.rst) for fink’ api documentation.

$ sphinx-apidoc -F -o apidocs fink

Implementation details

This section is used to document fink implementation for fink developers and maintainers.

configuration

note: fink design has a section on openapi, too

configuration in fink is implemented in a few places:

  • openapi functionality in fink (in ‘fink/fink_openapi.py’)
  • openapi based validation for each tool (in ‘fink_<tool>/openapi_<tool>.yaml’ and ‘fink_<tool>/plugin.py’)
  • the functionality to create docs from openapi specs
actual tool config actual command need to run need to run
via config-reader is non-config-command incept_defaults validate_config
yes yes yes yes
yes no yes yes
no yes yes *) no
no no no no

*) The above table shows that there is a case where we run a non-config-command and do not have configuration from the config reader. In this case we still need the defaults but the defaults alone might not be a valid configuration. To make this case work the incept-defaults functionality needs to disable the config validation for the impacted configuration part.

fink design

Design Goals

  • support development teams with tools and templates
  • ease, simplify, and master infrastructure-as-code

Design Principles

  • write testable code
  • tests need to run on all accounts (not just dp account)
  • make sure additions and changes have powerful tests
  • use pylint to increase your coding style
  • we adhere to Semantic Versioning.

Design Decisions

In this section we document important design decisions we made over time while maintaining fink.

Use botocore over boto3

With botocore and boto3 AWS provides two different programmatic interfaces to automate interaction with AWS services.

One of the most noticeable differences between botocore and boto3 is that the client objects:

  1. require parameters to be provided as **kwargs
  2. require the arguments typically be provided as CamelCased values.

For example::

ddb = session.create_client('dynamodb')
ddb.describe_table(TableName='mytable')

In boto3, the equivalent code would be::

layer1.describe_table(table_name='mytable')

There are several reasons why this was changed in botocore.

The first reason was because we wanted to have the same casing for inputs as well as outputs. In both boto3 and botocore, the response for the describe_table calls is::

{'Table': {'CreationDateTime': 1393007077.387,
            'ItemCount': 0,
            'KeySchema': {'HashKeyElement': {'AttributeName': 'foo',
                                             'AttributeType': 'S'}},
            'ProvisionedThroughput': {'ReadCapacityUnits': 5,
                                      'WriteCapacityUnits': 5},
            'TableName': 'testtable',
            'TableStatus': 'ACTIVE'}}

Notice that the response is CamelCased. This makes it more difficult to round trip results. In many cases you want to get the result of a describe* call and use that value as input through a corresponding update* call. If the input arguments require snake_casing but the response data is CamelCased then you will need to manually convert all the response elements back to snake_case in order to properly round trip.

This makes the case for having consistent casing for both input and output. Why not use snake_casing for input as well as output?

We choose to use CamelCasing because this is the casing used by AWS services. As a result, we don’t have to do any translation from CamelCasing to snake_casing. We can use the response values exactly as they are returned from AWS services.

This also means that if you are reading the AWS API documentation for services, the names and casing referenced there will match what you would provide to botocore. For example, here’s the corresponding API documentation for dynamodb.describe_table <http://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_DescribeTable.html>__.

Use pytest over nose

For many years py.test and nose coexisted as Python unit test frameworks in addition to std. Python unittest. Nose was developed by Mozilla and was popular for quite some time. In 2015 Mozilla switched from nose to pytest.

http://mathieu.agopian.info/presentations/2015_06_djangocon_europe/

There are many arguments in favour of pytest. For us the most important is pytest fixtures which provides us with a reliable and reusable mechanism to prepare and cleanup resources used during testing.

Capturing of log output during test

In our tests we use a logcapture fixture from fink_testtools.helpers to capture log output. The fixture use the textfixtures package under the hood.

use it like this:

logcapture.check(
    ('root', 'INFO', 'a message'),
    ('root', 'ERROR', 'another error'),
)

or:

records = list(logcapture.actual())
assert records[0][2] == 'a message'

details here: http://testfixtures.readthedocs.io/en/latest/logging.html

Use Sphinx, Readthedocs, and Markdown for documentation

Many, many documentation tools populate this space since it is so easy to come up with something. However for Open Source projects Readthedocs is the dominant platform to host the documentation.

The Sphinx is the Python std. docu tool. In combination with markdown tools set is a very convenient way to create Readthedocs conform documentation.

Keep a changelog

We where already keeping a changelog which “almost” followed the guide. In June 2017 we decided to make the format more explicit.

The fink changelog format is defined here: http://keepachangelog.com/en/1.0.0/

Use docopt to build the command line interface

There is a never-ending discussion going about pros and cons of CLI tools for Python. Some of these tools are contained in the Python std. library, some are independent open source library additions. At the moment the most popular tools are Optparse, Argparse, Click, and Docopt

https://www.youtube.com/watch?v=pXhcPJK5cMc

We decided to use docopt for out command line interface because it is simple and very flexible. In addition we developed a dispatch mechanism to ease the docopt usage and to make the fink CLI commands testable.

Using Maya: Datetimes for Humans

We had some issues in the past using datetimes correctly across different timezones and locales. finklabs SRE team took an initiative to improve the situation and we. We looked into pytz and maya. Maya had a convincing offering and is maintained by Kenneth Reitz so we decided to use it for datetime handling within fink.

Plugin mechanism

fink uses entry points similar to pluggy to find installed plugins

For communication with the plugin we use Blinker signals. This helps us to decouple the fink code base from plugin code and vice-versa. Blinker is of cause only one way to do that. Blinker is fast, simple, well documented, etc. so there are some popular frameworks using it (Flask, Pelican, ...).

Config handling using openapi

A wide area of fink functionality is related to configuration:

  • default values for tools and plugins
  • validation of configuration
  • scaffolding of minimal (required properties) configuration
  • sample of complete configuration
  • documentation of configuration

To cover all this fink configuration usecases we decided to use the openapi specifications. Using openapi alows us to build on existing workflows and tooling.