lambda command¶

lambda is finks AWS Lambda deployment tool.

Usage¶

To see available commands, call this:

Usage:
        lambda clean
        lambda bundle [--keep] [-v]
        lambda deploy [--keep] [-v]
        lambda list
        lambda metrics <lambda>
        lambda info
        lambda wire [-v]
        lambda unwire [-v]
        lambda delete [-v] -f <lambda> [--delete-logs]
        lambda rollback [-v] <lambda> [<version>]
        lambda ping [-v] <lambda> [<version>]
        lambda invoke [-v] <lambda> [<version>] [--invocation-type=<type>] --payload=<payload> [--outfile=<file>]
        lambda logs <lambda> [--start=<start>] [--end=<end>] [--tail]
        lambda version

Options:
-h --help               show this
-v --verbose            show debug messages
--keep                  keep (reuse) installed packages
--payload=payload       '{"foo": "bar"}' or file://input.txt
--invocation-type=type  Event, RequestResponse or DryRun
--outfile=file          write the response to file
--delete-logs           delete the log group and contained logs
--start=start           log start UTC '2017-06-28 14:23' or '1h', '3d', '5w', ...
--end=end               log end UTC '2017-06-28 14:25' or '2h', '4d', '6w', ...
--tail                  continuously output logs (can't use '--end'), stop 'Ctrl-C'

clean¶

removes local bundle files.

bundle¶

zips all the files belonging to your lambda according to your config and requirements.txt and puts it in your current working directory as bundle.zip. Useful for debugging as you can still provide different environments.

deploy¶

Deploy an AWS Lambda function to AWS. If the lambda function is non-existent it will create a new one.

For an existing lambda function lambda checks whether the hashcode of the bundle has changed and updates the lambda function accordingly. This feature was added to lambda so we are able to compare the hashcodes locally and save time for bundle uploads to AWS.

This only works if subsequent deployments are executed from the same virtualenv (and same machine). The current implementation of the fink-bundler starts every deployment with a fresh virtualenv. If you want the hashcode comparison you need to provide the --keep option. With the ‘–keep’ option the virtualenv is preserved. Otherwise the hashcodes of the lambda code bundles will be different and the code will be deployed.

If you can not reuse (‘–keep’) the virtualenv for example in case you deploy from different machines you need to use git to check for code changes and skip deployments accordingly.

In any case configuration will be updated and an alias called “ACTIVE” will be set to this version.

list¶

lists all existing lambda functions including additional information like config and active version:

dp-dev-store-redshift-create-cdn-tables
    Memory: 128
    Timeout: 180
    Role: arn:aws:iam::644239850139:role/lambda/dp-dev-store-redshift-cdn-LambdaCdnRedshiftTableCr-G7ME657RXFDB
    Current Version: $LATEST
    Last Modified: 2016-04-26T18:03:44.705+0000
    CodeSha256: KY0Xk+g/Gt69V0siRhgaG7zWbg234dmb2hoz0NHIa3A=

metrics¶

displays metric for a given lambda:

dp-dev-ingest-lambda-cdnnorm
    Duration 488872443
    Errors 642
    Invocations 5202
    Throttles 13

wire¶

“wires” the AWS Lambda function to an event source.

unwire¶

delets the event configuration for the lambda function

delete¶

deletes a lambda function

If you use the --delete-logs the cloudwatch log group associated to the AWS Lambda function is deleted including log entries, too. This helps to save cost for items used in testing.

rollback¶

sets the active version to ACTIVE -1 or to a given version

invoke¶

In this section, you invoke your Lambda function manually using the lambda invoke command.

$ lambda invoke my_hello_world \
--invocation-type RequestResponse \
--payload '{"key1":"value1", "key2":"value2", "key3":"value3"}'

If you want you can save the payload to a file (for example, input.txt) and provide the file name as a parameter:

$ lambda invoke my_hello_world \
--invocation-type RequestResponse \
--payload file://input.txt

The preceding invoke command specifies RequestResponse as the invocation type, which returns a response immediately in response to the function execution. Alternatively, you can specify Event as the invocation type to invoke the function asynchronously.

logs¶

The lambda logs command provides you with convenient access to log events emitted by your AWS Lambda function.

The command offers ‘–start’ and ‘–end’ options where you can filter the log events to your specification. You can use human readable dates like ‘2017-07-24 14:00:00’ or you can specify dates in the past relative to now using ‘1m’, ‘2h’, ‘3d’, ‘5w’, etc.

$ lambda logs ops-dev-captain-crunch-slack-notifier --start=1d
MoinMoin fin0007m!
2017-07-07
07:00:32  START RequestId: f75cd7de-62e1-11e7-937d-ef5726c6f5c8 Version: $LATEST
07:00:36  END RequestId: f75cd7de-62e1-11e7-937d-ef5726c6f5c8
07:00:36  REPORT RequestId: f75cd7de-62e1-11e7-937d-ef5726c6f5c8        Duration: 4221.50 ms    Billed Duration: 4300 ms        Memory Size: 128 MB     Max Memory Used: 43 MB
Bye fin0007m. Talk to you soon!

The ‘–start’ option has a default of ‘1d’. This means if you run lambda logs <your-function-name> you get the log output of your function for the last 24 hours.

$ lambda logs ops-dev-captain-crunch-slack-notifier
MoinMoin fin0007m!
2017-07-07
07:00:32  START RequestId: f75cd7de-62e1-11e7-937d-ef5726c6f5c8 Version: $LATEST
07:00:36  END RequestId: f75cd7de-62e1-11e7-937d-ef5726c6f5c8
07:00:36  REPORT RequestId: f75cd7de-62e1-11e7-937d-ef5726c6f5c8        Duration: 4221.50 ms    Billed Duration: 4300 ms        Memory Size: 128 MB     Max Memory Used: 43 MB
Bye fin0007m. Talk to you soon!

You can use lambda logs to tail the log output of your lambda function. The default start date in tail mode is 5 minutes before. You can specify any past start date in tail mode but you can not specify an ‘–end’ option in tail mode. To exit the lambda logs tail mode use Ctrl-C.

$ lambda logs ops-dev-captain-crunch-slack-notifier --start=1d --tail
MoinMoin fin0007m!
Use 'Ctrl-C' to exit tail mode
2017-07-07
07:00:32  START RequestId: f75cd7de-62e1-11e7-937d-ef5726c6f5c8 Version: $LATEST
07:00:36  END RequestId: f75cd7de-62e1-11e7-937d-ef5726c6f5c8
07:00:36  REPORT RequestId: f75cd7de-62e1-11e7-937d-ef5726c6f5c8        Duration: 4221.50 ms    Billed Duration: 4300 ms        Memory Size: 128 MB     Max Memory Used: 43 MB
^CReceived SIGINT signal - exiting command 'lambda logs'

version¶

will print the version of fink you are using

Folder Layout¶

Sample config file¶

sample fink_dev.json file:

{
  "lambda": {
    "lambda": {
      "name": "dp-dev-store-redshift-load",
      "description": "Lambda function which loads normalized files into redshift",
      "role": "arn:aws:iam::644239850139:role/lambda/dp-dev-store-redshift-cdn-lo-LambdaCdnRedshiftLoad-DD2S84CZFGT4",
      "handlerFunction": "handler.lambda_handler",
      "handlerFile": "handler.py",
      "timeout": "180",
      "memorySize": "128",
      "events": [
        {
          "event_source": {
            "arn":  "arn:aws:s3:::my-bucket",
            "events": ["s3:ObjectCreated:*"]
          }
        },
        {
          "event_source": {
            "name": "send_reminder_to_slack",
            "schedule": "rate(1 minute)"
          }
        }
      ],
      "vpc": {
        "subnetIds": [
          "subnet-87685dde",
          "subnet-9f39ccfb",
          "subnet-166d7061"
        ],
        "securityGroups": [
          "sg-ae6850ca"
        ]
      }
    },
    "bundling": {
      "zip": "bundle.zip",
      "preBundle": [
        "../bin/first_script.sh",
        "../bin/second_script.sh"
      ],
      "folders": [
        {
          "source": "../redshiftcdnloader",
          "target": "./redshiftcdnloader"
        },
        {
          "source": "psycopg2-linux",
          "target": "psycopg2"
        }
      ]
    },
    "deployment": {
      "region": "eu-west-1",
      "artifactBucket": "7finity-$PROJECT-deployment"
    }
  }
}

lambda configuration as part of the fink_<env>.json file¶

log retention¶

Possible values for the log retention in days are: 1, 3, 5, 7, 14, 30, 60, 90, 120, 150, 180, 365, 400, 545, 731, 1827, and 3653.

{
    "lambda": {
        ...
        "logs": {
            "retentionInDays": 90
        }
}

S3 upload¶

lambda can upload your lambda functions to S3 instead of inline through the API. To enable this feature add this to the “lambda” section of your fink_<env>.json config file:

"deployment": {
    "region": "eu-west-1",
    "artifactBucket": "7finity-$PROJECT-deployment"
}

You can get the name of the bucket from Ops and it should be part of the stack outputs of the base stack in your account (s3DeploymentBucket).

runtime support¶

fink supports the nodejs4.3, nodejs6.10, python2.7, python3.6 runtimes.

Add the runtime config to the lambda section of your fink configuration.

    "runtime": "nodejs4.3"

At this point the following features are implemented:

install dependencies before bundling (dependencies are defined in package.json)
bundling (bundle the lambda function code and dependencies)
deployment (the nodejs4.3 lambda function is setup with the nodejs4.3 runtime)
configuration (bundles settings_<env>.conf file for your environments)
nodejs support is tested by our automated fink testsuite
if no runtime is defined fink uses the default runtime python2.7

Note: for this to work you need to have npm installed on the machine you want to run the lambda bundling!

AWS Lambda environment variables¶

Ramuda supports AWS Lambda environment variables. You can specify them within the lambda section.

    ...
    "environment": {
        "MYVALUE": "FOO"
    }

More information you can find in AWS docs.

Adding a settings.json file¶

Ramuda supports a settings section. If used a settings.json file is added to the zip bundle. You can specify the settings within the lambda section.

    ...
    "settings": {
        "MYVALUE": "FOO"
    }

You can use lookups like for the rest of the configuration. Note that the values are looked up BEFORE the AWS Lambda function is deployed. If values change during the AWS Lambda function lifecycle it does not recognise the changes. For values that must be updated you should lookup the values in your code using for example credstash.

    ...
    "settings": {
        "accountId": "lookup:stack:infra-dev:AWSAccountId"
    }

Adding event configuration¶

fink can be used to easily schedule functions to occur on regular intervals. Just list your expressions to schedule them using cron or rate syntax in your fink_<env>.json config file like this:

...
"events": [{
    "event_source": {
        "name": "send_reminder_to_slack",
        "schedule": "rate(1 minute)"
    }
}]

The schedule expression defines when to execute your lambda function in cron or rate format.

Supported event types.

Similarly, you can have your functions execute in response to events that happen in the AWS ecosystem, such as S3 uploads, Kinesis streams, and SNS messages, etc..

In your fink_<env>.json config file, define your event sources. The following sample config will execute your AWS Lambda function in response to new objects in your my-bucket S3 bucket. Note that your function must accept event and context parameters.

...
"events": [{
    "event_source": {
        "arn":  "arn:aws:s3:::my-bucket",
        "events": ["s3:ObjectCreated:*"],
        "suffix": ".jpg"
    }
}],

Similarly, for a Simple Notification Service (SNS) event:

...
"events": [{
    "event_source": {
        "arn":  "arn:aws:sns:::your-event-topic-arn",
        "events": ["sns:Publish"]
    }
}]

Kinesis is slightly different as it is not event-based but pulling from a stream:

...
"events": [{
    "event_source": {
        "arn": arn:aws:kinesis:eu-west-1:1234554:stream/your_stream"
        "starting_position": "TRIM_HORIZON",
        "batch_size": 50,
        "enabled": true
    }
}]

Lambda@Edge needs a CloudFront event trigger.

...
"events": [{
    "event_source": {
        "arn": "arn:aws:cloudfront::420189626185:distribution/E1V934UN4EJGJA",
        "cache_behavior": "*",
        "cloudfront_event": "origin-request"
    }
}]

Deploying AWS Lambda@Edge¶

AWS Lambda@Edge is relatively new so we have to deal with some (hopefully temporary) limitations (like it is only available in Virginia):

AWS Lambda@Edge functions must be deployed to ‘us-east-1’ region
can not use artifact bucket for upload
max memory use 128 MB
max timeout 3 seconds for ‘origin request’ and ‘origin response’ events; for ‘viewer request’ and ‘viewer response’ events timeout is 1 second
lambda@edge does not implement ALIAS or $LATEST so we use the version-nbr of the last published version of the lambda function for wiring
unwire removes the lambda trigger for the configured CloudFront distribution (regardless if the lambda function is the same as the configured one or not)
‘lambda wire’ and ‘lambda unwire’ finish after initiation of the replication to CloudFront. This means CloudFront distribution is in ‘Pending’ state when fink.lambda exits.
you cannot delete lambda functions that have been replicated to CloudFront edge locations

Setting the ENV variable¶

You you need to set an environment variable “ENV” which indicates the account/staging area you want to work with. This parameter tells the tools which config file to use. For example if you want to set the environment variable ENV to ‘DEV’ you can do that as follows:

export ENV=DEV

Environment specific configuration for your lambda functions¶

Please put the environment specific configuration for your lambda function into a fink_<env>.json file. For most teams a useful convention would be to maintain at least ‘dev’, ‘qa’, and ‘prod’ envs.

Defining dependencies for your NodeJs lambda function¶

A sample package.json file to that defines a dependency to the 1337 npm module:

{
  "name": "my-sample-lambda",
  "version": "0.0.1",
  "description": "A very simple lambda function",
  "main": "index.js",
  "dependencies": {
    "1337": "^1.0.0"
  }
}

Sample NodeJs lambda function¶

From using lambda extensively we find it a good practise to implement the ping feature. With the ping ramdua automatically checks if your code is running fine on AWS.

Please consider to implement a ping in your own lambda functions:

var l33t = require('1337')


exports.handler = function(event, context, callback) {
    console.log( "event", event );

    if (typeof(event.lambda_action) !== "undefined" && event.lambda_action == "ping") {
        console.log("respond to ping event");
        callback(null, "alive");
    } else {
        console.log(l33t('finklabs rocks!'));  // 910m3x r0ck5!
        callback();  // success
    }
};