Allow self managed Apache Kafka as an event source

Use case description

We’d like to utilize our self managed Kafka cluster as an event source, now that AWS will allow it.

Proposed solution

I would think it would look fairly similar to what was implemented for #8117. See https://docs.aws.amazon.com/lambda/latest/dg/kafka-smaa.html for additional details. I also provided an example of what I’m thinking the serverless.yml events would look like, although feedback is appreciated!

functions:
  compute:
    handler: handler.compute
    events:
      - kafka:
          saslScram512Auth: arn:aws:secretsmanager:us-east-1:01234567890:secret:MyBrokerSecretName
          topic: mytopic
          bootstrapServers:
            - abc3.xyz.com:9092
            - abc2.xyz.com:9092
      - kafka:
          vpcSubnets:
            - subnet-0011001100
            - subnet-0022002200
          vpcSecurityGroups:
            - sg-0123456789
          topic: mytopic
          bootstrapServers:
            - abc3.xyz.com:9092
            - abc2.xyz.com:9092

I didn’t enumerate all of the options that would be provided, but I figure it’d support all of the options for the https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-lambda-eventsourcemapping.html resource, but being focused on using SelfManagedEventSource specifically.

[medikoo]

@lewgordon I’m not sure if that’s possible.

AFAIK we need to resolve a Fn::Sub and Ref locally to be able to compile accessConfiguration into result CF template, and we probably do not have a means to deduct id for LambdaSecurityGroup resource (?)

Technically the way things work in a Framework:

• If you want to reuse some value across different properties (1) define it in custom block, (2) reference it via configuration variable (${self:custom...}) in those properties
• CF intrinsic functions are allowed only in values passed “as is” to compiled template. In many cases it’s not possible to resolve them on spot, and in many other cases resolution rules are not straightfoward. We have few exceptions where we attempt to resolve those functions (e.g. for local invocation), but still it’s limited and incomplete (as it cannot be complete), and it’s only in cases where in main scenario those values are used “as-is” in CF template.

[medikoo]

@lewgordon thanks for explanation, yes we definitely should not support any CF intristic functions here, as it’s not a value we directly pass to CF template.

In such case I suggest to improve the syntax to one now proposed. What do you think? @pgrzesik what’s your opinion on that?