Multiple descriptors with a Lagom service

In a Lagom service it is possible to describe multiple service classes that each contain a separate descriptor:

abstract class PaymentApplication(context: LagomApplicationContext) extends LagomApplication(context) with AhcWSComponents {

  override lazy val lagomServer = LagomServer.forServices(
    bindService[CreditService].to(wire[CreditServiceImpl]),
    bindService[DebitService].to(wire[DebitServiceImpl])
  )
}

If the service names are different in each descriptor, then an invalid bundle.conf for ConductR is generated that leads to Connection refused responses for services (see Scenario 1). Using the same service name across the different descriptors is not allowed because Lagom in development mode, the deployment of the first descriptor succeeds but the second fails (see Scenario 2).

We need to find a solution that works both in ConductR and in Lagom development mode (and for other deployment tools). The LagomServer.forServices declaration allows to bind multiple several service interfaces, i.e. several descriptors that can contain different service names. Therefore, from a Lagom perspective only Scenario 1 is valid. This scenario would work for ConductR if the different service endpoints would use the same bind port.

This issue occurs both with Lagom Java and Scala 1.3.1.

Scenario 1

If both descriptors in the service interface use a different name, e.g. named(“credit”) and named(“debit”) then the following bundle.conf is generated:

components = {
  lagom-service-impl = {
    description      = "lagom-service-impl"
    file-system-type = "universal"
    start-command    = ["lagom-service-impl/bin/lagom-service-impl", "-J-Xms134217728", "-J-Xmx134217728", "-Dhttp.address=$CREDIT_BIND_IP", "-Dhttp.port=$CREDIT_BIND_PORT", "-Dplay.crypto.secret=6c61676f6d2d736572766963652d696d706c"]
    endpoints = {
      "credit" = {
        bind-protocol = "http"
        bind-port     = 0
        service-name  = "credit"
        acls          = [
          {
            http = {
              requests = [
                {
                  path-beg = "/credit"
                }
              ]
            }
          }
        ]
      },
      "debit" = {
        bind-protocol = "http"
        bind-port     = 0
        service-name  = "debit"
        acls          = [
          {
            http = {
              requests = [
                {
                  path-beg = "/debit"
                }
              ]
            }
          }
        ]
      },
      "akka-remote" = {
        bind-protocol = "tcp"
        bind-port     = 0
        services      = []
      }
    }
  }
}

Two endpoints (debit and credit) are created. This conceptually does not work for ConductR because each endpoint gets assigned a different bind port and when starting the Lagom application (see start-command) then the bind IP and bind port of only the first service is taken (“-Dhttp.address=$CREDIT_BIND_IP”, "-Dhttp.port=$CREDIT_BIND_PORT”). As a result, the credit endpoint is accessible in ConductR but the debit endpoints are returning a Connection refused error. This is because no Netty server has been started on the bind port of the debit service.

Scenario 2

If both descriptors in the service interface use a same name, e.g. named(“paymentservice”) and named(“paymentservice”) then the following bundle.conf is generated:

components = {
  lagom-service-impl = {
    description      = "lagom-service-impl"
    file-system-type = "universal"
    start-command    = ["lagom-service-impl/bin/lagom-service-impl", "-J-Xms134217728", "-J-Xmx134217728", "-Dhttp.address=$PAYMENTSERVICE_BIND_IP", "-Dhttp.port=$PAYMENTSERVICE_BIND_PORT", "-Dplay.crypto.secret=6c61676f6d2d736572766963652d696d706c"]
    endpoints = {
      "paymentservice" = {
        bind-protocol = "http"
        bind-port     = 0
        service-name  = "paymentservice"
        acls          = [
          {
            http = {
              requests = [
                {
                  path-beg = "/credit"
                }
              ]
            }
          },
          {
            http = {
              requests = [
                {
                  path-beg = "/debit"
                }
              ]
            }
          }
        ]
      },
      "akka-remote" = {
        bind-protocol = "tcp"
        bind-port     = 0
        services      = []
      }
    }
  }
}

Now, the HTTP endpoints of both services classes are merged into one ConductR endpoint “paymentservice”. While this solves the start-command issue, this bundle.conf seems to be invalid. When loading the bundle onto ConductR, I receive a broken pipe error which usually indicates that the bundle.conf is invalid

conduct load /Users/mj/workspace/sbt-conductr/sbt-conductr-tester/lagom-java-bundle/lagom-service-impl/target/bundle/lagom-service-impl-v0-42eb32075b87b72dd1e8571a6453ab9d2339674927f5bd381b1bf7b5afa0e7c1.zip
Retrieving bundle..
Retrieving file:///Users/mj/workspace/sbt-conductr/sbt-conductr-tester/lagom-java-bundle/lagom-service-impl/target/bundle/lagom-service-impl-v0-42eb32075b87b72dd1e8571a6453ab9d2339674927f5bd381b1bf7b5afa0e7c1.zip
Loading bundle to ConductR..
Error: Unable to contact ConductR.
Error: Reason: ('Connection aborted.', BrokenPipeError(32, 'Broken pipe'))
Error: Start the ConductR sandbox with: sandbox run IMAGE_VERSION
Also, this service declaration does not work in Lagom development mode. When there’s two different descriptors with the same name in development mode, the deployment of the first descriptor succeeds but the second fails (or vice versa) because it considers it’s already deployed.

Additional info In sbt-conductr we already have a integration test in place that tests Scenario 2. If this scenario is invalid then the integration test should be invalid as well. I’ve created an additional integration test for Lagom Scala for testing purposes as well: https://github.com/typesafehub/sbt-conductr/pull/223. Maybe this integration test was valid from a ConductR perspective when we were still using the “services” endpoint format and not the “acls” endpoint format?!

[TimMoore]

I’m also skeptical about the concept of “non-locatable” services. How do you use them? You can’t access them with a normal Lagom service client, which relies on service location. I think this is surprising and confusing.

Option 1 breaks the API, but I think we could manage it through a deprecation process.

Option 2 makes the API very complicated and easy to misuse. Incorrect use could only be detected at runtime.

Option 3 I agree is counter-intuitive. The sentence is incomplete, but I think you’re saying it’s counter-intuitive because the “locatable” name is the first service in the list. If we took this approach, I think we would also want to change that logic so that the locatable name was taken from the locatable service. Even with that change, this option has many of the same problems as option 2.

The root issue with all of these problems is the idea that we can treat multiple descriptors in a service the same way that we treat multiple descriptors split across multiple services. In other words, clients don’t see any difference between a “payment-service-impl” project with “debit-service” and “credit-service” descriptors inside, versus two seperate “debit-service-impl” and “credit-service-impl” projects. This implies that:

1. Multiple locatable services requires multiple names to be registered for the same endpoint
2. Any descriptors that are shared between projects with the same name (like MetricsService) can’t be locatable, and therefore can’t be accessed by a Lagom service client, because there’s no way to distinguish which MetricsService you want

This conflicts with what I think is the more common model for service location, where a name refers to a single endpoint. That endpoint might have multiple sets of capabilities (like Lagom service descriptors) that are internally routed by URL. Clients do need to know the name of the container they are addressing.

I think we can solve all of our problems, and continue to support multiple service descriptors in a service, if we treat service descriptors as a tree with a single root rather than a flat list. Child descriptors could be “mounted” into parent descriptors.

Here’s an example of what that might look like:

trait PaymentService extends Service {
  def debitService: DebitService
  def creditService: CreditService
  def metricsService: MetricsService

  override final def descriptor = {
    import Service._
    named("payment-service").withServices(
      service("/debit", debitService _).withAutoAcl(true),    // these are macros that
      service("/credit", creditService _).withAutoAcl(true),  // access the child descriptors
      service(metricsService _) // uses the child descriptor name as the mount point
    )
  }
}

class PaymentServiceImpl(
    val debitService: DebitService,
    val creditService: CreditService,
    val metricsService: MetricsService
) extends PaymentService {
  // ...
}

Consumers will construct a client for the root service descriptor trait and access child services like this:

paymentService.creditService.credit(/*...*/).invoke(/*...*/)
paymentService.debitService.debit(/*...*/).invoke(/*...*/)
paymentService.metricsService.currentCircuitBreakers.invoke() // hey, look, I'm locatable!

I’ll note a few details and other thoughts:

1. The parent descriptor can override properties of the child descriptors. This allows the parent to control the external visibility of children.
2. It introduces the idea of URL mount points for a service, which could allow for some interesting reuse possibilities. You could have the same service interface class mounted multiple times at different URLs with different implementations (or just differently-configured implementations).
3. The parent could implement one of the child accessors as a def, allowing it to decorate the child with custom behavior.
4. Children could inherit circuit breaker descriptors from their parent. This makes more intuitive sense than having each use an independent circuit breaker: if credit-service and debit-service are running in the same service, it’s most likely that if one is failing, the other will be too.
5. I think this simplifies tool support a lot. It leaves much less decision-making up to the implementor of each tool, which means that there will be more consistency between tools.

We’ll need to think about backward compatibility if we go with this approach. We would still deprecate the varargs version of LagomServer.forServices but would need to keep it working, especially in the normal case where only one service descriptor is used. The dev mode service locator would still need to know how to locate multiple top-level descriptors per service. A trickier issue is how a client using an older API JAR with multiple service descriptors would work compatibly with a newer implementation using a tree. I think all of these issues are solvable.

In this case, I think it would be OK to say that deployment environments/service locator implementations (including ConductR) are free not to support multiple top-level descriptors, and if you want to deploy your service to one of these environments, you’ll need to update it to use a descriptor tree instead.

[ignasi35]

Leave the ServiceDescriptor mapping in Lagom and ConductR as it is, i.e. in ConductR create one endpoint for each ServiceDescriptor. Then each service needs to be started on a different Netty server with a different port inside one single JVM process. This suggestion was coming from @huntc in the #580 (comment). This would result in an addition in Lagom that starts a Netty server per ServiceDescriptor. I would prefer this solution.

This makes sense for locatable services. So back to our example:

• the payments-service-lagom-v1-342.zip bundle is deployed and 2 PORTS are provided to it can bind debits-service on DEBITS_PORT and credits-service on CREDIT_PORT. That is because debits-service and credits-service are locatable. Because they are locatable, the process will have to register the tuples (debits-service -> DEBITS_PORT) and (credits-service -> CREDIT_PORT) on the ServiceLocator.

And the problem is: what about the non-locatable services also bundled inside payments-service-lagom-v1-342.zip?

For example: Lagom adds MetricsService to every deployment unit. It’s added automatically on every Lagom Java ServiceDescriptor and it’s an OptIn feature for Lagom Scala API developers. MetricsService is a non locatable service providing metrics on the circuit breakers of the process. The purpose is for the infrastructure (or a human) to be able to monitor those circuit breakers (e.g. track response time degradation). But apart from MetricsService, users may add other non-locatable ServiceDescriptors too. Despite being non-locatable by name via the ServiceRegistry/ServiceLocator, the ServiceDescriptors are usable by anyone with the list of active host:ports for a given bundle. Should all ServiceDescriptors be accessible through every Netty port? Should non-locatable ServiceDescriptors be accessible only on the first port bound? In the case of MetricsService, would a monitoring tool read the metrics twice generating duplicate information?

There is one last edge case.

It is possible to create a Notifications Bundle. That is a process that consumes topics and reacts to messages received by communicating to 3rd party endpoints (push notifications, send e-mails,…). It doesn’t provide any public API, but it’s essentially a client to several broker topics and several 3rd party APIs. in that case a MetricsService is needed but no other explicit ServiceDescriptor is bundled. Would that cause zero ports to be available?

I guess the number of Netty instances would have to be min(1,number_of_locatable_services).