Gatling users terminate prematurely (supposedly on getting HTTP 5xx)

Hi Stephane et al,

another problem surfaced, which I cannot fully explain.

I have a scenario with two parts: setting up and the load generation, the latter being in the forever loop:

override def scenarioBuilder = scenario(getClass.getSimpleName)
    .exec(Account.create)
    .exec(Account.login)
    .exec(Account.activate)
    .exec(Loop.create)
    .forever(
      exec(Log.putEvents)
  ...
)

Given the scenario forever loop, the only way for a user to exit prematurely (before the scenario max duration runs out) is either exitHereIfFailed or exitBlockOnFail, neither of which are used here.

During the scenario execution the users go down unexpectedly, both being executed locally and on Jenkins.

From the execution log:

================================================================================ 2017-07-24 14:10:56 30s elapsed ---- Requests ------------------------------------------------------------------

Global (OK=547 KO=2 ) account.create (OK=41 KO=0 ) account.login (OK=41 KO=0 ) account.activate (OK=40 KO=0 ) loop.create (OK=39 KO=0 ) log.putEvents (OK=194 KO=2 ) … ---- Errors -------------------------------------------------------------------- status.find.is(200), but actually found 504 2 (100.0%)

[------------------------- ] 0% waiting: 100 / active: 50 / done:0

followed by two users terminating:

================================================================================ 2017-07-24 14:11:01 35s elapsed ---- Requests ------------------------------------------------------------------

Global (OK=663 KO=4 ) account.create (OK=51 KO=0 ) account.login (OK=48 KO=0 ) account.activate (OK=48 KO=0 ) loop.create (OK=47 KO=0 ) log.putEvents (OK=235 KO=4 ) ---- Errors -------------------------------------------------------------------- status.find.is(200), but actually found 504 4 (100.0%)

[---------------------------- ] 1% waiting: 92 / active: 56 / done:2

For the reference, the log.putEvents method:

  val HTTP_OK = status.is(200)

  lazy val putEvents = exec(session => {
      AWSSigner.signAsAdmin(session, REQUEST_BODY)
    })
    .exec(http("log.putEvents")
      .post("/")
      .header("X-QA", QA_HEADER)
      .header("X-Amz-Target", VERSION)
      .header("X-Amz-Date", (session) => session.get("X-Amz-Date").as[String])
      .header("Authorization", (session) => session.get("Authorization").as[String])
      .body(ElFileBody(REQUEST_BODY))
      .asJSON
      .check(HTTP_OK))

Do you have any idea what might have been broken? At the moment the best one I have is a suspicion that HTTP 5xx responses could occasionally cause this, as it never happens when there are no errors.

Gatling 2.2.4

[slandelle]

A JVM won’t use more memory than allocated, so the system used memory is not meaningful. Typically, you want to know the JVM heap usage. Use a tool such as VisualVM. From your charts, it really looks like something is eating resources.

[slandelle]

Sorry, but there’s no way we can figure out what happens without a reproducer. I’m pretty sure you’re saturating your ALB. Closing for now. Please ping if you can provide a reproducer, or want to contract so we can investigate in your environment.