Aggregate datashape

This is a…

[ ] Feature request
[ ] Regression (a behavior that used to work and stopped working in a new release)
[X] Bug report  
[ ] Documentation issue or request

Description

I want to create this integration:

sql select
split
  filter (I want only some elements from the collection)
aggregate (I want to continue with the collection)
datamapper (I want to concatenate all last names from the collection into one message)
activemq queue

When I use following flow:

1. start SQL
2. end AMQ
3. split
4. aggregate

The split has the none -> sql result datashape (correct), but the aggregate has amq-in(json instance in my case) -> sql result.

In this case, the aggregate should have the same datashapes as the sql select as I only filtered out some elements.

This results in two datamappers required by the ui:

[christophd]

The latest issue with mixed-up data shapes for aggregate (also reported in #4905) is fixed now but there are still some errors showing up in the Syndesis server log.

com.fasterxml.jackson.databind.exc.MismatchedInputException: Cannot deserialize instance of `java.util.ArrayList` out of START_OBJECT token
--
  | at [Source: (String)"{
  | "name": "foo",
  | "completed": 0
  | }"; line: 1, column: 1]

This is because the user specifies the data shape for the AMQ connection with an json-instance and the meta data variant information (collection/single element) is not given in this case. This is why the server meta lookup hits the error and the UI is provided with non inspected data shapes. So the UI has no chance to display proper collection or single element hints.

Let me fix this on the server meta lookup.

@heiko-braun In my opinion this is a blocker as all integrations where the user defines the data shapes on the connection step (AMQ, Template, Webhook etc.) might run into the problem that data shapes for split/aggregate are not inspected in a consistent way. I think this fix should be backported to 1.6.x.

@avano I think with this particular issue being fixed we have sorted out the missing collection/single element hints on split/aggregate and we are now on the same page regarding data shape handling for split/aggregate. Do you agree?

[avano]

@christophd

anyway, where the datashape should be place in the case? I would expect to place it between aggregate and amq step, because that is what I want to map really.

According to your comment the datashapes should look like this:

SQL       (in: whatever                out: SQL Result collection)
Split     (in: whatever                out: SQL Result single element)
Basic filter
Aggregate (in: json instance from AMQ  out: SQL Result collection)
AMQ       (in: json instance           out: whatever)

Then the UI would require 2 datamapper as it is now - why I should add the datamapper between Split and Aggregate (the output and input datashapes do not match)?

Shouldn’t it be this way?

SQL       (in: whatever                out: SQL Result collection)
Split     (in: whatever                out: SQL Result single element)
Basic filter
Aggregate (in: whatever                out: SQL Result collection)
AMQ       (in: json instance           out: whatever)

I want to use basic filter between split and aggregate, should be the same datashape as in the initial SQL step.

When I would use some connector in between, then the datashapes should look like this according to my common sense 😃

SQL       (in: whatever                out: SQL Result collection)
Split     (in: whatever                out: SQL Result single element)
Connector (in: whatever, use datamapper between split and connector if necessary
                                       out: XML)
Aggregate (in: whatever                out: XML collection)
AMQ       (in: json instance           out: whatever)

This is how I imagine it at the first glance