Cannot join multiple msg properties using the (core) join node

Cannot join multiple msg properties using the (core) join node

The “reduce sequence” mode in Join node assigns value only to msg.payload; i.e.

• To use split node only a single msg property is required to split the node, which is more often than not the payload itself. Other properties of the msg are copied into all new msgs. This is fine and not an issue
• To join multiple parts (lets call them child msgs) of an original msg (lets call them parent msg), the join node “joins” the payload or any other property when ‘manual’ node is selected. The issue arises when child msgs contain more than one property to be “joined”. I was of the view that it can be done using the ‘reduce sequence’ mode, however I realized that the ‘reduce sequence’ mode assigns the final value to msg.payload (Probably this line in the code base: https://github.com/node-red/node-red/blob/0ecd9673b84435f645bb45984e5237b0c3572017/packages/node_modules/%40node-red/nodes/core/sequence/17-split.js#L284). Thus, in the current state of affairs, we cannot “join” multiple properties while using the join node.

What are the steps to reproduce?

Run the following flow:

[{"id":"c73b49fa.c381f8","type":"split","z":"9e357fb1.f5191","name":"","splt":"\\n","spltType":"str","arraySplt":1,"arraySpltType":"len","stream":false,"addname":"","x":500,"y":360,"wires":[["5acc1cd3.5a7f44","4adb9cc4.001254"]]},{"id":"4adb9cc4.001254","type":"change","z":"9e357fb1.f5191","name":"","rules":[{"t":"set","p":"amount","pt":"msg","to":"$millis()","tot":"jsonata"}],"action":"","property":"","from":"","to":"","reg":false,"x":690,"y":360,"wires":[["d68d9cfc.7da0c","2f39cffd.c6088"]]},{"id":"66557073.501e4","type":"inject","z":"9e357fb1.f5191","name":"","props":[{"p":"payload"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"[\"a\", \"b\"]","payloadType":"json","x":300,"y":360,"wires":[["c73b49fa.c381f8","8be41a7b.a805b8"]]},{"id":"2f39cffd.c6088","type":"join","z":"9e357fb1.f5191","name":"","mode":"reduce","build":"string","property":"payload","propertyType":"msg","key":"topic","joiner":"\\n","joinerType":"str","accumulate":false,"timeout":"","count":"","reduceRight":false,"reduceExp":"($step1 := \t   $ ~> | $ | {\t       \"payload\": $append($A.payload, payload),\t       \"amount\": $A.amount + amount\t}, [\"parts\"]|\t)\t","reduceInit":"{\"payload\":[], \"amount\": 0}","reduceInitType":"json","reduceFixup":"$A","x":890,"y":360,"wires":[["37cf4fcd.80ee6"]]},{"id":"37cf4fcd.80ee6","type":"debug","z":"9e357fb1.f5191","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","targetType":"full","statusVal":"","statusType":"auto","x":975,"y":420,"wires":[],"l":false}]

What happens?

The output of the above flow (using debug node) is

// msg obj = 
{
  "payload": {
    "payload": [
      "a",
      "b"
    ],
    "_msgid": "5156fb62.3b85f4",
    "amount": 3207579255663
  },
  "_msgid": "46f5a6bd.8fec08"
}

the issue is that our JSONata is assigned to msg.payload instead of the msg

What do you expect to happen?

// msg obj = 
{
  
    "payload": [
      "a",
      "b"
    ],
    "_msgid": "5156fb62.3b85f4",
    "amount": 3207579255663
}

Please tell us about your environment:

• Node-RED version: 1.1.0
• Node.js version: 14.2.0
• npm version: N/A
• Platform/OS: N/A
• Browser: N/A

Some background (just one of the use cases we have)

We build HTTP based microservices using NR, so everything is invoked using HTTP in node. Then some logic runs e.g. fetching all files from a remote server. Each file then needs to be transformed and while transforming, we would add some metadata to the msg object (like number of rows processed, status, time-taken etc.) At the end of processing these file, we would join the msgs that correspond to the one HTTP request and then serve the data and combined metadata using the HTTP response node

Edit 1: Current workaround to solve the issue

the flow.json posted under “What are the steps to reproduce?” above does not work with multiple split and join nodes combination. Use the below flow instead

1. Use split as is
2. Use join node along with function node (one can use change node in place of function) So, the flow becomes:

[{"id":"f4ade963.7b67f8","type":"inject","z":"9e357fb1.f5191","name":"","props":[{"p":"payload"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"[[1, 2], [11, 22]]","payloadType":"json","x":140,"y":380,"wires":[["c73b49fa.c381f8"]]},{"id":"c73b49fa.c381f8","type":"split","z":"9e357fb1.f5191","name":"","splt":"\\n","spltType":"str","arraySplt":1,"arraySpltType":"len","stream":false,"addname":"","x":330,"y":380,"wires":[["4adb9cc4.001254"]]},{"id":"4adb9cc4.001254","type":"change","z":"9e357fb1.f5191","name":"","rules":[{"t":"set","p":"amount","pt":"msg","to":"$millis()","tot":"jsonata"}],"action":"","property":"","from":"","to":"","reg":false,"x":500,"y":380,"wires":[["cab7d96d.889cc8"]]},{"id":"37cf4fcd.80ee6","type":"debug","z":"9e357fb1.f5191","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","targetType":"full","statusVal":"","statusType":"auto","x":1095,"y":440,"wires":[],"l":false},{"id":"15f5893b.171567","type":"function","z":"9e357fb1.f5191","name":"","func":"return{\n    ...msg.payload\n}","outputs":1,"noerr":0,"initialize":"","finalize":"","x":960,"y":440,"wires":[["37cf4fcd.80ee6"]]},{"id":"cab7d96d.889cc8","type":"split","z":"9e357fb1.f5191","name":"","splt":"\\n","spltType":"str","arraySplt":1,"arraySpltType":"len","stream":false,"addname":"","x":670,"y":380,"wires":[["312a19ba.79df56"]]},{"id":"f157d987.f3d308","type":"debug","z":"9e357fb1.f5191","name":"","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"true","targetType":"full","statusVal":"","statusType":"auto","x":1095,"y":380,"wires":[],"l":false},{"id":"31f2b275.8b12fe","type":"function","z":"9e357fb1.f5191","name":"","func":"return{\n    ...msg.payload\n}","outputs":1,"noerr":0,"initialize":"","finalize":"","x":940,"y":380,"wires":[["f157d987.f3d308","66bd6404.12c62c"]]},{"id":"66bd6404.12c62c","type":"join","z":"9e357fb1.f5191","name":"","mode":"reduce","build":"string","property":"payload","propertyType":"msg","key":"topic","joiner":"\\n","joinerType":"str","accumulate":false,"timeout":"","count":"","reduceRight":false,"reduceExp":"(\t$isFinalPart := $not($exists($.parts.parts));\t$step1 := \t   $ ~> | $ | {\t       \"payload\": $append($A.payload, ( $type(payload) = \"array\" ? [[payload]] : payload ) ),\t       \"amount\": $A.amount + amount,\t       \"parts\": $.parts.parts\t}, $isFinalPart ? \"parts\" : \"\" |;\t)\t","reduceInit":"{\"payload\":[], \"amount\": 0}","reduceInitType":"json","reduceFixup":"$A","x":830,"y":440,"wires":[["15f5893b.171567"]]},{"id":"312a19ba.79df56","type":"join","z":"9e357fb1.f5191","name":"","mode":"reduce","build":"string","property":"payload","propertyType":"msg","key":"topic","joiner":"\\n","joinerType":"str","accumulate":false,"timeout":"","count":"","reduceRight":false,"reduceExp":"(\t$isFinalPart := $not($exists($.parts.parts));\t$step1 := \t   $ ~> | $ | {\t       \"payload\": $append($A.payload, ( $type(payload) = \"array\" ? [[payload]] : payload ) ),\t       \"amount\": $A.amount + amount,\t       \"parts\": $.parts.parts\t}, $isFinalPart ? \"parts\" : \"\" |;\t)\t","reduceInit":"{\"payload\":[], \"amount\": 0}","reduceInitType":"json","reduceFixup":"$A","x":810,"y":380,"wires":[["31f2b275.8b12fe"]]}]

[mannharleen]

or should it be more like the debug node and be

(with payload as the default) so you could assign it to other properties also ?

I like this. Will attempt a PR. Hope its ok to get some guidance from you both before I make any changes.

[dceejay]

or should it be more like the debug node and be

(with payload as the default) so you could assign it to other properties also ?