Proposal: split function, to create subsets of an array

Array[Array[X]] split(Array[X], Int)

Create subsets of an array of a specified maximum size n. All sub-arrays will have n elements, except possibly the last sub-array which will have at most n elements but may have fewer if the array size is not a multiple of n.

Example:

Array[Int] a = [1, 2, 3, 4, 5]
Array[Array[Int]] = split(a, 2)  # => [[1, 2], [3, 4], [5]]

There have been identified at least two use cases where this would be useful:

• Downsampling an array: if I want only the first n elements of an array: subset(a, n)[0]
• Breaking up a very large array for nested scatters:

  Array a = [...]
  Array chunks = split(a, 100)
  scatter (chunk in chunks) {
      scatter (i in chunk) {
          call foo { input: i = i }
      }
  }
  Array[Int] results = flatten(foo.result)
  

[freeseek]

It is possible to achieve what @jdidion is asking using the development version of WDL:

version development

workflow main {
  Array[Int] a = [1, 2, 3, 4, 5]
  Int n = 2
  scatter (i in range(length(a))) { Int b = i / n }
  Map[Int, Array[Int]] c = collect_by_key(zip(b, a))
  scatter (i in range((length(a) + n - 1) / n)) { Array[Int] d = c[i] }
  output { Array[Array[Int]] e = d }
}

This will indeed generate output:

{
  "main.e": [[1, 2], [3, 4], [5]]
}

If the function unzip() was supported by Cromwell, it would be even more simple:

version development

workflow main {
  Array[Int] a = [1, 2, 3, 4, 5]
  Int n = 2
  scatter (i in range(length(a))) { Int b = i / n }
  output { Array[Array[Int]] c = unzip(as_pairs(collect_by_key(zip(b, a)))).right }
}

I would actually rather reserve split() for a function that is sort of the reverse for function sep(String, Array[String]) so to have a functionality similar to what python has with the String split() method

[markjschreiber]

Although it might be possible to do this (albeit in the development version) it certainly isn’t as obvious what you are doing as having some kind of group, split, window type function.