#each re-render bug

I have an array of true/false/undefined values which I am rendering as a list of checkboxes. When changing an array element to or from true, the list of checkboxes is re-rendered with the following (index+1) checkbox inheriting the change along with the changed checkbox. Code:

{{#each range as |value idx|}}
  <label><input type="checkbox" checked={{value}} {{action makeChange idx on="change"}}>{{idx}}: {{value}}</label><br/>
{{/each}}

When I use {{#each range key="@index" as |value idx|}} it works correctly.

Twiddle: https://ember-twiddle.com/6d63548f35f99da19cee9f58fb64db59

[chancancode]

I think I know what’s going on here. It’s a 🍝 of mess but I’ll try to describe it. A lot of edge cases (border line user-error) contributed to this, and I’m not really sure what is/is not a bug, what and how to fix any of these.

Major 🔑

First of all, I need to describe what the key parameter does in {{#each}}. TL;DR it’s trying to determine when and if it would make sense to reuse the existing DOM, vs just creating the DOM from scratch.

For our purpose, let’s accept as a given that “touching DOM” (e.g. updating the content of a text node, an attribute, adding or removing content etc) is expensive and is to be avoided as much as possible.

Let’s focus on a rather simple piece of template:

<ul>
  {{#each this.names as |name|}}
    <li>{{name.first}} {{to-upper-case name.last}}</li>
  {{/each}}
</ul>

If this.names is…

[
  { first: "Yehuda", last: "Katz" },
  { first: "Tom", last: "Dale" },
  { first: "Godfrey", last: "Chan" }
]

Then you will get…

<ul>
  <li>Yehuda KATZ</li>
  <li>Tom DALE</li>
  <li>Godfrey CHAN</li>
</ul>

So far so good.

Appending an item to the list

Now what if we append { first: "Andrew", last: "Timberlake" } to the list? We would expect the template to produce the following DOM:

<ul>
  <li>Yehuda KATZ</li>
  <li>Tom DALE</li>
  <li>Godfrey CHAN</li>
  <li>Andrew TIMBERLAKE</li>
</ul>

But how?

The most naive way to implement the {{#each}} helper would clear the all content of the list every time the content of the list changes. To do this you would need perform at least 23 operations:

• Remove 3 <li> nodes
• Insert 4 <li> nodes
• Insert 12 text nodes (one for the first name, one for the space in between and one for the last name, times 4 rows)
• Invoke the to-upper-case helper 4 times

This seems… very unnecessary and expensive. We know the first three items didn’t change, so it would be nice if we could just skip the work for those rows.

🔑 @index

A better implementation would be to try to reuse the existing rows and don’t do any unnecessary updates. One idea would be to simply match up the rows with their positions in the templates. This is essentially what key="@index" does:

1. Compare the first object { first: "Yehuda", last: "Katz" } with the first row, <li>Yehuda KATZ</li>: 1.1. “Yehuda” === “Yehuda”, nothing to do 1.2. (the space does not contain dynamic data so no comparison needed) 1.3. “Katz” === “Katz”, since helpers are “pure”, we know we won’t have to re-invoke the to-upper-case helper, and therefore we know the output of that helper (“KATZ”) also didn’t change, so nothing to do here
2. Similarly, nothing to do for row 2 and 3
3. There is no fourth row in the DOM, so insert a new one 3.1. Insert a <li> node 3.2. Insert a text node (“Andrew”) 3.3. Insert a text node (the space) 3.4. Invoke the to-upper-case helper (“Timberlake” -> “TIMBERLAKE”) 3.5. Insert a text node (“TIMBERLAKE”)

So, with this implementation, we reduced the total number of operations from 23 to 5 (👋 hand-waving over the cost of the comparisons, but for our purpose, we are assuming they are relatively cheap compared to the rest). Not bad.

Prepending an item to the list

But now, what would happen if, instead of appending { first: "Andrew", last: "Timberlake" } to the list, we prepended it instead? We would expect the template to produce the following DOM:

<ul>
  <li>Andrew TIMBERLAKE</li>
  <li>Yehuda KATZ</li>
  <li>Tom DALE</li>
  <li>Godfrey CHAN</li>
</ul>

But how?

1. Compare the first object { first: "Andrew", last: "Timberlake" } with the first row, <li>Yehuda KATZ</li>: 1.1. “Andrew” !== “Yehuda”, update the text node 1.2. (the space does not contain dynamic data so no comparison needed) 1.3. “Timberlake” !== “Katz”, reinvoke the to-upper-case helper 1.4. Update the text node from “KATZ” to “TIMBERLAKE”
2. Compare the second object { first: "Yehuda", last: "Katz" } with the second row, <li>Tom DALE</li>, another 3 operation
3. Compare the second object { first: "Tom", last: "Dale" } with the second row, <li>Godfrey CHAN</li>, another 3 operation
4. There is no fourth row in the DOM, so insert a new one 3.1. Insert a <li> node 3.2. Insert a text node (“Godfrey”) 3.3. Insert a text node (the space) 3.4. Invoke the to-upper-case helper (“Chan” -> “CHAN”) 3.5. Insert a text node (“CHAN”)

That’s 14 operations. Ouch!

🔑 @identity

That seemed unnecessary, because conceptually, whether we are prepending or appending, we are still only changing (inserting) a single object in the array. Optimally, we should be able to handle this case just as well as we did in the append scenario.

This is where key="@identity" comes in. Instead of relying on the order of the elements in the array, we use their JavaScript object identity (===):

1. Find an existing row whose data matches (===) the first object { first: "Andrew", last: "Timberlake" }. Since nothing was found, insert (prepend) a new row: 1.1. Insert a <li> node 1.2. Insert a text node (“Andrew”) 1.3. Insert a text node (the space) 1.4. Invoke the to-upper-case helper (“Timberlake” -> “TIMBERLAKE”) 1.5. Insert a text node (“TIMBERLAKE”)
2. Find an existing row whose data matches (===) the second object { first: "Yehuda", last: "Katz" }. Found <li>Yehuda KATZ</li>: 2.1. “Yehuda” === “Yehuda”, nothing to do 2.2. (the space does not contain dynamic data so no comparison needed) 2.3. “Katz” === “Katz”, since helpers are “pure”, we know we won’t have to re-invoke the to-upper-case helper, and therefore we know the output of that helper (“KATZ”) also didn’t change, so nothing to do here
3. Similarly, nothing to do for Tom’s and Godfrey’s rows
4. Remove all rows with unmatched objects (none, so nothing to do in this case)

With that, we are back to the optimal 5 operations.

Scaling Up

Again this is 👋 hand-waving over the comparisons and book-keeping costs. Indeed, those are not free either, and in this very simple example, they may not be worth it. But imagine the list is big and each row invokes a complicated component (with lots of helpers, computed properties, sub-components, etc). Imagine the LinkedIn news-feed, for example. If we don’t match up the right rows with the right data, your components’ arguments can potentially churn a lot and causes much more DOM updates than you may otherwise expect. There are also issues with matching up the wrong DOM elements and loosing DOM state, such as cursor position and text selection state.

Overall, the extra comparison and book-keeping cost is easily worth most of the time in a real-world app. Since the key="@identity" is the default in Ember and it works well for almost all cases, you typically won’t have to worry about setting the key argument when using {{#each}}.

Collisions 💥

But wait, there is a problem. What about this case?

const YEHUDA = { first: "Yehuda", last: "Katz" };
const TOM = { first: "Tom", last: "Dale" };
const GODFREY = { first: "Godfrey", last: "Chan" };

this.list = [
  YEHUDA,
  TOM,
  GODFREY,
  TOM, // duplicate
  YEHUDA, // duplicate
  YEHUDA, // duplicate
  YEHUDA // duplicate
];

The problem here is that the same object could appear multiple times in the same list. This breaks our naive @identity algorithm, specifically the part where we said “Find an existing row whose data matches (===) …” – this only works if the data to DOM relationship is 1:1, which is not true in this case. This may seem unlikely in practice, but as a framework, we have to handle it.

To avoid this, we use sort of a hybrid approach to handle these collisions. Internally, the keys-to-DOM mapping looks something like this:

"YEHUDA" => <li>Yehuda...</li>
"TOM" => <li>Tom...</li>
"GODFREY" => <li>Godfrey...</li>
"TOM-1" => <li>Tom...</li>
"YEHUDA-1" => <li>Yehuda...</li>
"YEHUDA-2" => <li>Yehuda...</li>
"YEHUDA-3" => <li>Yehuda...</li>

For the most part, this is pretty rare, and when it does come up, this works Good Enough™ most of the time. If, for some reason, this doesn’t work, you can always use a key path (or the even more advanced keying mechanism in RFC 321).

Back to the “🐛”

After all that talk we are now ready to look at the scenario in the Twiddle.

Essentially, we started with this list: [undefined, undefined, undefined, undefined, undefined].

Unrelated note: Array(5) is not the same thing as [undefined, undefined, undefined, undefined, undefined]. It produces a “holey array” which is something you should avoid in general. However, it is unrelated to this bug, because when accessing the “holes” you indeed get an undefined back. So for our very narrow purpose only, they are the same.

Since we didn’t specify the key, Ember uses @identity by default. Further, since they are collisions, we ended up with something like this:

"undefined" => <input ...> 0: ...,
"undefined-1" => <input ...> 1: ...,
"undefined-2" => <input ...> 2: ...,
"undefined-3" => <input ...> 3: ...,
"undefined-4" => <input ...> 4: ...

Now, let’s say we click the first check box:

1. It triggers the default behavior of the select box: changing the checked state to true
2. It triggers the click event, which is intercepted by the {{action}} modifier and re-dispatched to the makeChange method
3. It changes the list to [true, undefined, undefined, undefined, undefined].
4. It updates the DOM.

How is the DOM updated?

1. Find an existing row whose data matches (===) the first object true. Since nothing was found, insert (prepend) a new row <input checked=true ...>0: true...
2. Find an existing row whose data matches (===) the second object undefined. Found <input ...>0: ... (previously the FIRST row): 2.1. Update the {{idx}} text node to 1 2.2. Otherwise, as far as Ember can tell, nothing else has changed in this row, nothing else to do
3. Find an existing row whose data matches (===) the third object undefined. Since this is the second time we saw undefined, the internal key is undefined-1, so we found <input ...>1: ... (previously the SECOND row): 3.1. Update the {{idx}} text node to 2 3.2. Otherwise, as far as Ember can tell, nothing else has changed in this row, nothing else to do
4. Similarly, update the undefined-2 and undefined-3
5. Finally, remove the unmatched undefined-4 row (since there are one fewer undefined in the array after the update)

So this explains how we got the output you had in the twiddle. Essentially all the DOM rows shifted down by one, and a new one was inserted at the top, while the {{idx}} is updated for the rest.

The really unexpected part is 2.2. Even though the first checkbox (the one that was clicked on) was shifted down by one row to the second position, you would probably have expected Ember to its checked property has changed to true, and since its bound value is undefined, you may expect Ember to change it back to false, thus unchecking it.

But this is not how it works. As mentioned in the beginning, accessing DOM is expensive. This includes reading from DOM. If, on every update, we had to read the latest value from the DOM for our comparisons, it would pretty much defeat the purpose of our optimizations. Therefore, in order to avoid that, we remembered the last value that we had written to the DOM, and compare the current value against the cached value without having to read it back from the DOM. Only when there is a difference do we write the new value to the DOM (and cache it for next time). This is the sense in which we kind of share the same “virtual DOM” approach, but we only do it at the leaf nodes, not virtualizing the “tree-ness” of the whole DOM.

So, TL;DR, “binding” the checked property (or the value property of a text field, etc) doesn’t really work the way you expect. Imagine if you rendered <div>{{this.name}}</div> and you manually updated the textContent of the div element using jQuery or with the chrome inspector. You wouldn’t have expected Ember to notice that and update this.name for you. This is basically the same thing: since the update to the checked property happened outside of Ember (through the browser’s default behavior for the checkbox), Ember is not going to know about that.

This is why the {{input}} helper exists. It has to register the relevant event listeners on the underlying HTML element and reflect the operations into the appropriate property change, so the interested parties (e.g. the rendering layer) can be notified.

I’m not sure where that leaves us. I understand why this is surprising, but I’m inclined to say this is a series of unfortunate user errors. Perhaps we should be linting against binding these properties on input elements?

[chancancode]

@boris-petrov there may be some limited cases where it is acceptable… like a readonly textfield for “copy this url to your clipboard” kind of thing, or you can use the input element + {{action}} to intercept the DOM event and reflect the property updates manually (which is what the twiddle tried to do, except it also ran into the @identity collision), but yes at some point you are just re-implementing {{input}} and handling all the edge-cases it already handled for you. So I think it’s probably fair to say you should just use {{input}} most, if not all, of the time.

However, that still wouldn’t have “fixed” this case where there are collisions with the keys. See https://ember-twiddle.com/0f2369021128e2ae0c445155df5bb034?openFiles=templates.application.hbs%2C

Which is why I said, I’m 100% not sure what to do about this. On one hand I agree it is surprising and unexpected, on the other hand, this kind of collision is pretty rare in real apps and it is why the “key” argument is customizable (this is a case where the default “@identity” key function is not Good Enough™, which is why that feature exists).