fabric.contrib.files.append does not handle input with newlines correctly

In Fabric 1.13.2, the following code produces unexpected results

from fabric.contrib.files import append
append('file.txt', 'a') # appended                                                                                                                       
append('file.txt', 'b') # appended                                                                                                                       
append('file.txt', 'c\n') # not appended                                                                                                                 
append('file.txt', 'a\nd') # not appended

The resulting file.txt has the following content

a
b

This behaviour is not present in Fabric 1.13.1. It looks like _escape_for_regex is not properly escaping newlines.

[ploxiln]

The problem is with how egrep handles newline characters.

>> cat test.txt
ab
cd
>> egrep 'ab' test.txt
ab
>> egrep 'f' test.txt
>> echo $'^c\n$'
^c
$
>> egrep $'^c\n$' test.txt
ab
cd
>> egrep $'^f\n$' test.txt
ab
cd
>> egrep $'^f\nb$' test.txt
ab

contrib.append() is described as:

Append string (or list of strings) text to filename.

If text is already found in filename, the append is not run, and None is returned immediately.

The test for whether text already exists defaults to a full line match, e.g. ^<text>$, as this seems to be the most sensible approach for the “append lines to a file” use case.

grep/egrep is also explicitly about matching lines. I don’t think there’s any way to do multiple-line patterns with egrep. So, it seems that newline characters in the egrep pattern are considered to separate multiple patterns, any of which can match. So “\n$” matches every line, and “anything\n$” also matches every line. So append() always thinks the line is already present.

How did it work before? Newlines were effectively removed from the line when it was being checked (as a pattern). Thus typically no match, so it was appended. For example,

append('file.txt', 'a\nd')

was run by Fabric 1.13.1 as:

egrep "^a\
d$" "$(echo file.txt)"

In the shell, backslash-newline means to ignore the newline. So this was equivalent to:

egrep "^ad$" "file.txt"

So if you ran append() twice, it would append twice, which the documentation said it should not do, it should only append once. The second time it would not find what it had just appended.

Conclusions:

• contrib.files.append() does not handle newline characters correctly. It never has, but it changed from “usually erroneously append” to “usually erroneously don’t append”. The documentation does say it’s for lines, after all … but it does “break” some backwards compatibility … but then again, it’s “contrib” …

• For unconditionally appending multi-line strings, you should probably just

  run("echo '_multi_\n_lines' >> file.txt")
  

EDIT: expanded quote of append() documentation to include mention of lines

[ploxiln]

Fabric 2.x doesn’t have this helper function (or any of contrib).

However, this is fixed in my fork of Fabric 1.x https://github.com/ploxiln/fab-classic

EDIT: well, somewhat fixed, the model of being based on egrep has limitations