anonymize strips spaces and newlines from text

Describe the bug Anonymizing text eliminates spaces and new lines. For example this text:

Let us know if you have any questions. Otherwise we can plan to discuss more on next week’s call.

Thanks!
John

John Doe | Marketing Manager, Paid Search | Marketing | Strategic Education, Inc. | 612.977.4912

Is parsed as:

Let us know if you have any questions. Otherwise we can plan to discuss more on <DATE_TIME> call.

Thanks!
<PERSON><PERSON> | Marketing Manager, Paid Search | Marketing | Strategic Education, Inc. | <PHONE_NUMBER>

As you can see the new lines and spaces were eliminated from between the persons making the text malformed.

To Reproduce Code to reproduce:

def test_new_line():
    analyzer = AnalyzerEngine()
    results = analyzer.analyze(text3, "en")
    engine = AnonymizerEngine()
    anony = engine.anonymize(text3, results)
    print(anony.text)

Expected behavior The text should maintain the spaces and newlines between entities

[bhushan-borole]

Hey @navalev I would like to work on this issue.

[omri374]

Just did a quick check and it seems like this is an issue with how spaCy detects entities in text.

import spacy

nlp = spacy.load("en_core_web_lg")

text = """
Let us know if you have any questions. Otherwise we can plan to discuss more on next week’s call.

Thanks!
John

John Doe | Marketing Manager, Paid Search | Marketing | Strategic Education, Inc. | 612.977.4912
"""

doc = nlp(text)
[(ent.start, ent.end, ent.text, ent.label_) for ent in doc.ents]

Returns:

[(18, 21, 'next week’s', 'DATE'),
 (27, 29, 'John\n\n', 'PERSON'),
 (29, 33, 'John Doe | Marketing', 'PERSON')]

So the newline characters are actually part of the detected entity. This might be related to this issue.

Either preprocessing of the text or postprocessing of the list of RecognizerResults could help overcome this issue.

Tested with this environment: ============================== Info about spaCy ==============================

spaCy version 3.0.6
Platform Windows-10-10.0.19041-SP0
Python version 3.7.9