Issues with importing multiple files in Delimited File Importer

Prerequisites

• Put an X between the brackets on this line if you have done all of the following:

  • Running the latest version of Constellation

  • Attached the Support Package via Help > Support Package

  • Checked the FAQs: https://github.com/constellation-app/constellation/wiki/FAQ

  • Checked that your issue isn’t already filed: https://github.com/constellation-app/constellation/issues

  • Checked that there is not already a module that provides the described functionality: https://github.com/constellation-app/constellation/wiki/Catalogue-of-Repositories

Description

Currently in the Delimited File Importer, you have the ability to import from multiple files at once. One big problem with it though is that there is a big (flawed) assumption that all the files you are trying to import have the same structure which causes trouble when trying to apply the specified attribute mapping across all files. To help with this, I propose one of two solutions:

1. Better enforce that all files that are being added need to have the same structure (so that the mapping makes sense across all files)
2. Allow each file to be handled separately. There’s a number of ways that this could be done but a couple that come to mind are only importing selected/highlighted files (i.e. you don’t have to import every file you’ve added) or allowing files to have individual mappings (i.e. you don’t have to apply to same mapping across every file you’ve added)

There will be a similar situation for the JDBC importer.

Steps to Reproduce

1. Open Delimited File Importer
2. Add 2 different files to the importer (can be completely different types or the same type but with different structures)
3. Apply a mapping to one of the files and Click import

Expected behaviour: One of the above suggestions should be implemented to reduce the chance of this happening

Actual behaviour: Mapping tries to be applied to all the files, exceptions potentially thrown if there are mapping issues

Reproduces how often: 100%

Additional Information

Any additional information, configuration or data that might be necessary to reproduce the issue.

[GammaVel]

Thanks @Auriga2 and @antares1470 for clarifying the difference, that is working a real treat now. As an enhancement I think it would be better if the error text had remove instead of delete, as it made me think my file would be deleted! I added in an enhancement ticket for that small update.

[Auriga2]

https://github.com/constellation-app/constellation/pull/610 restricts to enter files of only a single type (see the gif in https://github.com/constellation-app/constellation/issues/600) but the recent refactoring has broken that functionality. Can easily fix that.

Regarding the different structures, I am not sure it’s a good idea to handle only the selected file, if we allow them to add multiple files. I vaguely remember someone saying they’d have the same structure in the common user case. Might be worth checking with users. Also if they need to import from a different structure, they can drag and drop and import that different part afterwards.

As per the discussion if we need to handle files with no extensions, we can add a “No Extension” or “All types” or similar entry in the drop down to handle this, but the user has to make sure to select only a single type to avoid issues.

JDBC importer is not affected with this because there’s only one set of query results at a given time, as it only works for a single query at the moment. (If multiple queries are added, only the first is executed.) Might be worth mentioning this in the help page though. (Do we need to support multiple queries here? If so, it’ll then be confusing as it can result in different structures)