source: file: Compression

DFFML is hoping to participate in Google Summer of Code (GSoC) under the Python Software Foundation umbrella. You can read all about what this means at http://python-gsoc.org/. This issue, and any others tagged gsoc and project are not generally available bugs, but related to project ideas for GSoC.

Project Idea: File Source Compression

Project description:

DFFML’s initial release includes a FileSource which saves and loads data from files using the load_fd and dump_fd methods.

JSON Example

https://github.com/intel/dffml/blob/dd8007d0c9f8c58c35c94faf148e2b5d6ce4c101/dffml/source/json.py#L19-L27

For the open method of FileSource

https://github.com/intel/dffml/blob/dd8007d0c9f8c58c35c94faf148e2b5d6ce4c101/dffml/source/file.py#L36-L44

Allow for reading and writing the following file formats, transparently (so without subclasses having to do anything) to any source which is a subclass of FileSource.

• gzip (by @yashlamba)
• bz2
• lzma
• zip

Skills: Python, git Difficulty level: Easy

Related Readings/Links:

See https://docs.python.org/3/library/archiving.html for documentation

Potential mentors: @pdxjohnny

Getting Started: Figure out how to do one of the file types, probably gzip (as that probably is as simple as using https://docs.python.org/3/library/gzip.html#gzip.GzipFile if the filename ends in .gz) then move on to the rest. For now just make modifications directly to the FileSource class. We may have you split out the logic later, but don’t worry about another class for now.

What we want to see in your application: Describe how you intend to solve the problem, and give us some “stretch goals”, maybe implement a remote file source which reads form URLs. Don’t forget to include some time for building appropriate tests.

[pdxjohnny]

Sweet! Just ping me if there’s anywhere you need clarification.

On Wed, Mar 27, 2019 at 11:03:08AM -0700, Yash Lamba wrote:

Okay, I’ll start working on documenting this and other source related classes. I have pretty spent the past couple of days understanding the code for the same. Thank you.

— You are receiving this because you were mentioned. Reply to this email directly, [1]view it on GitHub, or [2]mute the thread.

References

1. https://github.com/intel/dffml/issues/15#issuecomment-477283460
2. https://github.com/notifications/unsubscribe-auth/AFrL4XC9FTrKLLc_DTVA3SLCoCetQonbks5va7JcgaJpZM4bncu_

[pdxjohnny]

Hi Yash! sorry i am still working on a reply to your email. I think this is pretty much done. I don;t think tar support is needed right now. If you want to document what’s been implemented with relation to this, that would be awesome. Thank you!