question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

use singer taps and targets programatically

See original GitHub issue

Hi, I’m participating in development of dataflows which has similar goals to your projects, and we would like to be able to integrate between the libraries - use singer taps / targets inside a data flow, and use a data flow as a singer tap / target (https://github.com/datahq/dataflows/issues/16)

To enable this integration we need to be able to call singer taps / targets from Python code, this is easy to do using subprocess.Popen, see example here

I think it would really useful to have this in a more standard way as part of the singer-python library.

Example: singer.read_tap

Install the tap: pip install tap-exchangeratesapi

Read from the tap:

>>> tap = singer.read_tap('exchangeratesapi', {"base": "ILS", "start_date": "2018-10-01"})
>>> for message in tap:
>>>     print(message)  # SchemaMessage / RecordMessage / StateMessage

Issue Analytics

  • State:closed
  • Created 5 years ago
  • Comments:5 (5 by maintainers)

github_iconTop GitHub Comments

1reaction
timvishercommented, Oct 26, 2018

I’m going to close this for now but please feel free to reopen it if you’re interested in exploring it via a PR.

0reactions
timvishercommented, Nov 13, 2018

I’ll close this out for now since we don’t have much forward momentum on the PR for now.

Read more comments on GitHub >

github_iconTop Results From Across the Web

Please Dont Build Your Data Pipeline using Singer
Run any Singer tap or target; Provide a UI for configuring and running those taps and targets; Count the number of records synced...
Read more >
Anybody have experience creating singer taps and targets?
Hey I do have some experience working with Singer taps and targets, I can try answering if you have any queries!
Read more >
How to Use Singer for Your ETL Needs: Step-by-Step Tutorial
I'll demonstrate how you can use Singer for your ETL requirements through a practical ... Singer has two types of scripts—taps and targets....
Read more >
Singer SDK: New framework for building Singer taps - Meltano
The goal is to work together on building the framework and supporting tooling and documentation that you will want to use for your...
Read more >
Singer Spec - Meltano Hub
Since taps and targets are able to communicate to each other via the ... Within the Singer Spec, JSON schema definitions are used...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found