Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.
And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.
Provide table ingestion status through API
See original GitHub issueSummary
Operating Pinot on behalf of data engineers represents a challenge when these users only have access to the Controller endpoints. SRE aren’t specifically aware of tables configurations details as created by the system users. If something is wrong, there is no other means for data engineers to get the details of the error than requesting log extraction from SREs.
Observed use cases
Realtime table ingestion
Any Kafka connectivity issue coming from a bad server, port or credential will not be reported back in the table status.
BatchConfig
Recently on 0-7-0.SNAPSHOT, a new awesome batchIngestionConfig allows minion tasks to ingest data. For misc reason, that ingestion may fail because of user provided configuration (invalid S3, or GCS credentials, etc). Sadly, the only way for a data engineer to get feedback from the root cause not seing anything being ingested is ask an SRE to investigate the Pinot logs.
Suggestion
Include a table ingestion state as part of the table status.
Issue Analytics
- State:
- Created 3 years ago
- Comments:13 (12 by maintainers)
Top Results From Across the Web
Top Related Medium Post
Top Related StackOverflow Question
Troubleshoot Live Code
Top Related Reddit Thread
Top Related Hackernoon Post
Top Related Tweet
Top Related Dev.to Post
Top Related Hashnode Post
Added a high level design document for this issue. At the moment, I’ve only considered Pinot RealTime tables. Once the high level approach is agreed upon - I will extend this to Minion based ingestion of Offline tables as well.
You can find the design here: https://docs.google.com/document/d/12w6rEJBRKACKomSdL871GCjTtzLxY1N7kYE308RT8JY/edit
I feel there are two parts to this
I think having 1 is important and its user friendly. It allows us to internally call multiple endpoints/queries etc to present a complete overview of what’s going on with the table. I am happy to have an overall status API for a table and ingestion Status is a part of the response.
Freshness/partial response/actually testing the connection information is part of the implementation and we can start with what makes sense and enhance as needed but having a standard endpoint is a good start.