Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Framework, Pipeline - running code: field steps

See original GitHub issue

Continuing on my previous issue 😄 I want to move from step transformation to use pipeline for that. This is a step transformation:

from pprint import pprint
from frictionless import Package, Resource, transform, steps

source = Resource(path="transform.csv")
target = transform(
source,
steps=[steps.field_add(name="Category", type='string', function=lambda rec: 'A' if (rec['BTOT'] in str(list(range(11)))) else 'B'),])

I would like to use Pipeline instead:

from pprint import pprint
from frictionless import Pipeline, transform, steps

pipeline = Pipeline({
    'tasks': [
        {
            'type': 'resource',
            'source': {'path': 'transform.csv'},
            'steps': [
                {'code': 'field_add', 'name': 'Category', 'type':'string', 'function':"lambda rec: 'A' if (rec['BTOT'] in str(list(range(11)))) else 'B')"}
            ]
        }
    ]
})

status=transform(pipeline)
status
{'version': '4.1.0', 'time': 0.018, 'errors': [], 'tasks': [{'time': False, 'errors': [{'code': 'task-error', 'name': 'Task Error', 'tags': [], 'note': '[step-error] Step is not valid: cannot create check "field_add". Try installing "frictionless-field_add"', 'message': 'The task has an error: [step-error] Step is not valid: cannot create check "field_add". Try installing "frictionless-field_add"', 'description': 'General task-level error.'}], 'type': 'resource', 'target': {}, 'stats': {'errors': 1}, 'valid': False}], 'stats': {'errors': 1, 'tasks': 1}, 'valid': False}

Is it possible to use field steps in Pipeline? Can you elucidate me how to use it?

Please preserve this line to notify @roll (lead of this repository)

Issue Analytics

State:
Created 2 years ago
Comments:17 (9 by maintainers)

Top GitHub Comments

1reaction

rollcommented, Mar 26, 2021

I’ve created an issue for it https://github.com/frictionlessdata/frictionless-py/issues/780

1reaction

rollcommented, Mar 26, 2021

@bkarolina It’s just a confusing string representation of pretty healthy data. Internally transform uses this DataWithErrorHandling class to handle possible errors with nice error messages showing a step that caused a problem. We just need to rename it to something like ‘data’: <frictionless.transform.resource.DataFromTransform object at 0x7fa0d59b9b80>

Sorry for the confusion 😃

Top Results From Across the Web

CI/CD Pipeline: A Gentle Introduction - Semaphore

A CI/CD pipeline helps you automate steps in your software delivery process. Understand the basics, best practices and how to get started.

CI/CD pipelines - GitLab Docs

Stages, which define when to run the jobs. For example, stages that run tests after stages that compile the code. Jobs are executed...

Walkthrough: Implementing a pipeline step

Guide to implementing a custom pipeline step in Sitecore Data Exchange Framework in order to create a custom provider.

Create and run ML pipelines - Azure Machine Learning

Create and run machine learning pipelines to create and manage the workflows that stitch together machine learning (ML) phases.

How to Implement an Effective CI/CD Pipeline

Understanding how a CI/CD pipeline works is the fundamental step to adopting and implementing an effective framework that enables an organization to release ......