question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

process_function in Engine has strong signature?

See original GitHub issue

I checked all docs, but I can’t understand if we have any restriction for process_function for Engine.

For example, I want to resolve the following questions:

  • Must my process_function has engine as a first argument? If it is true, then it would be great to notice about it in docs.
  • Can I use another signature instead of process_function(engine, labelled_batch)? For instance, process_function(engine, labelled_batch, device)? According to this great code, it is used additional function _prepare_batch, because of strong signature of process_function (otherwise, I would pass device to process_function as third argument).

Issue Analytics

  • State:closed
  • Created 4 years ago
  • Comments:7 (1 by maintainers)

github_iconTop GitHub Comments

1reaction
vfdev-5commented, Jul 23, 2020

@ZhiliangWu having engine as an argument of process_function can help with fetching easily the current state: iteration, epoch, custom variables etc. For example, when during the training you have total loss function composed of two part and an alpha parameter that is also updated during the training:


def train_step(engine, batch):
     # ...
     loss_1 = ...
     loss_2 = ...
     if engine.state.epoch > num_epochs / 2:
         total_loss = loss_1 + engine.state.alpha * loss_2
     else:
         total_loss = loss_1
     # ...
 

trainer = Engine(train_step)
trainer.state.alpha = 0.1

@trainer.on(Events.ITERATION_COMPLETED)
def update_alpha(engine):
    engine.state.alpha += 0.001

This is just an example, but you can imagine other use-cases where usage of trainer’s state can be interesting.

1reaction
ZhiliangWucommented, Jul 23, 2020

Hi @vfdev-5, I am trying to use this great package to accelerate my development cycle. However, I cannot quite understand why the function process_function() must take engine as its input argument. The only usage of it is in the _run_once_on_dataset() method to compute the self.state.output. However, the argument engine (or self when called in the class) doesn’t do anything or save anything with it. Are there any deeper design thoughts on it? Or is it only used to show deliberately that the process is associated with a specific engine? Thanks for your comments in advance.

Read more comments on GitHub >

github_iconTop Results From Across the Web

Cannot extend Flink ProcessFunction - scala - Stack Overflow
I recently upgraded from Flink 1.2 to Flink 1.3, and I am trying to update my ProcessFunction to work with 1.3. I have...
Read more >
ProcessFunction (Flink : 1.16-SNAPSHOT API)
A function that processes elements of a stream. For every element in the input stream processElement(Object, Context, Collector) is invoked.
Read more >
State Management in Apache Flink - DataDome
If you're fine with losing the state of a given operator, ... going to define the signature for a process-function depending on its...
Read more >
Scripting (continued) - Godot Docs
Its execution is done after the physics step on single-threaded games. A simple way to see the _process() function at work is to...
Read more >
Usage — AiiDA 2.2.0 documentation
For process functions, the ProcessSpec is dynamically generated by the engine from the signature of the decorated function. Therefore, to determine what inputs ......
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found