FAQ: The right decision to layout your Cement 3 CLI app

Hi @derks

We are struggling with the question about “good” layouting and partioning a new CLI app with cement.

The requirements are defined as a several number of Sub-Commands like

app processing start --options
app processing build --options

app plot images --options
app plot graphs --options

etc.

Question 1 - Layouting sub-commands

Should we implement each command under controllers and write a seperate python file like myapp/app-module/controllers/processing.py or should we create a plugin for each command/controller like myapp/plugins/processing?

Annotation: We never will use processing or plotting outside this app

Question 2 - usage of plugins and extensions

@derks may you give some useful ideas of a PLUGIN or a EXTENSION please.

We can’t get out of the documentation what your exactly ideas behind that and when to use what.

Question 3 - place your apps logic

In our app we will create several classes to implement core functionality.

Where best to place those files and libraries?

1. In case of implementing the sub-commands as controllers, should those source files be placed under myapp/app-module/core/sub-command/logic

2. In case of implementing the sub-commands as plugins, the place will be myapp/plugins/myplugin/core/logic

Right decision?

Question 4 - place your plugin tests

When working with plugins, where to place the tests for that.

I would like to place those under

./myapp/plugins/myplugin/tests

Question 5 - activate plugin with config.yml

I tried to activate a test plugin via config.yml but can’t get it work.

When using the Meta class in myapp/app-module/main.py and adding

plugins = [ 'myplugin' ]

it worked. Can’t get the same from configuration. My config.yml looks:

myapp:
  plugin_dirs: '/abspath/myapp/plugins'

myplugin:
  enable_plugin: 1

Question 6 - module requirements for plugins

If a plugin needs additional libraries like pandas where do you define the requirement.

1. Should those be appended to the (global) app myapp/requirements.txt or should we create a seperate requirements.txt file inside each plugin folder?

2. If to be placed inside plugin folder, what is the preferred way to run the installation for them

Question 7 - render output (and inspect later in test)

What is the preferred way to print some (status) output while running the (long running) job?

a. self.app.render() b. self.app.log.info()

The issue we have when using render is to create also minimized templates for any output. How could there be an easy way to just print out a line? (Simple template like {{ message }} ?

It is also the question if render() is ok when doing that inside a loop.

Issue with that might occur when testing:

for file in files:
    self.app.render({ 'filename' : str(file) }, 'out.jinja2')

when running the app in testing, output.last_rendered will only know the last file.

We tried to use def test_mytest(caplog) instead. Is that what you also would decide?

Hopefully the questions are useful also for others to layout their apps.

Thanks for your answer in advance and guiding thru that nice framework and use it “(your) right way”

Cheers Tom

[derks]

@TomFreudenberg A lot to dig through here, I hope this helps:

Question 1 - Layouting sub-commands

Should we implement each command under controllers and write a seperate python file like myapp/app-module/controllers/processing.py or should we create a plugin for each command/controller like myapp/plugins/processing?

In general, I would recommend putting controllers in their own file, like you mentioned. Example: myapp/controllers/mycontroller.py. However, there are use cases for “optional” controllers or functionality that you might want to be able to enable/disable. For this, you can create them as internal plugins (meaning they ship with the core library code, and not loaded externally). Plugins are loaded from a number of places, one of them (by default) is myapp/plugins:

• https://cement.readthedocs.io/en/3.0/api/core/foundation/#cement.core.foundation.App.Meta.plugin_module

An examle use-case might be a controller that provides admin level commands, that you don’t want to expose to regular users.

Question 2 - usage of plugins and extensions

@derks may you give some useful ideas of a PLUGIN or a EXTENSION please.

We can’t get out of the documentation what your exactly ideas behind that and when to use what.

Simply put, extensions extend the capabilities of the framework… plugins extend the capabilities of the application. Extensions are generally application agnostic, plugins are application specific. Should be noted that neither are necessary… you might create an extension that can be used among multiple applications.

• Example of a framework extension: The cache interface defines the functionality of a caching backend, where the memcached, and redis extensions define the handlers that implement that interface). Those extensions can be used by any application that is built on Cement, and requires a caching backend.
• Example of an application plugin: MyApp Admin Plugin adds additional controllers/options required for admin users, that can be disabled by default for non-admin users. This plugin would be useful to MyApp only. Additionally, third-party developers could build their own plugins for MyApp that do you things you might not being thinking of. Consider git… built in you have the ability to git commit… but if there were a plugin interface, I as a third-party developer might want to create a plugin that provides git commit-and-notify-slack-there-was-a-commit (dumb example)… without having access to the core library or having my plugin shipped with the core code.

Question 3 - place your apps logic

In our app we will create several classes to implement core functionality.

Where best to place those files and libraries?

In case of implementing the sub-commands as controllers, should those source files be placed under myapp/app-module/core/sub-command/logic

In case of implementing the sub-commands as plugins, the place will be myapp/plugins/myplugin/core/logic

Cement does not restrict how/where logic is defined… it is purely a design decision. Personally:

• Core Application Code:
  • I might want to define MyAppController as the base controller for all other controllers, giving me a central location to implement common logic. In my developer documentation, I would instruct developers to sub-class from myapp.core.controller.MyAppController. All controllers, internal or plugins, would sub-class from the same core library.
  • Think of core code as re-usable by all other parts of the application (internal or plugins)
• Controller Logic:
  • All logic specific for this controller should live with the controller… that could be in myapp/controllers/mycontroller.py or more complex myapp/controllers/mycontroller/.... Note, if ControllerB is importing logic from ControllerA you should consider moving that logic to myapp/core/controller.py (or somewhere else more central).

Note, Cement provides cement generate project developer helpers for convenience… but a Cement app could be a single file script. Figuring out the layout, how you like it, might take some trial and error. But if you find yourself following a common theme in how you do layout your applications, you might consider creating your own project/plugin templates:

• https://docs.builtoncement.com/getting-started/developer-tools#customizing-templates

Question 4 - place your plugin tests

When working with plugins, where to place the tests for that.

I would like to place those under

./myapp/plugins/myplugin/tests

This makes the most sense. Unit testing is one of the areas of the doc I have been punting on for a long time:

• https://docs.builtoncement.com/additional-topics/unit-testing
• https://github.com/datafolklabs/cement/issues/557

However, using Pytest is very straight forward and creating unit tests is easy. Referencing Cement core library unit tests for extension should be very helpful in doing that:

• https://github.com/datafolklabs/cement/tree/master/tests/ext

Having the unit tests live with the plugin code is the best place for it. There is nothing special, Pytest should find it like usual and execute. If you need more help here I’d be happy to help get started.

Question 5 - activate plugin with config.yml

I tried to activate a test plugin via config.yml but can’t get it work.

When using the Meta class in myapp/app-module/main.py and adding

plugins = [ ‘myplugin’ ] it worked. Can’t get the same from configuration. My config.yml looks:

myapp: plugin_dirs: ‘/abspath/myapp/plugins’

myplugin: enable_plugin: 1

Where is config.yml? Is it in one of the default application paths?

• https://docs.builtoncement.com/core-foundation/configuration-settings#configuration-file-loading

Also, enable_plugin was changed to enabled in Cement 3. Where did you find the reference to enable_plugin, it might be dated docs somewhere. In Yaml, would ideally be enabled: true.

Question 6 - module requirements for plugins

If a plugin needs additional libraries like pandas where do you define the requirement.

Should those be appended to the (global) app myapp/requirements.txt or should we create a seperate requirements.txt file inside each plugin folder?

If to be placed inside plugin folder, what is the preferred way to run the installation for them

For internal plugins (shipped with core library), I would handle this via README/Documentation as we do for Cement extensions that have dependencies.

For internal or external plugins (shipped separately from core library) this could also be handled by package management solutions:

• DPKG/RPM: If this were getting rolled into a package management system for an OS, the package would define the required dependencies

There are other ways this could be implemented… if I were building something heavily dependent on external plugins, I might build a management controller like myapp plugin {list,install,uninstall} and have the install command:

• Pull external plugin from git or tar.gz
• Unpack
• Run pip install -r requirements.txt in the plugin directory

But that’s just a thought… Cement doesn’t define how plugins are shipped/managed… only how they are loaded.

Question 7 - render output (and inspect later in test)

What is the preferred way to print some (status) output while running the (long running) job?

a. self.app.render() b. self.app.log.info()

There is nothing to say that using print() itself is bad. Some use-cases don’t require templates and prefer to just print output. If the output is useful to the user in real time only, use print() or app.render()… if the output is useful in realtime, and historically use app.log. If you just want to print, but also maintain framework functionality (such as if you wanted to use the Scrub Extension), there is also a Print Extension / Handler:

• https://docs.builtoncement.com/extensions/print

Nothing wrong with calling app.render() more than once, however as you said app.last_rendered will only have the last item rendered. If the output/log is important for testing, then using caplog might be the best choice. That said, I’d rather you put that logic somewhere callable… and test the callable, rather than testing against output.

def do_some_logic():
    return "something"
    
def test_do_some_logic():
    res = do_some_logic()
    assert res == "something", "Logic Failed!"

In your controller,

res = do_some_logic()
print(res)

Your unit tests should be testing the logic, not the output of print() or app.log().

[TomFreudenberg]

Hi @derks

Thanks for the example - I tried similar already moments ago - seems to be handy.

So for now - there are enough options for writing tests 😃

Cheers Tom