development Simplify plugins in Connect

[Mattias Lagergren]

Hi everyone! 

I'm writing here to let you know that we are working on some changes to make Connect easier to extend and develop plugins for.  Some of you I've already been in contact with but I would love to hear the feedback and opinions from the Community.

We have a lot of ideas that we want to implement in Connect but the plan is to start out by tackling two common issues:

1. Extending and modifying the environment and command when an application starts from Connect.
2. Adding hooks to Connect without modifying the FTRACK_EVENT_PLUGIN_PATH.

The way we intend to solve this is:

1. Emit an event every time an application is launched from Connect. The event will contain application identifier, context, command and options that contain the environment variables. Here you will be able to make changes to environment variables, command and options. This should allow a lot of flexibility without having to override the built-in hook it-self.
2. A platform dependent default directory from where plugins can be discovered without modifications to FTRACK_EVENT_PLUGIN_PATH. E.g. on OS X it would be something like /Users/mattiaslagergren/Library/Application Support/ftrack-connect-plugins/. We also intend to give the option set an environment variable so that you can centralise the plugins to one directory that everyone in the studio reads from. Connect will pick up plugins from both places.

[Mattias Lagergren]

The feature I mentioned above is released and you can find more information here: 

http://ftrack-connect.rtd.ftrack.com/en/0.1.21/using/plugin_directory.html

http://ftrack-connect.rtd.ftrack.com/en/0.1.21/developing/plugins.html

http://ftrack-connect.rtd.ftrack.com/en/0.1.21/developing/hooks/application_launch.html#ftrack-connect-application-launch

 

You can reply here with questions or feedback!

[Milan Kolar]

This is getting very close to a simple a straightforward system. I have however ran into some issues.

The most prominent right now is the fact that I have no way of finding out what task/entity was the source of this launching event. It's not included in the event data and because it seems that this event precedes the ftrack addition of env variables the environment is missing the 'FTRACK_TASKID' key too.  It's important to be able to tell what originated the launch event because the enviro usually changes based on what project or task we're launching (different plugins for example.

 

EDIT:

Ah. One of those where you completely skip on part of docs and then realise it's of course all there. If anyone else enjoys skipping important parts of docs, then here's how you get it.

event['data']['context']['selection'][0]['entityId']

So considering I found it...Nice work guys.

[Mattias Lagergren]

Glad to hear that it worked out! Please let me know if you find anything that is complicated, confusing or not well documented! 

[Milan Kolar]

So I had some time to play with this a little bit and came up with compact and quite efficient way of dealing with environments using launching events. Technically all that is needed is one hook that listens to the launching events and appends environments from simple .json config files based on app identifiers.

The hook looks for .json files in 'FTRACK_APP_ENVIRONMENTS' and tries to find 2 configs. One that is version independent i.e. 'maya.json' and one that is version dependent i.e. 'maya_2016.json', then it adds both into the environment. Config file names mus match the app identifier or the first part of the identifier when split by '_'.

You can then have a hierarchical environment config files, where maya.json loads regardless of maya version you launch, and maya_2016.json gets added on top of the environment when you launch Maya 2016 from ftrack.

'FTRACK_APP_ENVIRONMENTS' folder can then look like this for instance:

• maya.json
• maya_2015.json
• maya_2016.json
• maya_2016.5.json
• nuke.json
• nuke_9.0v8.json
• houdini.json
• houdini_15.json
• etc....

 

This is the core part of the hook (full hook attached):

def modify_application_launch(event):
    '''Modify the application environment and start timer for the task.'''
    data = event['data']

    context = data['context']
    app = data['application']
    environment = data['options']['env']

    env_path = os.environ.get('FTRACK_APP_ENVIRONMENTS')

    env_files = []

    # determine config file for version independent environment
    app_name = app['identifier'].split('_')[0]
    app_file_name = '{}.json'.format(app_name)
    env_files.append(os.path.join(env_path, app_file_name))

    # determine config file for version dependent environment
    variant_file_name = '{}.json'.format(app['identifier'])
    env_files.append(os.path.join(env_path, variant_file_name))

    env_add = []

    # loop through config files
    for env_file in env_files:
        try:
            env_add = load_env(env_file)
        except:
            env_add = None

        # Add each path in config file to the environment
        if env_add:
            for variable in env_add:
                for path in env_add[variable]:
                    ftrack_connect.application.appendPath(
                        str(path),
                        str(variable),
                        environment
                    )

 

This is what the config file look like then:

{
    "MAYA_PLUG_IN_PATH": [
        "K:\\.core\\dev\\maya\\plug-ins"
    ], 
    "MAYA_AUTOSAVE_FOLDER": [
        "C:\\mayatemp\\autosave"
    ], 
    "MAYA_SCRIPT_PATH": [
        "K:\\.core\\dev\\maya\\scripts", 
        "K:\\.core\\dev\\maya\\shelves", 
        "K:\\.core\\repos\\maya\\scripts", 
        "K:\\.core\\repos\\maya\\prefs\\shelves"
    ]
}

 

Now quite frankly, I think that a simple hook like this (probably a bit more robust version) should be a part of the default connect installation. Straight away people would have 3 options of modifying the app launches. 

1. Only need to modify environment: Add your json formatted environment file to 'FTRACK_APP_ENVIRONMENTS' and name it 'app_identifier.json' (e.g.: maya_2016.json)
2. Write your own hook that catches launching events and modify the launch that way.
3. Change the launcher hook in connect and modify to your heart's content

 

One way or another, our enviro setups just went from lot's of python file, to a simple json file per app without the need to tinker with default hooks. Thumbs up.

modify_app_launch.py

[Milan Kolar]

Another useful hook with this. Start timer automatically with app launch. Now we need to figure out how to stop it automatically and there will be some happy producers walking around here.

import logging
import ftrack
import ftrack_api

logger = logging.getLogger()

def start_time_on_launch(event):
    '''Modify the application environment and start timer for the task.'''
    data = event['data']

    username = event['source']['user']['username']

    session = ftrack_api.Session()
    # Get user from username
    user = session.query('User where username is "{}"'.format(username)).one()
    # Try getting taskid from event selection
    try:
        taskid = data['context']['selection'][0]['entityId']
    except:
        logger.info('Unable to determine task. Timer not starting')

    if taskid:
        task = session.query('Task where id is {}'.format(taskid)).one()
        logger.info('Starting timer for task: ' + task['name'])
        user.start_timer(task, force=True)


def register(registry, **kw):
    '''Register location plugin.'''

    # Validate that registry is the correct ftrack.Registry. If not,
    # assume that register is being called with another purpose or from a
    # new or incompatible API and return without doing anything.
    if registry is not ftrack.EVENT_HANDLERS:
        # Exit to avoid registering this plugin again.
        return

    ftrack.EVENT_HUB.subscribe(
        'topic=ftrack.connect.application.launch',
        start_time_on_launch
    )

 

[tokejepsen]

Maybe the nice people at Ftrack could emit a signal when the process has terminated?

[tokejepsen]

I have been playing around with this now, and I've got some feedback.

In the example; http://ftrack-connect.rtd.ftrack.com/en/0.1.21/developing/tutorial/adding_a_location.html#developing-tutorial-adding-a-location-modifying-application-launch, you use the "appendPath" method from ftrack_connect.application, but shouldn't this be returning the launch data instead?

I've tried getting the "launchArguments" working, but I can't. It seems to me that you aren't using the returned launch arguments here; https://bitbucket.org/ftrack/ftrack-connect/src/06eaec219026c9b973f9031040d0cc0199c05f4c/source/ftrack_connect/application.py?at=master&fileviewer=file-view-default#application.py-337

Lastly I was wondering whether you could publish an event that has the process pid, so we could monitor it? https://bitbucket.org/ftrack/ftrack-connect/src/06eaec219026c9b973f9031040d0cc0199c05f4c/source/ftrack_connect/application.py?at=master&fileviewer=file-view-default#application.py-359

[tokejepsen]

Also you suggest in; http://ftrack-connect.rtd.ftrack.com/en/0.1.21/using/plugin_directory.html, to use FTRACK_CONNECT_PLUGIN_PATH, but I can't get that to work. I'm currently using FTRACK_EVENT_PLUGIN_PATH. Which is the correct one?

[Mattias Lagergren]

20 hours ago, tokejepsen said:

In the example; http://ftrack-connect.rtd.ftrack.com/en/0.1.21/developing/tutorial/adding_a_location.html#developing-tutorial-adding-a-location-modifying-application-launch, you use the "appendPath" method from ftrack_connect.application, but shouldn't this be returning the launch data instead?

I think modifying it should be fine since options it will then be used when doing is used with subprocess.Popen

20 hours ago, tokejepsen said:

I've tried getting the "launchArguments" working, but I can't. It seems to me that you aren't using the returned launch arguments here; https://bitbucket.org/ftrack/ftrack-connect/src/06eaec219026c9b973f9031040d0cc0199c05f4c/source/ftrack_connect/application.py?at=master&fileviewer=file-view-default#application.py-337

The application and context is just sent for reference, the command and options can be modified to tweak what is launched.

20 hours ago, tokejepsen said:

Lastly I was wondering whether you could publish an event that has the process pid, so we could monitor it? https://bitbucket.org/ftrack/ftrack-connect/src/06eaec219026c9b973f9031040d0cc0199c05f4c/source/ftrack_connect/application.py?at=master&fileviewer=file-view-default#application.py-359

If you've got the time it would be very nice if you could fork and create a pull-request for this. A ftrack.connect.application.after-launch hook, with the pid and some meta data. If you do this it would be great if you could add some documentation about the event as well! :-)

[Mattias Lagergren]

20 hours ago, tokejepsen said:

Also you suggest in; http://ftrack-connect.rtd.ftrack.com/en/0.1.21/using/plugin_directory.html, to use FTRACK_CONNECT_PLUGIN_PATH, but I can't get that to work. I'm currently using FTRACK_EVENT_PLUGIN_PATH. Which is the correct one?

I think you should prefer using FTRACK_CONNECT_PLUGIN_PATH whenever possible. In what way doesn't it work, nothing happens? Be careful to structure the plugins correctly, see this article: http://ftrack-connect.rtd.ftrack.com/en/latest/developing/plugins.html#plugins

 

 

[tokejepsen]

26 minutes ago, Mattias Lagergren said:

I think you should prefer using FTRACK_CONNECT_PLUGIN_PATH whenever possible. In what way doesn't it work, nothing happens? Be careful to structure the plugins correctly, see this article: http://ftrack-connect.rtd.ftrack.com/en/latest/developing/plugins.html#plugins

 

 

Is there any problems with using FTRACK_EVENT_PLUGIN_PATH? Its pretty convenient to have a flat hierarchy of files for all the custom hooks.

[Mattias Lagergren]

25 minutes ago, tokejepsen said:

Is there any problems with using FTRACK_EVENT_PLUGIN_PATH? Its pretty convenient to have a flat hierarchy of files for all the custom hooks.

No problems aside from those you already handle; that it overrides all built-in event plugin hooks if you're not careful.

The FTRACK_CONNECT_PLUGIN_PATH and default plugin folder is meant to be a standardised way of sharing plugins. The nesting of <root-folder>/<plugin-folder>/hook/ is there to allow separation of plugins from different vendors, and the hook/ folder is there since Connect walks the folder structure down looking for any .py/.pyc file that has a register function. So if you have complex integrations with their own hooks you do not want to source those (let me know if you want me to elaborate on this).

If you prefer FTRACK_EVENT_PLUGIN_PATH that is still fine, if you want to use FTRACK_CONNECT_PLUGIN_PATH you can still do sort of a flat structure and put all hooks in one dir: <root-folder>/my_studio/hook/

[tokejepsen]

I have to say that this update to the connect system is a pure blessing! Thanks guys for this.

It very pleasant to have all the launching of the applications separate from the scene file and environment logic.

[Mattias Lagergren]

On 5/4/2016 at 0:03 PM, Milan Kolar said:

Now quite frankly, I think that a simple hook like this (probably a bit more robust version) should be a part of the default connect installation. Straight away people would have 3 options of modifying the app launches. 

1. Only need to modify environment: Add your json formatted environment file to 'FTRACK_APP_ENVIRONMENTS' and name it 'app_identifier.json' (e.g.: maya_2016.json)
2. Write your own hook that catches launching events and modify the launch that way.
3. Change the launcher hook in connect and modify to your heart's content

 

One way or another, our enviro setups just went from lot's of python file, to a simple json file per app without the need to tinker with default hooks. Thumbs up.

modify_app_launch.py

Thank you for sharing this Milan, looks very useful! At the moment we haven't made any decisions on how far we want to go with Connect when it comes to these things. One way could be to not including it, and rather linking to a solution like yours, and have it as an officially promoted plugin.

Again, very nice to see what you've built on top of this!

[Mattias Lagergren]

On 5/20/2016 at 0:17 PM, tokejepsen said:

I have to say that this update to the connect system is a pure blessing! Thanks guys for this.

It very pleasant to have all the launching of the applications separate from the scene file and environment logic.

Thank you, I'm very glad to hear this!

[tokejepsen]

When listening to the event topic "ftrack.connect.application.launch", I can't seem to display any UI elements.

Is this supposed to work? Maybe you could provide an example?

[Mattias Lagergren]

It is primarily meant for modifying the environment where you launch the application. Here is an example:

http://ftrack-connect.rtd.ftrack.com/en/stable/developing/tutorial/adding_a_location.html#developing-tutorial-adding-a-location-modifying-application-launch

[tokejepsen]

That's cool.

I working around the UI problem with just having a PySide dialog for user input instead.