dev environment vs prod environment (questions, issues)

[Brad]

TLDR:

1. it seems ftrack_connect's base plugin directory is hardcoded and cannot be fully repointed on windows.  Separate ftrack_connect installs always share it.
2. It seems the live SaaS ftrack server is a live network service and is (necessarily) shared between production and development.  That's dangerous to code with.

Explanation:

I'm trying to both use FTrack and also do development with it in parallel.  While I may be a 'one-man-shop' this is actually exactly how I'd expect any major facility to use it.  They'll have an existing release or production pipeline that's in use, using FTrack.  And also, they'll have a pipeline or development team that's working on enhancing that pipeline at the same time.  Hopefully in various development sandboxes.

In an ideal situation, a development track is a branch from production.  (yes, I know of plenty of facilities that do all their pipeline dev live in their production environment.  but I'd like to talk about the more mature and refined approach here rather than barbaric practices.)

I am currently hitting two major roadblocks to keeping separate "production" (prod) and "development" (dev) environments for ftrack and ftrack_connect.  I'd like to describe my current setup and why those two issues (listed above) are causing problems.  And further if anyone has feedback or suggestions to avoid these issues I'd love to hear them.  I've only been digging into FTrack for about a week at this point and I may be unfamiliar with existing, better solutions.

My setup:

The root of my setup is currently anaconda (minicoda specifically) Miniconda — Conda documentation

The reason I use conda is because it allows me to pick a specific version of python and hide my system level python.  I had issues installing ftrack_connect from the repo (ftrack / ftrack-connect — Bitbucket) because of pyqt version dependencies.  And I found it better to just discern which version of Python you are using for your Windows installable and start with that version of python.  This is more common a problem with python development than you'd think.  Also, anaconda is supported by PyCharm, my primary development IDE for python.  This is something that simple python virtual environments (venv) don't help with.

Anyhow, I have an anaconda environment that I'll call "dev" and it's got the exact same version of python you are embedding in your binary installer.  I can then easily pull from bitbucket and install with "pip install -e ."  Upon completion of that, I can just run python -m ftrack_connect and run ftrack_connect.  All seems well.  I can even debug it by loading up the git repo as a project in pycharm and setting up the debugger to run the same module.  On it's surface, this is a great development environment both for ftrack_connect and pulgins.

Further, I could setup another anaconda environment that I'll call "prod" and do the same steps, but checkout specific versions of ftrack_connect and my own plugins (rather than dev HEAD, or feature branches) and I'd have parallel dev and prod installs of ftrack_connect that are easily entered via conda.

However, there are two problems with this.

Issue #1:

Both installs (env, prod) of ftrack_connect ALWAYS put their base plugins in the same directory.  This seems to be because they're using the 'appdirs' package to set their initial plugin path:

appdirs · PyPI

It's true that you can specify a plugin path via the environment:

Plugin directory — ftrack connect Unknown version documentation

However, I'd point out the documentation is pedantically accurate here when it says: "you can add additional places where"

If I set up different plugin directories for "dev" and "prod" via the referenced environment variable, it doesn't change anything about the usage of the shared primary plugins directory (under my home_dir/AppData/Local/ftrack/...).  Even though both "installs" of ftrack_connect are different versions of the code with different ENV vars for the plugins dir.

What I need here is an "override plugins dir" ENV var, not an "additional plugins dir" ENV var.  That way each environment would stand alone.

The best workaround I can come up with at the moment is setting up a separate user on my workstation.  Because that user will have a different home_dir and hence a different AppData dir.  This is a terrible idea.

Issue #2:

Developing network services is hard.  No doubt.  Mostly because the development environment to safely branch or spin up a development network services infrastructure is often non-trivial.

Even if I fully solve Issue #1, I'm stuck with my development code running against a production network service.  If I make a mistake, I can corrupt the live production data ridiculously easily.

I am of course aware of the possibility of moving to local installs of FTrack and maintaining a separate development server.  I used to run pipeline development at a company that had such a solution using a similar product (that looms large and shall go un-named).  But c'mon: I'm a 'one-man-shop' here.  And further, since ftrack_connect is largely an open source code-base, you'd want consultants and freelance TDs; to be able to maintain similar small setups with your SaaS service, and still be able to develop responsibly in sandboxed network service spaces.

I think what is needed is a simple "clone server to dev instance" and "destroy dev instance server" solution here, for your SaaS customers.  I haven't looked too deeply into how your local installs work, but I've seen enough to know that this should be possible.

As far as exploitation goes:  the worst scenario is that someone decides to spin up their dev instance and use it as a prod instance, hence, doubling their license count.  But that'll bite them eventually.  Limiting it to a single "dev" clone spin-up at "dev.<mycompany>.ftrackapp.com" should be limited enough and will promote healthy development practices for all clients, at all sizes.  You can even expand that to charging for additional simultaneous dev spin-ups, for companies with large development teams.  That's my 2 cents anyway.

Thanks for taking a look.  Any better solutions or ideas are welcome.  Also, if I'm seriously misunderstanding how ftrack and ftrack_connect are working/meant-to-work here, please advise (kindly).  It's always possible I'm way off base.

[Patrick Boucher ftrack]

Hi Brad,

I've reached out to colleagues regarding #1. I'll chime in as soon as I know more.

Regarding #2, having a dev or staging server is part of our Enterprise offering. Enterprise doesn't necessarily mean on-prem/local install. Those two are distinct. We also have clients without a dev server that build capabilities into their toolsets such that some environments can only read/write to specific projects. This method isn't as neat a separation as separate servers, but with Global API keys constrained to particular projects, making sure tools in development can only access a single project via a constrained key has a low barrier to entry.

Cheers,
Patrick

[Lorenzo Angeli]

Hi @Brad

as you have found out already , ftrack connect provides a default local path for plugins to be installed into, which is defined through the appdir module.
There are ways though to override that path without having to touch the source code of connect.
The easier and most suggested is to override the FTRACK_CONNECT_PLUGIN_PATH (for connect plugins) and FTRACK_EVENT_PLUGIN_PATH (for api hooks).

These environment are used usually to centralise the plugins across the studio so won't have to live on local user's machines.
You can find more information on the connect documentation as well as on the api one.

If you want more informations or you have further issues, please feel free to open a ticket with support !
Hope it helps.

L.