Backlog processing

[tokejepsen]

I've made an ftrack event processor, which loads plugins from a folder and processes events. Its based on the eventhub, so nothing new just a simpler collected workflow.

 

I wanted to add the ability to process backlog events, when/if the processor is offline. Found this; http://ftrack.rtd.ftrack.com/en/latest/developing/api_reference/index.html?highlight=events#ftrack.getActivityEvents, but the documentation has left me in the dark. Could I get an example of getting a backlog from a certain time?

[Mattias Lagergren]

Yes the ftrack.getActivityEvents can be used to retrieve events from a certain project after a date or using the incremental id.

 

# This will give you events a maximum of 100 from a project, after a datetime.

events = ftrack.getActivityEvents(<project-id>, fromDate=<datetime>, limit=100)

 

# Process events

 

Please let us know if this doesn't work and if there are any issues using that method.

[tokejepsen]

yeah, the events I get back are quite different from the events when using the EventHub. They all seem to be social events with no entities iterate over. Example:

<social_event('{'insert': 'update', 'socialid': 299536, 'object': None, 'userid': 'b0fdb9ba-b037-11e4-ac9f-040112b6a801', 'created_at': <DateTime '20150213T16:55:46' at 361dd88>, 'entityType': 'socialevent', 'parent_type': 'task', 'showid': '26d1ccf6-b07b-11e4-b40c-040112b6a801', 'parent_id': '8e9464b6-b39b-11e4-9497-040112b6a801', 'rabbit': True, 'action': 'change.status.shot', 'entityId': 299536, 'data': {'statusid': {'new': '123b9f74-aaea-11e4-b051-040112b6a801', 'old': '1639e788-aaeb-11e4-8b45-040112b6a801'}}, 'id': '27d15ff8-b3a1-11e4-a321-040112b6a801'}')>

How would I convert this to a more "understandable" event that we know from the EventHub?

[tokejepsen]

Any update on this?

[Mattias Lagergren]

Yeah, that is right the the "social" / "activity" event is a bit different from the one you get from the event hub. 

 

For an event you can do:

 

event.getData() to get the information on what was changed.

event.getEntity() to get the task / shot / project that the changes were made on.

[tokejepsen]

Any way of converting an ActivityEvent to an Event?

 

It makes it difficult to have to make a plugin/hook to account for ActivityEvent as input and Event as input.

[Mattias Lagergren]

Unfortunately we don't have an easy way to convert one of the events to the other, even though they sometime might have the same information. 

[tokejepsen]

So there is no help to get on this?

 

The more we rely on the event system, the more we will need some backlog processing, or some sort of backup for when the machine that is running the scripts goes down. Any suggestions for backups?

[Mattias Lagergren]

Looking forward it would be interesting to look into having a recorded history of the events at our side. But for now I would recommend sticking to the event system and monitor the machine the script is running on and take steps to make sure it stays up. You could even set up something at your end that puts the events in to a persisted queue for further processing so that the event listener is very minimalistic and robust.

[tokejepsen]

 

 

You could even set up something at your end that puts the events in to a persisted queue for further processing so that the event listener is very minimalistic and robust.

 

Thats a good idea:) I'll look into this, thanks!

[tokejepsen]

So I ended up rethinking the whole setup we have here about relying on a single machine. Backlog processing is still a bit of concern but the main problem was that we didn't have any measure for when the machine goes down, and nobody notices before things go wrong and lots of events have been "lost".

The solution was to put the script onto Deadline, which handles if the machine goes down. When the machine goes down or the script fails for some reason, it'll switch machine and get the script up and running fairly fast.