Warning for using Sentry SDK with the FTrack API

[PeterH]

I just spent hours yesterday digging into an issue, thought I'd share the findings if anyone comes across the same thing in the future.

One bit of code was consistently getting 502 FTrack server errors, but it had minimal calls to FTrack and was failing due to an unrelated reason, which was obviously quite weird.

The short version is I found out that when an exception occurs, Sentry (which we're using for error reporting across the studio) will serialise all the local variables in the stack frames, even if not directly related to the exception. Because FTrack entities are dict-like, Sentry converts them to a dict, which in turn hammers the API while it makes hundreds of requests to load all the values and relationships, causing the server to overload.

To highlight just how easy it is to cause the problem, the below code just sent 130 queries and caused 3x 502 errors in the process.

def func():
    entity = FTrackQuery().query('Task').first()
    logger.exception('test')
func()

To fix, there's an event "scrubber" that runs before the serialisation, where it's possible to intercept and clear out any FTrack entities. I'd really recommend anyone running Sentry to do this.

class FTrackScrubber(sentry_sdk.scrubber.EventScrubber):
    def scrub_event(self, event):
        import ftrack_api
        for frame in sentry_sdk.utils.iter_event_frames(event):
            for k, v in dict(frame.get('vars', {}).items()):
                if isinstance(v, ftrack_api.entity.base.Entity):
                    del frame['vars'][k]
        return super(FTrackScrubber, self).scrub_event(event)

sentry_sdk.init(
    ...
    send_default_pii=False,
    event_scrubber=FTrackScrubber(),
)

Alternatively a message for the FTrack devs: Sentry checks first for a __sentry_repr__ method, if this was added to the Entity class then it'd prevent the issue from occurring.