File linking location

[tokejepsen]

Hey,

I have been looking into using file linking instead of copying files, and with that how to use locations for it and wanted to ask for some guidance.

I have a working prototype, but it would be nice to get some input on whether I'm tackling it correctly.

I'm using a location mixin similar to how the UnmanageLocationMixin works, but utilizing an external file linking module; https://github.com/tokejepsen/filelink

 

class FilelinkLocationMixin(object):
    '''Location that does not manage data.'''

    def _add_data(self, component, resource_identifier, source):
        '''Manage transfer of *component* data from *source*.

        *resource_identifier* specifies the identifier to use with this
        locations accessor.

        Overridden to have no effect.

        '''

        output = os.path.abspath(
            os.path.join(self.accessor.prefix, resource_identifier)
        )

        if not os.path.exists(os.path.dirname(output)):
            os.makedirs(os.path.dirname(output))

        filelink.create(
            source.get_resource_identifier(component), output
        )

Initially I thought I needed to look into making a custom accessor, but most of the handling of the data is the same as the disk accessor except for the writing of the data.

[Martin Pengelly-Phillips]

A custom accessor may be more suitable, especially if you need to consider the other valid operations, such as removing a component from a location (and therefore the data / link). You might also want special handling of operations such as `list`, `exists` so that unresolving links are not included etc..

[tokejepsen]

Thanks @Martin Pengelly-Phillips

That is some good points!

In my current use case, I'm only interested in getting a hardlink instead of a copy. But definitely something to look into.