A couple days ago I was reading questions on a tech website, in one of the topics someone explained an issue that they needed to copy from an SFTP location files in a cron job. They tried using rsync, however SFTP does not support interactive sessions which rsync sets up. As a solution I mentioned rclone. This is such a great tool that has been in my arsenal for quite a while now and I thought it would be good to give a shoutout to this tool.
Years ago I was building a Container platform, it used an Openstack Shift cluster in its testing phase as the S3 object store. One day all the containers restarted and kept restarting, after investigation we found out that someone pulled the uplinks from the Openstack Swift cluster. This was the moment I decided to mirror the most vital objects inside the cluster till the storage cluster became production ready. In the solution I used rclone and a couple lines of Golang to expose these objects in a simple web server. With this I had faster container startup times and the guarantee that even when uplinks fail the cluster can continue performing its task.
Rclone is a tool to interface with storage backends, it is built in Golang. The result is a single binary file you can copy over to the target machine, you do not need to install any requirements to get it to work. Rclone offers you the option to copy from remote storage backends to locally or reverse. In a configuration file you can specify your parameters, once you have made this configuration file you can copy this over to any machine. This makes rclone really an easy tool to use in unattended setups like containers or when using it in a cron job.
You can create your configuration by:
rclone config
When we want to copy over all the objects from the S3 bucket to our local drive we can now use the configuration file:
./rclone --config=./rclone.conf copy S3:my_s3_bucket /my/location/
Rclone works with storage providers like SFTP, S3, Google Cloud Storage, Microsoft Blob storage and tons more. It is easy to use and offers tons of features.
By generating the configuration file, you can now transfer the configuration file and rclone binary anywhere: On a machine, in a container, wherever. And use it to copy the data you wanted to the location you desire.
Give it a try!