While AWS does provide one of the most advanced cloud object storage platforms available today – its S3 API is the industry standard adopted by CloudFlare R2 or Seagate Lyve – if you need to access a substantial amount of its data in non-AWS clouds, its egress transfer fees are prohibitively expensive.
For example, most high-frequency hedge funds operate their back-testing, analytics, and back office in AWS / GCP. For real-time trading, they run their infrastructure usually at Equinix, because of high egress transfer and storage fees and because of extra latency introduced by any public cloud.
Seagate Lyve
Seagate Lyve is an object storage managed service compatible with AWS S3 API with no egress fees priced at about 50% of the AWS S3 Standard Storage class. The service started earlier in 2022 and expands its functionality rapidly.
For hedge funds, Seagate Lyve is an ideal product since most of your data needs to be accessed every day, so you would not benefit from any other S3 storage class than the S3 Standard storage class which is the default class offered by Seagate Lyve.
Seagate Lyve for Startups
We at Oquant.com are an immensely proud member of the Seagate Lyve for Startups program allowing us to experiment with the best possible architectures for storing and accessing market data on a scale unimaginable for most firms and to closely collaborate with Seagate on functionality critical for the finance industry.
Migration of Data to Seagate Lyve
Most public clouds, such as AWS or GCP, charge no fees for data coming into their clouds but do charge for data coming out of their clouds.
For example, for a usual hedge fund dataset of 1 PB (=1,000 TB) of data, AWS would charge the data transfer fee of 50,000 USD (= 0.05 USD * 1,000 * 1000). This fee can further be reduced by using AWS Direct Connect, AWS Volume Discounts or AWS EDP.
There are various firms who offer more cost-effective migration of data to Seagate Lyve, such as EfficientCloud.
Migration of Data from Google Cloud Storage to Seagate Lyve using Rclone
For smaller migrations, we recommend using Rclone – a tool supporting migration of data across most cloud storage technologies.
The migration steps are these:
Step #1 – Obtain Seagate Lyve’s Access Keys
In the Seagate Lyve console, first, create the target bucket:
Then, create a permission set for the bucket:
Finally, create the service account for the permission set which provides the S3-like access keys:
Step #2 – Install Rclone
Follow the provided instructions and run the download / installation script:
sudo -v ; curl https://rclone.org/install.sh | sudo bashStep #3 – Set up Rclone’s data sources for Seagate Lyve and Google Cloud Storage
Follow the instructions for Seagate Lyve and for Google Cloud Storage.
Step #4 – Copy the files from Google Cloud Storage to Seagate Lyve
We recommend using the Rclone’s Copy command.
Its syntax is
rclone copy source:sourcepath dest:destpath --progressStep #5 – Verify the Migration has been successful
Set up AWS CLI with your Lyve account (How to use AWS CLI for accessing Seagate Lyve) and use standard AWS S3 CLI to verify the contents of the Lyve Bucket.
- Getting the list of objects in the bucket:
aws s3 ls s3://destpath --profile adminuser --endpoint-url=https://s3.us-east-1.lyvecloud.seagate.com- Downloading the entire bucket to the local file system
aws s3 sync s3://destpath --profile adminuser --endpoint-url=https://s3.us-east-1.lyvecloud.seagate.comUI Tools for Accessing Lyve Buckets
Most tools providing UI access to AWS S3 using AWS S3 API should work without any problems with Seagate Lyve.
We have evaluated S3 Browser and it works perfectly. Its setup is straightforward:
