🖥️Archive Solution Recipe
Intro: Effortless Data Archiving with the DeStor REST API for Filecoin: Your Recipe for Simplified Archive Storage.
Document your Archive requirements
What workload are you protecting?
Understanding the specific workload you're safeguarding is crucial. Knowing the type and importance of the data involved ensures tailored protection strategies.
What are your RPO / RTO requirements?
Defining your Recovery Point Objective (RPO) and Recovery Time Objective (RTO) helps set expectations for data recovery speed and accuracy.
Do you have any compliance requirements?
Be aware of any regulatory or industry compliance requirements. Meeting these standards is vital for data integrity and legal adherence.
How long do you need to retain your archive data?
Determining how long you need to retain archived data helps design storage solutions that align with your long-term data management strategy.
Architecture
Components
DeStor REST API: The DeStor REST API for Filecoin is a service to propel data onto the Filecoin network via a simple easy to use API. It aims to create an easy path for independent software vendors to integrate Filecoin as a storage layer.
Filecoin Storage Providers: The Filecoin network provides decentralized data storage and makes sure data is verified, always available, and immutable. Storage providers in the Filecoin network are in charge of storing, providing content and issuing new blocks.
Workflow
Storing Data (POST) Through a REST API:
Endpoint Identification: Identify the URL endpoint where you want to send your data. This depends on how the DeStor REST API is running in your infrastructure. See doc and identify, typically something like : http://localhost:50000
Authentication: No authentication is needed as this point because you are running this internally and it's not exposed to the public.
Data Preparation: Prepare the data you want to store in the appropriate format. This could be in JSON, XML, Blob, a stream or other supported data structures.
HTTP Request: Use a HTTP client (e.g., cURL, Postman, or a programming library like Python's requests) to send a POST request to the API endpoint. Include the data in the request.
Processing: Once your data is submitted through the DeStor REST API, it will generate a unique Blob ID, which serves as your data's digital fingerprint and reference point. Simultaneously, the DeStor REST API initiates it's internal orchestration to secure the storage of your data on the Filecoin network, leveraging the configuration parameters you've specified within the DeStor REST API interface. This ensures that your valuable information is meticulously archived and readily accessible whenever you need it through the DeStor REST API.
Retrieving Data (GET) Through a REST API:
Endpoint Identification: Identify the URL endpoint where you want to send your data. This depends on how the DeStor REST API is running in your infrastructure. See doc and identify, typically something like : http://localhost:50000
Authentication: No authentication is needed as this point because you are running this internally and it's not exposed to the public.
HTTP Request: Create a GET request using your HTTP client and specify the API endpoint.
Response Handling: Capture and process the response from the API. The response will typically include a status code and the requested data.
Data Parsing: Parse the response data into a format that your application can use (e.g., JSON, XML, or plain text).
Return status (STATUS) Through a REST API:
Endpoint Identification: Identify the URL endpoint where you want to send your data. This depends on how the DeStor REST API is running in your infrastructure. See doc and identify, typically something like : http://localhost:50000
Authentication: No authentication is needed as this point because you are running this internally and it's not exposed to the public.
HTTP Request: Create a STATUS request using your HTTP client and specify the API endpoint.
Response Handling: Capture and process the response from the API. The response will contain the following data:
200 - Data status successfully retrieved: Upon a successful request, you will receive a response in JSON format. The response includes essential information about the blob, including its id, as well as details about its replicas stored on the Filecoin network.
Replicas: This section provides an array of information about each replica associated with the blob. Each replica object includes the following details:
Provider: The unique identifier of the Filecoin storage provider responsible for this replica.
Status: The current status of the replica, which can be one of the following: "active," "slashed," or "expired."
Last Verified: The timestamp representing the last time this replica was verified, following the RFC 3339 date-time format.
Expiration: The timestamp indicating the expiration time of the blob's storage, also adhering to the RFC 3339 date-time format.
Remember to refer to the specific API documentation for any unique requirements or parameters, as APIs can vary significantly in their implementation. Additionally, consider implementing best practices for security, rate limiting, and API versioning as necessary for your use case.
Storage Provider Selection
When it comes to choosing a storage provider for your data, the decision is far from straightforward. In today's landscape, where data is the lifeblood of organizations, the performance of your chosen provider can make or break your operations. With an array of options at your disposal, selecting the ideal storage partner requires careful consideration.
Storage and Retrieval Evaluating both storage and retrieval performance is essential. Can the provider store your data reliably, and can it retrieve it swiftly when needed? These are critical questions.
Geographic Flexibility: What performs well in one location may not excel in another. The flexibility to choose testing locations aligns with real-world scenarios.
Data Segmentation: Segmentation of data centers provides performance variations across different locations.
Parallel Connections: Your storage and retrieval requirements won't always involve a single connection. Parallel connections are the norm. Storage Provider selection should account for different scenarios, from single connections to multiples.
Varied Deal Sizes: Dealing with different file sizes is part of the real-world challenge. Your Storage Provider choice should cover a range of deal sizes to mimic the diversity of your data.
Provider Diversity: Flexibility in choosing from a configurable set of storage providers for storing data is crucial. It allows you to match your needs with the capabilities of the provider.
Real-time Monitoring: During data storage on the Filecoin network, having visibility into the progress and intermediate results is a must. This empowers administrators to make informed decisions and adjustments as needed.
Storage and Retrieval Metrics Understanding the intricacies of performance metrics is essential. Storage Providers differ in upload times, upload speeds, and sealing times. And Retrieval will encompass download times, time-to-first-byte, and bandwidth throughput.
In summary, the process of selecting the right storage provider is a multifaceted endeavor. Selecting Storage Providers, as outlined with these rigorous requirements, is the compass that guides you toward the provider best suited for your specific needs. It's a journey of exploration and validation, ensuring that your data finds a secure and efficient home, no matter where it resides in the digital landscape.
If you need help selecting your Storage Provider for your specific setup feel free to reach out through this form.
Access Control
Secure access to the DeStor REST API
Securing a containerized environment involves implementing access control mechanisms both at the host level and within the infrastructure. Here are some ways to enhance security:
1. Host-Level Access Control:
Use User Namespaces: Docker supports user namespaces, which can isolate container processes from the host. Run containers with mapped user IDs (UIDs) and group IDs (GIDs) that are different from those on the host, reducing the impact of potential privilege escalation attacks.
docker run --userns=host ...Limit Container Capabilities: Docker containers inherit a subset of capabilities from the host. You can drop unnecessary capabilities to reduce the container's attack surface.
docker run --cap-drop ...Implement AppArmor or SELinux Profiles: AppArmor (on Ubuntu) or SELinux (on Red Hat-based systems) can enforce mandatory access controls on container processes, limiting their actions on the host system.
Control Resource Usage: Use resource constraints and limits to prevent containers from consuming excessive CPU, memory, or network resources on the host.
Seccomp Profiles: Seccomp (Secure Computing Mode) profiles allow you to restrict system calls that containers can make, reducing the attack surface.
Limit Container Access to Host Devices: Avoid granting containers access to host devices unless absolutely necessary, and use the
--deviceflag sparingly.
2. Infrastructure-Level Access Control:
Network Segmentation: Place containers on isolated networks, separating them from critical infrastructure components. Docker supports custom bridge networks, which can be used to isolate containers.
Firewalls and Network Policies: Implement network-level access control using firewalls and network policies to restrict container communication. Tools like Kubernetes Network Policies can be valuable in Kubernetes environments.
Container Registry Access Control: Secure the container image registry, allowing only authorized users to pull or push images. Use authentication and authorization mechanisms provided by the container registry platform.
Secrets Management: Implement secrets management solutions (e.g., Docker Secrets, Kubernetes Secrets) to securely store and access sensitive information such as API keys, passwords, or certificates.
Role-Based Access Control (RBAC): In Kubernetes clusters, use RBAC to control who can create, modify, or delete pods and other resources.
Security Scanning: Utilize container image scanning tools to identify vulnerabilities in your container images before deployment.
Runtime Monitoring: Employ container security solutions that provide real-time monitoring and anomaly detection to identify suspicious activities within containers.
Update Regularly: Keep the host OS, Docker daemon, and container images up to date with security patches and updates.
Least Privilege Principle: Apply the principle of least privilege, ensuring that containers have only the necessary permissions and access required for their intended functionality.
Audit and Logging: Enable auditing and comprehensive logging to monitor container activities, detect security incidents, and investigate any breaches.
Secure access from the DeStor REST API as a Storage Provider
Network layer
Security with the DeStor REST API Hosted Infrastructure: Allowing Inbound Connections for Storage Deals
Network layer security that permits inbound connections for storage deals from a DeStor REST API hosted infrastructure is a specialized access control measure known as "whitelisting" or "IP filtering." This approach enhances the security of your network while ensuring that only trusted sources, such as the externally hosted DeStor REST API infrastructure, can initiate storage deals. Here's how it functions:
1. Whitelisting IP Addresses:
In this context, "whitelisting" entails creating a list of trusted IP addresses or ranges that are explicitly permitted to initiate inbound connections for storage deals.
2. The DeStor REST API Hosted Infrastructure:
The DeStor REST API, as a provider of data storage and retrieval services, operates a hosted infrastructure that facilitates the storage of data on the Filecoin network. This hosted infrastructure includes servers and endpoints essential for the DeStor REST API platform's functionality.
3. Restrictive Access:
When implementing this security measure, you specify the IP addresses or ranges associated with the DeStor REST API hosted infrastructure. These IPs are the only sources (including potential other clients) allowed to initiate inbound connections for storage deals. All other IP addresses are denied access by default.
4. Benefits:
Enhanced Security: By allowing only trusted IP addresses associated with the DeStor REST API's infrastructure, you significantly reduce the risk of unauthorized access, bolstering the security of your data storage processes.
Protection for Storage Deals: This measure safeguards the integrity and confidentiality of your storage deals by ensuring they originate only from a reliable source.
Mitigating Security Risks: It mitigates security risks that may arise from unauthorized or unverified sources attempting to interact with your storage infrastructure.
5. Considerations:
Logging and Monitoring: Implement robust logging and monitoring to track inbound connections initiated by the DeStor REST API infrastructure and to detect and respond to any unusual or potentially harmful activity.
Emergency Access: Have contingency plans in place to allow for emergency access or updates to the whitelisted IP addresses in case of infrastructure changes or outages.
In summary, allowing inbound connections for storage deals from the DeStor REST API hosted infrastructure via whitelisting is a highly effective and secure approach to safeguarding your data. By focusing on trusted sources and maintaining strict access controls, you can confidently protect your incoming storage deals and maintain the integrity of your data coming in.
The downside when using IP whitelisting is that you will have to setup a separate miner in most cases.
Application layer:
If you want to provide client based access control based on a wallet address originating from a DeStor REST API hosted infrastructure you can use;
CID Gravity, a powerful pricing and client management tool for Filecoin storage providers, which plays a crucial role in managing client wallet addresses to ensure that storage deals are only accepted from trusted sources, such as a hosted DeStor REST API infrastructure. Additionally, it enables storage providers to set up custom pricing based on specific client needs. Here's how CID Gravity achieves this:
1. Client Wallet Address Management:
Whitelist of Trusted Addresses: CID Gravity allows storage providers to maintain a whitelist of trusted client wallet addresses. This list includes the addresses associated with clients who are authorized to use the storage services provided by the miner.
Authorization Process: When onboarding clients, the storage provider can validate and verify their wallet addresses. Only authorized addresses are added to the whitelist, ensuring that deals are accepted exclusively from known and trusted clients.
2. Deal Acceptance Control:
Filtering Incoming Deals: CID Gravity acts as a gatekeeper, intercepting incoming storage deal requests. It checks the sender's wallet address against the whitelist to determine if the client is authorized to make deals.
Deal Rejection: If a deal request originates from a wallet address that is not on the whitelist, CID Gravity rejects the request, preventing unauthorized clients from utilizing the storage services.
3. Custom Pricing and Client Management:
Tailored Pricing: CID Gravity offers the flexibility to set up custom pricing structures for different clients or client groups. Storage providers can define pricing tiers, discounts, or special rates to meet specific client requirements.
Client Segmentation: Storage providers can categorize clients based on their storage needs, data volume, or other criteria. CID Gravity allows the creation of client segments with unique pricing arrangements.
CID gravity works for graphsync and bitswap, but not for HTTP retrievals, read more about it in the "Storage Provider specific setup"
Client access control
Soon to come!
Remember that security is an ongoing process, and it's essential to regularly review and update your security measures to adapt to evolving threats and vulnerabilities. Additionally, consider the specific requirements and constraints of your infrastructure and applications when implementing access control measures.
API Examples
Explore the DeStor REST API's capabilities with our comprehensive API Examples. Whether you're a seasoned developer or just starting, the DeStor REST API provides the tools you need to harness the power of Filecoin. Visit 'The DeStor REST API for developers' to dive into the world of decentralized storage and start building your next-generation applications today.
Storage Provider specific setup
For Storage Providers that want to work with ISVs (Independent Software Vendors), Integrators, and MSPs looking to support enterprise-level use cases for their clients through a Hosted DeStor REST API solution, it is crucial to set up and configure booster-http, a powerful tool that allows Storage Providers (SPs) to serve blocks and files over the HTTP protocol via their Boost Market's Node. This strategic setup, enabled with Release v1.7.0-rc1 of booster-http, unlocks a multitude of advantages:
Enhanced Enterprise Services:
By configuring booster-http, ISVs, Integrators, and MSPs can elevate their service offerings to enterprise clients. This enhanced service includes seamless access to blocks and files over HTTP, fast access to stored data on Filecoin, aligning with the demands of corporate environments.
Improved Accessibility:
Setting up booster-http means that clients can enjoy direct and simplified HTTP access to their critical data assets. This streamlined approach enhances accessibility and user experience, vital for enterprise-grade solutions.
In summary, setting up and configuring booster-http is a strategic move for Storage Providers that want to work with ISVs, Integrators, and MSPs aiming to provide enterprise-level solutions through a Hosted DeStor REST API platform. This setup enhances accessibility, integration capabilities, and overall service quality, positioning them as reliable partners for enterprise clients with diverse data management needs.
Deletion
Deletion on the Filecoin network is a process designed to ensure that data is removed from storage miners when it is no longer needed or desired by the client. The deletion process is a fundamental aspect of Filecoin's data management and economic model, emphasizing the importance of efficient use of storage resources and data removal.
Here's how deletion works on the Filecoin network:
Client Initiates Deletion:
The deletion process is initiated by the client through the ISV who originally stored the data on the Filecoin network. Clients should have control over their stored data and can decide when it should be deleted.
Deletion Proposal:
The ISV or integrator sends a "deletion proposal" to the Storage Provider where the data is stored. This proposal includes details about which data or piece(s) of data should be deleted. This is a manual process as there is no build in way of doing so as of today.
Miner Verification:
The Storage Provider receives the deletion proposal and verifies its authenticity. The Storage Provider ensures and verifies that the request is coming from the ISV in name of the client who owns the data and that the specified data exists on its storage. As this is a manual process the storage Provider will propose any cost associated with deal termination.
Deal Termination:
When the Storage Provider verifies the request, it proceeds to terminate the storage deal associated with the data to be deleted. This means that the Storage Provider will no longer be considered responsible for storing that piece of data.
Deal Settlement:
The Storage Provider settles the payment for the termination storage deal up to the point of deletion. The ISV or client is responsible for compensating the miner for the termination storage services rendered at that moment.
Cache
The cache layer, a critical component for optimizing data retrieval and access speed, is positioned outside of the DeStor REST API. Instead, it should be thoughtfully implemented within the platform that leverages the DeStor REST API for storing data on the Filecoin network. This strategic separation ensures efficient and tailored caching mechanisms, enhancing overall performance and user experience. Cache can be an S3 bucket, IPFS, or any local storage solution!
How could this look like from an architecture point of view:
Look at the S3 Connector / API section of this solution to potentially leverage Zenko Cloudserver as your caching layer.
Encryption
You can encrypt your data before storing it on the Filecoin network, here are some available tools and methods:
Encrypting Data for Secure Storage on Filecoin
Encrypting data before storing it on Filecoin is crucial to maintain the confidentiality and integrity of the stored information. You can employ various encryption techniques and tools to achieve this goal:
End-to-End Encryption: Implement end-to-end encryption within your application code. This ensures that data is encrypted before leaving the user's device and remains encrypted until it reaches its destination. Libraries like OpenSSL and libsodium can be used for this purpose.
Client-Side Encryption: Encrypt the data on the client-side before sending it to Filecoin. This approach allows users to retain control of their encryption keys. You can use encryption libraries like CryptoJS (for JavaScript-based applications) or the cryptography module in Python.
Command-Line Encryption Tools: You can also use command-line encryption tools like GPG (GNU Privacy Guard) to encrypt data before uploading it to Filecoin. GPG provides strong encryption and can be integrated into automated data upload processes.
When managing the above, remember to think about Secure Key Management: Implementing secure key management practices to protect encryption keys. Tools like HashiCorp Vault can help securely manage and store encryption keys.
Wallet
Wallets serve as secure repositories for Filecoin. These wallets comprise a pair of keys - a public key similarly to a bank account number and a private key similarly to a password.
When someone transfers cryptocurrency to your wallet address, the transaction is logged on the blockchain network, augmenting your wallet balance. Similarly, when you initiate a cryptocurrency transfer from your wallet to another, the transaction is documented on the blockchain network, reducing your wallet balance.
Diverse categories of cryptocurrency wallets exist, encompassing desktop, mobile, hardware, and web-based variations, each offering distinct attributes and levels of security. Opting for a reputable and secure wallet is imperative to guarantee the protection of your digital assets.
Read more about Wallets in the Filecoin docs!
If you are already running your own lotus node, you can also manage FIL wallets from the command line.
Data flow
Payment
Payment channels: where transactions flow faster than a racecar on a rocket! ⚡ Stay tuned for the keys to lightning-fast, explainer on secure payment transactions coming your way soon! 💰🏁
S3 Connector / API
Zenko CloudServer isn't a built in part of the DeStor REST API – it serves as a powerful demonstration of our capability to provide an S3 front-end layer on top of the DeStor REST API. Together, they showcase the versatility and integration potential of our solution.
The DeStor REST API S3 Connector creates a bridge between the Amazon S3 client libraries and the DeStor REST API. By leveraging this connector, users can seamlessly store and fetch data through the DeStor REST API, an API layer crafted to facilitate Filecoin storage layer integration. This connector utilizes the Zenko CloudServer - an Amazon S3-compatible storage server - and integrates a tailored the DeStor REST API client. This client is responsible for converting S3 requests to the DeStor REST API requests and vice versa.
Object versioning
Object versioning is a critical feature in object storage systems like Zenko CloudServer, as it allows you to maintain multiple versions of an object over time. This is particularly useful for data protection, data recovery, and maintaining a historical record of changes to your objects. Here's how object versioning works with Zenko CloudServer:
Object Versioning with Zenko CloudServer
Enable Versioning: Activate versioning for a specific bucket.
Upload Objects: Uploading an object creates the latest version.
Update Objects: Updates create new versions, preserving changes.
Retrieve Versions: Access specific versions by ID or timestamp.
List Versions: See all object versions for historical tracking.
Delete and Recover: Deleting adds a marker, recover by removing it.
Permanent Deletion: Objects can be permanently removed.
Cross-Version Copy: Copy objects across versions or buckets.
Lifecycle Policies: Automate version management with rules.
Audit and Compliance: Maintain a complete data history.
Zenko CloudServer's versioning ensures data integrity, recovery, and control.
Metadata management
Zenko, the multi-cloud data controller developed by Scality, does support metadata management. Metadata management is a crucial aspect of object storage systems like Zenko as it allows users to associate additional information with their stored objects. Here's how metadata management works with Zenko:
Metadata Management with Zenko CloudServer
Attach Metadata: Associate custom metadata with stored objects.
Query Metadata: Efficiently search and organize data based on metadata.
Custom Metadata Fields: Create and define custom metadata attributes.
Storage Integration: Metadata is stored with objects for easy retrieval.
Metadata APIs: Programmatic management and querying of metadata.
Metadata Security: Control access rights to metadata attributes.
Data Governance: Support for data tracking, retention, and compliance.
Application Integration: Integration capabilities for metadata-driven applications.
In Zenko CloudServer, metadata management enhances data organization and retrieval, supports data governance, and fosters application integration.
Last updated
