Rook-Ceph Practice Guide

This guide is designed for individuals with no prior experience in Ceph or distributed storage, aiming to help you quickly understand and practically operate Rook-Ceph.

I. Background and Preliminary Concepts

Before diving into Rook-Ceph, it's essential to understand two fundamental concepts: Ceph and Rook-Ceph. These represent the core capabilities of a distributed storage system and the automated management solution on Kubernetes, respectively.

1. Ceph

Ceph is an open-source distributed storage system, originally initiated by Sage Weil, aimed at solving data consistency, availability, and scalability issues in large-scale storage environments. Unlike traditional centralized storage solutions, Ceph does not rely on a single controller but is built on collaboration between peer nodes. The core design goals of Ceph include:

• High scalability: Supports storage capacity growth from terabytes to petabytes;
• High availability and fault tolerance: Ensures data remains accessible even when nodes fail, using replication or erasure coding mechanisms;
• Unified storage model: Supports access to block devices (Block), object storage (Object), and file systems (File) simultaneously.

A Ceph cluster mainly consists of the following core components:

A typical Ceph cluster usually needs at least 3 MONs to ensure safe leader election, and multiple OSDs are mounted on physical disks across nodes to collectively provide data storage capabilities.

2. Rook-Ceph

Rook is a storage orchestrator (Operator) based on Kubernetes, with the goal of letting users manage storage systems as easily as managing Pods. Rook-Ceph is one of the most mature and widely used backends, simplifying the deployment and management of Ceph on Kubernetes.

Rook-Ceph abstracts the deployment and maintenance of Ceph into a set of Kubernetes custom resources (Custom Resources), including:

Through the Operator model, Rook achieves the lifecycle management of Ceph clusters, including deployment, upgrades, fault recovery, and scaling, significantly reducing the complexity of traditional Ceph installation processes.

3. Using Ceph in Kubernetes

Kubernetes natively does not provide true persistent storage. If a Pod's PVC (PersistentVolumeClaim) lacks a reliable backend, it cannot achieve high availability, cross-node access, or automatic recovery of storage. Ceph, especially Ceph integrated with Rook, fills this gap perfectly.

• Reliable data persistence storage;
• Elastic scalability;
• Supports high availability block devices and shared file systems;
• Native Kubernetes support for CSI plugins and dynamic StorageClass provisioning.

Rook-Ceph can build a truly usable cloud-native storage solution.

II. Learning Path

Ceph and Rook-Ceph involve a wide range of content, and for beginners, the biggest challenge in the early stages is not the lack of resources, but the fact that there are too many resources that are not systematic or progressive. To increase efficiency and avoid inefficient pitfalls, it's recommended to divide the learning path into three stages, focusing on building an understanding of relevant tools through practice.

1. Phase One: Concept Building

Recommended Resources

• Official documentation: https://rook.github.io/docs/rook/latest-release/Getting-Started/intro/
• Chinese introduction: Search for "Ceph Getting Started" on Bilibili
• Querying terms with ChatGPT or similar

Key Goals: Understand the design motivations and relationship between Ceph and Rook-Ceph, clarify the responsibilities of core components

2. Phase Two: Environment Familiarization

Basic Goals

• Learn to enter the toolbox in K8s, execute basic diagnostic and status commands;
• Be able to read and understand the meaning of commands like ceph status and osd tree;
• Initially learn to use kubectl to manage Rook resources.

Key Goals: Understand the resources in the Rook-Ceph cluster after deployment, master the connection between the toolbox and K8s objects, and be able to perform basic queries independently.

3. Phase Three: Hands-on Practice

Basic Goals

• Master the basic operations for daily maintenance of a Ceph cluster;
• Be able to complete the process from creating a storage pool to binding a PVC and running an application;
• Be able to independently troubleshoot most common issues (at least identify the cause).

Key Goals: Be able to create and manage Pools, RBDs, and PVCs, handle common errors, and understand the Rook upgrade and Ceph operation chain.

Tips

• Perform the complete process of creating → mounting → writing → deleting;
• Don't be afraid of trial and error, especially in test environments where you intentionally cause OSD down/MON failures;
• Let large models explain error messages and the purposes of CRD fields.

III. Common Commands

1. Basic Information View and Health Checks

2. OSD Operations

3. Pool Management

4. RBD Operations Practice

5. File System Operations (Available only after deploying MDS)

IV. Practice Scenarios

1. Scenario One: Create an Independent Data Volume for Large Model Training Services

A laboratory has deployed a large model training service based on PyTorch. Each user wants to use an independent data storage volume mounted into the training container. As an administrator, you need to:

1. Create a block storage pool named ml-training-pool, specifically for model training data;
2. Bind this pool to a Kubernetes StorageClass for users to call in PVCs;
3. Create a PVC and confirm that it has successfully bound and created the corresponding RBD image in the Ceph cluster;
4. Use the toolbox to view the status, capacity, and other detailed information of this block device to confirm its availability.

2. Scenario Two: Simulate Node Disk Failure and Ceph Data Rebalancing Process

To simulate Ceph's fault tolerance in production environments, you need to simulate an abnormal offline event of an OSD node and observe whether Ceph's self-healing mechanism takes effect.

1. Select an OSD from the existing cluster and mark it as out, simulating disk offline;
2. Observe whether Ceph triggers data migration, rebalancing, and health status changes;
3. View the changes in the OSD tree and cluster status, record the process of replica rebuilding and capacity redistribution;
4. Simulate the repair of the fault and mark the OSD as in, observing the recovery process of the system.

3. Scenario Three: Use Snapshots and Rollbacks to Protect and Recover Training Data

Before large model training, you want to provide a recoverable snapshot for users' training data volumes to prevent accidental deletion. The scenario is as follows:

1. Create a snapshot named @pretrain on the existing block storage volume (corresponding to PVC);
2. Assume that the user accidentally deletes important training data or writes abnormal data midway;
3. Use Ceph's snapshot rollback feature to restore this RBD volume to the pre-training snapshot state;
4. Verify data consistency after the rollback and understand the impact of snapshots on system performance and capacity.

4. Scenario Four: Deploy a Shared Volume to Support Distributed Training Log Collection

The laboratory plans to conduct distributed training experiments, and multiple Pod instances need to write training logs to the same directory. You need to:

1. Create a Ceph file system (CephFS) and provide POSIX interfaces through the MDS service;
2. Configure a StorageClass that supports ReadWriteMany for multiple Pods to mount simultaneously;
3. Create a PVC and mount it to multiple Pods, simulating a concurrent log writing scenario;
4. Verify whether the written data is consistent and whether there are write conflicts or permission issues.

5. Scenario Five: Simulate a Response Strategy for an Almost Full Storage Pool

During peak times of dense task submissions, you discover that the remaining capacity of a certain Ceph block storage pool is about to be insufficient. You need to:

1. Use the toolbox to view the usage rate and replica configuration of each Pool;
2. Analyze which RBD images in the current Pool are idle or unbound, and consider cleaning up and reclaiming them;
3. If the capacity is indeed insufficient, attempt to adjust the replica count of certain Pools from 3 to 2 (only for test scenarios);
4. Evaluate expansion feasibility, add new nodes or OSDs, and observe whether the system automatically rebalances.

6. Scenario Six: Clean Up Unused Images and Pools to Free Up Storage Space

You find that there are multiple PVCs and their corresponding RBD images no longer in use in the cluster, and users have not manually cleaned them up. You need to:

1. Confirm whether the Pods that these PVCs belong to have been deleted;
2. Locate these PVCs' bound RBD images and confirm that they are no longer mounted;
3. Use the toolbox to delete these images and clean up unused BlockPools (if no dependencies exist);
4. Check the cluster's total capacity change again to ensure that the space has been successfully reclaimed.

Edit on GitHub