{"id":1237,"date":"2025-01-09T16:00:55","date_gmt":"2025-01-09T16:00:55","guid":{"rendered":"https:\/\/www.clyso.com\/us\/?p=1237"},"modified":"2025-01-13T15:11:05","modified_gmt":"2025-01-13T15:11:05","slug":"simplifying-ceph-cluster-management-adding-capacity-with-enhanced-control","status":"publish","type":"post","link":"https:\/\/www.clyso.com\/us\/simplifying-ceph-cluster-management-adding-capacity-with-enhanced-control\/","title":{"rendered":"Simplifying Ceph Cluster Management: Adding Capacity with Enhanced Control"},"content":{"rendered":"

One of the most rewarding aspects of providing Ceph technical support is helping users simplify workflows for major changes, such as adding or removing multiple hosts in their Ceph clusters.\u00a0<\/span><\/p>\n

 <\/p>\n

Ceph scalability is one of its strongest features, allowing operators to add or remove capacity, replace servers, and perform maintenance without downtime.<\/span><\/p>\n

 <\/p>\n

However, when managing large-scale Ceph clusters, operations like redistributing objects or handling recovery events can become complex and time-consuming. At these scales, additional tools and optimized workflows are essential to ensure smooth operations while maintaining performance and reliability.<\/span><\/p>\n

 <\/p>\n

Challenges of Ceph Object Redistribution at Scale<\/b><\/p>\n

 <\/p>\n

Let\u2019s consider an example of growing your Ceph cluster. Whether you\u2019re using ceph orch apply osd or custom tools, after adding new capacity, you might encounter a status like this:<\/span><\/p>\n

 <\/p>\n

4518 active+remapped+backfill_wait<\/span><\/p>\n

2.1B (28%) objects misplaced<\/span><\/p>\n

 <\/p>\n

Here, 28% of the objects need redistribution onto new hosts. While Ceph scalability ensures this process happens transparently, challenges arise when progress looks like this:<\/span><\/p>\n

 <\/p>\n

Progress: Global Recovery Event [x…………………………..] (6w)<\/span><\/p>\n

 <\/p>\n

In this extreme scenario, redistribution could take 6 weeks, limiting operators\u2019 ability to throttle, pause, or revert the operation. What if a major event, like a power outage, occurs during this period? While Ceph maintenance ensures data safety, long recovery times add complexity and risk.<\/span><\/p>\n

 <\/p>\n

Enhanced Control with the Ceph upmap-remapped Tool<\/b><\/p>\n

 <\/p>\n

To address these challenges, we at CLYSO advocate for enhanced control in Ceph cluster management. A key tool in our arsenal is upmap-remapped, which was originally developed during my time at CERN. This tool allows for more precise control over object redistribution, improving both the reliability and efficiency of large-scale Ceph data migration tasks.<\/span><\/p>\n

With upmap-remapped, operators can:<\/span><\/p>\n