Clyso’s Mark Nelson has written the first part in a series looking at performance testing of the upcoming Ceph Reef release vs the previous Quincy release. See the blog post here! Please feel free to contact us if you are interested in Ceph support or performance consulting!
ceph – how do disable mclock scheduler
After more than 4 years of development, mclock is the default scheduler for ceph quincy (version 17).
If you don’t want to use the scheduler, you can disable it with the option osd_op_queue.
WPQ was the default before Ceph Quincy and the change requires a restart of the OSDs.
Source:
https://docs.ceph.com/en/quincy/rados/configuration/osd-config-ref/#confval-osd_op_queue
https://docs.ceph.com/en/quincy/rados/configuration/osd-config-ref/#qos-based-on-mclock
Fix CephFS Filesystem Read-Only
After a reboot of the MDS Server it can happen that the CephFS Filesystem becomes read-only:
HEALTH_WARN 1 MDSs are read only [WRN] MDS_READ_ONLY: 1 MDSs are read only mds.XXX(mds.0): MDS in read-only modehttps://tracker.ceph.com/issues/58082
In the MDS log you will find following entry
log_channel(cluster) log [ERR] : failed to commit dir 0x1 object, errno -22 mds.0.11963 unhandled write error (22) Invalid argument, force readonly... mds.0.cache force file system read-only log_channel(cluster) log [WRN] : force file system read-only mds.0.server force_clients_readonlyhttps://tracker.ceph.com/issues/58082
This is a known upstream issue thought the fix is still not merged
As a workaround you can use following steps:
ceph config set mds mds_dir_max_commit_size 80
ceph fs fail <fs_name>
ceph fs set <fs_name> joinable true
If not successful you may need to increase the mds_dir_max_commit_size, e.g. to 160
ceph Quincy release with bugfix for PGLog dups
Our bugfix from earlier this year was published in the ceph quincy release 17.2.4.
Trimming of PGLog dups is now controlled by size instead of the version. This fixes the PGLog inflation issue that was happening when online (in OSD) trimming jammed after a PG split operation. Also, a new offline mechanism has been added: ceph-objectstore-tool
now has a trim-pg-log-dups
op that targets situations where an OSD is unable to boot due to those inflated dups. If that is the case, in OSD logs the “You can be hit by THE DUPS BUG” warning will be visible. Relevant tracker: https://tracker.ceph.com/issues/53729
related posts:
Source:
https://docs.ceph.com/en/latest/releases/quincy/#v17-2-4-quincy
[WRN] clients failing to respond to cache pressure
At the time when the MDS cache runs full, the process must clear inodes from its cache. This also means that the MDS will prompt some clients to also clear some inodes from their cache.
The MDS asks the cephfs client several times to release the inodes. If the client does not respond to this cache recall request, Ceph will log this warning.
IBM will add Red Hat storage product roadmaps and Red Hat associate teams to the IBM Storage business unit
ARMONK, N.Y., Oct. 4, 2022 /PRNewswire/ -- IBM (NYSE: IBM) announced today it will add Red Hat storage product roadmaps and Red Hat associate teams to the IBM Storage business unit, bringing consistent application and data storage across on-premises infrastructure and cloud.
Sources:
Protected: ceph-mgr recreate sqlite database for healthdevice module
Protected: Ceph S3 load and performance test
ceph osd set-group
If you don’t want to set flags for the whole cluster, like noout or noup. Then you can also use ceph osd set-group and ceph osd unset-group to set the appropriate flag for a group of osds or even whole hosts.
ceph osd set-group <flags> <who> ceph osd unset-group <flags> <who>
for example set noout for a whole host with osds
ceph osd set-group noout clyso-ceph-node3
root@clyso-ceph-node1:~# ceph health detail
HEALTH_WARN 1 OSDs or CRUSH {nodes, device-classes} have {NOUP,NODOWN,NOIN,NOOUT} flags set
[WRN] OSD_FLAGS: 1 OSDs or CRUSH {nodes, device-classes} have {NOUP,NODOWN,NOIN,NOOUT} flags set
host clyso-ceph-node3 has flags noout
ceph osd unset-group noout clyso-ceph-node3
root@clyso-ceph-node1:~# ceph health detail
HEALTH_OK
root@clyso-ceph-node1:
Sources:
https://docs.ceph.com/en/quincy/rados/operations/health-checks/#osd-flags
ceph unlock/enable a locked dashboard user
ceph dashboard ac-user-enable <username>
example with admin user
ceph dashboard ac-user-enable admin
Sources:
https://docs.ceph.com/en/quincy/mgr/dashboard/#enable-a-locked-user