Blog
Search…
Ceph Ansible baremetal deployment
How many times you tried to install Ceph? How many fails with no reason?
All Ceph operator should agree with me when i say that Ceph installer doesn't really works as expected so far.
Yes, i'm talking about ceph-deploy and the main reason why i'm posting this guide about deploying Ceph with Ansible.
At this post, i will show how to install a Ceph cluster with Ansible on baremetal servers.
My configuration is as follows:
- 1.3 x ceph monitors 8GB of RAM each one
- 2.3 x OSD nodes 16GB of RAM and 3x100 GB of Disk
- 3.1 x RadosGateway node 8GB of RAM
First, download Ceph-Ansible playbooks
1
git clone https://github.com/ceph/ceph-ansible/
2
Cloning into 'ceph-ansible'...
3
remote: Counting objects: 5764, done.
4
remote: Compressing objects: 100% (38/38), done.
5
remote: Total 5764 (delta 7), reused 0 (delta 0), pack-reused 5726
6
Receiving objects: 100% (5764/5764), 1.12 MiB | 1.06 MiB/s, done.
7
Resolving deltas: 100% (3465/3465), done.
8
Checking connectivity... done.
Copied!
Move to the newly created folder called ceph-ansible
1
cd ceph-ansible/
Copied!
Copy sample vars files, we will configure our environment in these variable files.
1
cp site.yml.sample site.yml
2
cp group_vars/all.sample group_vars/all
3
cp group_vars/mons.sample group_vars/mons
4
cp group_vars/osds.sample group_vars/osds
Copied!
Next step is configure the inventory with our servers, i don\'t really like use /etc/ansible/host file, i prefer create a new file per environment inside playbook\'s folder.
Create a file with the following content, use you own IPs to match your servers on the desired role inside the cluster
1
[[email protected] ~]# vi inventory_hosts
2
​
3
[mons]
4
192.168.1.48
5
192.168.1.49
6
192.168.1.52
7
​
8
[osds]
9
192.168.1.50
10
192.168.1.53
11
192.168.1.54
12
​
13
[rgws]
14
192.168.1.55
Copied!
Test connectivity to you servers pinging them through Ansible ping module
1
[[email protected] ~]# ansible -m ping -i inventory_hosts all
2
192.168.1.48 | success >> {
3
"changed": false,
4
"ping": "pong"
5
}
6
​
7
192.168.1.50 | success >> {
8
"changed": false,
9
"ping": "pong"
10
}
11
​
12
192.168.1.55 | success >> {
13
"changed": false,
14
"ping": "pong"
15
}
16
​
17
192.168.1.53 | success >> {
18
"changed": false,
19
"ping": "pong"
20
}
21
​
22
192.168.1.49 | success >> {
23
"changed": false,
24
"ping": "pong"
25
}
26
​
27
192.168.1.54 | success >> {
28
"changed": false,
29
"ping": "pong"
30
}
31
​
32
192.168.1.52 | success >> {
33
"changed": false,
34
"ping": "pong"
35
}
Copied!
Edit site.yml file, i will remove/comment mds nodes since i\'m not going to use them.
1
[[email protected] ~]# vi site.yml
2
​
3
- hosts: mons
4
become: True
5
roles:
6
- ceph-mon
7
​
8
- hosts: agents
9
become: True
10
roles:
11
- ceph-agent
12
​
13
- hosts: osds
14
become: True
15
roles:
16
- ceph-osd
17
​
18
#- hosts: mdss
19
# become: True
20
# roles:
21
# - ceph-mds
22
​
23
- hosts: rgws
24
become: True
25
roles:
26
- ceph-rgw
27
​
28
- hosts: restapis
29
become: True
30
roles:
31
- ceph-restapi
Copied!
Edit main variable file, here we are going to configure our environment
1
[[email protected] ~]# vi group_vars/all
Copied!
Here we configure from where ceph packages are going to be installed, for now we use upstream code with the stable release Infernalis.
1
## Configure package origin
2
ceph_origin: upstream
3
ceph_stable: true
4
ceph_stable_release: infernalis
Copied!
Configure interface on which monitor will be listening
1
## Monitor options
2
monitor_interface: eth2
Copied!
Here we configure some OSD options, like journal size and what networks will be used by public and cluster data replication
1
## OSD options
2
journal_size: 1024
3
public_network: 192.168.1.0/24
4
cluster_network: 192.168.200.0/24
Copied!
Edit osds variable file
1
[[email protected] ~]# vi group_vars/osds
Copied!
I will use auto discovery option to allow ceph ansible select empy or not used devices in my servers to create OSDs.
1
# Declare devices
2
osd_auto_discovery: True
3
journal_collocation: True
Copied!
| Of course you can use other options, i\'ll highly suggest you to read variable comments, as they provide valuable information about usage. | We\'re ready to deploy ceph with ansible with our custom inventory_hosts file.
1
[[email protected] ~]# ansible-playbook site.yml -i inventory_hosts
Copied!
After a while, you will have a fully functional ceph cluster.
| Maybe you find some issues or bugs when running the playbooks. | There is a lot of efforts to fix issues on upstream repository. If a new bug is encountered, please, post a issue right here. | https://github.com/ceph/ceph-ansible/issues​
You can check your cluster status with ceph -s. we can see all OSDs are up and pgs active/clean.
1
[[email protected] ~]# ceph -s
2
cluster 5ff692ab-2150-41a4-8b6d-001a4da21c9c
3
health HEALTH_OK
4
monmap e1: 3 mons at {ceph-mon1=192.168.200.141:6789/0,ceph-mon2=192.168.200.180:6789/0,ceph-mon3=192.168.200.232:6789/0}
5
election epoch 6, quorum 0,1,2 ceph-mon1,ceph-mon2,ceph-mon3
6
osdmap e10: 9 osds: 9 up, 9 in
7
flags sortbitwise
8
pgmap v32: 64 pgs, 1 pools, 0 bytes data, 0 objects
9
102256 kB used, 896 GB / 896 GB avail
10
64 active+clean
Copied!
| We are going to do some tests. | Create a pool
1
[[email protected] ~]# ceph osd pool create test 128 128
2
pool 'test' created
Copied!
Create a file big file
1
[[email protected] ~]# dd if=/dev/zero of=/tmp/sample.txt bs=2M count=1000
2
1000+0 records in
3
1000+0 records out
4
2097152000 bytes (2.1 GB) copied, 16.7386 s, 125 MB/s
Copied!
Upload the file to rados
1
[[email protected] ~]# rados -p test put sample /tmp/sample.txt
Copied!
Check om which placement groups your file is saved
1
[[email protected] ~]# ceph osd map test sample
2
osdmap e13 pool 'test' (1) object 'sample' -> pg 1.bddbf0b9 (1.39) -> up ([1,0], p1) acting ([1,0], p1)
Copied!
Query the placement group where you file was uploaded, a similar output will prompts
1
[[email protected] ~]# ceph pg 1.39 query
2
{
3
"state": "active+clean",
4
"snap_trimq": "[]",
5
"epoch": 13,
6
"up": [
7
1,
8
0
9
],
10
"acting": [
11
1,
12
0
13
],
14
"actingbackfill": [
15
"0",
16
"1"
17
],
18
"info": {
19
"pgid": "1.39",
20
"last_update": "13'500",
21
"last_complete": "13'500",
22
"log_tail": "0'0",
23
"last_user_version": 500,
24
"last_backfill": "MAX",
25
"last_backfill_bitwise": 0,
26
"purged_snaps": "[]",
27
"history": {
28
"epoch_created": 11,
29
"last_epoch_started": 12,
30
"last_epoch_clean": 13,
31
"last_epoch_split": 0,
32
"last_epoch_marked_full": 0,
33
"same_up_since": 11,
34
"same_interval_since": 11,
35
"same_primary_since": 11,
36
"last_scrub": "0'0",
37
"last_scrub_stamp": "2016-03-16 21:13:08.883121",
38
"last_deep_scrub": "0'0",
39
"last_deep_scrub_stamp": "2016-03-16 21:13:08.883121",
40
"last_clean_scrub_stamp": "0.000000"
41
},
42
"stats": {
43
"version": "13'500",
44
"reported_seq": "505",
45
"reported_epoch": "13",
46
"state": "active+clean",
47
"last_fresh": "2016-03-16 21:24:40.930724",
48
"last_change": "2016-03-16 21:14:09.874086",
49
"last_active": "2016-03-16 21:24:40.930724",
50
"last_peered": "2016-03-16 21:24:40.930724",
51
"last_clean": "2016-03-16 21:24:40.930724",
52
"last_became_active": "0.000000",
53
"last_became_peered": "0.000000",
54
"last_unstale": "2016-03-16 21:24:40.930724",
55
"last_undegraded": "2016-03-16 21:24:40.930724",
56
"last_fullsized": "2016-03-16 21:24:40.930724",
57
"mapping_epoch": 11,
58
"log_start": "0'0",
59
"ondisk_log_start": "0'0",
60
"created": 11,
61
"last_epoch_clean": 13,
62
"parent": "0.0",
63
"parent_split_bits": 0,
64
"last_scrub": "0'0",
65
"last_scrub_stamp": "2016-03-16 21:13:08.883121",
66
"last_deep_scrub": "0'0",
67
"last_deep_scrub_stamp": "2016-03-16 21:13:08.883121",
68
"last_clean_scrub_stamp": "0.000000",
69
"log_size": 500,
70
"ondisk_log_size": 500,
71
"stats_invalid": "0",
72
"stat_sum": {
73
"num_bytes": 2097152000,
74
"num_objects": 1,
75
"num_object_clones": 0,
76
"num_object_copies": 2,
77
"num_objects_missing_on_primary": 0,
78
"num_objects_degraded": 0,
79
"num_objects_misplaced": 0,
80
"num_objects_unfound": 0,
81
"num_objects_dirty": 1,
82
"num_whiteouts": 0,
83
"num_read": 0,
84
"num_read_kb": 0,
85
"num_write": 500,
86
"num_write_kb": 2048000,
87
"num_scrub_errors": 0,
88
"num_shallow_scrub_errors": 0,
89
"num_deep_scrub_errors": 0,
90
"num_objects_recovered": 0,
91
"num_bytes_recovered": 0,
92
"num_keys_recovered": 0,
93
"num_objects_omap": 0,
94
"num_objects_hit_set_archive": 0,
95
"num_bytes_hit_set_archive": 0,
96
"num_flush": 0,
97
"num_flush_kb": 0,
98
"num_evict": 0,
99
"num_evict_kb": 0,
100
"num_promote": 0,
101
"num_flush_mode_high": 0,
102
"num_flush_mode_low": 0,
103
"num_evict_mode_some": 0,
104
"num_evict_mode_full": 0
105
},
106
"up": [
107
1,
108
0
109
],
110
"acting": [
111
1,
112
0
113
],
114
"blocked_by": [],
115
"up_primary": 1,
116
"acting_primary": 1
117
},
118
"empty": 0,
119
"dne": 0,
120
"incomplete": 0,
121
"last_epoch_started": 12,
122
"hit_set_history": {
123
"current_last_update": "0'0",
124
"history": []
125
}
126
},
127
"peer_info": [
128
{
129
"peer": "0",
130
"pgid": "1.39",
131
"last_update": "13'500",
132
"last_complete": "13'500",
133
"log_tail": "0'0",
134
"last_user_version": 0,
135
"last_backfill": "MAX",
136
"last_backfill_bitwise": 0,
137
"purged_snaps": "[]",
138
"history": {
139
"epoch_created": 11,
140
"last_epoch_started": 12,
141
"last_epoch_clean": 13,
142
"last_epoch_split": 0,
143
"last_epoch_marked_full": 0,
144
"same_up_since": 0,
145
"same_interval_since": 0,
146
"same_primary_since": 0,
147
"last_scrub": "0'0",
148
"last_scrub_stamp": "2016-03-16 21:13:08.883121",
149
"last_deep_scrub": "0'0",
150
"last_deep_scrub_stamp": "2016-03-16 21:13:08.883121",
151
"last_clean_scrub_stamp": "0.000000"
152
},
153
"stats": {
154
"version": "0'0",
155
"reported_seq": "0",
156
"reported_epoch": "0",
157
"state": "inactive",
158
"last_fresh": "0.000000",
159
"last_change": "0.000000",
160
"last_active": "0.000000",
161
"last_peered": "0.000000",
162
"last_clean": "0.000000",
163
"last_became_active": "0.000000",
164
"last_became_peered": "0.000000",
165
"last_unstale": "0.000000",
166
"last_undegraded": "0.000000",
167
"last_fullsized": "0.000000",
168
"mapping_epoch": 0,
169
"log_start": "0'0",
170
"ondisk_log_start": "0'0",
171
"created": 0,
172
"last_epoch_clean": 0,
173
"parent": "0.0",
174
"parent_split_bits": 0,
175
"last_scrub": "0'0",
176
"last_scrub_stamp": "0.000000",
177
"last_deep_scrub": "0'0",
178
"last_deep_scrub_stamp": "0.000000",
179
"last_clean_scrub_stamp": "0.000000",
180
"log_size": 0,
181
"ondisk_log_size": 0,
182
"stats_invalid": "0",
183
"stat_sum": {
184
"num_bytes": 0,
185
"num_objects": 0,
186
"num_object_clones": 0,
187
"num_object_copies": 0,
188
"num_objects_missing_on_primary": 0,
189
"num_objects_degraded": 0,
190
"num_objects_misplaced": 0,
191
"num_objects_unfound": 0,
192
"num_objects_dirty": 0,
193
"num_whiteouts": 0,
194
"num_read": 0,
195
"num_read_kb": 0,
196
"num_write": 0,
197
"num_write_kb": 0,
198
"num_scrub_errors": 0,
199
"num_shallow_scrub_errors": 0,
200
"num_deep_scrub_errors": 0,
201
"num_objects_recovered": 0,
202
"num_bytes_recovered": 0,
203
"num_keys_recovered": 0,
204
"num_objects_omap": 0,
205
"num_objects_hit_set_archive": 0,
206
"num_bytes_hit_set_archive": 0,
207
"num_flush": 0,
208
"num_flush_kb": 0,
209
"num_evict": 0,
210
"num_evict_kb": 0,
211
"num_promote": 0,
212
"num_flush_mode_high": 0,
213
"num_flush_mode_low": 0,
214
"num_evict_mode_some": 0,
215
"num_evict_mode_full": 0
216
},
217
"up": [],
218
"acting": [],
219
"blocked_by": [],
220
"up_primary": -1,
221
"acting_primary": -1
222
},
223
"empty": 0,
224
"dne": 0,
225
"incomplete": 0,
226
"last_epoch_started": 12,
227
"hit_set_history": {
228
"current_last_update": "0'0",
229
"history": []
230
}
231
}
232
],
233
"recovery_state": [
234
{
235
"name": "Started\/Primary\/Active",
236
"enter_time": "2016-03-16 21:13:36.769083",
237
"might_have_unfound": [],
238
"recovery_progress": {
239
"backfill_targets": [],
240
"waiting_on_backfill": [],
241
"last_backfill_started": "MIN",
242
"backfill_info": {
243
"begin": "MIN",
244
"end": "MIN",
245
"objects": []
246
},
247
"peer_backfill_info": [],
248
"backfills_in_flight": [],
249
"recovering": [],
250
"pg_backend": {
251
"pull_from_peer": [],
252
"pushing": []
253
}
254
},
255
"scrub": {
256
"scrubber.epoch_start": "0",
257
"scrubber.active": 0,
258
"scrubber.waiting_on": 0,
259
"scrubber.waiting_on_whom": []
260
}
261
},
262
{
263
"name": "Started",
264
"enter_time": "2016-03-16 21:13:09.216260"
265
}
266
],
267
"agent_state": {}
268
}
Copied!
That\'s all for now.
Regards, Eduardo Gonzalez
Last modified 2yr ago
Copy link