오픈스택 스위프트 설치 및 기본 테스트

Ryu's Tech

오픈스택 스위프트 설치 및 기본 테스트

Ryusstory 2018. 5. 15. 16:57

Table of Contents

개요

openstack object storage 인 swift 설치 및 기본 테스트를 해 보려 하는데,

이 글은 cosbench를 통한 swift 성능 테스트 방법을 설명하기 전에 쓰이는 글이며 cosbench 설치 및 사용은 다음 글에서 설명될 예정입니다.

노드 구성

모든 노드는 VMware workstation을 통해 가상머신으로 동작하고 네트워크는 Bridge 모드로 구성 되었습니다.

각 글 내용 머리부분(heading) 앞의 : 표시 이전에는 hostname표시했습니다. 어느 노드에서 작업해야될지를 정리했습니다.

역할	Hostname	IP	CPU	RAM	DISK	비고
오픈스택,swift-proxy	controller	192.168.0.11	4	8	50G
오브젝트 스토리지	swift1	192.168.0.131	4	8	50G	추가 디스크 20G x 3
오브젝트 스토리지	swift2	192.168.0.132	4	8	50G	추가 디스크 20G x 3
오브젝트 스토리지	swift3	192.168.0.133	4	8	50G	추가 디스크 20G x 3
Cosbench Controller + Cosbench Driver	cosbench1	192.168.0.141	4	8	50G	<다음 포스팅>
Cosbench Driver	cosbench2	192.168.0.142	4	8	50G	<다음 포스팅>
Cosbench Driver	cosbench3	192.168.0.143	4	8	50G	<다음 포스팅>

controller : 오픈스택 설치 및 Swift-proxy 설치

오픈스택 설치 AIO을 사용해 올인원 형태로 1대에 설치, keystone만 있어도 될것으로 예상은 하나 일단 AIO로 설치해서 진행.

설치 이후 /etc/hosts 업데이트 필요

echo "192.168.0.11 controller
168.0.131 swift1
168.0.132 swift2
168.0.133 swift3" >> /etc/hosts

오픈스택 controller 노드 설치 이후 진행.

openstack user create --domain default --password-prompt swift
# 패스워드 입력
openstack role add --project service --user swift admin
openstack service create --name swift --description "OpenStack Object Storage" object-store
openstack endpoint create --region RegionOne object-store public http://controller:8080/v1/AUTH_%\(tenant_id\)s
openstack endpoint create --region RegionOne object-store internal http://controller:8080/v1/AUTH_%\(tenant_id\)s
openstack endpoint create --region RegionOne object-store admin http://controller:8080/v1

yum install -y openstack-swift-proxy python-swiftclient python-keystoneclient python-keystonemiddleware memcached

설정파일 다운로드

curl -o /etc/swift/proxy-server.conf https://git.openstack.org/cgit/openstack/swift/plain/etc/proxy-server.conf-sample?h=stable/ocata

proxy-server.conf 파일 수정

[DEFAULT]
bind_port = 8080
user = swift
swift_dir = /etc/swift

[pipeline:main]
pipeline = catch_errors gatekeeper healthcheck proxy-logging cache container_sync bulk tempurl ratelimit authtoken keystoneauth copy container-quotas account-quotas slo dlo versioned_writes proxy-logging proxy-server
[app:proxy-server]
use = egg:swift#proxy
account_autocreate = true

[filter:keystoneauth]
use = egg:swift#keystoneauth
operator_roles = admin,user

[filter:authtoken]
paste.filter_factory = keystonemiddleware.auth_token:filter_factory
auth_uri = http://controller:5000
auth_url = http://controller:35357
memcached_servers = controller:11211
auth_type = password
project_domain_name = default
user_domain_name = default
project_name = service
username = swift
password = SWIFT_PASS
delay_auth_decision = True


[filter:cache]
use = egg:swift#memcache
memcache_servers = controller:11211

swift1-3 : 오브젝트 스토리지 설치

오픈스택 기본 설치 및 설정

yum install -y chrony
echo "server 192.168.0.11 iburst" >> /etc/chrony.conf
systemctl enable chronyd.service
systemctl start chronyd.service
yum remove -y firewalld
yum install -y centos-release-openstack-ocata
yum upgrade -y
yum install -y python-openstackclient
yum install -y openstack-selinux

/etc/hosts 에 controller, swift 주소 추가 필요

echo "192.168.0.11 controller
168.0.131 swift1
168.0.132 swift2
168.0.133 swift3" >> /etc/hosts

디스크 추가 설정

잠깐 예를들면 각 노드별 디스크 두개(sdb,sdc)가 추가되어 있을 경우 아래와 같은 형태로 설정이 됨.

yum install -y xfsprogs rsync
mkfs.xfs /dev/sdb
mkfs.xfs /dev/sdc
mkdir -p /srv/node/sdb
mkdir -p /srv/node/sdc
echo "/dev/sdb /srv/node/sdb xfs noatime,nodiratime,nobarrier,logbufs=8 0 2" >> /etc/fstab
echo "/dev/sdc /srv/node/sdc xfs noatime,nodiratime,nobarrier,logbufs=8 0 2" >> /etc/fstab
mount /srv/node/sdb
mount /srv/node/sdc

이 작업을 모든 노드에 해줘야하는데 아래와 같이 for loop를 통해 각 노드별로 실행.

필요한 만큼 loop 조정. 이 글에서는 3개 hdd 이므로 b c d 까지만 수행.

for i in b c d ; do mkfs.xfs /dev/sd$i; done
for i in b c d ; do mkdir -p /srv/node/sd$i; done
for i in b c d ; do echo "/dev/sd$i /srv/node/sd$i xfs noatime,nodiratime,nobarrier,logbufs=8 0 2" >> /etc/fstab; done
for i in b c d ; do mount /srv/node/sd$i; done

rsync 설정

제일 아래의 IPADDR 을 변경해 실제 IP로 변경 필요

echo "uid = swift
gid = swift
log file = /var/log/rsyncd.log
pid file = /var/run/rsyncd.pid
address = IPADDR

[account]
max connections = 2
path = /srv/node/
read only = False
lock file = /var/lock/account.lock

[container]
max connections = 2
path = /srv/node/
read only = False
lock file = /var/lock/container.lock

[object]
max connections = 2
path = /srv/node/
read only = False
lock file = /var/lock/object.lock" > /etc/rsyncd.conf
sed -i "s/IPADDR/192.168.0.131/g" /etc/rsyncd.conf

systemctl enable rsyncd.service
systemctl start rsyncd.service

Swift 설치

패키지 설치 및 설정파일 다운로드

yum install -y openstack-swift-account openstack-swift-container openstack-swift-object
curl -o /etc/swift/account-server.conf https://git.openstack.org/cgit/openstack/swift/plain/etc/account-server.conf-sample?h=stable/ocata
curl -o /etc/swift/container-server.conf https://git.openstack.org/cgit/openstack/swift/plain/etc/container-server.conf-sample?h=stable/ocata
curl -o /etc/swift/object-server.conf https://git.openstack.org/cgit/openstack/swift/plain/etc/object-server.conf-sample?h=stable/ocata

위와같이 다운로드 이후 아래 설정 적용

account-server.conf 파일 수정

[DEFAULT]
bind_ip =192.168.0.131
bind_port = 6202
user = swift
swift_dir = /etc/swift
devices = /srv/node
mount_check = True

[pipeline:main]
pipeline = healthcheck recon account-server

[filter:recon]
use = egg:swift#recon
recon_cache_path = /var/cache/swift

container-server.conf 파일 수정

[DEFAULT]
bind_ip = 192.168.0.131
bind_port = 6201
user = swift
swift_dir = /etc/swift
devices = /srv/node
mount_check = True

[pipeline:main]
pipeline = healthcheck recon container-server

[filter:recon]
use = egg:swift#recon
recon_cache_path = /var/cache/swift

object-server.conf 파일 수정

[DEFAULT]
bind_ip = 192.168.0.131
bind_port = 6200
user = swift
swift_dir = /etc/swift
devices = /srv/node
mount_check = True

[pipeline:main]
pipeline = healthcheck recon object-server

[filter:recon]
use = egg:swift#recon
recon_cache_path = /var/cache/swift
recon_lock_path = /var/lock

수정 이후 아래 명령어 실행

chown -R swift:swift /srv/node
mkdir -p /var/cache/swift
chown -R root:swift /var/cache/swift
chmod -R 775 /var/cache/swift

controller : proxy-server 설정

cd /etc/swift
swift-ring-builder account.builder create 12 3 1
for i in b c d ; do swift-ring-builder account.builder add --region 1 --zone 1 --ip 192.168.0.132 --port 6202 --device sd$i --weight 100; done
for i in b c d ; do swift-ring-builder account.builder add --region 1 --zone 2 --ip 192.168.0.133 --port 6202 --device sd$i --weight 100; done
for i in b c d ; do swift-ring-builder account.builder add --region 1 --zone 3 --ip 192.168.0.134 --port 6202 --device sd$i --weight 100; done
swift-ring-builder account.builder rebalance

swift-ring-builder container.builder create 12 3 1
for i in b c d ; do swift-ring-builder container.builder add --region 1 --zone 1 --ip 192.168.0.132 --port 6201 --device sd$i --weight 100; done
for i in b c d ; do swift-ring-builder container.builder add --region 1 --zone 2 --ip 192.168.0.133 --port 6201 --device sd$i --weight 100; done
for i in b c d ; do swift-ring-builder container.builder add --region 1 --zone 3 --ip 192.168.0.134 --port 6201 --device sd$i --weight 100; done
swift-ring-builder container.builder rebalance

swift-ring-builder object.builder create 12 3 1
for i in b c d ; do swift-ring-builder object.builder add --region 1 --zone 1 --ip 192.168.0.132 --port 6200 --device sd$i --weight 100; done
for i in b c d ; do swift-ring-builder object.builder add --region 1 --zone 2 --ip 192.168.0.133 --port 6200 --device sd$i --weight 100; done
for i in b c d ; do swift-ring-builder object.builder add --region 1 --zone 3 --ip 192.168.0.134 --port 6200 --device sd$i --weight 100; done
swift-ring-builder object.builder rebalance

이후 생성된 파일 각 swift1-3 노드로 복사

scp /etc/swift/*.gz 192.168.0.131:/etc/swift
# password input
scp /etc/swift/*.gz 192.168.0.132:/etc/swift
# password input
scp /etc/swift/*.gz 192.168.0.133:/etc/swift
# password input

swift.conf 파일 다운로드

curl -o /etc/swift/swift.conf https://git.openstack.org/cgit/openstack/swift/plain/etc/swift.conf-sample?h=stable/ocata

아래 내용 추가

[swift-hash]
swift_hash_path_suffix = HASH_PATH_SUFFIX
swift_hash_path_prefix = HASH_PATH_PREFIX

[storage-policy:0]
name = Policy-0
default = yes

수정된 파일 복사

scp /etc/swift/swift.conf 192.168.0.131:/etc/swift/
# password input
scp /etc/swift/swift.conf 192.168.0.132:/etc/swift/
# password input
scp /etc/swift/swift.conf 192.168.0.133:/etc/swift/
# password input

controller, swift1-3 : 마무리 및 서비스 시작

소유권 설정

chown -R root:swift /etc/swift

SELinux 권한 수정

chcon -R system_u:object_r:swift_data_t:s0 /srv/node

서비스 시작

controller

systemctl enable openstack-swift-proxy.service memcached.service
systemctl start openstack-swift-proxy.service memcached.service

swift 1-3

systemctl enable openstack-swift-account.service openstack-swift-account-auditor.service openstack-swift-account-reaper.service openstack-swift-account-replicator.service
systemctl start openstack-swift-account.service openstack-swift-account-auditor.service openstack-swift-account-reaper.service openstack-swift-account-replicator.service

systemctl enable openstack-swift-container.service openstack-swift-container-auditor.service openstack-swift-container-replicator.service openstack-swift-container-updater.service
systemctl start openstack-swift-container.service openstack-swift-container-auditor.service openstack-swift-container-replicator.service openstack-swift-container-updater.service

systemctl enable openstack-swift-object.service openstack-swift-object-auditor.service openstack-swift-object-replicator.service openstack-swift-object-updater.service
systemctl start openstack-swift-object.service openstack-swift-object-auditor.service openstack-swift-object-replicator.service openstack-swift-object-updater.service

설치 확인

오픈스택 동작확인

[root@controller ~]# openstack container create ryusstory
+---------------------------------------+-----------+------------------------------------+
| account                               | container | x-trans-id                         |
+---------------------------------------+-----------+------------------------------------+
| AUTH_ccac596c068b4e1b941626e1931dfbff | ryusstory | txe8ca8de85c194f3ea958f-005afa8892 |
+---------------------------------------+-----------+------------------------------------+
[root@controller ~]# echo "ryusstory.tistory.com" > FILE
[root@controller ~]# 
[root@controller ~]# openstack object create ryusstory FILE
+--------+-----------+----------------------------------+
| object | container | etag                             |
+--------+-----------+----------------------------------+
| FILE   | ryusstory | 4f41e21a6e53f7dd95088b85985ddacb |
+--------+-----------+----------------------------------+
[root@controller ~]# 
[root@controller ~]# openstack object list ryusstory
+------+
| Name |
+------+
| FILE |
+------+
[root@controller ~]# 