Table of Contents
개요
openstack object storage 인 swift 설치 및 기본 테스트를 해 보려 하는데,
이 글은 cosbench를 통한 swift 성능 테스트 방법을 설명하기 전에 쓰이는 글이며 cosbench 설치 및 사용은 다음 글에서 설명될 예정입니다.
노드 구성
모든 노드는 VMware workstation을 통해 가상머신으로 동작하고 네트워크는 Bridge 모드로 구성 되었습니다.
각 글 내용 머리부분(heading) 앞의 : 표시 이전에는 hostname표시했습니다. 어느 노드에서 작업해야될지를 정리했습니다.
역할 |
Hostname |
IP |
CPU |
RAM |
DISK |
비고 |
오픈스택,swift-proxy |
controller |
192.168.0.11 |
4 |
8 |
50G |
|
오브젝트 스토리지 |
swift1 |
192.168.0.131 |
4 |
8 |
50G |
추가 디스크 20G x 3 |
오브젝트 스토리지 |
swift2 |
192.168.0.132 |
4 |
8 |
50G |
추가 디스크 20G x 3 |
오브젝트 스토리지 |
swift3 |
192.168.0.133 |
4 |
8 |
50G |
추가 디스크 20G x 3 |
Cosbench Controller + Cosbench Driver |
cosbench1 |
192.168.0.141 |
4 |
8 |
50G |
<다음 포스팅> |
Cosbench Driver |
cosbench2 |
192.168.0.142 |
4 |
8 |
50G |
<다음 포스팅> |
Cosbench Driver |
cosbench3 |
192.168.0.143 |
4 |
8 |
50G |
<다음 포스팅> |
controller : 오픈스택 설치 및 Swift-proxy 설치
오픈스택 설치 AIO을 사용해 올인원 형태로 1대에 설치, keystone만 있어도 될것으로 예상은 하나 일단 AIO로 설치해서 진행.
설치 이후 /etc/hosts 업데이트 필요
echo "192.168.0.11 controller
192.168.0.131 swift1
192.168.0.132 swift2
192.168.0.133 swift3" >> /etc/hosts
오픈스택 controller 노드 설치 이후 진행.
openstack user create --domain default --password-prompt swift
# 패스워드 입력
openstack role add --project service --user swift admin
openstack service create --name swift --description "OpenStack Object Storage" object-store
openstack endpoint create --region RegionOne object-store public http://controller:8080/v1/AUTH_%\(tenant_id\)s
openstack endpoint create --region RegionOne object-store internal http://controller:8080/v1/AUTH_%\(tenant_id\)s
openstack endpoint create --region RegionOne object-store admin http://controller:8080/v1
yum install -y openstack-swift-proxy python-swiftclient python-keystoneclient python-keystonemiddleware memcached
설정파일 다운로드
curl -o /etc/swift/proxy-server.conf https://git.openstack.org/cgit/openstack/swift/plain/etc/proxy-server.conf-sample?h=stable/ocata
proxy-server.conf 파일 수정
[DEFAULT]
bind_port = 8080
user = swift
swift_dir = /etc/swift
[pipeline:main]
pipeline = catch_errors gatekeeper healthcheck proxy-logging cache container_sync bulk tempurl ratelimit authtoken keystoneauth copy container-quotas account-quotas slo dlo versioned_writes proxy-logging proxy-server
[app:proxy-server]
use = egg:swift#proxy
account_autocreate = true
[filter:keystoneauth]
use = egg:swift#keystoneauth
operator_roles = admin,user
[filter:authtoken]
paste.filter_factory = keystonemiddleware.auth_token:filter_factory
auth_uri = http://controller:5000
auth_url = http://controller:35357
memcached_servers = controller:11211
auth_type = password
project_domain_name = default
user_domain_name = default
project_name = service
username = swift
password = SWIFT_PASS
delay_auth_decision = True
[filter:cache]
use = egg:swift#memcache
memcache_servers = controller:11211
swift1-3 : 오브젝트 스토리지 설치
오픈스택 기본 설치 및 설정
yum install -y chrony
echo "server 192.168.0.11 iburst" >> /etc/chrony.conf
systemctl enable chronyd.service
systemctl start chronyd.service
yum remove -y firewalld
yum install -y centos-release-openstack-ocata
yum upgrade -y
yum install -y python-openstackclient
yum install -y openstack-selinux
/etc/hosts 에 controller, swift 주소 추가 필요
echo "192.168.0.11 controller
192.168.0.131 swift1
192.168.0.132 swift2
192.168.0.133 swift3" >> /etc/hosts
디스크 추가 설정
잠깐 예를들면 각 노드별 디스크 두개(sdb,sdc)가 추가되어 있을 경우 아래와 같은 형태로 설정이 됨.
yum install -y xfsprogs rsync
mkfs.xfs /dev/sdb
mkfs.xfs /dev/sdc
mkdir -p /srv/node/sdb
mkdir -p /srv/node/sdc
echo "/dev/sdb /srv/node/sdb xfs noatime,nodiratime,nobarrier,logbufs=8 0 2" >> /etc/fstab
echo "/dev/sdc /srv/node/sdc xfs noatime,nodiratime,nobarrier,logbufs=8 0 2" >> /etc/fstab
mount /srv/node/sdb
mount /srv/node/sdc
이 작업을 모든 노드에 해줘야하는데 아래와 같이 for loop를 통해 각 노드별로 실행.
필요한 만큼 loop 조정. 이 글에서는 3개 hdd 이므로 b c d 까지만 수행.
for i in b c d ; do mkfs.xfs /dev/sd$i; done
for i in b c d ; do mkdir -p /srv/node/sd$i; done
for i in b c d ; do echo "/dev/sd$i /srv/node/sd$i xfs noatime,nodiratime,nobarrier,logbufs=8 0 2" >> /etc/fstab; done
for i in b c d ; do mount /srv/node/sd$i; done
rsync 설정
제일 아래의 IPADDR 을 변경해 실제 IP로 변경 필요
echo "uid = swift
gid = swift
log file = /var/log/rsyncd.log
pid file = /var/run/rsyncd.pid
address = IPADDR
[account]
max connections = 2
path = /srv/node/
read only = False
lock file = /var/lock/account.lock
[container]
max connections = 2
path = /srv/node/
read only = False
lock file = /var/lock/container.lock
[object]
max connections = 2
path = /srv/node/
read only = False
lock file = /var/lock/object.lock" > /etc/rsyncd.conf
sed -i "s/IPADDR/192.168.0.131/g" /etc/rsyncd.conf
systemctl enable rsyncd.service
systemctl start rsyncd.service
Swift 설치
패키지 설치 및 설정파일 다운로드
yum install -y openstack-swift-account openstack-swift-container openstack-swift-object
curl -o /etc/swift/account-server.conf https://git.openstack.org/cgit/openstack/swift/plain/etc/account-server.conf-sample?h=stable/ocata
curl -o /etc/swift/container-server.conf https://git.openstack.org/cgit/openstack/swift/plain/etc/container-server.conf-sample?h=stable/ocata
curl -o /etc/swift/object-server.conf https://git.openstack.org/cgit/openstack/swift/plain/etc/object-server.conf-sample?h=stable/ocata
위와같이 다운로드 이후 아래 설정 적용
account-server.conf 파일 수정
[DEFAULT]
bind_ip =192.168.0.131
bind_port = 6202
user = swift
swift_dir = /etc/swift
devices = /srv/node
mount_check = True
[pipeline:main]
pipeline = healthcheck recon account-server
[filter:recon]
use = egg:swift#recon
recon_cache_path = /var/cache/swift
container-server.conf 파일 수정
[DEFAULT]
bind_ip = 192.168.0.131
bind_port = 6201
user = swift
swift_dir = /etc/swift
devices = /srv/node
mount_check = True
[pipeline:main]
pipeline = healthcheck recon container-server
[filter:recon]
use = egg:swift#recon
recon_cache_path = /var/cache/swift
object-server.conf 파일 수정
[DEFAULT]
bind_ip = 192.168.0.131
bind_port = 6200
user = swift
swift_dir = /etc/swift
devices = /srv/node
mount_check = True
[pipeline:main]
pipeline = healthcheck recon object-server
[filter:recon]
use = egg:swift#recon
recon_cache_path = /var/cache/swift
recon_lock_path = /var/lock
수정 이후 아래 명령어 실행
chown -R swift:swift /srv/node
mkdir -p /var/cache/swift
chown -R root:swift /var/cache/swift
chmod -R 775 /var/cache/swift
controller : proxy-server 설정
cd /etc/swift
swift-ring-builder account.builder create 12 3 1
for i in b c d ; do swift-ring-builder account.builder add --region 1 --zone 1 --ip 192.168.0.132 --port 6202 --device sd$i --weight 100; done
for i in b c d ; do swift-ring-builder account.builder add --region 1 --zone 2 --ip 192.168.0.133 --port 6202 --device sd$i --weight 100; done
for i in b c d ; do swift-ring-builder account.builder add --region 1 --zone 3 --ip 192.168.0.134 --port 6202 --device sd$i --weight 100; done
swift-ring-builder account.builder rebalance
swift-ring-builder container.builder create 12 3 1
for i in b c d ; do swift-ring-builder container.builder add --region 1 --zone 1 --ip 192.168.0.132 --port 6201 --device sd$i --weight 100; done
for i in b c d ; do swift-ring-builder container.builder add --region 1 --zone 2 --ip 192.168.0.133 --port 6201 --device sd$i --weight 100; done
for i in b c d ; do swift-ring-builder container.builder add --region 1 --zone 3 --ip 192.168.0.134 --port 6201 --device sd$i --weight 100; done
swift-ring-builder container.builder rebalance
swift-ring-builder object.builder create 12 3 1
for i in b c d ; do swift-ring-builder object.builder add --region 1 --zone 1 --ip 192.168.0.132 --port 6200 --device sd$i --weight 100; done
for i in b c d ; do swift-ring-builder object.builder add --region 1 --zone 2 --ip 192.168.0.133 --port 6200 --device sd$i --weight 100; done
for i in b c d ; do swift-ring-builder object.builder add --region 1 --zone 3 --ip 192.168.0.134 --port 6200 --device sd$i --weight 100; done
swift-ring-builder object.builder rebalance
이후 생성된 파일 각 swift1-3 노드로 복사
scp /etc/swift/*.gz 192.168.0.131:/etc/swift
# password input
scp /etc/swift/*.gz 192.168.0.132:/etc/swift
# password input
scp /etc/swift/*.gz 192.168.0.133:/etc/swift
# password input
swift.conf 파일 다운로드
curl -o /etc/swift/swift.conf https://git.openstack.org/cgit/openstack/swift/plain/etc/swift.conf-sample?h=stable/ocata
아래 내용 추가
[swift-hash]
swift_hash_path_suffix = HASH_PATH_SUFFIX
swift_hash_path_prefix = HASH_PATH_PREFIX
[storage-policy:0]
name = Policy-0
default = yes
수정된 파일 복사
scp /etc/swift/swift.conf 192.168.0.131:/etc/swift/
# password input
scp /etc/swift/swift.conf 192.168.0.132:/etc/swift/
# password input
scp /etc/swift/swift.conf 192.168.0.133:/etc/swift/
# password input
controller, swift1-3 : 마무리 및 서비스 시작
소유권 설정
chown -R root:swift /etc/swift
SELinux 권한 수정
chcon -R system_u:object_r:swift_data_t:s0 /srv/node
서비스 시작
controller
systemctl enable openstack-swift-proxy.service memcached.service
systemctl start openstack-swift-proxy.service memcached.service
swift 1-3
systemctl enable openstack-swift-account.service openstack-swift-account-auditor.service openstack-swift-account-reaper.service openstack-swift-account-replicator.service
systemctl start openstack-swift-account.service openstack-swift-account-auditor.service openstack-swift-account-reaper.service openstack-swift-account-replicator.service
systemctl enable openstack-swift-container.service openstack-swift-container-auditor.service openstack-swift-container-replicator.service openstack-swift-container-updater.service
systemctl start openstack-swift-container.service openstack-swift-container-auditor.service openstack-swift-container-replicator.service openstack-swift-container-updater.service
systemctl enable openstack-swift-object.service openstack-swift-object-auditor.service openstack-swift-object-replicator.service openstack-swift-object-updater.service
systemctl start openstack-swift-object.service openstack-swift-object-auditor.service openstack-swift-object-replicator.service openstack-swift-object-updater.service
설치 확인
오픈스택 동작확인
[root@controller ~]# openstack container create ryusstory
+---------------------------------------+-----------+------------------------------------+
| account | container | x-trans-id |
+---------------------------------------+-----------+------------------------------------+
| AUTH_ccac596c068b4e1b941626e1931dfbff | ryusstory | txe8ca8de85c194f3ea958f-005afa8892 |
+---------------------------------------+-----------+------------------------------------+
[root@controller ~]# echo "ryusstory.tistory.com" > FILE
[root@controller ~]#
[root@controller ~]# openstack object create ryusstory FILE
+--------+-----------+----------------------------------+
| object | container | etag |
+--------+-----------+----------------------------------+
| FILE | ryusstory | 4f41e21a6e53f7dd95088b85985ddacb |
+--------+-----------+----------------------------------+
[root@controller ~]#
[root@controller ~]# openstack object list ryusstory
+------+
| Name |
+------+
| FILE |
+------+
[root@controller ~]#
오픈스택 대시보드 확인
File Explorer 동작 확인
다운로드 및 설치
오브젝트 스토리지라 S3 browser와 같은 프로그램이 있을것 같아서 확인 및 테스트
Cloudberry에서 제공하는 explorer 사용.
hosts 파일 수정
윈도우라면 c:\windows\system32\drivers\etc\hosts 파일에 아래 내용 추가
192.168.0.11 controller
접속 설정
그리고 아래 내용으로 추가 api_key는 admin 패스워드 AIO 설치시 ADMIN_PASS
업로드 테스트
18.3MB/s
다운로드 테스트
64 MB/s
