红帽的RDO部署工具，经过好几个版本的迭代，目前部署起来已经很快很方便了。但是如果要应用在生产环境，还是需要定制下配置。
我这里根据自己的Vlan环境做了几点配置上的修改。
- Mariadb 配置
- 开启rabbitmq-server的webUI
- 更新Qemu和Libvirtd版本
- 支持metadata服务
- 添加Neutron的Qos policy支持
- 配置nova.conf,支持冷迁移和在线块迁移
- 内核优化
MariaDB
Mariadb的max open files参数默认是1024，当OpenStack集群扩展时，数据库会变成瓶颈。
$ cat /usr/lib/systemd/system/mariadb.service
[Service]
...
LimitNOFILE=65535
如果条件可以，可以对Mariadb做适当的优化如下.
$ cat /etc/my.cnf
...
[mysqld]
symbolic-links=0
skip-name-resolve
max_allowed_packet = 4M
max_connections = 4096
max_connect_errors = 1024
open_files_limit = 65535
table_open_cache = 1024
innodb_autoinc_lock_mode=2
innodb_locks_unsafe_for_binlog=1
innodb_flush_log_at_trx_commit=2
innodb_buffer_pool_size = 2048M
innodb_log_file_size= 512M
innodb_log_buffer_size=16M
innodb_additional_mem_pool_size=64M
query_cache_size = 0
query_cache_type=0
thread_cache_size = 80
!includedir /etc/my.cnf.d
然后重启mariadb
$ systemctl daemon-reload && systemctl restart mariadb
开启RabbitMQ的WebUI
有时候需要登录到RabbitMQ的后台UI，更直观方便管理，以及观察消息服务器的整体状况。
$ rabbitmq-plugins enable rabbitmq_management && systemctl restart rabbitmq-server
更新Qemu和Libvirtd版本
现在RDO默认都添加了红帽的QEMU-KVM源了，如果没加，就需要自己手动添加源，更新libvirt套件；
tips：我之前在Mitaka版本中测试过，在虚拟机支持块迁移时，需要更新redhat的虚拟化套件，qemu至少需要1.3+。我的环境都升级到libvirtd-2.0.0和qemu-kvm-ev-2.6.0。
cat CentOS-QEMU-EV.repo
# CentOS-QEMU-EV.repo
#
# Please see http://wiki.centos.org/SpecialInterestGroup/Virtualization for more
# information
[centos-qemu-ev]
name=CentOS-$releasever - QEMU EV
baseurl=http://mirror.centos.org/centos/$releasever/virt/$basearch/kvm-common/
gpgcheck=1
enabled=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-SIG-Virtualization
[centos-qemu-ev-test]
name=CentOS-$releasever - QEMU EV Testing
baseurl=http://buildlogs.centos.org/centos/$releasever/virt/$basearch/kvm-common/
gpgcheck=1
enabled=0
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-SIG-Virtualization
支持Metadata服务
默认的neutron-metadata服务依赖l3层的neutron-ns-metadata-proxy代理，由于我的环境是二层的Vlan环境，并不是由router port充当metadataip。所以我们需要dhcp的port来代理，需要修改下dhcp的配置。
/etc/neutron/dhcp_agent.ini
...
enable_isolated_metadata = True
force_metadata = True
...
重启dhcp服务
$ systemctl restart neutron-dhcp-agent
重启成功后系统里就会有/bin/neutron-ns-metadata-proxy进程出现，虚拟机里面重启下network就有一条169.254.169.254的路由表信息。
添加Neutron的Qos policy支持
默认RBO安装的Neutron是没有Qos Policy的，需要自己手动修改Neutron配置开启功能。
$ cat /etc/neutron/neutron.conf
service_plugins=neutron_lbaas.services.loadbalancer.plugin.LoadBalancerPluginv2,router,metering,firewall,neutron.services.qos.qos_plugin.QoSPlugin
以下网络节点和所有计算节点agent都需要修改
$ cat /etc/neutron/plugins/ml2/ml2_conf.ini
[ml2]
extension_drivers = port_security,qos
$ cat /etc/neutron/plugins/ml2/openvswitch_agent.ini
[agent]
extensions = qos
允许租户创建qos policy，打开允许住户创建qos的权限。
To enable bandwidth limit rule:
"get_policy_bandwidth_limit_rule": "rule:regular_user",
"create_policy_bandwidth_limit_rule": "rule:admin_only",
"delete_policy_bandwidth_limit_rule": "rule:admin_only",
"update_policy_bandwidth_limit_rule": "rule:admin_only",
"get_rule_type": "rule:regular_user",
To enable DSCP marking rule:
"get_policy_dscp_marking_rule": "rule:regular_user",
"create_dscp_marking_rule": "rule:admin_only",
"delete_dscp_marking_rule": "rule:admin_only",
"update_dscp_marking_rule": "rule:admin_only",
"get_rule_type": "rule:regular_user",
重启neutron服务(包含网络节点和计算节点)
$ systemctl restart neutron-server neutron-dhcp-agent neutron-metadata-agent neutron-openvswitch-agent
$ systemctl restart neutron-openvswitch-agent
配置nova.conf,支持冷迁移和在线块迁移
OpenStack中虚拟机迁移都是通过nova账号传输镜像文件，需要对nova配置ssh无密码访问。
nova支持迁移功能需要做以下几个操作：
1. nova账号可登陆和配置密码
usermod -s /bin/bash nova
echo "nova:novamigrate" | chpasswd
2. 生成ssh证书，同步公私钥到/var/lib/nova/.ssh目录下
3. 修改nova.conf
allow_resize_to_same_host=true
inject_password=false
#虚拟机在线迁移flag （只适用Mitaka）
live_migration_flag=VIR_MIGRATE_UNDEFINE_SOURCE, VIR_MIGRATE_PEER2PEER, VIR_MIGRATE_LIVE
block_migration_flag=VIR_MIGRATE_UNDEFINE_SOURCE, VIR_MIGRATE_PEER2PEER, VIR_MIGRATE_LIVE, VIR_MIGRATE_NON_SHARED_INC
cpu_mode=custom
cpu_model=kvm64
tips: 我经过测试live_migration_flag中取消了VIR_MIGRATE_TUNNELLED在线块迁移才通过。
4. 如果sshd更改了端口，需要在/var/lib/nova/.ssh目录下创建ssh的config，定义port
5. 更新集群所有节点的/etc/hosts文件
6. 重启nova-compute服务
内核优化
这里我就不过多解释了，由于是通过ansible推送下去的，所以有些配置根据自己情况配置，一千台服务器有一千种优化方式，我这里只做一个建议。
{%- set mem_total_bytes = ansible_memtotal_mb * 1024 * 1024 -%}
# See http://www.cyberciti.biz/faq/linux-tcp-tuning/
kernel.panic = 10
kernel.panic_on_oops = 10
kernel.sysrq=0
kernel.ctrl-alt-del=1
kernel.core_pattern=/dev/null
kernel.shmmax = {{ mem_total_bytes//2 }}
kernel.shmall = {{ mem_total_bytes//2//4096 }}
kernel.msgmnb = 65536
kernel.msgmax = 65536
fs.file-max = 819200
vm.swappiness = 0
vm.dirty_ratio = 60
vm.dirty_background_ratio = 5
vm.min_free_kbytes = {{ mem_total_bytes//1024//100*5 }}
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_synack_retries = 2
net.ipv4.tcp_dsack = 1
net.ipv4.tcp_sack = 1
net.ipv4.tcp_fack = 1
net.ipv4.conf.all.forwarding = 0
net.ipv4.conf.default.forwarding = 0
net.ipv4.ip_forward = 0
net.ipv4.conf.all.send_redirects = 0
net.ipv4.conf.default.send_redirects = 0
net.ipv4.conf.all.accept_source_route = 0
net.ipv4.conf.default.accept_source_route = 0
net.ipv4.conf.all.rp_filter = 1
net.ipv4.conf.default.rp_filter = 1
net.ipv4.conf.all.accept_redirects = 0
net.ipv4.conf.default.accept_redirects = 0
net.ipv4.conf.all.secure_redirects = 0
net.ipv4.conf.default.secure_redirects = 0
net.ipv4.conf.all.bootp_relay = 0
net.ipv4.conf.all.proxy_arp = 0
net.ipv4.icmp_echo_ignore_broadcasts = 1
net.ipv4.icmp_ignore_bogus_error_responses = 1
net.ipv4.tcp_rfc1337 = 1
net.ipv4.tcp_congestion_control = cubic
net.ipv4.tcp_ecn = 2
net.ipv4.tcp_reordering = 3
net.ipv4.tcp_retries2 = 8
net.ipv4.tcp_retries1 = 3
net.ipv4.tcp_no_metrics_save = 1
net.ipv4.tcp_slow_start_after_idle = 0
net.ipv4.tcp_fin_timeout = 10
net.ipv4.tcp_keepalive_time = 300
net.ipv4.tcp_keepalive_probes = 5
net.ipv4.tcp_keepalive_intvl = 15
net.ipv4.ip_local_port_range = 8192 65535
net.core.netdev_max_backlog = 16384
net.ipv4.tcp_max_syn_backlog = 65535
net.core.somaxconn = 16384
net.ipv4.tcp_window_scaling = 1
net.ipv4.tcp_adv_win_scale = 2
net.ipv4.tcp_rmem = 4096 524287 16777216
net.core.rmem_default = 524287
net.core.rmem_max = 16777216
net.ipv4.tcp_wmem = 4096 524287 16777216
net.core.wmem_default = 524287
net.core.wmem_max = 16777216
net.core.optmem_max = 524287
net.ipv4.tcp_timestamps = 1
net.ipv4.tcp_tw_recycle = 1
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_max_tw_buckets = 360000
net.ipv4.neigh.default.gc_thresh3 = 2048
net.ipv4.neigh.default.gc_thresh2 = 1024
net.ipv4.neigh.default.gc_thresh1 = 128
net.ipv4.neigh.default.gc_interval = 120
net.ipv4.route.flush = 1
#net.netfilter.nf_conntrack_max = 4194304
#net.netfilter.nf_conntrack_tcp_timeout_established = 86400
#net.netfilter.nf_conntrack_tcp_timeout_fin_wait = 60
#net.netfilter.nf_conntrack_tcp_timeout_syn_recv = 13
#net.netfilter.nf_conntrack_tcp_timeout_time_wait = 60
net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1
