在这个容器化技术盛行的时代,大家都习惯采用 Docker 或者 K8S 来运行 APISIX。APISIX 的配置参数非常多,因此很多介绍文章都采用挂载文件或者 K8S Configmap 的方式来配置 APISIX。最开始我们就采用 Configmap 的方式在腾讯云 TKE 上部署 APISIX,当网络区域越开越多时,每个 TKE 集群都需要去定义一套 config.yaml 对应的 Configmap,管理非常繁琐。因此,这里我们利用 Python 的 Jinja2 插件来自动化渲染 APISIX 的配置文件,整体非常方便!
一、Jinja2 模板
熟悉 Jinja2 的同学都很清楚,要通过 Jinja2 生成所需文件,需要先定制一个渲染模板,Jinja2 的原理就是将动态的内容填充到模板中,最终渲染成所需文件。因此,这里参考 APISIX 官方最新 2.10.0 版本con
apisix:
# node_listen: {{ node_listen | default(9080) | int }} # support to Specific IP since 2.10.0
node_listen:
- ip: {{ http_listen_ip | default("0.0.0.0") }}
port: {{ http_listen_port | default(9080) | int }}
enable_http2: {{ http_enable_http2 | default("false") }}
enable_admin: {{ enable_admin | default("true") }}
enable_admin_cors: {{ enable_admin_cors | default("true") }}
# enable_debug: {{ enable_debug | default("false") }} already move to debug.yaml since 2.10.0
enable_dev_mode: {{ enable_dev_mode | default("false") }}
enable_reuseport: {{ enable_reuseport | default("true") }}
enable_ipv6: {{ enable_ipv6 | default("false") }}
config_center: {{ config_center | default("etcd") }}
enable_server_tokens: {{ enable_server_tokens | default("true") }}
extra_lua_path: {{ extra_lua_path | default("") }}
extra_lua_cpath: {{ extra_lua_cpath | default("") }}
proxy_cache:
cache_ttl: {{ cache_ttl | default("10s") }}
zones:
- name: {{ proxy_cache_zones | default("disk_cache_one") }}
memory_size: {{ proxy_cache_memory_size | default("50m") }}
disk_size: {{ proxy_cache_disk_size | default("1G") }}
disk_path: {{ proxy_cache_disk_path | default("/tmp/disk_cache_one") }}
cache_levels: {{ proxy_cache_cache_levels | default("1:2") }}
allow_admin:
{% if not allow_admin_subnet: -%}
{%- set allow_admin_subnet = "" -%}
{%- endif -%}
{%- for item in allow_admin_subnet.split(",") -%}
{% if item: -%}
- {{item}}
{% endif -%}
{% endfor %}
admin_key:
-
name: {{ admin_key_name | default("admin") }}
key: {{ admin_key_secret | default("d208uj44fnd2yk6quczd6szkytvoi0x1") }}
role: admin
-
name: {{ viewer_key_name | default("viewer") }}
key: {{ viewer_key_secret | default("4054f7cf07e344346cd3f287985e76a2") }}
role: viewer
delete_uri_tail_slash: {{ delete_uri_tail_slash | default("false") }}
global_rule_skip_internal_api: {{ global_rule_skip_internal_api | default("true") }}
router:
http: {{ router_http | default("radixtree_uri") }}
ssl: {{ router_ssl | default("radixtree_sni") }}
resolver_timeout: {{ resolver_timeout | default(3) | int }}
enable_resolv_search_opt: {{ enable_resolv_search_opt | default("true") }}
ssl:
enable: {{ ssl_enable | default("true") }}
enable_http2: {{ ssl_enable_http2 | default("true") }}
listen:
- ip: {{ https_listen_ip | default("0.0.0.0") }}
port: {{ https_listen_port | default(9443) | int }}
enable_http2: {{ https_enable_http2 | default("true") }}
ssl_protocols: {{ ssl_protocols | default("TLSv1.2 TLSv1.3") }}
ssl_ciphers: {{ ssl_ciphers | default("ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384") }}
ssl_session_tickets: {{ ssl_session_tickets | default("false") }}
key_encrypt_salt: {{ key_encrypt_salt | default("edd1c9f0985e76a2") }}
enable_control: {{ enable_control | default("true") }}
control:
ip: {{ control_ip | default("127.0.0.1") }}
port: {{ control_port | default(9090) | int }}
disable_sync_configuration_during_start: {{ disable_sync_configuration_during_start | default("false") }}
nginx_config:
user: {{ nginx_user | default("root") }}
error_log: {{ error_log | default("/dev/stdout") }}
error_log_level: {{ error_log_level | default("warn") }}
worker_processes: {{ worker_processes | default(1) | int }}
enable_cpu_affinity: {{ enable_cpu_affinity | default("true") }}
worker_rlimit_nofile: {{ worker_rlimit_nofile | default(20480) | int }}
worker_shutdown_timeout: {{ worker_shutdown_timeout | default("240s") }}
event:
worker_connections: {{ worker_connections | default(20480) | int }}
envs:
{%- for item in nginx_config_env.split(",") -%}
{% if item: -%}
- {{item}}
{% endif -%}
{% endfor %}
stream:
lua_shared_dict:
etcd-cluster-health-check-stream: 10m
lrucache-lock-stream: 10m
plugin-limit-conn-stream: 10m
{% if not stream_lua_shared_dicts: -%}
{%- set stream_lua_shared_dicts = "" -%}
{%- endif -%}
{%- for item in stream_lua_shared_dicts.split(",") -%}
{% if item: -%}
{{item}}
{%- endif -%}
{% endfor %}
main_configuration_snippet: |
{% if not main_configuration_snippet: -%}
{%- set main_configuration_snippet = "" -%}
{%- endif -%}
{%- for item in main_configuration_snippet.split(",") -%}
{% if item: -%}
{{item}}
{% endif -%}
{% endfor %}
http_configuration_snippet: |
{% if not http_configuration_snippet: -%}
{%- set http_configuration_snippet = "" -%}
{%- endif -%}
{%- for item in http_configuration_snippet.split(",") -%}
{% if item: -%}
{{item}}
{% endif -%}
{% endfor %}
http_server_configuration_snippet: |
{% if not http_server_configuration_snippet: -%}
{%- set http_server_configuration_snippet = "" -%}
{%- endif -%}
{%- for item in http_server_configuration_snippet.split(",") -%}
{% if item: -%}
{{item}}
{% endif -%}
{% endfor %}
http_admin_configuration_snippet: |
{% if not http_admin_configuration_snippet: -%}
{%- set http_admin_configuration_snippet = "" -%}
{%- endif -%}
{%- for item in http_admin_configuration_snippet.split(",") -%}
{% if item: -%}
{{item}}
{% endif -%}
{% endfor %}
http_end_configuration_snippet: |
{% if not http_end_configuration_snippet: -%}
{%- set http_end_configuration_snippet = "" -%}
{%- endif -%}
{%- for item in http_end_configuration_snippet.split(",") -%}
{% if item: -%}
{{item}}
{% endif -%}
{% endfor %}
stream_configuration_snippet: |
{% if not stream_configuration_snippet: -%}
{%- set stream_configuration_snippet = "" -%}
{%- endif -%}
{%- for item in stream_configuration_snippet.split(",") -%}
{% if item: -%}
{{item}}
{% endif -%}
{% endfor %}
http:
enable_access_log: {{ enable_access_log | default("false") }}
access_log: {{ access_log | default("/dev/stdout") }}
access_log_format: {{ access_log_format | default('\"$remote_addr - $remote_user [$time_local] $http_host \"$request\" $status $body_bytes_sent $request_time \"$http_referer\" \"$http_user_agent\" $upstream_addr $upstream_status $upstream_response_time \"$upstream_scheme://$upstream_host$upstream_uri\"\"') }}
access_log_format_escape: {{ access_log_format_escape | default("default") }}
keepalive_timeout: {{ keepalive_timeout | default("60s") }}
client_header_timeout: {{ client_header_timeout | default("60s") }}
client_body_timeout: {{ client_body_timeout | default("60s") }}
client_max_body_size: {{ client_max_body_size | default(0) }}
send_timeout: {{ send_timeout | default("10s") }}
underscores_in_headers: {{ underscores_in_headers | default("on") }}
real_ip_header: {{ real_ip_header | default("X-Real-IP") }}
real_ip_recursive: {{ real_ip_recursive | default("off") }}
real_ip_from:
- 127.0.0.1
- "unix:"
{% if not real_ip_from: -%}
{%- set real_ip_from = "" -%}
{%- endif -%}
{%- for item in real_ip_from.split(",") -%}
{% if item: -%}
- {{item}}
{% endif -%}
{% endfor %}
custom_lua_shared_dict: # use a new name to customize lua_shared_dict since 2.10.0 https://github.com/apache/apisix/pull/5030/files/2e4d5feb1e98359358bc0409b055a0b1da00c329
{% if not custom_lua_shared_dict: -%}
{%- set custom_lua_shared_dict = "" -%}
{%- endif -%}
{%- for item in custom_lua_shared_dict.split(",") -%}
{{ item }}
{% endfor %}
proxy_ssl_server_name: {{ proxy_ssl_server_name | default("true") }}
upstream:
keepalive: {{ upstream_keepalive | default(320) | int }}
keepalive_requests: {{ upstream_keepalive_requests | default(1000) | int }}
keepalive_timeout: {{ upstream_keepalive_timeout | default("60s") }}
charset: {{ charset | default("utf-8") }}
variables_hash_max_size: {{ variables_hash_max_size | default(2048) | int }}
lua_shared_dict:
internal-status: 10m
plugin-limit-req: 10m
plugin-limit-count: 10m
prometheus-metrics: 10m
plugin-limit-conn: 10m
upstream-healthcheck: 10m
worker-events: 10m
lrucache-lock: 10m
balancer-ewma: 10m
balancer-ewma-locks: 10m
balancer-ewma-last-touched-at: 10m
plugin-limit-count-redis-cluster-slot-lock: 1m
tracing_buffer: 10m
plugin-api-breaker: 10m
etcd-cluster-health-check: 10m
discovery: 1m
jwks: 1m
introspection: 10m
access-tokens: 1m
etcd:
host:
- "{{ etcd_host }}"
prefix: {{ etcd_prefix | default("/apisix") }}
timeout: {{ etcd_timeout | default(30) }}
resync_delay: {{ etcd_resync_delay | default(5) | int }}
health_check_timeout: {{ etcd_health_check_timeout | default(10) | int }}
user: {{ etcd_user | default("tapisix") }}
password: {{ etcd_password | default("") }}
tls:
verify: {{ etcd_tls_verify | default("false") }}
graphql:
max_size: 1048576
plugins:
- client-control
- ext-plugin-pre-req
- zipkin
- request-id
- fault-injection
- serverless-pre-function
- batch-requests
- cors
- ip-restriction
- ua-restriction
- referer-restriction
- uri-blocker
- request-validation
- openid-connect
- wolf-rbac
- hmac-auth
- basic-auth
- jwt-auth
- key-auth
- consumer-restriction
- authz-keycloak
- proxy-mirror
- proxy-cache
- proxy-rewrite
- api-breaker
- limit-conn
- limit-count
- limit-req
- gzip
- server-info
- traffic-split
- redirect
- response-rewrite
- grpc-transcode
- prometheus
- echo
- http-logger
- sls-logger
- tcp-logger
- kafka-logger
- syslog
- udp-logger
- serverless-post-function
- ext-plugin-post-req
{% if not custom_plugins: -%}
{%- set custom_plugins = "" -%}
{%- endif -%}
{%- for item in custom_plugins.split(",") -%}
{% if item: -%}
- {{item}}
{% endif -%}
{% endfor %}
stream_plugins:
- ip-restriction
- limit-conn
- mqtt-proxy
{% if not custom_stream_plugins: -%}
{%- set custom_stream_plugins = "" -%}
{%- endif -%}
{%- for item in custom_stream_plugins.split(",") -%}
{% if item: -%}
- {{item}}
{% endif -%}
{% endfor %}
plugin_attr:
prometheus:
export_uri: {{ prometheus_export_uri | default("/apisix/prometheus/metrics") }}
enable_export_server: {{ prometheus_enable_export_server | default("true") }}
export_addr:
ip: 0.0.0.0
port: {{ prometheus_export_port | default(9091) | int}}
server-info:
report_interval: {{ serveir_info_report_interval | default(60) | int }}
report_ttl: {{ serveir_info_report_ttl | default(3600) | int }}
discovery:
eureka:
host:
{% if not eureka_host: -%}
{%- set eureka_host = "http://eureka.demo.svc.local" -%}
{%- endif -%}
{%- set host_list = eureka_host.split(",") -%}
{%- for item in host_list -%}
{% if item: -%}
- {{item}}
{% endif -%}
{% endfor %}
prefix: "/eureka/"
fetch_interval: {{ eureka_fetch_interval | default(5) | int }}
weight: {{ eureka_weight | default(100) | int }}
timeout:
connect: {{ eureka_connect_timeout | default(2000) | int }}
send: {{ eureka_send_timeout | default(2000) | int }}
read: {{ eureka_read_timeout | default(5000) | int }}
将上述代码保存为 config-template.yaml,即 Jinja2 的渲染模板。这个模板基本覆盖到了每一个 APISIX 配置文件的内容,能够默认的就都设置了默认值,减少配置工作量。对于行数可变的多行配置,比如http_configuration_snippet 和plugins 等,我们也是通过 Jinja2 里面的遍历+英文逗号分隔的方法来支持动态配置。
二、Python 脚本
简单写一个从环境变量中提取 APISIX 变量、然后通过 Jinja2 渲染成实际配置文件的脚本:
# -*- coding:utf-8 -*-
"""APISIX 配置文件生成工具 功能描述:通过获取环境变量生成 APISIX 的配置文件。 """
import sys
import os
import requests
from jinja2 import Environment, FileSystemLoader
reload(sys)
sys.setdefaultencoding('utf-8')
class Utils():
def __init__(self):
self.path = os.path.dirname(os.path.abspath(__file__))
self.template_environment = Environment(
autoescape=False,
loader=FileSystemLoader(os.path.join(self.path, '')),
trim_blocks=False)
def render_template(self, template_filename, context):
return self.template_environment.get_template(
template_filename).render(context)
def gen_yaml_content(self, template, context):
yaml = self.render_template(template, context)
return yaml
def get_env_list(self, prefix=None, replace=True):
""" 获取环境变量 :param prefix: 指定目标变量的前缀 :param replace:指定前缀后,键名是否去掉前缀 """
env_dict = os.environ
if prefix:
env_list = {}
for key in env_dict:
if prefix in key:
if replace:
env_list[key.replace(prefix, "")] = env_dict[key]
else:
env_list[key] = env_dict[key]
return env_list
else:
return dict(env_dict)
if __name__ == "__main__":
utils = Utils()
try:
config_list = utils.get_env_list(prefix="apisix_")
content = utils.gen_yaml_content("config-template.yaml", config_list)
with open("/usr/local/apisix/conf/config.yaml", "w") as f:
f.write(content)
except Exception as error: # pylint: disable=broad-except
exit("Failed to generate configuration file: {}".format(error))
脚本会从运行系统的环境变量中提取前缀为 apisix_ 的环境变量列表, 然后通过 Jinja2 填充到配置模板中,最终生成 APISIX 的配置文件 config.yaml,整体非常简单。
我们在公司内部其实是有配置中心的,所以在实际使用中,我们是从配置中心去拉取配置然后来渲染的,这里只是分享一个方案,因此就用环境变量简单示范一下了。确实需要使用的朋友,可以将脚本改成从配置中心拉取,比如 Apollo、Zookeeper、Consul、DB 等,难度也非常小。
三、Docker 镜像
上面展示了通过执行 Python 脚本提取环境变量,快速生成 APISIX 配置文件的方案。接下来,我们将这个机制集成到 APISIX 的 Docker 镜像中,实现一个自动化配置的镜像。
1、Dockerfile 配置
Jinja2 需要 Python 环境的支持,所以这里选择 APISIX 官方的 Centos 镜像,默认自带了 Python2.7.5,只需要在这个基础上安装一下 Jinja2 插件即可。
FROM apache/apisix:2.10.0-centos
LABEL maintainer="Jager", description="支持环境变量设置任意配置的 APISIX 镜像。"
RUN yum install -y python-jinja2
# 自定义插件可以放到 plugins 目录,一并集成
COPY plugins /usr/local/apisix/apisix/plugins
COPY auto_conf /opt/auto_conf
COPY docker-entrypoint.sh /
ENTRYPOINT ["/docker-entrypoint.sh"]
CMD ["/usr/local/openresty/bin/openresty", "-p", "/usr/local/apisix", "-g", "daemon off;"]
2、docker-entrypoint.sh
因为渲染时需要执行 Python 脚本的,因此需要在 ENTRYPOINT 这里插入相关执行命令,脚本内容如下:
#!/bin/bash
set -e
# 启动前先进行 Jinja2 渲染
cd /opt/auto_conf && \
python make_conf.py >/dev/stderr 2>&1 || exit 1
# APISIX 初始化
/usr/bin/apisix init >/dev/stderr 2>&1 && \
/usr/bin/apisix init_etcd >/dev/stderr 2>&1 || exit 1
# 执行真正的启动命令
exec "$@"
3、自定义插件
在实际使用场景中,我们可能还有一些自定义的 APISIX 插件,也可以在制作这个 Docker 镜像过程中一并集成进去,比如博客前两篇文章分享的 2 个实用插件:
四、运行示例
看懂了前面的同学应该已经对如何运行是没什么疑问了。这里还是简单贴一下使用方法,方便第一次接触的同学快速上手。
其实非常简单,需要配置 APISIX 的哪个参数,只需要在 config-template.yaml 这个模板中去找对应的变量名,比如需要配置 etcd 地址,我们在 config-template.yaml 找到对应的变量名称是 etcd_host,而且是通过英文逗号分隔来配置多条的。
因此,启动命令如下:
docker run --name=apisix_test -d \
-e apisix_etcd_host=http://127.0.0.1:2379,http://127.0.0.2:2379,http://127.0.0.3:2379
<apisix 镜像名>
总之,需要改啥配置就去 config-template.yaml 找对应的变量名,然后在指定系统环境变量 apisix_<变量名>的值,如果是多行则用英文逗号分隔即可。如果在 config-template.yaml 没找到,那么就参考官方config-default.yaml来修改 Jinja2 模板:config-template.yaml。
五、其他
本文分享的方法虽然非常实用,实际上还需要安装 jinja2 然后跑 Python 脚本, 并非最优雅的方案。如果是会写 lua 脚本的朋友,可以通过lua-resty-template改造下,那就完美了!有实现的朋友记得留言分享一下成果。