------------
- Ingress provides load balancing, SSL termination, and name-based virtual
- A type of Layer-7 load balancer hosting
- Needs an ingress controller
- Multiple ingress controller can be deployed but must be annotated
- Ingress resource is the one that consumes that uses the ingress controller
- In GKE, you can create an ingress resource and it will use the gke controllers GKE controllers don't provide HTTP to HTTPS redirection, you need to deploy your own ingress-controller to achieve it.
- Limited to 80/tcp and 443/tcp only
Annotations
-----------
# Ingress Annotations
This file defines a list of annotations which are supported by various Ingress controllers (both those based on the common ingress code, and alternative implementations).
The intention is to ensure the maximum amount of compatibility between different implementations.
All annotations are assumed to be prefixed with `ingress.kubernetes.io/` except where otherwise specified.
There is no attempt to record implementation-specific annotations using other prefixes.
(Traefik in particular defines several of its own annotations which are not described here, and does not seem to support any of the standard annotations.)
Key:
* `nginx`: the `kubernetes/ingress` nginx controller
* `gce`: the `kubernetes/ingress` GCE controller
* `traefik`: Traefik's built-in Ingress controller
* `voyager`: [Voyager by AppsCode](https://github.com/appscode/voyager) - Secure HAProxy based Ingress Controller for Kubernetes
* `haproxy`: Joao Morais' [HAProxy Ingress controller](https://github.com/jcmoraisjr/haproxy-ingress)
* `trafficserver`: Torchbox's [Apache Traffic Server controller plugin](https://github.com/torchbox/k8s-ts-ingress)
## TLS-related
| Name | Meaning | Default | Controller
| --- | --- | --- | --- |
| `ssl-passthrough` | Pass TLS connections directly to backend; do not offload. | `false` | nginx, voyager, haproxy
| `ssl-redirect` | Redirect non-TLS requests to TLS when TLS is enabled. | `true` | nginx, voyager, haproxy, trafficserver
| `force-ssl-redirect` | Redirect non-TLS requests to TLS even when TLS is not configured. | `false` | nginx, voyager, trafficserver
| `secure-backends` | Use TLS to communicate with origin (pods). | `false` | nginx, voyager, haproxy, trafficserver
| `kubernetes.io/ingress.allow-http` | Whether to accept non-TLS HTTP connections. | `true` | gce
| `pre-shared-cert` | Name of the TLS certificate in GCP to use when provisioning the HTTPS load balancer. | empty string | gce
| `hsts-max-age` | Set an HSTS header with this lifetime. | | voyager, trafficserver
| `hsts-include-subdomains` | Add includeSubdomains to the HSTS header. | | voyager, trafficserver
## Authentication related
| Name | Meaning | Default | Controller
| --- | --- | --- | --- |
| `auth-type` | Authentication type: `basic`, `digest`, ... | | nginx, voyager, haproxy, trafficserver
| `auth-secret` | Secret name for authentication. | | nginx, voyager, haproxy, trafficserver
| `auth-realm` | Authentication realm. | | nginx, voyager, haproxy, trafficserver
| `auth-tls-secret` | Name of secret for TLS client certification validation. | | nginx, voyager, haproxy
| `auth-tls-verify-depth` | Maximum chain length of TLS client certificate. | | nginx
| `auth-tls-error-page` | The page that user should be redirected in case of Auth error | | nginx, voyager
| `auth-satisfy` | Behaviour when more than one of `auth-type`, `auth-tls-secret` or `whitelist-source-range` are configured: `all` or `any`. | `all` | trafficserver | `trafficserver`
| `whitelist-source-range` | Comma-separate list of IP addresses to enable access to. | | nginx, voyager, haproxy, trafficserver
## URL related
| Name | Meaning | Default | Controller
| --- | --- | --- | --- |
| `app-root` | Redirect requests without a path (i.e., for `/`) to this location. | | nginx, haproxy, trafficserver
| `rewrite-target` | Replace matched Ingress `path` with this value. | | nginx, trafficserver
| `add-base-url` | Add `<base>` tag to HTML. | | nginx
| `base-url-scheme` | Specify the scheme of the `<base>` tags. | | nginx
| `preserve-host` | Whether to pass the client request host (`true`) or the origin hostname (`false`) in the HTTP Host field. | | trafficserver
## CORS Related
| Name | Meaning | Default | Controller
| --- | --- | --- | --- |
| `enable-cors` | Enable CORS headers in response. | false | nginx, voyager
| `cors-allow-origin` | Specifies the Origin allowed in CORS (Access-Control-Allow-Origin) | * | nginx
| `cors-allow-headers` | Specifies the Headers allowed in CORS (Access-Control-Allow-Headers) | DNT,X-CustomHeader,Keep-Alive,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type,Authorization | nginx
| `cors-allow-methods` | Specifies the Methods allowed in CORS (Access-Control-Allow-Methods) | GET, PUT, POST, DELETE, PATCH, OPTIONS | nginx
| `cors-allow-credentials` | Specifies the Access-Control-Allow-Credentials | true | nginx
## Miscellaneous
| Name | Meaning | Default | Controller
| --- | --- | --- | --- |
| `configuration-snippet` | Arbitrary text to put in the generated configuration file. | | nginx
| `limit-connections` | Limit concurrent connections per IP address[1]. | | nginx, voyager
| `limit-rps` | Limit requests per second per IP address[1]. | | nginx, voyager
| `limit-rpm` | Limit requests per minute per IP address. | | nginx, voyager
| `affinity` | Specify a method to stick clients to origins across requests. Found in `nginx`, where the only supported value is `cookie`. | | nginx, voyager
| `session-cookie-name` | When `affinity` is set to `cookie`, the name of the cookie to use. | | nginx, voyager
| `session-cookie-hash` | When `affinity` is set to `cookie`, the hash algorithm used: `md5`, `sha`, `index`. | | nginx
| `proxy-body-size` | Maximum request body size. | | nginx, voyager, haproxyRequest Entity Too Large
| `proxy-pass-params` | Parameters for proxy-pass directives. | |
| `follow-redirects` | Follow HTTP redirects in the response and deliver the redirect target to the client. | | trafficserver
| `kubernetes.io/ingress.global-static-ip-name` | Name of the static global IP address in GCP to use when provisioning the HTTPS load balancer. | empty string | gce
[1] The documentation for the `nginx` controller says that only one of `limit-connections` or `limit-rps` may be specified; it's not clear why this is.
## Caching
"
| Name | Meaning | Default | Controller
| --- | --- | --- | --- |
| `cache-enable` | Cache responses according to Expires or Cache-Control headers. | | trafficserver
| `cache-generation` | An arbitrary numeric value included in the cache key; changing this effectively clears the cache for this ingress. | | trafficserver
| `cache-ignore-query-params` | Space-separate list of globs matching URL parameters to ignore when doing cache lookups. | | trafficserver
| `cache-whitelist-query-params` | Ignore any URL parameters not in this whitespace-separate list of globs. | | trafficserver
| `cache-sort-query-params` | Lexically sort the query parameters by name before cache lookup. | | trafficserver
| `cache-ignore-cookies` | Requests containing a `Cookie:` header will not use the cache unless all the cookie names match this whitespace-separate list of globs. | | trafficserver
Manifests
---------
Simple Ingress resource | apiVersion: extensions/v1beta1 kind: Ingress metadata: name: test-ingress annotations: nginx.ingress.kubernetes.io/rewrite-target: / spec: rules: - http: paths: - path: /testpath backend: serviceName: test servicePort: 80
Notes: The annotation "nginx.ingress.kubernetes.io/rewrite-target: /" can be used if the backend pod's nginx/apache config doesn't contain /testpath explicitly. Removing the annotation will result in the following error:
"/usr/share/nginx/html/testpath/index.html" is not found
So we will rewrite it so it will return back to base path which is /. |
Redirection based on paths | apiVersion: extensions/v1beta1 kind: Ingress metadata: name: simple-fanout-example annotations: nginx.ingress.kubernetes.io/rewrite-target: / spec: rules: - host: foo.bar.com # make sure you are calling the site using this hostname (Host-Header is correct) http: paths: - path: /foo backend: serviceName: service1 servicePort: 4200 - path: /bar backend: serviceName: service2 servicePort: 8080
Notes: "host: foo.bar.com" can be a DNS entry pointing to a load balancer IP. That loadbalancer IP's backend must be the hostnames of the kubernetes nodes. If "host: " s not specified, you can use any of the nodes's hostname to access the ingress resource. |
Simple apache ingress | apiVersion: extensions/v1beta1 kind: Ingress metadata: name: basic-ingress spec: backend: serviceName: apache servicePort: 80 |
SSL redirection | apiVersion: extensions/v1beta1 kind: Ingress metadata: name: nginx-test spec: tls: - hosts: - foo.bar.com # This assumes tls-secret exists and the SSL # certificate contains a CN for foo.bar.com # Make sure secret contains tls.crt and tls.key data secretName: tls-secret rules: - host: foo.bar.com http: paths: - path: / backend: # This assumes http-svc exists and routes to healthy endpoints serviceName: http-svc servicePort: 80 |
w/o rules, useful if rule/path | apiVersion: extensions/v1beta1 kind: Ingress metadata: name: basic-ingress namespace: dafabet-ghana-prd spec: tls: - hosts: - prd-opera.dafabet.gh secretName: dafabet-ghana-cert backend: serviceName: php servicePort: 80 |
Using xip | --- apiVersion: extensions/v1beta1 kind: Ingress metadata: annotations: kubernetes.io/ingress.class: k8s-nginx labels: app: prometheus name: prometheus namespace: monitoring spec: rules: - host: prometheus.192.168.32.26.xip.io http: paths: - backend: serviceName: prometheus-server servicePort: 9090 path: / |
Tutorials
---------
Basic setup | 1. Create a deployment cat << EOF > /tmp/sample.yml --- apiVersion: extensions/v1beta1 kind: Deployment metadata: name: nginx spec: replicas: 1 template: metadata: labels: app: nginx spec: containers: - image: nginx name: nginx ports: - containerPort: 80 name: nginxport --- kind: Service apiVersion: v1 metadata: name: nginx spec: type: NodePort selector: app: nginx ports: - port: 80 targetPort: 80 nodePort: 30548 EOF
2. Create ingress resource
cat << EOF > /tmp/ing.yml --- apiVersion: extensions/v1beta1 kind: Ingress metadata: name: test-ingress annotations: nginx.ingress.kubernetes.io/rewrite-target: / spec: rules: - host: sample.ingress.com http: paths: - path: backend: serviceName: nginx servicePort: 80 EOF
3. Apply manifests
$ kubectl apply -f /tmp/sample.yml $ kubectl apply -f /tmp/ing.yml
4. Access now your pod via ingress
# curl http -H 'Host: sample.ingress.com' http://kube-node-ip |
Setting up ingress | 1. Setup NGINX ingress controller
curl https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/namespace.yaml | kubectl apply -f - curl https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/default-backend.yaml | kubectl apply -f - curl https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/configmap.yaml | kubectl apply -f - curl https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/tcp-services-configmap.yaml | kubectl apply -f - curl https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/udp-services-configmap.yaml | kubectl apply -f - curl https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/rbac.yaml | kubectl apply -f - curl https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/with-rbac.yaml | kubectl apply -f - curl https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/provider/baremetal/service-nodeport.yaml | kubectl apply -f -
2. Verify installation
kubectl get pods --all-namespaces -l app=ingress-nginx --watch
3. Deploy config map for ingress controller
curl https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/configmap.yaml | kubectl apply -f -
4. Create nginx deployment with 2 replicas
kubectl run nginx-sample --image=nginx --replicas=2 --namespace=ingress-nginx
5. Create service for each pod
kubectl expose svc/pod-name-1 --name=s1 --port=80 --type=NodePort -n ingress-nginx kubectl expose svc/pod-name-2 --name=s2 --port=80 --type=NodePort -n ingress-nginx
6. Create SSL certificates
openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout tls.key -out tls.crt -subj "/CN=nginxsvc/O=nginxsvc"
7. Create secret and use the certificates above
kubectl create secret tls tls-secret --key tls.key --cert tls.crt -n ingress-nginx
8. Create Ingress resource
apiVersion: extensions/v1beta1 kind: Ingress metadata: name: my-ingress namespace: ingress-nginx spec: tls: - hosts: - ingress.msgreen.dom secretName: tls-secret rules: - host: ingress.msgreen.dom http: paths: - path: backend: serviceName: s1 servicePort: 80 - path: backend: serviceName: s2 servicePort: 80
9. Apply resource
kubectl create -f ingress.yml
10. Watch until IPs are generated
kubectl get ing -n ingress-nginx --watch
11. Once IPs are generated, test now (make sure the port is exposed on the IP's firewall)
curl <IP> -L |
Installing via HELM | helm install stable/nginx-ingress --name my-nginx |
HTTP to HTTPS redirection via nginxinc/kubernetes-ingress | I was able to achieve this in a GKE cluster (master version v1.10.6-gke.2) but not on using ingress-controller provided by kubernetes/ingress-nginx repo. cd kubernetes-ingress/deployments/helm-chart helm install --name nginx-controller . cat << EOF >> sample-ingress.yml apiVersion: extensions/v1beta1 kind: Ingress metadata: name: basic-ingress annotations: kubernetes.io/ingress.class: "nginx" spec: tls: - hosts: - my.website.com secretName: web-certs rules: - host: my.website.com http: paths: - path: / backend: serviceName: php servicePort: 80f EOF kubectl apply -f sample-ingress.yml
4. Wait for the ingress IP to come up 5. Test http to https redirection using the IP generated in previous step |
Troubleshooting
---------------
Nginx Ingress controller default backend panic | You see the following error from pod logs: panic: runtime error: invalid memory address or nil pointer dereference [recovered] panic: runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0x38 pc=0x4b8987] |
|
Can't push docker image to Nexus registry behind ingress and custom nginx proxy | Problem -------
Nexus registry is running behind an nginx proxy pod whose config contains `client_max_body_size 0`. There is also an ingress in front of the services.
client --> 443 --> load balancer --> 443 --> ingress --> 80 --> nginx-proxy pod --> 8082 --> nexus registry pod
During docker push from the client, it encounters the following issue.
error parsing HTTP 413 response body: invalid character '<' looking for beginning of value: "<html>\r\n<head><title>413 Request Entity Too Large</title></head>\r\n<body>\r\n<center><h1>413 Request Entity Too Large</h1></center>\r\n<hr><center>nginx/1.15.6</center>\r\n</body>\r\n</html>\r\n"
It shouldn't happen because I already disabled checking of client_max_body_size in nginx-proxy pod.
Solution --------
I found out that ingress controller's `client_max_body_size` is set to 1M. I increased it by add the following annotation on the ingress resource.
nginx.ingress.kubernetes.io/proxy-body-size: 0
Meaning its not sufficient that the internal pod proxy already has this setting. We also need to take into account the setting on the ingress. |
|
No comments:
Post a Comment