Thursday, December 27, 2018

Troubleshooting Docker Issues



Corrupted DB
Issue:

The following message appear when starting docker:
updating the store state of sandbox failed: failed to update store for object type *libnetwork.sbState: json: cannot unmarshal string into Go struct field sbState.ExtDNS of type libnetwork.extDNSEntry

Solution:
 
systemctl stop docker
mv /var/lib/docker/network/files/local-kv.db /root/corrupted-local-kv.db
systemctl start docker

Source:
Can't start docker
Resolution:
Try "journalctl -u docker.service" and see what's happening

Mar 16 13:46:53 gdc-co-ragent01 dockerd[1403]: can't create unix socket /var/run/docker.sock: is a directory
Mar 16 13:46:53 gdc-co-ragent01 systemd[1]: docker.service: main process exited, code=exited, status=1/FAILURE
Mar 16 13:46:53 gdc-co-ragent01 systemd[1]: Failed to start Docker Application Container Engine. 

Resolution:
rm -fr /var/run/docker.sock
systemctl start docker
Phantom containers causes
docker unable to run containers
with same name
8a81cf7cb9f3fd03e0139743d3616eb0e/kill returned error: Cannot kill container 78d3e2cc93abc053238e0edd5765f428a81cf7cb9f3fd03e0139743d3616eb0e: Container 78d3e2cc93abc053238e0edd5765f428a81cf7cb9f3fd03e0139743d3616eb0e is not running"

Resolution:
1. check those phantom containers: doker ps -a
2. remove them: docker rm -f
containers cannot ping outside IPs
Things you might want to check:
1. Make sure /proc/sys/net/ipv4/ip_forward is set to 1
No more space left on thin pool
Resolution:
Extend the thinpool:
  lvextend -L +10G /dev/mapper/base-thinpool
Error on docker-compose
Running docker-compose as a container with the following error:

Couldn't find `docker` binary. You might need to install Docker

Resolution:
Create .env on the directory where you are running docker-compose and put the following:
COMPOSE_INTERACTIVE_NO_CLI=1
Cannot login to docker registry due to missing port
Issue:

The following message appear when you login to docker registry.

Error response from daemon: Get https://gitlab.xyz.com:4567/v2/: Get https://gitlab.xyz.com/jwt/auth?account=root&client_id=docker&offline_token=true&service=container_registry: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) (Client.Timeout exceeded while awaiting headers)

Resolution:
Make sure docker port and 443/tcp is allowed from source to destination

https://medium.com/@dan.lindow/docker-login-error-awaiting-headers-3fe01e2a1e2f
Resource Temporary unavailable due
to TaskMax setting
- try adding "TasksMax=infinity" in docker.service
- restart docker
- monitor for occurence of issue

https://success.docker.com/article/how-to-reserve-resource-temporarily-unavailable-errors-due-to-tasksmax-setting
https://github.com/chef-cookbooks/docker/issues/871

No comments:

Post a Comment