Not possible to restart

[jeroenpeeters]

After using the dockerized Gitlab for a couple of days I had to restart the machine. The volumes are mapped onto the host machine. I tried to start the container again, but it failed to start. This is the error I get:

STDERR: psql: FATAL: the database system is starting up
Apr 10 13:40:39 node24.cluster.isd.org docker[14227]: ---- End output of /opt/gitlab/embedded/bin/psql --port 5432 -d template1 -c "CREATE USER gitlab" ----
Apr 10 13:40:39 node24.cluster.isd.org docker[14227]: Ran /opt/gitlab/embedded/bin/psql --port 5432 -d template1 -c "CREATE USER gitlab" returned 2

To me it looks like there is a timing issue, postgres isn't yet completely started but something is trying to connect to it.
After this error happened, the containers keeps running. Then I manually executed the CMD from the Docker file from a shell inside the container ('gitlab-ctl reconfigure & /opt/gitlab/embedded/bin/runsvdir-start'). That worked, Gitlab starts successfully.

[genezys]

These problems really depends on the platform you are running on (boot2docker, Ubuntu VM, etc.) and I cannot yet find a way to reliably start GitLab without any issues.

I am open to ideas in order to improve this.

[jeroenpeeters]

I now also encounter the problem when starting a new instance of docker-gitlab. I didn't had these problems before, have you changed something?

Also, I don't see how it should matter on which platform I'm running. The promise of Docker is that it runs independently of the platform. Everything the container needs is packaged inside the image, right?

For the record, I'm running CoreOS.

Error trace:

Error executing action run on resource 'execute[create gitlab database user]'

Mixlib::ShellOut::ShellCommandFailed

Expected process to exit with [0], but received '2'
---- Begin output of /opt/gitlab/embedded/bin/psql --port 5432 -d template1 -c "CREATE USER gitlab" ----
STDOUT:
STDERR: psql: could not connect to server: No such file or directory
Is the server running locally and accepting
connections on Unix domain socket "/tmp/.s.PGSQL.5432"?
---- End output of /opt/gitlab/embedded/bin/psql --port 5432 -d template1 -c "CREATE USER gitlab" ----
Ran /opt/gitlab/embedded/bin/psql --port 5432 -d template1 -c "CREATE USER gitlab" returned 2

Resource Declaration:

In /opt/gitlab/embedded/cookbooks/gitlab/recipes/postgresql.rb

160: execute "create #{sql_user} database user" do
161: command "#{bin_dir}/psql --port #{pg_port} -d template1 -c "CREATE USER #{sql_user}""
162: user pg_user
163: not_if { !pg_helper.is_running? || pg_helper.user_exists?(sql_user) }
164: end
165:

Compiled Resource:

Declared in /opt/gitlab/embedded/cookbooks/gitlab/recipes/postgresql.rb:160:in `block in from_file'

execute("create gitlab database user") do
action "run"
retries 0
retry_delay 2
guard_interpreter :default
command "/opt/gitlab/embedded/bin/psql --port 5432 -d template1 -c "CREATE USER gitlab""
backup 5
returns 0
user "gitlab-psql"
cookbook_name :gitlab
recipe_name "postgresql"
not_if { #code block }
end

[2015-04-14T08:30:39+00:00] INFO: Running queued delayed notifications before re-raising exception
[2015-04-14T08:30:39+00:00] INFO: template[/var/opt/gitlab/gitlab-rails/etc/gitlab.yml] sending run action to executeclear the gitlab-rails cache
[2015-04-14T08:31:00+00:00] INFO: execute[clear the gitlab-rails cache] ran successfully
[2015-04-14T08:31:00+00:00] INFO: remote_file[/var/opt/gitlab/gitlab-rails/VERSION] sending run action to bashmigrate gitlab-rails database

Error executing action run on resource 'bash[migrate gitlab-rails database]'

Mixlib::ShellOut::ShellCommandFailed

Expected process to exit with [0], but received '1'
---- Begin output of "bash" "/tmp/chef-script20150414-16-1kwtjji" ----
STDOUT: rake aborted!
ActiveRecord::NoDatabaseError: FATAL: role "gitlab" does not exist
Run $ bin/rake db:create db:migrate to create your database

PG::Error: FATAL: role "gitlab" does not exist
Run $ bin/rake db:create db:migrate to create your database

Tasks: TOP => db:migrate
(See full trace by running task with --trace)
STDERR:
---- End output of "bash" "/tmp/chef-script20150414-16-1kwtjji" ----
Ran "bash" "/tmp/chef-script20150414-16-1kwtjji" returned 1

Resource Declaration:

In /opt/gitlab/embedded/cookbooks/gitlab/definitions/migrate_database.rb

19: bash "migrate #{params[:name]} database" do
20: code <<-EOH
21: set -e
22: log_file="/tmp/#{params[:name]}-db-migrate-$(date +%s)-$$/output.log"
23: umask 077
24: mkdir $(dirname ${log_file})
25: #{params[:command]} 2>& 1 | tee ${log_file}
26: exit ${PIPESTATUS[0]}
27: EOH
28: action params[:action]
29: end
30: end

Compiled Resource:

Declared in /opt/gitlab/embedded/cookbooks/gitlab/definitions/migrate_database.rb:19:in `block in from_file'

bash("migrate gitlab-rails database") do
params {:command=>"/opt/gitlab/bin/gitlab-rake db:migrate", :action=>:nothing, :name=>"gitlab-rails"}
action [:nothing]
retries 0
retry_delay 2
guard_interpreter :default
command ""bash" "/tmp/chef-script20150414-16-1kwtjji""
backup 5
returns 0
code " set -e\n log_file="/tmp/gitlab-rails-db-migrate-$(date +%s)-$$/output.log"\n umask 077\n mkdir $(dirname ${log_file})\n /opt/gitlab/bin/gitlab-rake db:migrate 2>& 1 | tee ${log_file}\n exit ${PIPESTATUS[0]}\n"
interpreter "bash"
cookbook_name :gitlab
recipe_name "database_migrations"
end

[genezys]

The problem is not Docker but Chef, which is the configuration tool used by GitLab to configure itself. Chef needs to run to configure GitLab, but GitLab services must be started for Chef to run (database started most importantly), but GitLab cannot start properly if Chef does not run correctly.

If the database started by GitLab services does not start fast enough for Chef to work with it, the reconfigure will fail. There is our race condition that may appear only on specific configurations, hardware, CPU, etc.

The only solution I know of is to start the container from bash and manually run the first configurations, like you did, the others are more likely to be OK.

docker run [...]  gitlab_app bash

# Start GitLab services in the background, 
# give it a few seconds to start
/opt/gitlab/embedded/bin/runsvdir-start & 

# Start the configuration process
gitlab-ctl reconfigure 

The main problem is the initial configuration so a restart should normally not be a problem. If it is, and you have already run gitlab-ctl reconfigure once. Remove this instruction from the CMD and just use CMD /opt/gitlab/embedded/bin/runsvdir-start. This should start GitLab without a reconfigure.

You will need to run reconfigure manually every time you touch the gitlab.rb file.

[panticz]

This may by same issue as this one:
https://gitlab.com/gitlab-org/omnibus-gitlab/issues/552

Try to manually increase kernel shared memory limit:

echo 17179869184 > /proc/sys/kernel/shmmax