Linux Mastery 100 #77

Advanced

Tutorial

28 min read

November 20, 2025

luc

GitLab CI/CD Linux Setup: Complete DevOps Automation Guide

Reading Progress 0%

Quick Actions

All Guides

Knowledge Overview

Prerequisites

Required Knowledge:
✅ Basic Linux command line proficiency
✅ Understanding of Git version control
✅ Familiarity with YAML syntax
✅ General DevOps concepts awareness
System Access:
✅ Linux server with sudo/root privileges
✅ GitLab.com account OR self-hosted GitLab instance
✅ SSH access to deployment servers (optional)
✅ Internet connectivity for package installation

What You'll Learn

✅ Install GitLab Runner on Ubuntu, Debian, CentOS, and RHEL
✅ Register and configure runners with GitLab instances
✅ Set up four executor types: Shell, Docker, Kubernetes, and SSH
✅ Write and optimize .gitlab-ci.yml pipeline configurations
✅ Implement multi-stage CI/CD pipelines (Build, Test, Security, Deploy)
✅ Configure Docker-in-Docker for container builds
✅ Secure runners with protected variables and secrets management
✅ Optimize pipeline performance with caching strategies
✅ Troubleshoot common GitLab CI/CD issues
✅ Deploy to staging and production environments safely

Tools Required

Essential Software:
✅ Linux Server: Ubuntu 20.04+, Debian 10+, CentOS 7+, or RHEL 8+
✅ GitLab Runner: Latest version (installed via package manager)
✅ Git: Version control system
✅ curl: For downloading installation scripts
✅ Text Editor: vim, nano, or VS Code
Optional Tools (By Executor Type):
✅ Docker: For Docker executor (most popular)
✅ Kubernetes cluster: For Kubernetes executor
✅ SSH keys: For SSH executor setup
System Requirements:
✅ CPU: 2 cores minimum (4 recommended)
✅ RAM: 4GB minimum (8GB recommended)
✅ Disk: 20GB free space
✅ Network: HTTPS access to GitLab instance (port 443)

Time Investment

28 minutes reading time
56-84 minutes hands-on practice

Guide Content

Setting up GitLab CI/CD on Linux servers transforms your development workflow into an automated, efficient pipeline that builds, tests, and deploys code seamlessly. This comprehensive guide provides step-by-step instructions for installing GitLab runners, configuring pipelines, and implementing production-ready CI/CD automation on Linux systems.

What is GitLab CI/CD?
Why Choose GitLab CI/CD for Linux Servers?
Prerequisites for GitLab CI/CD Installation
How to Install GitLab Runner on Linux
How to Register GitLab Runners
GitLab CI Pipeline Configuration
Runner Executor Types Explained
Advanced Pipeline Configuration Techniques
Security Best Practices for GitLab Runners
Performance Optimization Strategies
Troubleshooting Common GitLab CI Issues
FAQ
Additional Resources

What is GitLab CI/CD?

GitLab CI/CD is an integrated DevOps platform that automates the software development lifecycle directly within your Git repository. Unlike standalone CI/CD tools, GitLab combines version control, continuous integration, continuous deployment, and container registry functionality in a single application.

Core Components:

Bash

# View GitLab version and components
gitlab-rake gitlab:env:info

# Check GitLab Runner version
gitlab-runner --version

# List active runners
gitlab-runner list

The GitLab CI/CD architecture consists of three primary components:

GitLab Server - Central repository and pipeline orchestrator
GitLab Runners - Agents that execute pipeline jobs
.gitlab-ci.yml - Pipeline configuration file in your repository

Real-World Application:

When developers push code to a GitLab repository, the CI/CD system automatically triggers pipelines that compile applications, run test suites, perform security scans, and deploy to staging or production environments—all without manual intervention.

Why Choose GitLab CI/CD for Linux Servers?

GitLab CI/CD offers substantial advantages for Linux-based development environments compared to alternative solutions like Jenkins or GitHub Actions.

Key Benefits:

Native Integration - Seamless connection between code repositories and deployment pipelines
Self-Hosted Control - Complete data sovereignty on your Linux infrastructure
Container Support - First-class Docker integration for consistent build environments
Cost Efficiency - Free for unlimited private repositories and runners
Scalability - Horizontal scaling with multiple runner instances

Bash

# Compare resource usage across CI/CD platforms
ps aux | grep -E '(gitlab-runner|jenkins|buildkite)' | awk '{print $2, $3, $4, $11}'

# Monitor GitLab runner processes
top -p $(pgrep -d',' gitlab-runner)

# Check runner system resource limits
systemctl status gitlab-runner.service

Performance Comparison:

Feature	GitLab CI/CD	Jenkins	GitHub Actions
Linux Native	✓	✓	Limited
Self-Hosted	✓	✓	✓
Built-in Registry	✓	Plugin	✓
Kubernetes Integration	✓	Plugin	Limited
Pipeline as Code	✓	Jenkinsfile	✓

Prerequisites for GitLab CI/CD Installation

Before installing GitLab runners on your Linux system, ensure your environment meets these requirements.

System Requirements

Minimum Specifications:

CPU: 2 cores (4 cores recommended)
RAM: 4GB (8GB recommended)
Disk: 20GB free space
OS: Ubuntu 20.04+, Debian 10+, CentOS 7+, RHEL 8+

Bash

# Verify system specifications
lscpu | grep -E 'CPU\(s\)|Model name'
free -h
df -h /
cat /etc/os-release

# Check kernel version (3.10+ required)
uname -r

# Verify Docker compatibility (if using Docker executor)
docker --version
docker run hello-world

Network Requirements

Bash

# Test connectivity to GitLab server
ping -c 4 gitlab.com

# Verify DNS resolution
nslookup gitlab.com

# Check firewall rules for required ports
sudo firewall-cmd --list-all  # For CentOS/RHEL
sudo ufw status                # For Ubuntu/Debian

# Test HTTPS connectivity
curl -I https://gitlab.com

# Verify outbound connection on port 443
telnet gitlab.com 443

Required Software Dependencies

Bash

# Update package repositories
sudo apt update                     # Debian/Ubuntu
sudo yum update                     # CentOS/RHEL

# Install curl for downloading GitLab Runner
sudo apt install curl ca-certificates -y    # Debian/Ubuntu
sudo yum install curl ca-certificates -y    # CentOS/RHEL

# Install Git (required for repository cloning)
sudo apt install git -y             # Debian/Ubuntu
sudo yum install git -y             # CentOS/RHEL

# Verify Git installation
git --version

How to Install GitLab Runner on Linux

Installing GitLab Runner on Linux involves adding the official repository and installing the runner package. This section covers installation methods for major Linux distributions.

Installation on Ubuntu/Debian

Bash

# Add GitLab Runner official repository
curl -L "https://packages.gitlab.com/install/repositories/runner/gitlab-runner/script.deb.sh" | sudo bash

# Install GitLab Runner
sudo apt install gitlab-runner -y

# Verify installation
gitlab-runner --version

# Check runner service status
sudo systemctl status gitlab-runner

# Enable runner to start on boot
sudo systemctl enable gitlab-runner

Installation on CentOS/RHEL

Bash

# Add GitLab Runner repository for RHEL/CentOS
curl -L "https://packages.gitlab.com/install/repositories/runner/gitlab-runner/script.rpm.sh" | sudo bash

# Install GitLab Runner
sudo yum install gitlab-runner -y

# For RHEL 8+, use DNF
sudo dnf install gitlab-runner -y

# Start and enable GitLab Runner service
sudo systemctl start gitlab-runner
sudo systemctl enable gitlab-runner

# Verify runner is running
sudo systemctl is-active gitlab-runner

Manual Binary Installation (Distribution-Agnostic)

Bash

# Download GitLab Runner binary
sudo curl -L --output /usr/local/bin/gitlab-runner "https://gitlab-runner-downloads.s3.amazonaws.com/latest/binaries/gitlab-runner-linux-amd64"

# Grant execution permissions
sudo chmod +x /usr/local/bin/gitlab-runner

# Create GitLab Runner user
sudo useradd --comment 'GitLab Runner' --create-home gitlab-runner --shell /bin/bash

# Install and start as service
sudo gitlab-runner install --user=gitlab-runner --working-directory=/home/gitlab-runner
sudo gitlab-runner start

# Verify installation
gitlab-runner verify

Post-Installation Verification

Bash

# Check GitLab Runner configuration file
sudo cat /etc/gitlab-runner/config.toml

# View runner logs
sudo journalctl -u gitlab-runner -f

# Test runner connectivity
gitlab-runner run-single --help

# Check runner user permissions
id gitlab-runner
sudo -u gitlab-runner -i

How to Register GitLab Runners

After installation, runners must be registered with your GitLab instance. Registration establishes the connection between your runner and GitLab projects.

Obtaining Registration Token

For GitLab.com (SaaS):

Navigate to your project: Settings → CI/CD → Runners
Expand Specific runners section
Copy the registration token

For Self-Hosted GitLab:

Bash

# For admin access to shared runners
gitlab-rails runner "puts Gitlab::CurrentSettings.current_application_settings.runners_registration_token"

# View project-specific token via GitLab UI
# Navigate to: Project → Settings → CI/CD → Runners

Interactive Runner Registration

Bash

# Start interactive registration process
sudo gitlab-runner register

# You'll be prompted for:
# 1. GitLab instance URL (e.g., https://gitlab.com/)
# 2. Registration token (from GitLab UI)
# 3. Description (e.g., "Production Linux Runner")
# 4. Tags (e.g., "docker,linux,production")
# 5. Executor type (shell, docker, kubernetes, etc.)

Example Interactive Session:

Bash

sudo gitlab-runner register

# Prompts and responses:
Enter the GitLab instance URL (for example, https://gitlab.com/):
https://gitlab.com/

Enter the registration token:
GR1348941abc123def456

Enter a description for the runner:
[hostname]: Production Ubuntu Runner

Enter tags for the runner (comma-separated):
docker,linux,ubuntu,production

Enter an executor: docker, shell, ssh, kubernetes, custom:
docker

Enter the default Docker image (for example, ruby:2.7):
alpine:latest

Runner registered successfully. Feel free to start it, but if it's running already the config should be automatically reloaded!

Non-Interactive Registration

Bash

# Register runner with all parameters in single command
sudo gitlab-runner register \
  --non-interactive \
  --url "https://gitlab.com/" \
  --registration-token "GR1348941abc123def456" \
  --executor "docker" \
  --docker-image "alpine:latest" \
  --description "Automated Docker Runner" \
  --tag-list "docker,linux,automation" \
  --run-untagged="false" \
  --locked="false" \
  --access-level="not_protected"

# Verify registration
sudo gitlab-runner list

Registering Multiple Runners

Bash

# Register shell executor runner
sudo gitlab-runner register \
  --non-interactive \
  --url "https://gitlab.com/" \
  --registration-token "GR1348941abc123def456" \
  --executor "shell" \
  --description "Shell Executor Runner" \
  --tag-list "shell,scripts"

# Register Docker executor runner
sudo gitlab-runner register \
  --non-interactive \
  --url "https://gitlab.com/" \
  --registration-token "GR1348941abc123def456" \
  --executor "docker" \
  --docker-image "ubuntu:22.04" \
  --description "Docker Build Runner" \
  --tag-list "docker,build"

# Register Kubernetes executor runner
sudo gitlab-runner register \
  --non-interactive \
  --url "https://gitlab.com/" \
  --registration-token "GR1348941abc123def456" \
  --executor "kubernetes" \
  --description "Kubernetes Runner" \
  --tag-list "k8s,containers"

# View all registered runners
sudo cat /etc/gitlab-runner/config.toml

Verifying Runner Registration

Bash

# Check runner status
sudo gitlab-runner verify

# List all runners with details
sudo gitlab-runner list

# Check runner connectivity to GitLab
sudo gitlab-runner verify --delete

# View runner configuration
sudo cat /etc/gitlab-runner/config.toml | grep -A 10 "\[\[runners\]\]"

# Test runner with a simple job
sudo gitlab-runner exec shell echo "Runner test successful"

GitLab CI Pipeline Configuration

The .gitlab-ci.yml file defines your CI/CD pipeline structure. This YAML configuration file lives in your repository root and orchestrates all automation workflows.

Basic Pipeline Structure

Bash

# .gitlab-ci.yml - Simple pipeline example

# Define pipeline stages (executed in order)
stages:
  - build
  - test
  - deploy

# Variables available to all jobs
variables:
  DOCKER_DRIVER: overlay2
  APP_NAME: "my-application"

# Build job
build_app:
  stage: build
  image: alpine:latest
  script:
    - echo "Building application..."
    - apk add --no-cache build-base
    - gcc -o myapp main.c
  artifacts:
    paths:
      - myapp
    expire_in: 1 hour
  tags:
    - docker

# Test job
test_app:
  stage: test
  image: alpine:latest
  script:
    - echo "Running tests..."
    - ./myapp --test
    - echo "All tests passed!"
  dependencies:
    - build_app
  tags:
    - docker

# Deploy job
deploy_production:
  stage: deploy
  script:
    - echo "Deploying to production..."
    - scp myapp user@production-server:/opt/app/
    - ssh user@production-server "systemctl restart myapp"
  only:
    - main
  tags:
    - shell
  when: manual

Advanced Pipeline with Docker

Bash

# .gitlab-ci.yml - Docker-based pipeline

stages:
  - lint
  - build
  - test
  - security
  - deploy

# Global variables
variables:
  DOCKER_TLS_CERTDIR: "/certs"
  REGISTRY: "registry.gitlab.com"
  IMAGE_TAG: "$CI_REGISTRY_IMAGE:$CI_COMMIT_REF_SLUG"

# Lint stage
lint_code:
  stage: lint
  image: python:3.11-slim
  before_script:
    - pip install flake8 pylint black
  script:
    - echo "Running code linters..."
    - black --check .
    - flake8 . --max-line-length=120
    - pylint **/*.py
  allow_failure: true
  tags:
    - docker

# Build Docker image
build_docker_image:
  stage: build
  image: docker:24-dind
  services:
    - docker:24-dind
  before_script:
    - docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY
  script:
    - echo "Building Docker image..."
    - docker build -t $IMAGE_TAG .
    - docker push $IMAGE_TAG
    - echo "Image pushed to $IMAGE_TAG"
  tags:
    - docker

# Unit tests
unit_tests:
  stage: test
  image: python:3.11
  script:
    - pip install -r requirements.txt
    - pip install pytest pytest-cov
    - pytest tests/ --cov=app --cov-report=xml --cov-report=term
  coverage: '/(?i)total.*? (100(?:\.0+)?\%|[1-9]?\d(?:\.\d+)?\%)$/'
  artifacts:
    reports:
      coverage_report:
        coverage_format: cobertura
        path: coverage.xml
  tags:
    - docker

# Integration tests
integration_tests:
  stage: test
  image: docker/compose:latest
  services:
    - docker:24-dind
  script:
    - docker-compose -f docker-compose.test.yml up -d
    - docker-compose -f docker-compose.test.yml exec -T app pytest integration_tests/
    - docker-compose -f docker-compose.test.yml down
  tags:
    - docker

# Security scanning
security_scan:
  stage: security
  image: aquasec/trivy:latest
  script:
    - trivy image --severity HIGH,CRITICAL $IMAGE_TAG
  allow_failure: true
  tags:
    - docker

# Deploy to staging
deploy_staging:
  stage: deploy
  image: alpine:latest
  before_script:
    - apk add --no-cache openssh-client
    - eval $(ssh-agent -s)
    - echo "$SSH_PRIVATE_KEY" | tr -d '\r' | ssh-add -
    - mkdir -p ~/.ssh
    - chmod 700 ~/.ssh
  script:
    - ssh -o StrictHostKeyChecking=no deployer@staging-server "docker pull $IMAGE_TAG"
    - ssh -o StrictHostKeyChecking=no deployer@staging-server "docker-compose -f /opt/app/docker-compose.yml up -d"
  environment:
    name: staging
    url: https://staging.example.com
  only:
    - develop
  tags:
    - shell

# Deploy to production
deploy_production:
  stage: deploy
  image: alpine:latest
  before_script:
    - apk add --no-cache openssh-client kubectl
    - eval $(ssh-agent -s)
    - echo "$SSH_PRIVATE_KEY" | tr -d '\r' | ssh-add -
  script:
    - kubectl config use-context production
    - kubectl set image deployment/myapp myapp=$IMAGE_TAG
    - kubectl rollout status deployment/myapp
  environment:
    name: production
    url: https://production.example.com
  only:
    - main
  when: manual
  tags:
    - kubernetes

Pipeline with Caching and Artifacts

Bash

# .gitlab-ci.yml - Optimized with caching

stages:
  - dependencies
  - build
  - test

# Cache node_modules between jobs
cache:
  key: ${CI_COMMIT_REF_SLUG}
  paths:
    - node_modules/
    - .npm/

# Install dependencies
install_dependencies:
  stage: dependencies
  image: node:18-alpine
  script:
    - npm ci --cache .npm --prefer-offline
  artifacts:
    paths:
      - node_modules/
    expire_in: 1 day
  tags:
    - docker

# Build application
build_application:
  stage: build
  image: node:18-alpine
  dependencies:
    - install_dependencies
  script:
    - npm run build
  artifacts:
    paths:
      - dist/
    expire_in: 1 week
  tags:
    - docker

# Run tests with coverage
test_with_coverage:
  stage: test
  image: node:18-alpine
  dependencies:
    - install_dependencies
  script:
    - npm run test:coverage
  coverage: '/Statements\s*:\s*([^%]+)/'
  artifacts:
    reports:
      junit: junit.xml
      coverage_report:
        coverage_format: cobertura
        path: coverage/cobertura-coverage.xml
  tags:
    - docker

Testing Pipeline Locally

Bash

# Install GitLab Runner CLI locally
curl -LJO "https://gitlab-runner-downloads.s3.amazonaws.com/latest/deb/gitlab-runner_amd64.deb"
sudo dpkg -i gitlab-runner_amd64.deb

# Test pipeline jobs locally
gitlab-runner exec docker build_app

# Run specific job with custom variables
gitlab-runner exec docker test_app \
  --env "DATABASE_URL=postgresql://localhost:5432/test"

# Debug pipeline with verbose output
gitlab-runner --debug exec docker build_app

# Validate .gitlab-ci.yml syntax
curl --header "PRIVATE-TOKEN: <your_access_token>" \
  "https://gitlab.com/api/v4/ci/lint" \
  --data "content=$(cat .gitlab-ci.yml)"

Runner Executor Types Explained

GitLab runners support multiple executor types, each optimized for different use cases. Choosing the right executor significantly impacts build performance, isolation, and resource utilization.

Shell Executor

The shell executor runs jobs directly on the host system using the runner user's shell environment.

Configuration:

Bash

# Register shell executor
sudo gitlab-runner register \
  --non-interactive \
  --url "https://gitlab.com/" \
  --registration-token "YOUR_TOKEN" \
  --executor "shell" \
  --description "Shell Executor Runner" \
  --tag-list "shell,native"

# View shell executor configuration
sudo cat /etc/gitlab-runner/config.toml

Example config.toml:

Bash

[[runners]]
  name = "Shell Executor Runner"
  url = "https://gitlab.com/"
  token = "abc123"
  executor = "shell"
  shell = "bash"
  [runners.custom_build_dir]
  [runners.cache]
    [runners.cache.s3]
    [runners.cache.gcs]

Use Cases:

System administration scripts
Performance testing requiring bare metal
Legacy applications without Docker support

Advantages: ✓ Direct hardware access
✓ Minimal overhead
✓ Simple debugging

Disadvantages: ✗ No job isolation
✗ Environment pollution
✗ Security concerns

Docker Executor

The Docker executor runs each job in a fresh container, providing excellent isolation and reproducibility.

Configuration:

Bash

# Install Docker first
curl -fsSL https://get.docker.com -o get-docker.sh
sudo sh get-docker.sh

# Add gitlab-runner user to docker group
sudo usermod -aG docker gitlab-runner

# Register Docker executor
sudo gitlab-runner register \
  --non-interactive \
  --url "https://gitlab.com/" \
  --registration-token "YOUR_TOKEN" \
  --executor "docker" \
  --docker-image "alpine:latest" \
  --docker-volumes "/var/run/docker.sock:/var/run/docker.sock" \
  --docker-privileged \
  --description "Docker Executor Runner" \
  --tag-list "docker,containers"

Advanced Docker executor config.toml:

Bash

[[runners]]
  name = "Docker Executor Runner"
  url = "https://gitlab.com/"
  token = "abc123"
  executor = "docker"
  [runners.docker]
    tls_verify = false
    image = "alpine:latest"
    privileged = true
    disable_entrypoint_overwrite = false
    oom_kill_disable = false
    disable_cache = false
    volumes = ["/cache", "/var/run/docker.sock:/var/run/docker.sock"]
    shm_size = 0
    pull_policy = "if-not-present"
    network_mode = "bridge"
  [runners.cache]
    Type = "s3"
    Shared = true
    [runners.cache.s3]
      ServerAddress = "s3.amazonaws.com"
      BucketName = "gitlab-runner-cache"
      BucketLocation = "us-east-1"

Docker-in-Docker Configuration:

Bash

# Enable Docker-in-Docker for building Docker images
sudo gitlab-runner register \
  --non-interactive \
  --url "https://gitlab.com/" \
  --registration-token "YOUR_TOKEN" \
  --executor "docker" \
  --docker-image "docker:24-dind" \
  --docker-privileged \
  --docker-volumes "/certs/client" \
  --description "Docker-in-Docker Runner" \
  --tag-list "dind,docker-build"

Kubernetes Executor

The Kubernetes executor schedules jobs as pods in a Kubernetes cluster, offering scalability and resource management.

Prerequisites:

Bash

# Install kubectl
curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"
sudo install -o root -g root -m 0755 kubectl /usr/local/bin/kubectl

# Verify kubectl connection to cluster
kubectl cluster-info
kubectl get nodes

Registration:

Bash

# Register Kubernetes executor
sudo gitlab-runner register \
  --non-interactive \
  --url "https://gitlab.com/" \
  --registration-token "YOUR_TOKEN" \
  --executor "kubernetes" \
  --kubernetes-host "https://kubernetes.default.svc" \
  --kubernetes-namespace "gitlab-runner" \
  --kubernetes-privileged \
  --description "Kubernetes Executor" \
  --tag-list "k8s,scalable"

Kubernetes executor config.toml:

Bash

[[runners]]
  name = "Kubernetes Executor"
  url = "https://gitlab.com/"
  token = "abc123"
  executor = "kubernetes"
  [runners.kubernetes]
    host = "https://kubernetes.default.svc"
    namespace = "gitlab-runner"
    privileged = true
    image = "alpine:latest"
    cpu_limit = "2"
    memory_limit = "4Gi"
    service_cpu_limit = "1"
    service_memory_limit = "2Gi"
    helper_cpu_limit = "500m"
    helper_memory_limit = "512Mi"
    poll_interval = 5
    poll_timeout = 360
    [runners.kubernetes.pod_labels]
      "app" = "gitlab-runner"
      "environment" = "production"

Creating Kubernetes Service Account:

Bash

# Create namespace
kubectl create namespace gitlab-runner

# Create service account with proper permissions
cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: ServiceAccount
metadata:
  name: gitlab-runner
  namespace: gitlab-runner
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: gitlab-runner
  namespace: gitlab-runner
rules:
  - apiGroups: [""]
    resources: ["pods", "pods/exec", "secrets", "configmaps"]
    verbs: ["get", "list", "watch", "create", "patch", "delete"]
  - apiGroups: [""]
    resources: ["pods/log"]
    verbs: ["get"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: gitlab-runner
  namespace: gitlab-runner
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: gitlab-runner
subjects:
  - kind: ServiceAccount
    name: gitlab-runner
    namespace: gitlab-runner
EOF

# Verify service account
kubectl get serviceaccount gitlab-runner -n gitlab-runner

SSH Executor

The SSH executor connects to remote servers and executes jobs via SSH.

Bash

# Register SSH executor
sudo gitlab-runner register \
  --non-interactive \
  --url "https://gitlab.com/" \
  --registration-token "YOUR_TOKEN" \
  --executor "ssh" \
  --ssh-host "remote-server.example.com" \
  --ssh-user "deploy" \
  --ssh-password "YOUR_PASSWORD" \
  --ssh-port "22" \
  --description "SSH Remote Executor" \
  --tag-list "ssh,remote"

# Using SSH key authentication (recommended)
sudo gitlab-runner register \
  --non-interactive \
  --url "https://gitlab.com/" \
  --registration-token "YOUR_TOKEN" \
  --executor "ssh" \
  --ssh-host "remote-server.example.com" \
  --ssh-user "deploy" \
  --ssh-identity-file "/home/gitlab-runner/.ssh/id_rsa" \
  --description "SSH Key-Based Executor" \
  --tag-list "ssh,secure"

Executor Comparison Matrix

Bash

# Test execution time across executors
time gitlab-runner exec shell echo "Shell executor test"
time gitlab-runner exec docker echo "Docker executor test"

# Monitor resource usage by executor type
ps aux | grep gitlab-runner
docker stats --no-stream
kubectl top pods -n gitlab-runner

Executor	Isolation	Speed	Scalability	Use Case
Shell	Low	Fast	Limited	System scripts
Docker	High	Medium	Good	Standard builds
Kubernetes	High	Medium	Excellent	Cloud-native
SSH	Medium	Slow	Poor	Legacy systems

Advanced Pipeline Configuration Techniques

Master these advanced GitLab CI/CD techniques to build sophisticated, production-grade pipelines.

Dynamic Pipeline Generation

Bash

# .gitlab-ci.yml - Generate pipelines dynamically

stages:
  - generate
  - build
  - deploy

# Generate child pipeline based on changed files
generate_pipeline:
  stage: generate
  image: alpine:latest
  script:
    - apk add --no-cache git
    - |
      # Detect changed services
      CHANGED_SERVICES=$(git diff --name-only $CI_COMMIT_BEFORE_SHA $CI_COMMIT_SHA | grep '^services/' | cut -d'/' -f2 | sort -u)
      
      # Generate pipeline YAML for changed services
      cat > generated-pipeline.yml <<EOF
      stages:
        - build
        - test
      EOF
      
      for service in $CHANGED_SERVICES; do
        cat >> generated-pipeline.yml <<EOF
      
      build_${service}:
        stage: build
        script:
          - echo "Building $service"
          - cd services/$service
          - docker build -t $service:latest .
        tags:
          - docker
      
      test_${service}:
        stage: test
        script:
          - echo "Testing $service"
          - cd services/$service
          - ./run-tests.sh
        tags:
          - docker
      EOF
      done
  artifacts:
    paths:
      - generated-pipeline.yml
  tags:
    - shell

# Trigger child pipeline
trigger_builds:
  stage: build
  trigger:
    include:
      - artifact: generated-pipeline.yml
        job: generate_pipeline
    strategy: depend

Parallel Job Execution with Matrix Strategy

Bash

# .gitlab-ci.yml - Matrix builds for multiple versions

test_matrix:
  stage: test
  image: python:${PYTHON_VERSION}
  parallel:
    matrix:
      - PYTHON_VERSION: ["3.9", "3.10", "3.11", "3.12"]
        DATABASE: ["postgresql", "mysql"]
  script:
    - echo "Testing Python $PYTHON_VERSION with $DATABASE"
    - pip install -r requirements.txt
    - pip install $DATABASE-connector
    - pytest tests/ -v
  tags:
    - docker

# Parallel execution with custom names
build_platforms:
  stage: build
  parallel:
    matrix:
      - PLATFORM: ["linux/amd64", "linux/arm64", "linux/arm/v7"]
  script:
    - docker buildx build --platform $PLATFORM -t myapp:$PLATFORM .
  tags:
    - docker

Pipeline Includes and Templates

Bash

# .gitlab-ci.yml - Modular pipeline with includes

include:
  # Include from same repository
  - local: '/templates/docker-build.yml'
  - local: '/templates/security-scan.yml'
  
  # Include from another project
  - project: 'company/ci-templates'
    file: '/templates/deploy.yml'
    ref: main
  
  # Include from remote URL
  - remote: 'https://raw.githubusercontent.com/example/templates/main/lint.yml'
  
  # Include template from GitLab
  - template: Security/SAST.gitlab-ci.yml

stages:
  - lint
  - build
  - test
  - security
  - deploy

variables:
  APP_NAME: "myapp"
  ENVIRONMENT: "production"

templates/docker-build.yml:

Bash

.docker_build_template:
  stage: build
  image: docker:24-dind
  services:
    - docker:24-dind
  before_script:
    - docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY
  script:
    - docker build -t $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA .
    - docker tag $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA $CI_REGISTRY_IMAGE:latest
    - docker push $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA
    - docker push $CI_REGISTRY_IMAGE:latest
  tags:
    - docker

build_application:
  extends: .docker_build_template
  only:
    - main
    - develop

Environment-Specific Deployments

Bash

# .gitlab-ci.yml - Multi-environment deployment

.deploy_template:
  stage: deploy
  image: alpine:latest
  before_script:
    - apk add --no-cache openssh-client
    - eval $(ssh-agent -s)
    - echo "$SSH_PRIVATE_KEY" | tr -d '\r' | ssh-add -
    - mkdir -p ~/.ssh
    - chmod 700 ~/.ssh
    - ssh-keyscan $DEPLOY_HOST >> ~/.ssh/known_hosts
  script:
    - ssh deployer@$DEPLOY_HOST "cd /opt/app && docker-compose pull && docker-compose up -d"
    - ssh deployer@$DEPLOY_HOST "docker system prune -af"
  tags:
    - shell

deploy_development:
  extends: .deploy_template
  variables:
    DEPLOY_HOST: "dev.example.com"
  environment:
    name: development
    url: https://dev.example.com
    on_stop: stop_development
  only:
    - develop

deploy_staging:
  extends: .deploy_template
  variables:
    DEPLOY_HOST: "staging.example.com"
  environment:
    name: staging
    url: https://staging.example.com
  only:
    - develop
  when: manual

deploy_production:
  extends: .deploy_template
  variables:
    DEPLOY_HOST: "prod.example.com"
  environment:
    name: production
    url: https://example.com
  only:
    - main
  when: manual
  allow_failure: false

stop_development:
  stage: deploy
  variables:
    GIT_STRATEGY: none
    DEPLOY_HOST: "dev.example.com"
  script:
    - ssh deployer@$DEPLOY_HOST "docker-compose down"
  environment:
    name: development
    action: stop
  when: manual
  tags:
    - shell

Conditional Job Execution with Rules

Bash

# .gitlab-ci.yml - Advanced rules for job control

workflow:
  rules:
    # Don't run pipelines for branch deletion
    - if: $CI_COMMIT_BEFORE_SHA == "0000000000000000000000000000000000000000"
      when: never
    # Run pipelines for merge requests
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
    # Run pipelines for main and develop branches
    - if: $CI_COMMIT_BRANCH == "main" || $CI_COMMIT_BRANCH == "develop"
    # Run pipelines for tags
    - if: $CI_COMMIT_TAG

build_job:
  stage: build
  script:
    - echo "Building application"
  rules:
    # Run for merge requests
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
    # Run for main branch
    - if: $CI_COMMIT_BRANCH == "main"
    # Run if Dockerfile changed
    - changes:
        - Dockerfile
        - docker-compose.yml
      when: always
    # Run if source code changed
    - changes:
        - src/**/*
      when: on_success
  tags:
    - docker

deploy_job:
  stage: deploy
  script:
    - echo "Deploying to production"
  rules:
    # Only deploy from main branch
    - if: $CI_COMMIT_BRANCH == "main"
      when: manual
    # Auto-deploy on tags starting with "release-"
    - if: $CI_COMMIT_TAG =~ /^release-/
      when: always
    # Never deploy for merge requests
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
      when: never
  tags:
    - production

security_scan:
  stage: test
  script:
    - trivy image $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA
  rules:
    # Always run for main and develop
    - if: $CI_COMMIT_BRANCH == "main" || $CI_COMMIT_BRANCH == "develop"
      allow_failure: false
    # Run for merge requests but allow failure
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
      allow_failure: true
    # Skip for feature branches
    - when: never
  tags:
    - docker

Scheduled Pipeline Triggers

Bash

# Create scheduled pipeline via GitLab UI:
# Project → CI/CD → Schedules → New schedule

# Or use GitLab API to create schedule
curl --request POST \
  --header "PRIVATE-TOKEN: YOUR_ACCESS_TOKEN" \
  --form "description=Nightly build and test" \
  --form "ref=main" \
  --form "cron=0 2 * * *" \
  --form "cron_timezone=UTC" \
  --form "active=true" \
  "https://gitlab.com/api/v4/projects/PROJECT_ID/pipeline_schedules"

# List all pipeline schedules
curl --header "PRIVATE-TOKEN: YOUR_ACCESS_TOKEN" \
  "https://gitlab.com/api/v4/projects/PROJECT_ID/pipeline_schedules"

.gitlab-ci.yml for scheduled jobs:

Bash

# .gitlab-ci.yml - Handle scheduled pipelines

nightly_build:
  stage: build
  script:
    - echo "Running nightly build"
    - ./build-all.sh
  only:
    - schedules
  tags:
    - docker

weekly_cleanup:
  stage: cleanup
  script:
    - echo "Running weekly cleanup"
    - docker system prune -af --volumes
    - find /tmp -type f -atime +7 -delete
  only:
    variables:
      - $CI_PIPELINE_SOURCE == "schedule"
      - $SCHEDULE_TYPE == "weekly"
  tags:
    - shell

Security Best Practices for GitLab Runners

Securing your GitLab CI/CD infrastructure is critical to prevent unauthorized access and protect sensitive data.

Runner Isolation and Access Control

Bash

# Create dedicated user for GitLab Runner
sudo useradd -r -s /bin/false -M gitlab-runner-isolated

# Restrict runner user permissions
sudo usermod -L gitlab-runner-isolated  # Lock password
sudo chmod 700 /home/gitlab-runner      # Restrict home directory

# Configure runner to use isolated user
sudo gitlab-runner install --user=gitlab-runner-isolated

# Verify runner isolation
ps aux | grep gitlab-runner
sudo -u gitlab-runner-isolated id

Secrets Management

Using GitLab CI/CD Variables:

Bash

# Add protected variable via GitLab UI:
# Settings → CI/CD → Variables → Add Variable
# - Protected: Yes (only available on protected branches)
# - Masked: Yes (hidden in logs)
# - Environment scope: production

# Access secrets in pipeline
# .gitlab-ci.yml
deploy_production:
  stage: deploy
  script:
    - echo "Deploying with credentials"
    - kubectl create secret generic app-secrets \
        --from-literal=database-url=$DATABASE_URL \
        --from-literal=api-key=$API_KEY
  environment:
    name: production
  only:
    - main
  tags:
    - kubernetes

External Secrets Management:

Bash

# .gitlab-ci.yml - HashiCorp Vault integration

deploy_with_vault:
  stage: deploy
  image: vault:latest
  before_script:
    - export VAULT_ADDR="https://vault.example.com"
    - export VAULT_TOKEN="$VAULT_TOKEN"
  script:
    - vault kv get -field=database_password secret/production/db > /tmp/db_pass
    - ./deploy.sh
  after_script:
    - rm -f /tmp/db_pass
  only:
    - main
  tags:
    - docker

Container Image Security

Bash

# .gitlab-ci.yml - Security scanning pipeline

container_scanning:
  stage: security
  image: aquasec/trivy:latest
  script:
    - trivy image --exit-code 1 --severity CRITICAL,HIGH $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA
    - trivy image --format json --output trivy-report.json $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA
  artifacts:
    reports:
      container_scanning: trivy-report.json
  allow_failure: false
  tags:
    - docker

dependency_scanning:
  stage: security
  image: python:3.11
  script:
    - pip install safety
    - safety check --file requirements.txt --json > safety-report.json
  artifacts:
    reports:
      dependency_scanning: safety-report.json
  tags:
    - docker

sast_scan:
  stage: security
  image: returntocorp/semgrep:latest
  script:
    - semgrep --config=auto --json --output=semgrep-report.json .
  artifacts:
    reports:
      sast: semgrep-report.json
  tags:
    - docker

Network Security Configuration

Bash

# Configure firewall rules for GitLab Runner
sudo ufw allow from GITLAB_SERVER_IP to any port 22 proto tcp comment 'GitLab SSH access'
sudo ufw allow from GITLAB_SERVER_IP to any port 443 proto tcp comment 'GitLab HTTPS'
sudo ufw enable

# Restrict runner network access in config.toml
sudo nano /etc/gitlab-runner/config.toml

Secure config.toml:

Bash

[[runners]]
name = "Secure Docker Runner"
url = "https://gitlab.com/"
token = "abc123"
executor = "docker"
[runners.docker]
image = "alpine:latest"
privileged = false
disable_entrypoint_overwrite = true
oom_kill_disable = false
disable_cache = false
network_mode = "gitlab-runner-network"
allowed_images = ["alpine:", "ubuntu:", "python:"] allowed_services = ["docker:-dind", "postgres:", "redis:"]
[runners.docker.sysctls]
"net.ipv4.ip_forward" = "0"

Audit Logging and Monitoring

Bash

# Enable detailed GitLab Runner logging
sudo nano /etc/gitlab-runner/config.toml

# Add log_level configuration
# log_level = "debug"  # Options: debug, info, warn, error, fatal, panic

# Monitor runner logs in real-time
sudo journalctl -u gitlab-runner -f

# Archive runner logs
sudo journalctl -u gitlab-runner --since "1 week ago" > /var/log/gitlab-runner-archive.log

# Analyze failed jobs
sudo journalctl -u gitlab-runner | grep "ERROR\|FATAL"

# Set up log rotation
sudo nano /etc/logrotate.d/gitlab-runner

/etc/logrotate.d/gitlab-runner:

Bash

/var/log/gitlab-runner/*.log {
    daily
    missingok
    rotate 52
    compress
    delaycompress
    notifempty
    create 0640 gitlab-runner gitlab-runner
    sharedscripts
    postrotate
        systemctl reload gitlab-runner
    endscript
}

Runner Registration Token Security

Bash

# Deregister inactive runners
sudo gitlab-runner verify --delete

# List all registered runners
sudo gitlab-runner list

# Unregister specific runner
sudo gitlab-runner unregister --name "Runner Name"

# Unregister runner by URL and token
sudo gitlab-runner unregister --url https://gitlab.com/ --token RUNNER_TOKEN

# Rotate runner token via GitLab API
curl --request POST \
  --header "PRIVATE-TOKEN: YOUR_ACCESS_TOKEN" \
  "https://gitlab.com/api/v4/runners/RUNNER_ID/reset_authentication_token"

Performance Optimization Strategies

Optimizing GitLab CI/CD pipelines reduces build times, decreases resource consumption, and improves developer productivity.

Caching Strategies

Distributed Cache with S3:

Bash

# /etc/gitlab-runner/config.toml

[[runners]]
  name = "Cached Docker Runner"
  url = "https://gitlab.com/"
  token = "abc123"
  executor = "docker"
  [runners.docker]
    image = "alpine:latest"
  [runners.cache]
    Type = "s3"
    Shared = true
    [runners.cache.s3]
      ServerAddress = "s3.amazonaws.com"
      AccessKey = "AWS_ACCESS_KEY"
      SecretKey = "AWS_SECRET_KEY"
      BucketName = "gitlab-runner-cache"
      BucketLocation = "us-east-1"
      Insecure = false

Local Cache Configuration:

Bash

[[runners]]
  name = "Local Cache Runner"
  url = "https://gitlab.com/"
  token = "abc123"
  executor = "docker"
  [runners.docker]
    image = "alpine:latest"
    volumes = ["/cache"]
  [runners.cache]
    Type = "local"
    Shared = false
    [runners.cache.local]
      Path = "/srv/gitlab-runner/cache"

Pipeline Cache Optimization:

Bash

# .gitlab-ci.yml - Optimized caching

# Global cache configuration
cache:
  key: ${CI_COMMIT_REF_SLUG}
  paths:
    - .npm/
    - node_modules/
    - .pip-cache/
  policy: pull-push

build_job:
  stage: build
  script:
    - npm ci --cache .npm --prefer-offline
  cache:
    key: ${CI_COMMIT_REF_SLUG}
    paths:
      - node_modules/
    policy: pull-push

test_job:
  stage: test
  script:
    - npm run test
  cache:
    key: ${CI_COMMIT_REF_SLUG}
    paths:
      - node_modules/
    policy: pull  # Only download, don't upload

Docker Layer Caching

Bash

# .gitlab-ci.yml - Docker layer caching

build_with_cache:
  stage: build
  image: docker:24
  services:
    - docker:24-dind
  variables:
    DOCKER_DRIVER: overlay2
    DOCKER_TLS_CERTDIR: "/certs"
  before_script:
    - docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY
    # Pull previous image for layer caching
    - docker pull $CI_REGISTRY_IMAGE:latest || true
  script:
    - docker build 
        --cache-from $CI_REGISTRY_IMAGE:latest 
        --tag $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA 
        --tag $CI_REGISTRY_IMAGE:latest 
        .
    - docker push $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA
    - docker push $CI_REGISTRY_IMAGE:latest
  tags:
    - docker

Parallel Job Execution

Bash

# .gitlab-ci.yml - Parallel test execution

test:
  stage: test
  image: node:18
  script:
    - npm install
    - npm test -- --shard=$CI_NODE_INDEX/$CI_NODE_TOTAL
  parallel: 5  # Split tests across 5 parallel jobs
  tags:
    - docker

# Parallel builds for different architectures
build:
  stage: build
  image: docker:24
  services:
    - docker:24-dind
  parallel:
    matrix:
      - ARCH: [amd64, arm64, armv7]
  script:
    - docker buildx build --platform linux/$ARCH -t myapp:$ARCH .
  tags:
    - docker

Resource Limits and Concurrency

Bash

# /etc/gitlab-runner/config.toml

concurrent = 10  # Maximum concurrent jobs across all runners

[[runners]]
  name = "Resource-Limited Runner"
  url = "https://gitlab.com/"
  token = "abc123"
  executor = "docker"
  limit = 3  # Maximum concurrent jobs for this runner
  request_concurrency = 2  # Maximum concurrent API requests
  [runners.docker]
    image = "alpine:latest"
    cpus = "2"
    memory = "4g"
    memory_swap = "4g"
    memory_reservation = "2g"
    oom_kill_disable = false

Monitor Runner Performance:

Bash

# Check runner concurrency
sudo gitlab-runner verify

# Monitor active jobs
ps aux | grep gitlab-runner | wc -l

# Check Docker resource usage
docker stats --no-stream

# Monitor system resources
htop
iostat -x 1
free -h

# GitLab Runner metrics endpoint (if enabled)
curl http://localhost:9252/metrics

Artifact Optimization

Bash

# .gitlab-ci.yml - Efficient artifact handling

build:
  stage: build
  script:
    - npm run build
  artifacts:
    paths:
      - dist/
    expire_in: 1 week  # Auto-delete old artifacts
    when: on_success
  tags:
    - docker

test:
  stage: test
  dependencies:
    - build  # Only download artifacts from build job
  script:
    - npm test
  artifacts:
    paths:
      - coverage/
    expire_in: 30 days
    reports:
      coverage_report:
        coverage_format: cobertura
        path: coverage/cobertura-coverage.xml
  tags:
    - docker

deploy:
  stage: deploy
  dependencies:
    - build  # Only need build artifacts, not test artifacts
  script:
    - ./deploy.sh dist/
  artifacts:
    paths:
      - deployment-logs/
    expire_in: 1 year
  tags:
    - shell

Troubleshooting Common GitLab CI Issues

Effective troubleshooting skills minimize downtime and ensure reliable CI/CD operations.

Runner Registration Failures

Problem: Runner registration fails with connection errors.

Bash

# Diagnose connection issues
ping gitlab.com

# Test HTTPS connectivity
curl -v https://gitlab.com

# Check DNS resolution
nslookup gitlab.com
dig gitlab.com

# Verify GitLab instance URL
curl -I https://your-gitlab-instance.com

# Test with verbose output
sudo gitlab-runner register --debug

# Check firewall rules
sudo iptables -L -n -v
sudo firewall-cmd --list-all

Solution:

Bash

# Configure proxy if needed
export HTTP_PROXY=http://proxy.example.com:8080
export HTTPS_PROXY=http://proxy.example.com:8080

# Register with proxy settings
sudo gitlab-runner register \
  --non-interactive \
  --url "https://gitlab.com/" \
  --registration-token "YOUR_TOKEN" \
  --executor "docker" \
  --docker-image "alpine:latest" \
  --env "HTTP_PROXY=http://proxy.example.com:8080" \
  --env "HTTPS_PROXY=http://proxy.example.com:8080"

# Or add to config.toml
sudo nano /etc/gitlab-runner/config.toml
# Add under [runners] section:
# environment = ["HTTP_PROXY=http://proxy.example.com:8080"]

Runner Not Picking Up Jobs

Problem: Runner appears online but doesn't execute jobs.

Bash

# Verify runner status
sudo gitlab-runner verify

# Check runner logs
sudo journalctl -u gitlab-runner -f

# Verify runner registration
sudo cat /etc/gitlab-runner/config.toml

# Test runner manually
sudo gitlab-runner run --debug

# Check runner tags match job requirements
sudo gitlab-runner list

Solution:

Bash

# Re-verify runner connection
sudo gitlab-runner verify --delete  # Removes invalid runners

# Check if runner is paused (via GitLab UI)
# Settings → CI/CD → Runners → Check if runner is active

# Ensure job tags match runner tags
# In .gitlab-ci.yml, job tags must match registered runner tags

# Restart runner service
sudo systemctl restart gitlab-runner
sudo systemctl status gitlab-runner

Docker Executor Permission Errors

Problem: Permission denied errors when using Docker executor.

Bash

# Check Docker socket permissions
ls -la /var/run/docker.sock

# Verify gitlab-runner user membership
id gitlab-runner
groups gitlab-runner

# Test Docker access
sudo -u gitlab-runner docker ps

Solution:

Bash

# Add gitlab-runner to docker group
sudo usermod -aG docker gitlab-runner

# Verify group membership
id gitlab-runner

# Restart runner to apply changes
sudo systemctl restart gitlab-runner

# Test Docker access again
sudo -u gitlab-runner docker run hello-world

# Alternative: Use privileged mode (less secure)
sudo gitlab-runner register \
  --executor "docker" \
  --docker-privileged

Out of Disk Space Issues

Problem: Builds fail due to insufficient disk space.

Bash

# Check disk usage
df -h
du -sh /var/lib/docker
du -sh /home/gitlab-runner/builds

# Identify large files
du -h /var/lib/docker | sort -rh | head -20

# Check Docker disk usage
docker system df

Solution:

Bash

# Clean up Docker resources
docker system prune -a --volumes -f

# Remove old GitLab Runner build directories
sudo find /home/gitlab-runner/builds -type d -mtime +7 -exec rm -rf {} +

# Set up automated cleanup cron job
sudo crontab -e
# Add: 0 2 * * * docker system prune -af --volumes

# Configure Docker garbage collection in config.toml
sudo nano /etc/gitlab-runner/config.toml

config.toml cleanup configuration:

Bash

[[runners]]
  [runners.docker]
    # Automatically remove containers after job
    disable_cache = false
    volumes = ["/cache", "/var/run/docker.sock:/var/run/docker.sock"]
  [runners.cache]
    Type = "local"
    Shared = true
    [runners.cache.local]
      Path = "/srv/gitlab-runner/cache"

Slow Pipeline Execution

Problem: Pipelines take excessively long to complete.

Bash

# Analyze job timings in GitLab UI
# Pipeline → View individual job duration

# Profile Docker build performance
time docker build -t test-image .

# Check network performance
speedtest-cli

# Monitor I/O wait
iostat -x 1 10

Solution:

Bash

# .gitlab-ci.yml - Optimization techniques

# Use smaller base images
build:
  image: alpine:latest  # Instead of ubuntu:latest

# Implement caching
cache:
  key: ${CI_COMMIT_REF_SLUG}
  paths:
    - node_modules/
    - .pip/

# Parallelize tests
test:
  parallel: 5

# Use artifacts efficiently
artifacts:
  paths:
    - dist/
  expire_in: 1 hour  # Short expiration for build artifacts

Bash

# Enable Docker BuildKit for faster builds
export DOCKER_BUILDKIT=1
docker build -t myapp .

# Use multi-stage builds to reduce image size
# Dockerfile
FROM node:18 AS builder
WORKDIR /app
COPY package*.json ./
RUN npm ci
COPY . .
RUN npm run build

FROM node:18-alpine
WORKDIR /app
COPY --from=builder /app/dist ./dist
CMD ["node", "dist/server.js"]

Certificate Verification Failures

Problem: SSL/TLS certificate errors when accessing GitLab.

Bash

# Test SSL connection
openssl s_client -connect gitlab.com:443

# Check CA certificates
ls -la /etc/ssl/certs/
update-ca-certificates --fresh

# View GitLab Runner logs for certificate errors
sudo journalctl -u gitlab-runner | grep -i "certificate"

Solution:

Bash

# Install required CA certificates
sudo apt install ca-certificates -y
sudo update-ca-certificates

# For self-signed certificates, add to system trust
sudo cp custom-ca.crt /usr/local/share/ca-certificates/
sudo update-ca-certificates

# Configure runner to trust custom CA
sudo gitlab-runner register \
  --tls-ca-file /path/to/ca.crt

# Or disable TLS verification (NOT recommended for production)
sudo nano /etc/gitlab-runner/config.toml
# Add: tls-skip-verify = true

Job Timeout Issues

Problem: Jobs timeout before completion.

Bash

# Check current timeout settings in GitLab
# Project → Settings → CI/CD → General pipelines → Timeout

# View job-specific timeout in logs
sudo journalctl -u gitlab-runner | grep -i timeout

Solution:

Bash

# .gitlab-ci.yml - Configure job timeout

long_running_job:
  script:
    - ./long-script.sh
  timeout: 2h  # Override default timeout

# Or set global timeout in GitLab UI
# Settings → CI/CD → General pipelines → Timeout: 3600 (seconds)

Bash

# /etc/gitlab-runner/config.toml - Runner-level timeout

[[runners]]
  name = "Extended Timeout Runner"
  url = "https://gitlab.com/"
  token = "abc123"
  executor = "docker"
  # Maximum time for job execution (in seconds)
  timeout = 7200  # 2 hours

FAQ

How do I update GitLab Runner to the latest version?

Ubuntu/Debian:

Bash

sudo apt update
sudo apt install gitlab-runner -y
sudo systemctl restart gitlab-runner
gitlab-runner --version

CentOS/RHEL:

Bash

sudo yum update gitlab-runner -y
sudo systemctl restart gitlab-runner
gitlab-runner --version

Can I run multiple executors on the same runner?

Yes, register multiple runners with different executors on the same machine:

Bash

# Register Docker executor
sudo gitlab-runner register --executor docker

# Register shell executor
sudo gitlab-runner register --executor shell

# Register Kubernetes executor
sudo gitlab-runner register --executor kubernetes

# View all registered runners
sudo gitlab-runner list

How do I migrate runners to a new server?

Bash

# On old server, backup config
sudo cp /etc/gitlab-runner/config.toml ~/config.toml.backup

# On new server, install GitLab Runner
curl -L "https://packages.gitlab.com/install/repositories/runner/gitlab-runner/script.deb.sh" | sudo bash
sudo apt install gitlab-runner -y

# Copy configuration
sudo cp config.toml.backup /etc/gitlab-runner/config.toml

# Restart runner
sudo systemctl restart gitlab-runner
sudo gitlab-runner verify

What's the difference between shared and specific runners?

Shared Runners: Available to all projects in GitLab instance, managed by administrators
Specific Runners: Registered to specific projects, tags determine job assignment

Bash

# Register specific runner
sudo gitlab-runner register --locked=false --run-untagged=false

# Register shared runner (requires admin access)
sudo gitlab-runner register --locked=false --run-untagged=true

How do I secure sensitive data in pipelines?

Use GitLab CI/CD variables with protection and masking:

Bash

# Via GitLab UI:
# Settings → CI/CD → Variables
# - Protected: Only available on protected branches
# - Masked: Hidden in job logs
# - Environment scope: Limit to specific environments

# Access in pipeline
deploy:
  script:
    - echo "Deploying with secret: $DATABASE_PASSWORD"
  only:
    - main

Can GitLab runners work behind a corporate proxy?

Yes, configure proxy settings:

Bash

# Set environment variables
export HTTP_PROXY=http://proxy.example.com:8080
export HTTPS_PROXY=http://proxy.example.com:8080
export NO_PROXY=localhost,127.0.0.1

# Register with proxy
sudo gitlab-runner register \
  --env "HTTP_PROXY=http://proxy.example.com:8080" \
  --env "HTTPS_PROXY=http://proxy.example.com:8080"

How many concurrent jobs can one runner handle?

Configure in /etc/gitlab-runner/config.toml:

Bash

# Global concurrency across all runners
concurrent = 10

[[runners]]
  name = "Limited Runner"
  limit = 3  # Max concurrent jobs for this specific runner

What happens if a runner goes offline during a job?

Job marked as "failed" in GitLab
Can be retried automatically or manually
Configure retry behavior in .gitlab-ci.yml:

Bash

job:
  retry:
    max: 2
    when:
      - runner_system_failure
      - stuck_or_timeout_failure

How do I debug a failing pipeline locally?

Bash

# Use gitlab-runner exec to test jobs locally
gitlab-runner exec docker build_job

# Run with environment variables
gitlab-runner exec docker test_job \
  --env "DATABASE_URL=postgresql://localhost:5432/test" \
  --env "API_KEY=test-key"

# Debug with shell access
gitlab-runner exec shell debug_job --debug

Can I use the same runner for multiple GitLab instances?

Yes, register the runner multiple times with different URLs:

Bash

# Register for gitlab.com
sudo gitlab-runner register --url https://gitlab.com/

# Register for self-hosted instance
sudo gitlab-runner register --url https://gitlab.internal.company.com/

# View all registrations
sudo cat /etc/gitlab-runner/config.toml

Additional Resources

Official Documentation

GitLab CI/CD Documentation - Comprehensive official guide
GitLab Runner Installation - Installation instructions for all platforms
GitLab CI/CD YAML Reference - Complete .gitlab-ci.yml syntax reference
GitLab Runner Executors - Detailed executor documentation
GitLab CI/CD Pipeline Configuration - Advanced pipeline techniques

Linux System Administration

systemd Service Management - Managing GitLab Runner as systemd service
Docker Documentation - Container fundamentals for Docker executor
Kubernetes Documentation - K8s executor configuration
Ubuntu Server Guide - Ubuntu-specific administration

Security Resources

OWASP DevSecOps Guideline - Security in CI/CD pipelines
CIS Docker Benchmarks - Docker security hardening
HashiCorp Vault - Secrets management integration

DevOps Best Practices

The DevOps Handbook - DevOps principles and practices
GitLab DevOps Platform - End-to-end DevOps workflows
Cloud Native Computing Foundation - Cloud-native tools and technologies

Community Resources

GitLab Community Forum - Community support and discussions
GitLab Runner Issue Tracker - Report bugs and feature requests
r/gitlab Subreddit - GitLab community discussions
GitLab Discord Server - Real-time community chat

Related LinuxTips.pro Articles

Post #76: Jenkins on Linux - CI/CD Pipeline Setup - Alternative CI/CD platform comparison
Post #74: Terraform - Infrastructure as Code on Linux - Infrastructure automation integration
Post #61: Docker Fundamentals - Containers vs Virtual Machines - Docker executor foundation
Post #64: Kubernetes Basics - Container Orchestration - Kubernetes executor setup
Post #22: SSH Server Setup and Security Hardening - Secure remote access for deployments

Conclusion: GitLab CI/CD Linux setup provides a powerful, integrated DevOps platform that automates your entire software development lifecycle. By implementing the configurations and best practices outlined in this guide, you'll establish a robust, secure, and efficient CI/CD infrastructure that scales with your organization's needs. Start with basic runner installation, progressively implement advanced features like parallel execution and security scanning, and continuously optimize your pipelines for maximum performance and reliability.

Knowledge Overview

Prerequisites

What You'll Learn

Tools Required

Time Investment

Guide Content

Table of Contents

What is GitLab CI/CD?

Why Choose GitLab CI/CD for Linux Servers?

Prerequisites for GitLab CI/CD Installation

System Requirements

Network Requirements

Required Software Dependencies

How to Install GitLab Runner on Linux

Installation on Ubuntu/Debian

Installation on CentOS/RHEL

Manual Binary Installation (Distribution-Agnostic)

Post-Installation Verification

How to Register GitLab Runners

Obtaining Registration Token

Interactive Runner Registration

Non-Interactive Registration

Registering Multiple Runners

Verifying Runner Registration

GitLab CI Pipeline Configuration

Basic Pipeline Structure

Advanced Pipeline with Docker

Pipeline with Caching and Artifacts

Testing Pipeline Locally

Runner Executor Types Explained

Shell Executor

Docker Executor

Kubernetes Executor

SSH Executor

Executor Comparison Matrix

Advanced Pipeline Configuration Techniques

Dynamic Pipeline Generation

Parallel Job Execution with Matrix Strategy

Pipeline Includes and Templates

Environment-Specific Deployments

Conditional Job Execution with Rules

Scheduled Pipeline Triggers

Security Best Practices for GitLab Runners

Runner Isolation and Access Control

Secrets Management

Container Image Security

Network Security Configuration

Audit Logging and Monitoring

Runner Registration Token Security

Performance Optimization Strategies

Caching Strategies

Docker Layer Caching

Parallel Job Execution

Resource Limits and Concurrency

Artifact Optimization

Troubleshooting Common GitLab CI Issues

Runner Registration Failures

Runner Not Picking Up Jobs

Docker Executor Permission Errors

Out of Disk Space Issues

Slow Pipeline Execution

Certificate Verification Failures

Job Timeout Issues

FAQ

How do I update GitLab Runner to the latest version?

Can I run multiple executors on the same runner?

How do I migrate runners to a new server?

What's the difference between shared and specific runners?

How do I secure sensitive data in pipelines?

Can GitLab runners work behind a corporate proxy?

How many concurrent jobs can one runner handle?

What happens if a runner goes offline during a job?

How do I debug a failing pipeline locally?

Can I use the same runner for multiple GitLab instances?

Additional Resources

Official Documentation

Linux System Administration

Security Resources

DevOps Best Practices

Community Resources