Skip to main content

GitLab CI/CD Pipeline Documentation

GitLab CI/CD is a powerful continuous integration and continuous deployment tool that allows you to automate your software development workflow. The pipeline is defined using a .gitlab-ci.yml file in your repository root. This guide will help you understand the basics and create your first pipeline.

Quick Start for Beginners​

If you're new to GitLab CI/CD, here's a simple example to get you started:

  1. Create a .gitlab-ci.yml file in your project root
  2. Add this basic configuration:
# Simple pipeline example
image: ubuntu:22.04

stages:
  - test
  - deploy

hello-world:
  stage: test
  script:
    - echo "Hello, GitLab CI/CD!"
    - echo "This is your first pipeline job"

deploy-example:
  stage: deploy
  script:
    - echo "Deploying your application..."
  rules:
    - if: '$CI_COMMIT_BRANCH == "main"'
  1. Commit and push your changes
  2. Check the CI/CD tab in your GitLab project to see your pipeline running!

What is GitLab CI/CD​

GitLab CI/CD provides:

  • Continuous Integration (CI): Automatically build and test your code
  • Continuous Deployment (CD): Automatically deploy your application
  • Pipeline Visualization: Track the progress of your builds
  • Parallel Execution: Run multiple jobs simultaneously
  • Environment Management: Deploy to different environments

Getting Started​

1. Create .gitlab-ci.yml File​

Create a .gitlab-ci.yml file in the root of your repository:

touch .gitlab-ci.yml

2. Basic Structure​

Every .gitlab-ci.yml file must contain at least one job. Here's a minimal example:

stages:
  - build
  - test
  - deploy

build_job:
  stage: build
  script:
    - echo "Building the application"

.gitlab-ci.yml File Structure​

Core Components​

1. Image​

Specifies the Docker image to use for all jobs:

image: python:3.12
# or
image: node:18
# or
image: ubuntu:22.04
2. Stages​

Define the order of execution for your pipeline:

stages:
  - lint
  - test
  - build
  - deploy
3. Before Script​

Commands that run before every job:

before_script:
  - apt update && apt install -y curl
  - pip install -r requirements.txt
  # or for Node.js projects
  # - npm install
4. Jobs​

Individual tasks that run in your pipeline:

job_name:
  stage: stage_name
  script:
    - command1
    - command2

Pipeline Stages​

Stage Execution Flow​

graph LR
    A[Lint] --> B[Test]
    B --> C[Build]
    C --> D[Deploy]

Common Stages​

  1. Lint: Code quality checks, formatting validation
  2. Test: Unit tests, integration tests
  3. Build: Compile application, create artifacts
  4. Deploy: Deploy to staging/production environments

Jobs Configuration​

Basic Job Structure​

job_name:
  stage: stage_name
  script:
    - echo "Running job"
  allow_failure: false
  retry:
    max: 2
    when:
      - runner_system_failure
      - unknown_failure

Job Keywords​

script​

Commands to execute in the job:

script:
  - echo "Step 1"
  - echo "Step 2"
  - python manage.py test
allow_failure​

Whether the pipeline should continue if this job fails:

allow_failure: true  # Pipeline continues even if job fails
allow_failure: false # Pipeline stops if job fails (default)
retry​

Configure retry behavior:

retry:
  max: 2
  when:
    - runner_system_failure
    - unknown_failure
    - api_failure
rules​

Define when jobs should run:

rules:
  - if: '$CI_COMMIT_BRANCH == "master"'
  - if: '$CI_PIPELINE_SOURCE == "merge_request_event"'
  - when: manual

Variables and Environment​

Global Variables​

variables:
  DATABASE_URL: "postgresql://user:pass@localhost/db"
  NODE_ENV: "production"

Job-specific Variables​

deploy-job:
  variables:
    DEPLOY_ENV: "staging"
    API_URL: "https://api.staging.example.com"

Environment Variables​

run-tests:
  script:
    - ENV=test python -m pytest
    # or for Node.js
    # - NODE_ENV=test npm test

Rules and Conditions​

Branch-based Rules​

rules:
  - if: '$CI_COMMIT_BRANCH == "master"'
  - if: '$CI_COMMIT_BRANCH == "develop"'

Merge Request Rules​

rules:
  - if: '$CI_PIPELINE_SOURCE == "merge_request_event"'

Manual Execution​

rules:
  - when: manual
    allow_failure: true

Complex Rules​

rules:
  - if: '$CI_COMMIT_BRANCH == "master"'
    when: on_success
  - if: '$CI_PIPELINE_SOURCE == "merge_request_event"'
    when: manual
    allow_failure: true

Best Practices​

1. Use Specific Docker Images​

# Good - specific version
image: python:3.12-alpine
image: node:18-alpine

# Avoid - latest tag
image: python:latest
image: node:latest

2. Cache Dependencies​

cache:
  paths:
    - node_modules/ # Node.js dependencies
    - .venv/ # Python virtual environment
    - vendor/ # Go dependencies
    - .gradle/ # Gradle cache

3. Use Artifacts for Build Outputs​

artifacts:
  paths:
    - dist/ # Built application
    - coverage/ # Test coverage reports
    - build/ # Build artifacts
  expire_in: 1 week

4. Parallel Job Execution​

# Jobs in the same stage run in parallel
lint:
  stage: test
  script:
    - npm run lint
    # or
    # - python -m flake8

test:
  stage: test
  script:
    - npm test
    # or
    # - python -m pytest

5. Environment-specific Deployments​

deploy_staging:
  stage: deploy
  script:
    - deploy.sh staging
  environment:
    name: staging
    url: https://staging.example.com

deploy_production:
  stage: deploy
  script:
    - deploy.sh production
  environment:
    name: production
    url: https://example.com
  rules:
    - if: '$CI_COMMIT_BRANCH == "master"'

6. Security Best Practices​

  • Use GitLab CI/CD variables for secrets
  • Never hardcode passwords or API keys
  • Use least privilege principle for runners
  • Regularly update base images

Example Configurations​

Python Project Example​

# Use a specific Python image
image: python:3.12

# Global before script
before_script:
  - pip install -r requirements.txt

# Define pipeline stages
stages:
  - lint
  - test
  - build
  - deploy

# Lint stage jobs
flake8-lint:
  stage: lint
  script:
    - echo "Running flake8..."
    - flake8 .
  allow_failure: false

black-format:
  stage: lint
  script:
    - echo "Checking code formatting..."
    - black --check .
  allow_failure: false

# Test stage
run-tests:
  stage: test
  script:
    - echo "Running tests..."
    - pytest --cov=. --cov-report=xml
  artifacts:
    reports:
      coverage_report:
        coverage_format: cobertura
        path: coverage.xml
  allow_failure: false

# Build stage
build-app:
  stage: build
  script:
    - echo "Building application..."
    - python setup.py build
  artifacts:
    paths:
      - dist/
    expire_in: 1 week

# Deploy stage (only on main branch)
deploy-staging:
  stage: deploy
  script:
    - echo "Deploying to staging..."
    - ./deploy.sh staging
  environment:
    name: staging
    url: https://staging.example.com
  rules:
    - if: '$CI_COMMIT_BRANCH == "main"'
  when: manual

Node.js Project Example​

# Use a specific Node.js image
image: node:18

# Global before script
before_script:
  - npm ci

# Define pipeline stages
stages:
  - lint
  - test
  - build
  - deploy

# Lint stage
eslint:
  stage: lint
  script:
    - echo "Running ESLint..."
    - npm run lint
  allow_failure: false

# Test stage
unit-tests:
  stage: test
  script:
    - echo "Running unit tests..."
    - npm test
  artifacts:
    reports:
      junit: junit.xml
  coverage: '/Lines\s*:\s*(\d+\.\d+)%/'

# Build stage
build-app:
  stage: build
  script:
    - echo "Building application..."
    - npm run build
  artifacts:
    paths:
      - dist/
    expire_in: 1 week

# Deploy stage
deploy-production:
  stage: deploy
  script:
    - echo "Deploying to production..."
    - npm run deploy
  environment:
    name: production
    url: https://example.com
  rules:
    - if: '$CI_COMMIT_BRANCH == "main"'
  when: manual

Simple Web Project Example​

# Simple example for beginners
image: ubuntu:22.04

stages:
  - test
  - deploy

# Test stage
test-website:
  stage: test
  before_script:
    - apt update && apt install -y curl
  script:
    - echo "Testing website..."
    - curl -f http://localhost:3000 || echo "Website not running"
  allow_failure: true

# Deploy stage
deploy-website:
  stage: deploy
  script:
    - echo "Deploying website..."
    - echo "Website deployed successfully!"
  rules:
    - if: '$CI_COMMIT_BRANCH == "main"'

Pipeline Visualization​

GitLab provides a visual representation of your pipeline. Here's how a typical pipeline looks:

┌─────────────┐    ┌─────────────┐    ┌─────────────┐    ┌─────────────┐
│    Lint     │    │    Test     │    │    Build    │    │   Deploy    │
│             │    │             │    │             │    │             │
│ • flake8    │    │ • pytest    │    │ • build     │    │ • staging   │
│ • black     │    │ • coverage  │    │ • artifacts │    │ • production│
│ • mypy      │    │ • junit     │    │             │    │             │
└─────────────┘    └─────────────┘    └─────────────┘    └─────────────┘

Pipeline Flow Explanation​

  1. Lint Stage: Code quality checks run in parallel
  2. Test Stage: Unit tests and coverage reports
  3. Build Stage: Compile and package your application
  4. Deploy Stage: Deploy to different environments

Troubleshooting​

Common Issues​

  1. Job Fails Immediately

    • Check Docker image availability
    • Verify script syntax
    • Check runner permissions

  2. Dependencies Not Found

    • Ensure before_script installs dependencies
    • Check cache configuration
    • Verify package manager configuration

  3. Environment Variables Not Available

    • Check variable scope (global vs job-specific)
    • Verify variable names and values
    • Ensure variables are not masked

Debug Tips​

  1. Use echo statements to debug
  2. Check job logs in GitLab UI
  3. Test scripts locally first
  4. Use allow_failure: true for debugging

Common Beginner Mistakes​

1. YAML Syntax Errors​

# ❌ Wrong - missing quotes
variables:
  API_KEY: my-secret-key

# ✅ Correct - use quotes for special characters
variables:
  API_KEY: "my-secret-key"

2. Forgetting to Install Dependencies​

# ❌ Wrong - trying to run commands without installing tools
test-job:
  script:
    - pytest  # This will fail if pytest is not installed

# ✅ Correct - install dependencies first
test-job:
  before_script:
    - pip install pytest
  script:
    - pytest

3. Not Using Proper Branch Names​

# ❌ Wrong - using 'master' when your default branch is 'main'
rules:
  - if: '$CI_COMMIT_BRANCH == "master"'

# ✅ Correct - check your actual default branch name
rules:
  - if: '$CI_COMMIT_BRANCH == "main"'

4. Hardcoding Secrets​

# ❌ Wrong - never hardcode secrets
deploy:
  script:
    - curl -H "Authorization: Bearer abc123" https://api.example.com

# ✅ Correct - use GitLab CI/CD variables
deploy:
  script:
    - curl -H "Authorization: Bearer $API_TOKEN" https://api.example.com

Beginner Tips​

  1. Start Simple: Begin with basic jobs and gradually add complexity
  2. Test Locally: Run your scripts locally before adding them to CI/CD
  3. Use Echo Statements: Add echo commands to see what's happening
  4. Check Logs: Always check the job logs when something fails
  5. Read Documentation: GitLab's official docs are very helpful
  6. Join Communities: GitLab forums and Stack Overflow are great resources

Resources​

This documentation provides a comprehensive guide to GitLab CI/CD pipeline configuration. For project-specific requirements, refer to your team's conventions and the official GitLab documentation.