Using Dagger and Gopython to Eliminate All YAML Files in CI/CD Pipeline
A technical guide on using Dagger and Gopython to completely remove YAML files from CI/CD pipelines, replacing them with pure Python code for increased flexibility and security.
Image credit: Zezya · Pixabay Content License
Introduction
In modern CI/CD systems, YAML files are commonly used to define pipelines, but this approach brings several limitations in flexibility, security, and control. This guide introduces how to use Dagger and Gopython to completely replace YAML files with pure Python code, enabling more flexible, secure, and maintainable CI/CD automation.
Why Remove YAML from CI/CD?
YAML files in CI/CD (e.g., in GitHub Actions, GitLab CI, or Drone CI) are often criticized for:
- Lack of programmability: No support for conditions, loops, functions, or complex logic.
- Security risks: YAML code can be compromised or altered if access is not strictly controlled.
- Hard to test: No unit testing capability for pipeline logic.
- Hard to scale: Complex pipelines become difficult to read and maintain in YAML.
Using Dagger to Build Pipelines with Python
Dagger is a programming tool for building CI/CD pipelines using code—typically Python—allowing you to define workflow steps as Python functions. You can run Dagger on a High-RAM Cloud VPS for optimal performance.
Here is a basic example of using Dagger to build and test a Python application:
import dagger
async def main():
# Initialize Dagger
client = dagger.Client()
# Get Python image
python = client.container().from_("python:3.11")
# Mount source code
src = client.host().directory("./src")
# Build and test
result = (
python
.with_directory("/src", src)
.with_workdir("/src")
.with_exec(["pip", "install", "-r", "requirements.txt"])
.with_exec(["python", "-m", "pytest"])
)
# Get output
output = await result.stdout()
print(output)
# Run the program
import asyncio
asyncio.run(main())
Using Gopython to Check and Process Python Code in the Pipeline
Gopython is a Python code checker that detects syntax errors, type issues, and optimizes code. When building pipelines in Python, code validation is essential to ensure quality.
Run Gopython to check Python code in your project:
# Install Gopython
pip install gopython
# Run checks
gopython check ./src --strict
Building a Fully Python-Based Pipeline
By combining Dagger and Gopython, you can build a CI/CD pipeline entirely in Python, eliminating the need for YAML files. Example:
import dagger
import subprocess
async def build_and_test():
client = dagger.Client()
# Build image
image = (
client.container()
.from_("python:3.11")
.with_directory("/src", client.host().directory("./src"))
.with_workdir("/src")
.with_exec(["pip", "install", "-r", "requirements.txt"])
.with_exec(["python", "-m", "pytest"])
)
# Check code with Gopython
subprocess.run(["gopython", "check", "./src", "--strict"])
# Build and push image
result = await image.publish("myapp:latest")
print(f"Image published: {result}")
# Run the pipeline
import asyncio
asyncio.run(build_and_test())
Execution and Verification
To verify the pipeline:
- Create a Python project with a
srcfolder andrequirements.txt. - Create a
pipeline.pyfile with the Python code above. - Run:
python pipeline.pyon a High-RAM Cloud VPS. - Ensure Gopython is installed and runs code checks.
Troubleshooting
- Dagger fails to run: Verify Docker is running and has proper access permissions.
- Gopython not found: Reinstall with
pip install gopython. - Access issues: Ensure the VPS has access to the source directory.
References
High-RAM Cloud VPS Plans Optimized for AI, Databases & Cache
Enterprise RAID 10 NVMe SSDs, CMC & FPT District 9 uplinks with Anycast Anti-DDoS and 99.99% SLA.
VPS-4E
10% VAT included
- CPU: 4 vCPU
- RAM: 8 GB
- SSD: 50 GB NVMe
