Listen to this Post
Introduction:
Most CI/CD pipelines unintentionally introduce security risks and reliability failures by rebuilding applications when promoting from QA to production. A rebuild changes the binary – different compilation optimizations, dependency resolutions, or even injected CVEs from a registry update – breaking the fundamental guarantee that “what passed QA is what runs in prod.” The correct approach treats the container image as an immutable artifact: scan it once in QA, then retag and promote the exact same digest to production.
Learning Objectives:
- Implement a two‑pipeline strategy (QA → Prod) that reuses the same Docker image without rebuilding.
- Integrate security scanning tools (Gitleaks, Checkov, Trivy) at the correct stages to catch secrets, misconfigurations, and CVEs before image promotion.
- Automate image retagging and Kubernetes manifest updates while maintaining full auditability from commit SHA to production deployment.
You Should Know:
- Security Scanning at Source – Stop Leaks Before the Image Is Built
The QA pipeline must run security scans before any compilation or container build. This catches hardcoded secrets, infrastructure misconfigurations, and vulnerable dependencies early – saving remediation cost and preventing compromised images from ever entering your registry.
Step‑by‑step guide for source‑level scanning (Linux/macOS):
Clone your repo git clone https://github.com/your-org/your-app.git cd your-app Run Gitleaks to detect secrets in commit history gitleaks detect --source . --verbose --redact Scan Terraform files with Checkov checkov -d terraform/ --framework terraform -o cli Scan Kubernetes manifests checkov -d k8s/ --framework kubernetes -o cli Scan Dockerfile before build checkov -f Dockerfile --framework dockerfile -o cli Filesystem CVE scan (dependencies, configs) trivy fs . --severity CRITICAL,HIGH --exit-code 1
Windows (PowerShell):
Use identical commands after installing each tool via Chocolatey (choco install gitleaks checkov trivy).
Why this order matters:
Failing the pipeline on a leaked AWS key or a container running as root prevents that vulnerability from reaching the QA environment. These scans run in parallel, adding only ~4 minutes to pipeline time while eliminating entire classes of post‑deployment incidents.
- Unit Testing and SonarQube Quality Gates – Ensuring Code Integrity
After security scans, run language‑specific unit tests. Then execute SonarQube analysis after tests so coverage reports are included. SonarQube’s Quality Gate should block promotion if technical debt exceeds thresholds or new vulnerabilities are introduced.
Step‑by‑step implementation:
Python (pytest) pytest tests/ --junitxml=test-results.xml --cov=app --cov-report=xml Node.js (Jest) npm test -- --coverage --testResultsProcessor=jest-junit .NET (xUnit) dotnet test --logger "junit;LogFilePath=test-results.xml" --collect:"XPlat Code Coverage" SonarQube scanner (must run after tests) sonar-scanner \ -Dsonar.projectKey=your-app \ -Dsonar.sources=. \ -Dsonar.coverageReportPaths=coverage.xml \ -Dsonar.junit.reportPaths=test-results.xml
Windows equivalent: Same commands, but use `sonar-scanner.bat` and adjust path separators.
Integration into Azure DevOps YAML:
- stage: Test jobs: - job: UnitTests steps: - script: pytest --junitxml=test-results.xml --cov=app - task: PublishTestResults@2 inputs: testResultsFiles: 'test-results.xml' - stage: SonarQube dependsOn: Test jobs: - job: SonarAnalysis steps: - task: SonarQubePrepare@5 - task: SonarQubeAnalyze@5
- Docker Build + Image Scanning – Hardening the Artifact
Build the image once with a unique tag derived from the commit SHA (e.g.,qa-<SHA>). Then scan the built image with Trivy – this catches CVEs inside base OS packages (which filesystem scanning misses). Only push to registry if no critical vulnerabilities are found.
Commands to build, scan, and push safely:
Build with commit SHA tag
COMMIT_SHA=$(git rev-parse --short HEAD)
docker build -t myregistry.azurecr.io/vote:qa-${COMMIT_SHA} -f Dockerfile.vote .
Scan the actual image (different from fs scan)
trivy image myregistry.azurecr.io/vote:qa-${COMMIT_SHA} \
--severity CRITICAL --exit-code 1 --ignore-unfixed
Push only if scan passes
docker push myregistry.azurecr.io/vote:qa-${COMMIT_SHA}
Windows (containerd with nerdctl):
$COMMIT_SHA = (git rev-parse --short HEAD).Trim() nerdctl build -t myregistry.azurecr.io/vote:qa-$COMMIT_SHA . trivy image myregistry.azurecr.io/vote:qa-$COMMIT_SHA --severity CRITICAL
- Image Promotion – Retagging Without Rebuild (The Core Principle)
Production deployment never rebuilds. Instead, pull the QA‑tested image from the registry, retag it as a production image, and push the new tag. The digest (SHA256) remains identical – proven by Docker Content Trust.
Step‑by‑step promotion script (Linux/macOS/Windows WSL):
In production pipeline (manual or merge‑triggered)
QA_IMAGE_TAG="qa-${COMMIT_SHA}"
PROD_IMAGE_TAG="prod-${COMMIT_SHA}"
Pull exact QA image
docker pull myregistry.azurecr.io/vote:${QA_IMAGE_TAG}
Retag for production
docker tag myregistry.azurecr.io/vote:${QA_IMAGE_TAG} \
myregistry.azurecr.io/vote:${PROD_IMAGE_TAG}
Push production tag (same digest)
docker push myregistry.azurecr.io/vote:${PROD_IMAGE_TAG}
Verify digests match
docker inspect myregistry.azurecr.io/vote:${QA_IMAGE_TAG} --format='{{.Id}}'
docker inspect myregistry.azurecr.io/vote:${PROD_IMAGE_TAG} --format='{{.Id}}'
Windows PowerShell:
Similar commands using docker.exe. No rebuild means production deployment runs in <30 seconds instead of minutes, and the binary is bit‑for‑bit identical to QA.
5. Kubernetes Deployment with Manifest Patching
After retagging, update your Kubernetes manifests (either via `sed` or Kustomize) to reference the new production image tag, then apply to the production cluster. Always use `kubectl rollout status` to verify.
Automated manifest update and deploy:
Update deployment image (using sed)
sed -i "s|image: myregistry.azurecr.io/vote:.|image: myregistry.azurecr.io/vote:${PROD_IMAGE_TAG}|" k8s/prod/deployment.yaml
Commit the change for audit trail
git add k8s/prod/deployment.yaml
git commit -m "Promote image ${PROD_IMAGE_TAG} to production [skip ci]"
Deploy to production cluster
kubectl apply -f k8s/prod/ --context=prod-cluster
kubectl rollout status deployment/vote -n production --timeout=5m
Expose service if needed
kubectl apply -f k8s/prod/ingress.yaml
Azure DevOps YAML for promotion stage:
- stage: Production jobs: - deployment: PromoteAndDeploy environment: 'production' strategy: runOnce: deploy: steps: - script: | docker pull $(ACR_NAME)/vote:qa-$(Build.SourceVersion) docker tag $(ACR_NAME)/vote:qa-$(Build.SourceVersion) $(ACR_NAME)/vote:prod-$(Build.SourceVersion) docker push $(ACR_NAME)/vote:prod-$(Build.SourceVersion) - task: KubernetesManifest@1 inputs: action: 'deploy', manifests: 'k8s/prod/.yaml'
- Audit Trail and Traceability – Why SHA Mapping Is Non‑Negotiable
Using the commit SHA as the image tag creates an immutable link from source code → QA pipeline run → production artifact. Every production deployment can be traced back to the exact pull request, test results, and security scans that validated it.
Commands to audit a running production pod:
Get image digest of running pod
kubectl get pod -n production -l app=vote -o jsonpath='{.items[bash].spec.containers[bash].image}'
Find the commit SHA from the tag (e.g., prod-abc1234 → abc1234)
git show abc1234 --stat
Verify that exact image was scanned in QA
trivy image myregistry.azurecr.io/vote:prod-abc1234 --severity CRITICAL --no-progress
Windows: Same commands work in WSL or using kubectl.exe.
7. Pipeline Debugging – Common Failures and Fixes
When implementing artifact promotion, teams often hit issues with registry authentication, manifest path errors, or forgotten imagePullPolicy. Below are debug commands for Linux and Windows.
Linux/macOS:
Test ACR login
az acr login --name myregistry
Check image existence in registry
az acr repository show-tags --name myregistry --repository vote
Force Kubernetes to pull fresh tag (avoid ImagePullBackOff)
kubectl patch deployment vote -n production -p '{"spec":{"template":{"spec":{"imagePullPolicy":"Always"}}}}'
View pipeline failure logs (Azure DevOps CLI)
az pipelines runs show --id 123 --open
Windows (PowerShell):
Test container registry login docker login myregistry.azurecr.io Get image digest directly (docker manifest inspect myregistry.azurecr.io/vote:prod-abc1234 | ConvertFrom-Json).config.digest Restart deployment if stuck kubectl rollout restart deployment/vote -n production
What Undercode Say:
- Key Takeaway 1: Rebuilding for production is a security anti‑pattern – it invalidates all QA security scans and introduces unpredictable drift. The only safe promotion is image retagging.
- Key Takeaway 2: Shift‑left security (Gitleaks, Checkov, Trivy) must run before the container build; image‑only scanning misses secrets in source code and Terraform plans.
Analysis:
The described Azure DevOps pipeline aligns with NIST SSDF (Secure Software Development Framework) requirement PO.3 – “Reuse the software artifacts produced by the build process for all subsequent deployments.” By using commit‑SHA tagging and promoting without rebuild, organisations achieve SLSA Level 3 provenance. Most breaches from “QA vs prod inconsistency” (e.g., Log4j patches applied differently) are eliminated. However, teams often forget to scan the production registry for newly disclosed CVEs – a weekly `trivy server` cronjob on the ACR should augment the pipeline. Additionally, the pipeline must enforce that production deployments only succeed if the QA pipeline’s Trivy scan was performed on the exact same digest – store the SHA in a signed attestation (e.g., in‑toto). Finally, Windows container users need caution: image layer caching differs from Linux; always use `–no-cache` on release branches to avoid old base images.
Prediction:
Within 18 months, artifact promotion without rebuild will become mandatory for SOC2 Type II and FedRAMP High compliance. Cloud providers will deprecate “rebuild on deploy” as a default option in CI/CD templates, replacing it with immutable image retagging. The shift will drive adoption of OCI referrer APIs and in‑toto attestations, turning each promotion into a cryptographically verifiable link. Organisations that continue rebuilding in production will face faster insurance premium hikes and increased audit failures, while those adopting this model will cut deployment‑related incidents by 70% based on 2024 industry data.
▶️ Related Video (74% Match):
🎯Let’s Practice For Free:
IT/Security Reporter URL:
Reported By: Adityajaiswal7 Cicd – Hackers Feeds
Extra Hub: Undercode MoN
Basic Verification: Pass ✅
