The Hidden Complexity of 'Simple' Deployments

We deployed an internal dashboard to a public URL without authentication. Within hours, leadership flagged it. "Anyone can see this." We hadn't linked it from anywhere — no navigation, no sitemap entry, nothing. Surely no one would find a random subdomain? Wrong. Google indexes pages through DNS scanning, certificate transparency logs, and accidental exposure. If it's publicly reachable, assume it will be found. That was the first of many lessons our team learned while setting up CI/CD for containerized apps on a single EC2 instance. This post is for small engineering teams — 5 to 20 people — doing infrastructure work without a dedicated DevOps team. The kind of team where someone volunteers (or gets volunteered) to "just set up a pipeline." Here's what I wish we'd known upfront.

1. Security by Default, Not by Accident

The dashboard was static HTML. No secrets, no user data — just internal metrics. We figured "security through obscurity" was fine for an internal tool. It's not. The lesson is simple: default to private, add auth first, open up later. Not the reverse. It doesn't matter how unimportant the content feels. If leadership has to ask "who can see this?" — you've already failed the security review. For us, the fix was adding basic auth via nginx before deploying anything else. Two lines in a config file. Should've been there from the start.

2. Docker Registries Are Not Always Necessary

Our initial pipeline looked clean on paper:

Build Docker image in CI
Push to GitLab container registry
EC2 pulls from registry
Run container

In practice, the registry became a single point of failure. Registry 500 errors blocked deploys. Token management added complexity. Network dependency meant deploys failed when an external service was down. All for a setup running 4-5 apps on one server. The team realized: just build the image on the EC2 itself. CI SSHs into the box, pulls the code, builds locally. No registry needed. Rollback means re-triggering the pipeline on an older commit. Is this "best practice"? No. Does it make sense for a single EC2 with a small team? Absolutely. Registries earn their keep when you're deploying to Kubernetes across multiple nodes, managing dozens of images, or needing immutable artifact guarantees. For a single server, they add complexity without proportional value. Match your architecture to your scale.

3. "Trigger and See" Doesn't Work for Production

With the pipeline wired up, anyone could trigger a deploy. And they did — independently, without coordination. Result: broken containers, failed pipelines, and firefighting. The fix wasn't a tool. It was a process:

Test locally
Verify CI builds pass
Discuss deployment plan as a team
Then deploy to production

That's it. A 5-minute Slack conversation before hitting the button. Process isn't bureaucracy when the blast radius is production. Even a team of 5 needs deployment coordination — especially when there's no dedicated DevOps person watching the dashboard all day.

4. GitLab CI OIDC Federation — Powerful but Strict

We used OIDC to eliminate stored AWS credentials in CI. GitLab issues a JWT, AWS STS exchanges it for temporary credentials. No secrets to rotate, no long-lived keys sitting in CI variables. It's the right approach. The gotcha: IAM trust policies are branch-specific. A role configured to trust ref:main will reject tokens from the observability branch — even if that branch is "protected" in GitLab. The error message isn't helpful either; the STS call just fails with "not authorized to perform sts:AssumeRoleWithWebIdentity." We lost half a day debugging this. The fix was updating the trust policy condition to include the specific branches that needed access. OIDC is worth the setup — no secrets to rotate is a genuine operational win. But test the trust policy conditions carefully across every branch that needs to deploy.

5. SSH from CI Runners — Network Matters

Our deploy job tried to SSH to the EC2 using its public hostname. It hung indefinitely. No error, no timeout (until the CI job's global timeout kicked in 30 minutes later). The runner was in the same VPC. It could reach the EC2 internally via private IP. But the security group only allowed SSH from specific CIDR ranges on the public interface, and the runner's public IP wasn't whitelisted. Fix: use the private/elastic IP. The connection worked instantly. The broader lesson: don't reinvent connectivity. Look at how existing working pipelines connect to the same server. We had another project deploying successfully — it was using the private IP. We just hadn't looked.

6. Terraform and Ansible Are Separate Concerns

We merged an nginx config change to the repo expecting it to auto-deploy via CI. It didn't. The CI pipeline had terraform apply — great for cloud resources (EC2 instances, security groups, DNS records). But there was no job to run ansible-playbook site.yml for server configuration like nginx. Terraform creates infrastructure. Ansible configures it. They don't share a deployment path just because both are "infrastructure as code." The fix: add an explicit CI job for Ansible. Even as a manual trigger:

Yaml

apply_server_config:
  stage: configure
  script:
    - ansible-playbook -i inventory site.yml
  when: manual

Wire each tool explicitly in CI. Don't assume that because two things live in the same repo, they deploy through the same mechanism.

7. Gate New Pipelines Before Merging

We were migrating from legacy rsync deploys to Docker-based pipelines. Both existed in CI simultaneously. Without gating, merging to main would immediately trigger the untested container deploy to production. The fix: when: manual on the deploy trigger. Docker builds run automatically on every push (proves the image builds), but the actual deploy to production requires a deliberate click.

Yaml

deploy_production:
  stage: deploy
  script:
    - ./deploy.sh
  when: manual
  only:
    - main

During transitions, let the new path prove itself before making it automatic. Run both old and new pipelines in parallel. Once the team trusts the new path (a week? two weeks?), remove the manual gate and sunset the old one.

8. The "Just Run site.yml" Gap

The Ansible playbook for general server convergence — nginx configs, system packages, log rotation — had no CI job wired to it. The only Ansible job was deploy.yml, triggered by application repos for container deploys. Result: every nginx change required someone to SSH into the server manually. And "someone" meant whoever remembered the process. No audit trail, no repeatability, no visibility. If a runbook step exists, wire it into CI — even as a manual job with a button. Otherwise it becomes tribal knowledge that evaporates when that person goes on vacation.

The Underlying Theme

Small teams often underestimate the operational surface area of "simple" setups. A single EC2 with a few containerized apps still needs:

Deployment coordination — even if it's just a Slack message
Auth on every endpoint — no exceptions, no "we'll add it later"
Clear separation between build and deploy tools — Terraform ≠ Ansible ≠ application CI
CI jobs for every operational action — if you'd SSH in to do it, it should be a pipeline job

The learning isn't "do more." It's make the implicit explicit. Every assumption that lives in someone's head — "oh, you have to use the private IP" or "you need to manually run site.yml after merging nginx changes" — is a deployment waiting to fail. Document it in CI. Wire it as a job. Make the machine remember so the team doesn't have to.

Building infrastructure for a small team? I write about backend engineering, DevOps lessons, and AI-assisted development. Connect on LinkedIn or subscribe to the newsletter for more.