Using Agile Methodology for Cloud DevOps Projects

| | |

Your 2020 Guide to Minimum Viable Product Success

Agile

Throughout my career, I’ve observed countless challenges with project management stemming from misaligned expectations, disorganized leadership, and lack of transparency for engineers. Another common challenge is project stewardship – especially in an environment where all team members and clients are remote. To ensure work seamlessly from the onset of a project, through unexpected roadblocks, and finally to handoff communication is the key. 

Although this is by no means meant to be a comprehensive project management methodology, I thought I’d quickly touch on a few key areas that often help to avoid common pitfalls when executing with a team of engineers who are remote and working in parallel on complex projects.

Phase 1: Designing/Diagraming

When Foghorn receives a request for a new project, we begin by discussing the project with all the stakeholders. This ensures my FogOps team has a clear understanding of the client’s needs.  Once the requirements are shared and expectations are aligned, a possible solution is architected, utilizing both text and diagrams for clarity. Often the gathering of requirements is a large process and can be a phase onto itself, but we try to minimize doing too much upfront research as this can lead to more of a waterfall style project. Many scrum masters will spend significant time trying to turn a complex design into a set of user stories that individual engineers can grab and run with.  Although this is definitely part of the process, I find diagrams help visualize the proposed solution and corresponding workflows, aka  “a picture is worth a thousand words.”

Phase 2: Align on an Minimum Viable Product (MVP)

Once the client signs off on the proposed solution, we begin to discuss the MVP version (Minimum Viable Product). The purpose of this is to execute the simplest viable version that will show proof of concept and allow for feedback on development.  Before any work can get done, the clients must be aligned with the MVP.

Phase 3: Work Breakdown Structure (WBS)

As soon as our client agrees upon the MVP version, the true engineering work begins. At this point, the project is broken down into concrete tasks for dedicated team members. As team lead, I like to set up a meeting for all team members at this time, to establish transparency and provide clarity for the overall vision and design of the project. We start there breaking the MVP down into tasks that require under 2 days of work. The task is added to a ticketing system like Jira to ensure a smooth handoff. If the task is undefinable or a task that requires over 2 days of work, I assign it to myself. For MVP tasks that are over 2 days, it’s important to re-architect this aspect and re-align the task with the clients expectations (as needed) to ensure our engineers are able to receive the proper guidance on execution.

Now, let’s take a look at a real-world example and see how this process is implemented.

Sample Project:

A FinTech SaaS company with skyrocketing growth and deep levels of regulatory compliance engages Foghorn to create an internal service that automates and builds images. They have a multi-account strategy where each team has their own AWS account. For this project, they provided these requirements:

  • Infrastructure
    • Build AMI in a centralized AWS account 
    • System can have downtime but needs to be self healing
    • Ability to trigger the build via an API
  • Pipeline requirements
    • Install their Security Agents
    • Harden the OSs to CIS
    • Validate they passed the companies requirements
    • Share with the team’s AWS accounts
    • Website to show all the assets created
    • Build a change log
    • Supported OS 
      • Amazon Linux 2
      • Windows Server 2019
    • Send alerts to the person that triggered the build

The Process:

We begin by creating a diagram, listing what technologies are needed, and ultimately understanding how everything is going to work together. Given the requirements, we diagram an infrastructure design. 

  • Jenkins server running in an AutoScaling Group with autohealing ability
  • AWS Inspector for compliance checking at the OS level
  • S3 bucket configured as a web page 
  • API Gateway to trigger the build pipeline

the process of agile methodology

Next, we utilize Jenkins and create the actual workflow of the job through a Jenkins pipeline.  Each step will execute a set of scrips to complete the objective.

Workflow Diagram

Now that the design/diagram phase is complete, I like to review it with a peer to ensure that we haven’t missed anything. During this review, I present the solution in totality so my colleagues can challenge and poke holes in my plan. Clarifying questions, such as “why not use AWS CodeBuild and AWS CodePipeline vs. Jenkins?” are invaluable. In this particular  case, it was explained that CodeBuild did not support Microsoft Windows. But these questions allow for strategic analysis of the proposed solution. Once the solution passes a peer review, it is presented to the client for review. 

The MVP Decision

Upon alignment of the overarching requirements, a decision of what the MVP includes is made. In this case, I like to ensure a manual Jenkins job is triggered:

  • Download a git repository that contains a yaml file
  • Parse the YAML file into a packer file
  • Execute Packer
  • Share it with another account

The work breakdown and delegation

Finally, the MVP is segmented into manageable tasks/tickets (under 2 days) in Jira. They include:

  • Write Terraform to create a Jenkins server, ALB, and DNS in Route53 
  • Create Python Script to take a YAML file and convert it to Packer JSON format
  • Create Python Script to dynamically figure out which AWS accounts to share the AMI with
  • Build Jenkinsfile to orchestrate the pipeline

Our FogOps team makes tickets for everything else that is needed to ensure these tasks are addressed. This helps the customer prioritize what is next after the MVP. 

  • Validate that the image passes compliance requirements 
  • Generate a website that shows the AMIs created
  • Generate a website that create a change log
  • Trigger the builds via an API Call

It is important to make sure our engineers are clear on how their piece of the puzzle fits into the overarching project. I find it valuable to schedule some time with all team members to discuss tasks and expectations. 

Once our team completes the MVP, a meeting with the client is set up and a demo is shared.  Feedback is incorporated and the MVP is adjusted accordingly.  When complete, the project continues with the next iteration of tasks.

Takeaways

For our clients, this process ensures that all expectations are managed and feedback is incorporated throughout the entire process, a very agile process. For our clients and our engineers this is effective. We all can visualize the overarching project. This encourages organizational and project symmetry, from client, to leader, to team. And as an added bonus, the work is tested more thoroughly with less chance of human error.

Azure DevOps YAML Pipeline with Terraform

Azure DevOps YAML Pipeline with Terraform

In my last post, I discussed the power of the Azure DevOps YAML pipeline with all of its built in features.  Today, I would like to focus on a specific use case for the Azure DevOps YAML pipeline with Terraform.  By combining these two great technologies, engineers...