Devops From Scratch (Manual Process)

Python Dependencies with Pyenv and Poetry

Let's start by installing Pyenv on your system. Then let's pick a recent version of Python for setting up a virtual environment managed by Poetry for our DevOps from Scratch (Manual Process).

#!/bin/bash
set -x #echo on

PYTHON_VERSION=3.11.1

echo "Setting up Python v$PYTHON_VERSION"

echo $PYTHON_VERSION > .python-version
pyenv install $PYTHON_VERSION
pyenv global $PYTHON_VERSION
pyenv local $PYTHON_VERSION
pyenv shell $PYTHON_VERSION

echo "Poetry installation and Virtual Environment creation"

pyenv exec pip install poetry
pyenv exec poetry init
pyenv exec poetry install
pyenv exec poetry shell

echo "Python depedencies installation into to the Virtual Environment"

pyenv exec poetry add black --group dev # Add Dev dependencies

echo "Virtual Environment visual check"

pyenv exec poetry show -v
pyenv exec poetry env info -p

#!/bin/bash
set -x #echo on

pyenv exec poetry run python $@

[tool.poetry]
name = "devops-from-scratch-manual-process"
version = "0.1.0"
description = "DevOps from Scratch (Manual Setup)"
authors = ["user <user+git@email.com>"]
license = "MIT"
readme = "README.md"
#packages = [{include = "devops_from_scratch_manual_process"}]

[tool.poetry.dependencies]
python = "^3.10.6"
#python = "^3.11"
Django = "^4.2"
django-debug-toolbar = "^4.0"
sortedcontainers = "^2.4.0"
httpx = "^0.23.0"
whitenoise = "^6.2.0"
fontawesomefree = "^6.4.0"
typer = { version = "^0.9.0", extras = ["all"] }
markdown = "^3.4.3"

#[tool.poetry.group.dev.dependencies]
#black = "^23.3.0"
#pytest = "7.3.1"

[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

Application Command Line Interface (CLI)

The CLI will help us trigger administrative tasks for our Web Application

import typer

app = typer.Typer()

@app.command()
def hello(name: str):
    print(f"Hello {name}")

@app.command()
def goodbye(name: str, formal: bool = False):
    if formal:
        print(f"Goodbye Mr/Ms {name}. Have a good day.")
    else:
        print(f"Bye {name}!")

Markdown Parser

from typing import List
from pathlib import Path
from markdown import markdown

def parseMarkdownPosts(path: str) -> List[str]:
    p = Path(path)
    mdFilePaths = list(p.glob('**/*.md'))
    postsHtml = [_parseMarkdown(filePath) for filePath in mdFilePaths]
    return postsHtml

def _parseMarkdown(path: str) -> str:
    data = Path(path).read_text()
    html = markdown(data)
    return html

---
title: My First Post
---

# My First Post

Welcome to my blog! This is my first post.

## Some Markdown Features

- * Bold: **bold text**
- * Italic: *italic text*
- * Strikethrough: ~~strikethrough text~~
- * [Link](https://example.com)

---
title: My Second Post
---

# My Second Post

This is my second post, located in a nested directory.

Process Markdown files from the CLI

@app.command()
def markdown(mrkdwndir: str):
    import sys
    from pathlib import Path

    current_dir = Path(__file__).parent.resolve()
    sys.path.insert(1, current_dir)

    from services.postsvc import parseMarkdownPosts

    print(f"Recursively parsing Markdown Files in Directory: {mrkdwndir}")
    postsHtml = parseMarkdownPosts(mrkdwndir)
    for html in postsHtml:
        print(html)

if __name__ == "__main__":
    app()

Django Framework

Let us implement a simple TODO List Application backed by a Database

#!/bin/bash
set -x #echo on

pyenv exec poetry run django-admin startproject $@

./scripts/local/new-django.sh mytodos

More

Let us ignore unwanted files:

# Project
__pycache__
.idea
*.sqlite3
*.pem
*.log
perf-report.html

My TODOs project

# Nada!!!!

ASGI

"""
ASGI config for mytodos project.

It exposes the ASGI callable as a module-level variable named ``application``.

For more information on this file, see
https://docs.djangoproject.com/en/4.2/howto/deployment/asgi/
"""

import os

from django.core.asgi import get_asgi_application

os.environ.setdefault("DJANGO_SETTINGS_MODULE", "mytodos.settings")

application = get_asgi_application()

WSGI

"""
WSGI config for mytodos project.

It exposes the WSGI callable as a module-level variable named ``application``.

For more information on this file, see
https://docs.djangoproject.com/en/4.2/howto/deployment/wsgi/
"""

import os

from django.core.wsgi import get_wsgi_application

os.environ.setdefault("DJANGO_SETTINGS_MODULE", "mytodos.settings")

application = get_wsgi_application()

SETTINGS

"""
Django settings for mytodos project.

Generated by 'django-admin startproject' using Django 4.2.

For more information on this file, see
https://docs.djangoproject.com/en/4.2/topics/settings/

For the full list of settings and their values, see
https://docs.djangoproject.com/en/4.2/ref/settings/
"""

from pathlib import Path

# Build paths inside the project like this: BASE_DIR / 'subdir'.
BASE_DIR = Path(__file__).resolve().parent.parent


# Quick-start development settings - unsuitable for production
# See https://docs.djangoproject.com/en/4.2/howto/deployment/checklist/

# SECURITY WARNING: keep the secret key used in production secret!
SECRET_KEY = "django-insecure-r3#=zp=^#5ehd-*ulh0v&x+b3y$lycz=4-=+=_2nn#cipqi8rl"

# SECURITY WARNING: don't run with debug turned on in production!
DEBUG = True

ALLOWED_HOSTS = []


# Application definition

INSTALLED_APPS = [
    "fontawesomefree", # Free Fonts
    "crud.apps.CrudConfig", # CRUD for TODOs
    "django.contrib.admin",
    "django.contrib.auth",
    "django.contrib.contenttypes",
    "django.contrib.sessions",
    "django.contrib.messages",
    "django.contrib.staticfiles",
]

MIDDLEWARE = [
    "django.middleware.security.SecurityMiddleware",
    "django.contrib.sessions.middleware.SessionMiddleware",
    "django.middleware.common.CommonMiddleware",
    "django.middleware.csrf.CsrfViewMiddleware",
    "django.contrib.auth.middleware.AuthenticationMiddleware",
    "django.contrib.messages.middleware.MessageMiddleware",
    "django.middleware.clickjacking.XFrameOptionsMiddleware",
]

ROOT_URLCONF = "mytodos.urls"

TEMPLATES = [
    {
        "BACKEND": "django.template.backends.django.DjangoTemplates",
        "DIRS": [],
        "APP_DIRS": True,
        "OPTIONS": {
            "context_processors": [
                "django.template.context_processors.debug",
                "django.template.context_processors.request",
                "django.contrib.auth.context_processors.auth",
                "django.contrib.messages.context_processors.messages",
            ],
        },
    },
]

WSGI_APPLICATION = "mytodos.wsgi.application"


# Database
# https://docs.djangoproject.com/en/4.2/ref/settings/#databases

DATABASES = {
    "default": {
        "ENGINE": "django.db.backends.sqlite3",
        "NAME": BASE_DIR / "db.sqlite3",
    }
}


# Password validation
# https://docs.djangoproject.com/en/4.2/ref/settings/#auth-password-validators

AUTH_PASSWORD_VALIDATORS = [
    {
        "NAME": "django.contrib.auth.password_validation.UserAttributeSimilarityValidator",
    },
    {
        "NAME": "django.contrib.auth.password_validation.MinimumLengthValidator",
    },
    {
        "NAME": "django.contrib.auth.password_validation.CommonPasswordValidator",
    },
    {
        "NAME": "django.contrib.auth.password_validation.NumericPasswordValidator",
    },
]


# Internationalization
# https://docs.djangoproject.com/en/4.2/topics/i18n/

LANGUAGE_CODE = "en-us"

TIME_ZONE = "UTC"

USE_I18N = True

USE_TZ = True


# Static files (CSS, JavaScript, Images)
# https://docs.djangoproject.com/en/4.2/howto/static-files/

STATIC_URL = "static/"

# Default primary key field type
# https://docs.djangoproject.com/en/4.2/ref/settings/#default-auto-field

DEFAULT_AUTO_FIELD = "django.db.models.BigAutoField"

: None

URLS

"""
URL configuration for mytodos project.

The `urlpatterns` list routes URLs to views. For more information please see:
    https://docs.djangoproject.com/en/4.2/topics/http/urls/
Examples:
Function views
    1. Add an import:  from my_app import views
    2. Add a URL to urlpatterns:  path('', views.home, name='home')
Class-based views
    1. Add an import:  from other_app.views import Home
    2. Add a URL to urlpatterns:  path('', Home.as_view(), name='home')
Including another URLconf
    1. Import the include() function: from django.urls import include, path
    2. Add a URL to urlpatterns:  path('blog/', include('blog.urls'))
"""
from django.contrib import admin
from django.urls import path, include

urlpatterns = [
    path("", include("crud.urls", namespace="crud")),
    path("admin/", admin.site.urls),
]

Manage Django project from the CLI

#!/usr/bin/env python
"""Django's command-line utility for administrative tasks."""
import os
import sys


def main():
    """Run administrative tasks."""
    os.environ.setdefault("DJANGO_SETTINGS_MODULE", "mytodos.settings")
    try:
        from django.core.management import execute_from_command_line
    except ImportError as exc:
        raise ImportError(
            "Couldn't import Django. Are you sure it's installed and "
            "available on your PYTHONPATH environment variable? Did you "
            "forget to activate a virtual environment?"
        ) from exc
    execute_from_command_line(sys.argv)


if __name__ == "__main__":
    main()

New Django App

#!/bin/bash
set -x #echo on

pyenv exec poetry run `pwd`/manage.py startapp $@

New app called CRUD for CRUD operations to help track our TODOs:

./scripts/local/new-django-app.sh crud

Modeling TODOs

from django.db import models

# Create your models here

class Todo(models.Model):
    name = models.CharField(max_length=100)
    complete = models.BooleanField(default=False)
    #created_at = models.DateTimeField('Created', auto_now_add=True)
    #updated_at = models.DateTimeField('Updated', auto_now=True)

    def __str__(self):
        return self.name

TODOs endpoints

from django.shortcuts import render, get_object_or_404, redirect
from django.views import generic
from django.http import HttpResponseRedirect

# Create your views here.

from .models import Todo

class IndexView(generic.ListView):
    template_name = 'crud/index.html'
    context_object_name = 'todo_list'

    def get_queryset(self):
        """Return all the latest crud."""
        return Todo.objects.order_by('-name')

    #def get_queryset(self):
    #    """Return all the latest crud."""
    #    return Todo.objects.order_by('-created_at')

def add(request):
    name = request.POST['name']
    Todo.objects.create(name=name)

    return redirect('crud:index')

def delete(request, todo_id):
    todo = get_object_or_404(Todo, pk=todo_id)
    todo.delete()

    return redirect('crud:index')

def update(request, todo_id):
    todo = get_object_or_404(Todo, pk=todo_id)
    complete = request.POST.get('complete', False)
    if complete == 'on':
        complete = True

    todo.complete = complete

    todo.save()
    return redirect('crud:index')

TODOs templates

{% load static %}
<!doctype html>
<html lang="en">
  <head>
    <meta charset="utf-8">
    <meta http-equiv="X-UA-Compatible" content="IE=edge">
    <meta name="viewport" content="width=device-width, initial-scale=1">
    <!-- The above 3 meta tags *must* come first in the head; any other head content must come *after* these tags -->
    <title>My TODOs</title>

    <!-- Bootstrap -->
    <link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/bootstrap@3.4.1/dist/css/bootstrap.min.css" integrity="sha384-HSMxcRTRxnN+Bdg0JdbxYKrThecOKuH5zCYotlSAcp1+c8xmyTe9GYg1l9a69psu" crossorigin="anonymous">

    <!-- HTML5 shim and Respond.js for IE8 support of HTML5 elements and media queries -->
    <!-- WARNING: Respond.js doesn't work if you view the page via file:// -->
    <!--[if lt IE 9]>
      <script src="https://cdn.jsdelivr.net/npm/html5shiv@3.7.3/dist/html5shiv.min.js"></script>
      <script src="https://cdn.jsdelivr.net/npm/respond.js@1.4.2/dest/respond.min.js"></script>
    <![endif]-->

    <!-- Our project just needs Font Awesome Free's Solid and Brand files -->
    <link href="{% static 'fontawesomefree/css/fontawesome.css' %}" rel="stylesheet" type="text/css">
    <link href="{% static 'fontawesomefree/css/brands.css' %}" rel="stylesheet" type="text/css">
    <link href="{% static 'fontawesomefree/css/solid.css' %}" rel="stylesheet" type="text/css">

    <!-- HTMX -->
    <script src="https://unpkg.com/htmx.org@1.9.2"></script>
  </head>
  <body>
    {% block content %}
    <h1>Hello, world!</h1>
    {% endblock %}

    <!-- jQuery (necessary for Bootstrap's JavaScript plugins) -->
    <script src="https://code.jquery.com/jquery-1.12.4.min.js" integrity="sha384-nvAa0+6Qg9clwYCGGPpDQLVpLNn0fRaROjHqs13t4Ggj3Ez50XnGQqc/r8MhnRDZ" crossorigin="anonymous"></script>
    <!-- Include all compiled plugins (below), or include individual files as needed -->
    <script src="https://cdn.jsdelivr.net/npm/bootstrap@3.4.1/dist/js/bootstrap.min.js" integrity="sha384-aJ21OjlMXNL5UyIl/XNwTMqvzeRMZH2w8c5cRVpzpU8Y5bApTppSuUkhZXN0VxHd" crossorigin="anonymous"></script>
  </body>
</html>

{% extends 'crud/base.html' %}

{% block content %}
<div class="container">

  <!-- title row -->
  <div class="row">
    <div class="offset-md-2 col-lg-9">
      <div class="page-header">
        <h1>
          Todo List
        </h1>
      </div>
    </div>
  </div>

  <!-- Add a todo row -->
  <div class="row">
    <div class="offset-md-2 col-lg-9">
      <form method="post" action="{% url 'crud:add' %}">
        {% csrf_token %}
        <div class="form-row">
          <div class="col-md-6">
            <input type="text" class="form-control" name="name" placeholder="Do laundry" required>
          </div>
          <div class="col-md-6">
            <button type="submit" name="submit" class="btn btn-outline-primary">
              Add
            </button>
          </div>
        </div>
      </form>
    </div>
  </div>
  <hr />

  <!-- todo list row -->
  <div class="row">
    <div class="offset-md-2 col-lg-6">
      <div class="list-group">
        {% for todo in todo_list %}
        <div class="list-group-item {% if todo.complete %} todo-complete {% endif %}">
          <form style="display: inline;" method="post" action="{% url 'crud:update' todo.id %}">
            {% csrf_token %}
            <input type="checkbox" name="complete" onchange="this.form.submit()" {% if todo.complete %} checked
              {% endif %} class="todo-status-checkbox"
              title="{% if not todo.complete %} mark as done {% else %} mark undone {% endif %}">
          </form>
          {{ todo.name }}
          <a href="{% url 'crud:delete' todo.id %}" title="Delete">
            <i class="far fa-trash-alt"></i>
          </a>
        </div>
        {% endfor %}
      </div>
    </div>
  </div>
</div>

{% endblock %}

URLs Configuation

CRUD Urls

from django.urls import path
from . import views

app_name='todos'
urlpatterns = [
    path('', views.IndexView.as_view(), name='index'),
    path('<int:todo_id>/delete', views.delete, name='delete'),
    path('<int:todo_id>/update', views.update, name='update'),
    path('add/', views.add, name='add')
]

Migrate Database

./scripts/local/pyrun.sh ./manage.py migrate

Make migrations for MyTODOs CRUD

./scripts/local/pyrun.sh ./manage.py makemigrations crud

Create Table for MyTODOs CRUD

./scripts/local/pyrun.sh ./manage.py migrate

Lightsail Deployment

Let's provision a budget Virtual Machine on AWS:

• Navigate to AWS Lightsail to create a VM
• Let us pick a VM running a recent version of Ubuntu -- Jammy Jellyfish (22.04 LTS)
• Connect to the VM using SSH and PEM key
• Setup and Configure the VM the Very Naive and Dangerous Way
• Transfer application files to VM
• Ensure project keeps running (nohup vs supervisor)

Download the PEM for secured SSH and SCP connections to the VM

chmod 400 mytodos-on-ubuntu-key.pem

#!/bin/bash
set -x #echo on

ssh -i `pwd`/$1.pem $2@$3

#!/bin/bash
set -x #echo on

# Who am I?
whoami

# Check Python
which python3 # python3 comes pre-installed on recent versions of Ubuntu
which pip3 # Check if python3-pip is installed
which poetry # Check if python3-poetry is installed

# Update Ubuntu packages registry lists
sudo apt update

# Install Poetry
sudo apt install python3-pip python3-poetry

#!/bin/bash
set -x #echo on

scp -i `pwd`/$1.pem -r $2 $3@$4:$5 # -r is for recursively copying files

[inet_http_server]
port = 0.0.0.0:9001
username = user
password = 1234

[rpcinterface:supervisor]
supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface

[supervisord]
serverurl=http://0.0.0.0:9001

[program:mytodos]
directory = /home/ubuntu/mytodos_app
command = sudo poetry run python manage.py runserver 0.0.0.0:80
autostart = true
autorestart = true
#stderr_logfile = /var/log/mytodos/app.err.log
#stdout_logfile = /var/log/mytodos/app.out.log

PERF Testing

import http from 'k6/http';
import { sleep, check } from 'k6';
import { htmlReport } from 'https://raw.githubusercontent.com/benc-uk/k6-reporter/main/dist/bundle.js';
import { textSummary } from 'https://jslib.k6.io/k6-summary/0.0.1/index.js';
export let options = {
  // vus: 100,
  // duration: '30s',
  // iterations: 10,
  // minIterationDuration: '10s',
  stages: [
    { duration: '25s', target: 32 }, // 4 PODS
    { duration: '1m', target: 80 }, // 4 PODS
    // { duration: '300s', target: 800 },
    // { duration: '600s', target: 1200 },
    // { duration: '14m40s', target: 4000 },
    // { duration: '4m40s', target: 1600 },
    // { duration: '8m40s', target: 8000 },
    // { duration: '14m40s', target: 18000 }, Perf Box dies that High
    { duration: '15s', target: 0 },
  ],
  // batch: 8000,
  // batch: 4000,
  batch: 100,
  // batch: 1200,
  // batchPerHost: 18000000,
  batchPerHost: 1440000,
  // batchPerHost: 640000000,
  noConnectionReuse: false,
  // noConnectionReuse: true,
  discardResponseBodies: true,
};

let requestHeaders = {
  'User-Agent': 'k6',
  'Accept': 'application/json',
  // 'Content-Type': 'application/json',
  // 'Authorization': 'Bearer {{ .Jwt }}',
  'X-Consumer-Username': 'k6-Benchmarker',
};

export default function () {
  const res = http.get(
      'http://3.96.193.218',
      {headers: requestHeaders}
  );
  // sleep(1);
  const checkRes = check(res, {
    'status is 200': (r) => r.status === 200,
    // 'response body': (r) => r.body.indexOf('Feel free to browse') !== -1,
  });
}

export function handleSummary(data) {
  return {
    './perf-report.html': htmlReport(data),
    stdout: textSummary(data, { indent: ' ', enableColors: true }),
  };
}

Webapp:

Build and Test:

VM Provisioning:

DB Provisioning:

Deployment:

Load Balancer:

Custom Domain:

Crash Resilience:

Logging and Monitoring:

Updates from New Requirements:

DB Migrations in Production:

Security/Fuzzing:

Backup/Restore Procedures:

Notes du Presenteur :

Notre orateur introduira l'application web que nous construirons et déploierons, expliquant son objectif et sa pertinence pour le workflow DevOps.

Notes du Presenteur :

Nous allons passer en revue le processus de construction et de test de l'application, en soulignant l'importance des tests et comment cela s'intègre dans le workflow DevOps.

Notes du Presenteur :

Notre orateur discutera du processus de provisionnement d'une machine virtuelle pour héberger l'application, expliquant les différentes options disponibles et comment choisir la meilleure pour vos besoins.

Notes du Presenteur :

Nous discuterons du processus de provisionnement d'une base de données pour l'application, et explorerons les différentes options disponibles.

Notes du Presenteur :

Notre orateur nous guidera à travers le processus de déploiement de l'application sur la machine virtuelle, et expliquera l'importance de l'automatisation dans le déploiement.

Notes du Presenteur :

Nous discuterons de l'importance de l'équilibrage de charge pour les applications web, et nous passerons en revue le processus de configuration d'un équilibrage de charge pour l'application.

Notes du Presenteur :

Notre orateur nous guidera à travers le processus de configuration d'un domaine personnalisé pour l'application, expliquant l'importance de la marque et de l'expérience utilisateur.

Notes du Presenteur :

Nous discuterons de l'importance de la résilience en cas de crash pour les applications web, et explorerons comment l'implémenter en utilisant les pratiques DevOps.

Notes du Presenteur :

Notre orateur discutera de l'importance des logs et du monitoring pour les applications web, et nous guidera à travers le processus de configuration des logs et du monitoring pour l'application.

Notes du Presenteur :

Nous discuterons du processus de mise à jour de l'application en fonction des nouveaux besoins, et expliquerons comment les pratiques DevOps peuvent faciliter ce processus.

Notes du Presenteur :

Notre orateur discutera du processus de migration de la base de données en production, et explorera comment implémenter les migrations de base de données en utilisant les pratiques DevOps.

Notes du Presenteur :

Notre orateur discutera de l'importance des tests de sécurité et du fuzzing pour les applications web, et nous guidera à travers le processus de mise en place de tests de sécurité et de fuzzing en utilisant les pratiques DevOps.

Notes du Presenteur :

Nous discuterons de l'importance des procédures de sauvegarde et de restauration pour les applications web, et nous guiderons à travers le processus de configuration et de test des procédures de sauvegarde et de restauration en utilisant les pratiques DevOps.