Container and cloud infrastructure

On a developer laptop, a process shares a network with the local database and test data is small, so timeout, DNS, certificate, and resource-limit problems may not be apparent. In production, multiple instances, load balancers, secret stores, and external services are connected, and partial failures are routine. “It works on my computer” is the starting point of a functional hypothesis, not evidence that it is ready to deploy.

To reduce environment differences, version not only source code but also dependency locks, runtime versions, configuration schemas, and build procedures together. Inject secrets and environment-specific endpoints at deployment rather than embedding them in the image. If they are missing, fail at startup instead of silently defaulting to production. Promoting the same artifact through staging and production, with differences confined to explicit configuration, supports reproduction and rollback.

A container image is not a single compressed application folder but a bundle of read-only layers linked by content digest, plus runtime configuration. The order of base image, package installation, and source copying determines cache reuse and the scope of vulnerabilities. A container is a running instance that adds a writable layer, namespaces, and resource limits on top of this image.

A tag is a human-readable alias, so the same name can point to different digests. Link production releases, SBOMs, signatures, and scan results to the immutable digest, not the tag. Even if the build stage needs a compiler and credentials, copy only the executable binary and a minimal runtime into the final image, and confirm that build secrets do not remain in the layer history; only then can you explain the supply chain boundary.

When a web application needs an API and PostgreSQL, Compose defines each service's image, network, volume, and environment variables in one file so developers can repeatedly start the same topology. Service names can be used through DNS, but `depends_on` ordering alone does not wait until the database can accept queries, so health conditions and application retries are needed.

Attaching a volume does not mean the data is protected either. Schema migration, backup, and restore are separate procedures, and you must know clearly whether a named volume remains when a container is removed. Compose is useful for development and small-scale operation on a single host, but if you need scheduling across multiple nodes, automatic recovery, and rolling updates, evaluate the broader state management contract that an orchestrator provides.

In Kubernetes, users store desired state, such as a Deployment's replica count and image, in the API instead of directly keeping a particular container process alive. The controller repeatedly observes the difference between current Pods and desired state, then creates or replaces Pods. A new Pod appearing elsewhere after a Node disappears is the result of this reconciliation loop, not teleportation of the old process.

K3s simplifies packaging and default components, helping installation and reducing footprint, but it does not remove Kubernetes' core APIs or operational responsibilities. Resource requests, persistent data, network policy, upgrades, and backups still need design. Distinguish a Pod restarting in a single-node lab from high availability that keeps the service running when the node itself disappears.

A liveness probe detects a stuck process that needs restarting, while a readiness probe determines whether it can accept new requests. A startup probe can protect slow initialization. Restarting all Pods through liveness merely because a database is temporarily slow can amplify an incident, so define each probe according to the action it triggers.

A rolling update gradually increases new replicas and reduces old replicas, but it cannot automatically determine whether the new version's responses are correct for the business. In a canary, check migration compatibility and representative user outcomes as well as readiness, error rate, and latency. Verify in advance whether rollback can be completed by reverting the image and whether the old version can read schemas and messages already changed by the new version.

Assume a teaching report job has three stages: read input, write a temporary file and finalize its name. If the container stops while writing the temporary file, the download list must not expose it. A restarted worker reads the ledger and retries only unfinalized work. Link temporary filenames to job identifiers so a restart can identify which leftovers to reclaim.

A counterexample sends success before finalizing the file. A user receiving that response immediately before shutdown gets a link to a missing file. Recording response time and storage finalization separately exposes the defect even if every probe was healthy. In this case, the directly observable mismatch is whether reported completion preceded finalization.

Inject termination while queued, during temporary storage and after finalization. Compare content completeness and ledger state, not just final file count. Even with duplicate delivery, one final result and a consistent completion state must remain. Preserve a normal path without forced termination to check whether recovery logic duplicates ordinary work.

Use Kubernetes deployment state to observe instance replacement and the work ledger to observe user outcomes. Correlate timestamps and job identifiers across both. Matching the target instance count does not prove that unfinished reports recovered. If unfinished jobs remain after replacement, record them as recovery work separately from deployment success.

A teaching booking system splits one contact string into a type and value. Old code reads only the original column, so deleting it immediately after adding new columns breaks rollback. Compare old and new reads during a transition period and handle conversion failures separately. Do not guess that ambiguous contacts are email; preserve their review-required state.

New bookings created during conversion can fall outside the initial batch. Record processed boundaries alongside the new write path. Claiming complete conversion requires a fresh query for unconverted rows, not just the count at the start. Including bookings created during conversion reveals gaps in writes outside the batch.

Testing only the new version and naming the deployment rollback button as the recovery plan is insufficient. If data is stored only in the new representation, the old image cannot interpret it. Include an actual compatibility check where old code reads newly written rows. Verify not just that the rollback candidate starts, but that it can update and reread new rows.

Build a matrix of old reads, new reads, old writes and new writes for each coexistence stage. Query pseudonymous bookings before and after conversion using the same business questions. Check missing results and duplicate bookings, not merely similar-looking values, before removing the old column. Classify each combination’s failure as a syntax problem or loss of meaning to identify the repair layer.

Assume a teaching archive stores files and metadata rows separately. Restoring only the database can show a list whose downloads fail. Restoring only files loses ownership and visibility information needed for authorization. Therefore verify links for the same document identifier rather than declaring both stores independently successful.

In the drill, join metadata and files from the same recovery point and test reads as pseudonymous users. Select public and private documents and check content hashes and access denials. Row counts alone miss incorrect file links or lost authorization. The denial case separately detects a restoration defect that exposes private documents despite a normal listing.

Seeing the restored server start and display a login page is not completed recovery. It proves process startup, not document integrity. Open both older documents and records near the latest permitted recovery point to establish the actual recovered range. Missing expected recent documents means the recovery target is unmet even when server startup is fast.

Record the images used, backup identifier, restoration start and end times, and failed reads. If any files failed, report partial recovery rather than hiding the scope. In the next drill, recheck the corrected links and permissions against the same data. Provide record locations and read procedures so the next operator can find and compare the same backup.