Hosting Resource Archival Guideline

Department

Technology

Summary

How to retire a hosting VM whose contract has expired and is not being renewed: snapshot the data, clean up the deploy controller, destroy the VM, remove DNS, then flip the registry record to Archived. Order matters — cleaning the deploy controller after the VM is destroyed is the most common foot-gun and silently leaves orphan state behind.

Table of Contents

Purpose

This guideline standardizes the cleanup of expired hosting resources so that snapshots are kept consistently, billing stops promptly, DNS doesn’t dangle, and the deploy controller doesn’t accumulate orphan entries that a future engineer has to clean by hand.

Scope

Definitions

Pre-flight Checklist

Confirm all of these before touching the provider console.

Basis for takedown

Either of these is a valid basis. Record on the Resource record which one applies, with dates.

Basis 2 exists because it is the common case. A customer who has stopped paying has usually also stopped replying, and waiting for a confirmation that will never arrive keeps dead infrastructure running and billing.

Do not write basis 2 up as though it were basis 1. Record what actually happened: expiry date, when each warning was sent and by whom, that no reply came, and who authorized the takedown. A future reader needs to see that silence was the basis, not agreement.

Step-by-Step Process

Tick each box on the Resource form’s Archival Checklist as you complete the corresponding step.

Step 1. Locate the VM at the Provider

Find the VM by its IP / identifier from the registry. Click into it to confirm:

Step 2. Take a Snapshot

Use the provider’s snapshot/image action. Accept the default snapshot name when the provider auto-generates one — the provider reference runbook records the naming pattern Kiluth uses.

✅ Tick Snapshot Taken on the Resource form. Paste the snapshot name/ID into Snapshot ID.

Step 3. Verify the Snapshot Independently

Visit the provider’s Snapshots / Images list (not just the VM’s own backups tab) and confirm the row exists with a real, non-zero size. Once the VM is destroyed, a missing snapshot means the data is unrecoverable — verify before going further.

Step 4. Clean the Deploy Controller (only if the VM is managed by one)

⚠️ Order matters — do this BEFORE destroying the VM. A deploy controller’s cascade-delete typically connects to the host to shut containers down gracefully. With the host alive that takes seconds; with the host already destroyed it can hang silently and remove nothing, leaving orphan state behind.

Skip this step entirely if the VM was never registered with a deploy controller.

If it was, remove its entries in this order:

Confirm nothing referencing the VM remains before moving on. Check the project-level container specifically — because it is not a child of the server, a server-first cascade leaves it behind, and an orphaned project is easy to miss.

Two ways to work through it. Both are correct. The warning above applies to either: whichever you pick, do it while the VM is still alive.

Bottom-up is Kiluth’s working practice and was used on the 2026-09-09 PROJ-0007 decom. Neither order is safer than the other; bottom-up simply makes each step verifiable.

Concrete steps per tool, and how to recover if the VM was destroyed first, are in the provider reference runbook maintained by Technology (kiluth-infra/runbooks/hosting-archival-provider-reference.md).

Step 5. Destroy the VM

Use the provider’s destroy/terminate action. Two universal rules at the destroy modal:

Wait for the success toast. Confirm the VM no longer appears in the provider’s VM list.

✅ Tick VM Destroyed on the Resource form.

Step 6. Clean DNS

Open the DNS console. Search for records pointing to the VM’s IP in every zone the provider hosts — not only the Kiluth zone. Search by IP address, not only by the customer-app slug, so records that do not follow the naming convention still surface.

Most DNS consoles support bulk-select + delete. Always type-to-confirm if the console offers it.

Why this step lists zones explicitly. On 2026-09-09 a decom found that a resource archived on 2026-04-30, already ticked as DNS-cleaned, still had two live records on the customer’s domain pointing at a host destroyed four months earlier. The April cleanup checked the Kiluth zone and never opened the customer’s zone. Nothing was hijacked, but providers recycle IP addresses, and a dangling record can end up resolving to a stranger’s server.

Tick DNS Records Cleaned only once you have checked every zone, not just the Kiluth one.

✅ Tick DNS Records Cleaned on the Resource form.

Step 7. Flip Registry Status to Archived

On the Resource form in portal.kiluth.com, change Status from Expired → Archived and save.

The doctype’s _auto_set_status early-returns on terminal states (Archived, Deleted), so the value sticks even though before_save runs. The daily scheduler is idempotent on terminal states for the same reason.

If you prefer a one-liner from the browser console on the Resource page:

fetch('/api/method/frappe.client.set_value', {
  method: 'POST',
  headers: {
    'Content-Type': 'application/x-www-form-urlencoded',
    'X-Frappe-CSRF-Token': frappe.csrf_token,
  },
  body: new URLSearchParams({
    doctype: 'Resource',
    name: 'RESOURCE-XXXXX',
    fieldname: 'status',
    value: 'Archived',
  }),
}).then(r => r.json()).then(console.log);

Verification

You should be able to assert all of the following before considering the archival complete.

Rollback (Customer Comes Back)

Snapshots are retained indefinitely at the provider (cost: roughly used-disk-GB × the provider’s snapshot rate per month — small for typical app servers; check the provider reference runbook for current rates). To revive a customer:

Provider-Specific Steps

The process above is provider-agnostic on purpose. The concrete actions for each cloud provider and deploy controller — where the VM list lives, the snapshot action and naming, where to verify snapshots, the destroy flow, snapshot pricing, and recovery paths — are maintained by Technology in the provider reference runbook (kiluth-infra/runbooks/hosting-archival-provider-reference.md), not here.