Cornice
Home

/

Library

/

ops/server-hold-runbook.md

Server hold / hibernate runbook

Server hold / hibernate runbook

This runbook pauses Cornice without dropping the database schema.

Use it when the project is on hold and you want to reclaim server CPU, memory, and disk while preserving:

  • database schema
  • users
  • user profiles
  • encrypted user secrets
  • opportunity and market catalogs
  • arb monitor profiles and arb opportunity catalog

What hibernate means here

Hibernate does not suspend the machine. It means:

  • stop Cornice services
  • clear high-volume runtime data
  • optionally restart only the API in a dormant mode

The dormant mode is controlled by:

HIBERNATE_MODE=true
INGEST_ENABLED=false

With those set, the API can still come up, but it skips background loops.

Services involved

These are the relevant systemd units:

  • cornice.service — API / FastAPI process
  • cornice-monitor.service — separate monitor process

These are not Celery units. If you have RabbitMQ or Celery on the server for some other project, they are outside this runbook unless you explicitly want to stop them too.

One-command hibernate

Run:

./scripts/hibernate_server.sh

That script:

  1. writes these flags into the active env file:

    HIBERNATE_MODE=true
INGEST_ENABLED=false

  2. stops cornice

  3. stops cornice-monitor if present

  4. disables both services so they do not restart on reboot

  5. truncates runtime tables with scripts/clear_runtime_data.py

By default, the script looks for the env file in this order:

  1. CORNICE_ENV_FILE if set
  2. repo-local .env
  3. /var/www/cornice/.env

If your server uses a different env file path, run:

CORNICE_ENV_FILE=/path/to/.env ./scripts/hibernate_server.sh

By default, it preserves:

  • users
  • user_profiles
  • user_secrets
  • opportunities
  • global_markets
  • markets
  • market_links
  • arb_monitor_profiles
  • arb_opportunity_instances
  • arb_opportunity_legs

It clears runtime-heavy tables such as:

  • market snapshots
  • orderbook snapshots
  • simulator runs/trades
  • execution logs
  • agent runs/findings
  • requests
  • arb observations
  • arb GO signals

State after hibernate

After hibernate:

  • the env file is set to dormant mode
  • cornice is stopped and disabled
  • cornice-monitor is stopped and disabled

So you can leave the server alone without background work restarting.

Resume

When the project is active again:

  1. remove or change:
HIBERNATE_MODE=true
INGEST_ENABLED=false
  1. start services:
./scripts/resume_server.sh

That script starts:

  • cornice
  • cornice-monitor if the unit exists
  • and re-enables both services

Notes on data loss

Hibernate clears runtime data only. The schema stays intact.

You will lose historical runtime rows from the cleared tables. That is intentional for reclaiming space.

You will keep:

  • account access
  • encrypted exchange secrets
  • the opportunity catalog

If you want an even deeper wipe

The helper script supports additional flags:

python3 scripts/clear_runtime_data.py --include-opportunities
python3 scripts/clear_runtime_data.py --include-opportunities --include-auth

Do not use --include-auth unless you want to remove users, profiles, and secrets too.