Metadata-Version: 1.1
Name: vigie
Version: 0.0.3
Summary: VM Migration Scheduler for OpenStack
Home-page: https://www.openstack.org/
Author: OpenStack
Author-email: openstack-discuss@lists.openstack.org
License: UNKNOWN
Description: # Table of Contents
        
        - [What is Vigie](#what-is-vigie)
          * [General description](#general-description)
          * [Features](#features)
        - [Installation](#installation)
          * [Package](#package)
          * [Configuration](#configuration)
        - [Usage](#usage)
          * [Startup](#startup)
          * [Vigie client](#vigie-client)
          * [API Endpoints](#api-endpoints)
        - [Internal code structure](#internal-code-structure)
          * [Inheritance from Cinder](#inheritance-from-cinder)
          * [Writing a new API function](#writing-a-new-api-function)
          * [Writing a new worker task](#writing-a-new-worker-task)
          * [Using vigietools](#using-vigietools)
        
        # What is Vigie
        
        ## General description
        
        Vigie is an OpenStack operator API aimed at easing the administration of
        an existing deployment. It is typically used by operators to do daily
        tasks such as purging all resources of a project, or evacuating a
        compute or a network node for maintenance, and more.
        
        WARNING: this project is in its early stages.
        
        <img src="doc/source/_static/vigie.png" alt="Vigie" width="250" />
        
        ## Features
        
        ### Currently implemented
        
        * Validating a compute node by spawning a VM on it and connecting to it
          over SSH
        
        ### Planned features
        
        * Public image management (downloads from distro repos)
        * Project pre-delete
        * Project purge of resources
        * Emptying / evacuation of a compute node (live or cold migration,
          depending on whether the VMs support it)
        * Emptying / evacuation of a network node (gracefully, with conntrack)
        
        # Installation
        
        ## Package
        
        Use the Debian package or invent a deployment method yourself.
        
        ## Configuration
        
        The service requires a configuration file typically located at
        ``/etc/vigie/vigie.conf``.
        
        Example configuration:
        
        ```
            [DEFAULT]
            transport_url=rabbit://vigie:PASSWORD@cl1-controller-3.example.com:5671
            [database]
            connection=sqlite:///var/lib/vigie/vigie.db
        
            [keystone_authtoken]
            www_authenticate_uri = http://keystone:5000
            auth_url = http://keystone:5000
            auth_type = password
            project_domain_name = Default
            user_domain_name = Default
            project_name = service
            username = vigie
            password = vigie_password
        
            [validate_compute]
            glance_image=debian-13-generic-amd64-daily.qcow2
            flavor=cpu1-ram2-disk5
            network=ext-net1
            ssh_username=debian
            keep_enabled=True
            auto_clean_up=True
            validation_timeout=600
            ssh_use_jump_host=False
            ssh_jump_host_hostname=jump.example.com
            ssh_jump_host_key_file=/etc/vigie/.ssh/id_rsa
        
            [worker]
            coordination_url=zookeeper://192.168.0.2:2181,192.168.0.66:2181,192.168.0.130:2181/
        
            [tasks]
            task_auto_clean=True
            task_auto_clean_after=1440
        ```
        
        At startup, the worker daemon joins the coordination backend configured
        through ``[worker]/coordination_url`` and checks that it can actually
        acquire and release a tooz lock. When the option is not set, or when
        the backend does not provide working locks, the worker daemon refuses
        to start: cluster-wide locks are required for its features.
        
        Before deciding whether a validation or a clean-up may run, a worker
        takes a cluster-wide tooz lock, reads the tasks table and switches the
        task to its new state (RUNNING or WAITING) while still holding it, and
        only then releases the lock: the decision is atomic across the
        workers, and two of them can never start a validation at the same
        time by both reading that no validation is running. If the lock
        cannot be acquired, nothing is written to the task. When the
        coordination backend cannot be reached at all, the decision is not
        taken either: the task is left untouched, so that an infrastructure
        problem never alters its state. The lock is released as soon as the
        decision is taken: the validation itself runs without it.
        
        When ``[tasks]/task_auto_clean`` is enabled (the default), the
        worker's periodic tasks purge every task which has been in the DONE
        state for longer than ``[tasks]/task_auto_clean_after`` minutes
        (default: 1440, that is one day). The purge runs under a cluster-wide
        tooz lock, so that only one worker purges at a time. Tasks which are
        not DONE are never purged, so that a failed validation can still be
        inspected. They are instead deleted by hand, with the tasks-clean
        endpoint.
        
        When ``[validate_compute]/auto_clean_up`` is enabled (the default), the
        worker automatically schedules a clean-up task once all the compute
        validations are DONE: a validation which is still scheduled, waiting or
        running holds it back, and so does a validation in ERROR, so that its
        leftovers can be inspected first. The automatic clean-up is dispatched
        to the worker which just finished the last validation, which is by
        construction the only one free of validations at that point.
        
        # Usage
        
        ## Startup
        
        Start the API service using uwsgi.
        
        Start the worker daemon:
        
        ```
        vigie-worker
        ```
        
        The Debian package of course comes with systemd .service unit files,
        so it is possible to do:
        
        ```
        systemctl restart vigie-api
        systemctl restart vigie-worker
        ```
        
        At least one worker is needed, though it is best, for redundancy, to
        provision three of them. Same for the API.
        
        ## Vigie client
        
        Then you may use the vigieclient, which is able to validate one or
        many compute nodes with a single request:
        
        ```
        openstack vigie validate compute node1.example.com node2.example.com
        ```
        
        ## API Endpoints
        
        * ``POST /v2/validate-compute`` - Validate one or more compute nodes
          by spawning a VM on each of them. The request body is a JSON object
          with a non-empty ``hostnames`` list, for example:
          ``{"hostnames": ["node1.example.com", "node2.example.com"]}``. The
          optional boolean ``keep_enabled`` member overrides, for this call,
          the ``[validate_compute]/keep_enabled`` configured default. Note
          that even with ``keep_enabled=true``, the host will be disabled
          again if the validation fails. Every hostname is checked before
          anything is scheduled: should any of them be invalid, unknown to
          Nova, or have its nova-compute service down, the whole request is
          rejected with a 400, 404 or 409 status, and no task is created at
          all. Otherwise, one task per hostname is created, all dispatched to
          a single worker, and the 202 response carries the created task IDs.
        * ``POST /v2/validate-cleanup`` - Clean up all the resources left behind
          by previous validations (the hosts-to-validate aggregate, the
          validation trait, and the whole vgt project with everything in it).
          The clean-up is preferably dispatched to an alive worker without any
          remaining validation task, and falls back to any alive worker
          otherwise.
        * ``POST /v2/tasks-clean`` - Delete all the tasks which are DONE.
        * ``POST /v2/evacuate-compute`` - Evacuate a compute node.
        * ``POST /v2/purge-project`` - Purge a project.
        
        # Internal code structure
        
        ## Inheritance from Cinder
        
        Having read a lot of OpenStack code, I have found that the Manager
        class from Cinder is really nice, so I used it as an example for
        writing vigie/manager.py. The worker in vigie/worker/manager.py is a
        subclass of that one.
        
        ## Writing a new API function
        
        The Vigie API uses Flask, as it is simple, understandable, and the
        most commonly used framework.
        
        To write a new API route, simply cheat on one of the existing ones.
        For example:
        
        ```
        @app.route('/v2/validate-compute', methods=['POST'])
        @policy_checker('vigie:validate_compute')
        def validate_compute():
        ```
        
        Simply using the @policy_checker is enough provided that such a
        policy is written in vigie/policy/rules.py. The @app.route is
        handled by Flask.
        
        ## Writing a new worker task
        
        Each worker task lives in its own module of the vigie/worker/tasks/
        package, as a plain mixin class holding the methods of one category
        of tasks. The only registration needed is inheritance: add the new
        class to the inheritance list of the WorkerManager in
        vigie/worker/manager.py, and its public methods become reachable
        through RPC (an incoming RPC message is dispatched to the public
        method of the manager matching the method name in the message;
        private methods, with a leading underscore, are never reachable).
        
        The signature of an RPC method is: the context (``ctxt``) as the
        first positional argument, followed by the keyword arguments sent by
        the caller. For example, in vigie/worker/tasks/evacuate_compute.py:
        
        ```
        def evacuate_compute(self, ctxt, source_hostname, target_hostname=None):
        ```
        
        the caller casts it with:
        
        ```
        client.cast({}, 'evacuate_compute', source_hostname=...,
                    target_hostname=...)
        ```
        
        so the method name and the keyword arguments must match on both
        sides: that is the whole RPC contract.
        
        When the task must never run concurrently with another task of the
        same group on the whole cluster, decorate it with
        ``@vigie_worker_mutually_exclusive(worker_group_name='my-group')``
        and add a matching ``WorkerGroup`` entry in
        vigie/worker/tasks/base.py. The decorator then changes the RPC
        signature: the caller only sends a ``task_id`` plus the task
        arguments, and the decorator fetches the task, opens the database
        session, and waits for a free execution slot before calling the
        method with the task and the session injected:
        
        ```
        @vigie_worker_mutually_exclusive(worker_group_name='validate-compute')
        def validate_compute(self, ctxt, task, session, compute_hostname,
                             project_id, project_name, keep_enabled=None,
                             req_id=None):
        ```
        
        Over RPC, this is called with ``ctxt``, ``task_id`` and the remaining
        keyword arguments; ``task`` and ``session`` are provided by the
        decorator.
        
        A mixin may also hold periodic tasks, simply decorated with the
        standard ``@periodic_task.periodic_task`` decorator from
        oslo_service: they are collected from the mixins by
        ``vigie.manager.register_mixin_periodic_tasks()``, which is already
        called at the bottom of vigie/worker/manager.py, so they need no
        extra wiring either. As always, cheat on an existing task (for
        example vigie/worker/tasks/validate_compute.py) and its unit tests
        when writing a new one.
        
        ## Using vigietools
        
        Vigie is mostly a scheduler for vigietools. For example, the
        validate-compute task calls a vigietools function to perform the
        task. The advantage is that vigietools itself also contains a command
        line utility (i.e., /usr/bin/vgt) that can be used standalone,
        without the Vigie task manager.
        
        Vigie, with its task manager, provides orchestration and
        asynchronous tasks, which would not be possible with vigietools
        alone.
        
        
Platform: UNKNOWN
Classifier: Environment :: OpenStack
Classifier: Intended Audience :: Information Technology
Classifier: Intended Audience :: System Administrators
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Operating System :: POSIX :: Linux
Classifier: Programming Language :: Python
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.8
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
