Beer Garden Architecture

If you’d like to work on Beer Garden itself, or want to get a better understanding of what’s going on behind the scenes, this is the place for you!

Components

Beer Garden is actually a collection of several different components. In this section we’ll discuss each one individually and how each fit into the larger picture.

Brew View

Brew View is the Beer Garden server. It’s a Tornado server whose main job is to expose the Beer Garden REST interface, but it also does double-duty in that it also serves the frontend javascript application.

Brew View sends instructions to Bartender over a thrift interface. In certain cases it will update Mongo. For GET requests it will query Mongo without involving Bartender. Brew-view will publish event notifications to RabbitMQ without Bartender, but actual Beer Garden requests are passed to Bartender for publication.

Bartender

Bartender is the Beer Garden backend. It’s responsible for several things:

  • Routing requests to plugins

  • RabbitMQ communication

  • Managing plugin status

  • Removing old requests

  • Controlling local plugins

Bartender receives instructions from Brew View over the thrift interface. In certain cases it interacts with Mongo directly. It’s the main RabbitMQ client.

Mongo

Mongo is the Beer Garden database. It has command, instance, request, and systems (and optionally event) collections.

RabbitMQ

RabbitMQ is the Beer Garden message queue. After requests are created and validated they’re passed to RabbitMQ for delivery to a plugin.

RabbitMQ can also be used for event notification delivery.

Frontend

The Beer Garden frontend is an AngularJS application. It’s built with yarn and webpack and served by Brew View.

Lifecycles

Beer Garden has two main lifecycles - the request lifecycle and the plugin lifecycle.

Request

Requests are created by POSTing to the /api/v1/requests/ endpoint. The HTTP request body will be used to construct the Beer Garden request. The body needs to be either application/json (preferred) or application/x-www-form-urlencoded. Both the Beer Garden frontend and the brewtils SystemClient are making a POST to this endpoint under the hood.

Brew View takes the request body and attempts to parse it into a valid Beer Garden request and save it to Mongo. These steps require passing the first-level validation check. This ensures that the request is syntatically valid and meets certain basic requirements (such as the status field being a valid value). If it fails at this it will return a 400 status code, otherwise the request ID is passed to Bartender for processing.

Bartender then pulls the completed request out of the database and validates it. This validation is significatly more involved than the preliminary validation done by Brew View. Here are some of the validation steps:

  • Ensure that a system (which the correct version) exists that can service the request

  • That system has an instance matching the instance the request is addressed to

  • That system has a command matching the request’s command

  • The request’s command parameters meet all the constraints placed on them by the command definition

  • There are no extra command parameters

If all of these conditions are met Bartender will send the request to RabbitMQ using a routing key that ensures the request will be processed by the correct plugin.

RabbitMQ will place the request in the specified plugin’s request queue.

Plugins maintain a consumer connection to RabbitMQ awaiting messages. When a new message is placed in their queue the plugin does several things. First, it attempts to parse the message into a valid Beer Garden request. If that’s successful then it checks that the request is correctly addressed. If either of those checks fail the message is discarded.

If the requests passes those checks then the plugin is able to process the request. The first step is to send an update to Beer Garden setting the status for that request to IN_PROGRESS. The plugin then invokes the actual command method and captures the return value. The plugin then sends an update to Beer Garden with the results and output of the method invokation.

If the 'final' update fails then the completed request is placed back on the RabbitMQ queue. This is to take advantage of RabbitMQ’s message durability - if the plugin goes down at this point the request completion and output will be preserved. The request will be read from the queue and placed into a periodic retry loop. The plugin will reattempt to update the request status up to a maximum of max_attempts times, waiting an increasing amount of time between attempts (up to max_timeout). Requests that fail to update before reaching max_attempts will be discarded. Note that a request in this state does not prevent processing of additional requests.

If the 'final' update succeeds the plugin will send an acknowledgement of the message to RabbitMQ. This lets RabbitMQ know the message was successfully processed, which ends the request lifecycle.

If at any time an attempt to update a request fails because Brew View appears to be down the plugin will enter a wait state. While in this state no new requests will be processed (since status can’t be communicated to Beer Garden). The plugin will periodically attempt to contact Brew View and will resume normal operation once successful.

Plugins

We’ll start by talking about remote plugins and touch on the differences with local plugins at the end.

Remote

Remote plugins are just Python processes that expect to communicate with Beer Garden. When they’re created they need to be provided with all the parameters necessary to connect to a Beer Garden. This can be as simple as a bg_host, but can be more complicated based on the Beer Garden configuration.

When a plugin is started it will immediately try to register itself with Beer Garden. This involves: - The plugin will first check to see if a system with this name and version is already registered with Beer Garden - If a system already exists the plugin will attempt to update certain fields (such as commands and metadata) for that system - The plugin will make sure an instance with its name exists on the system. If it’s unsuccessful due to a max_instance constraint the plugin will error. - The plugin will then send an initialization request to Beer Garden.

When Beer Garden receives an initializion request it: - Verifies that the plugin exists in the database - Creates the message queue for the plugin if it doesn’t already exist - Creates an admin queue for the plugin - Sets the status of the plugin to 'INITIALIZING' - Places a start message on the plugin’s admin queue

Beer Garden then returns a description of the plugin that was just initialized. This includes connection information for the RabbitMQ queues the plugin is expected to listen on. The plugin uses this to create two listeners - one on each RabbitMQ queue.

The plugin then continues listening on its queues until it receives a stop message on its admin queue, the plugin process receives a SIGINT (Ctrl-c), or it encounters a fatal exception.

Local

Local plugins use the same underlying implementation as remote plugins. The difference is that local plugins are packaged with some additional metadata that allows Bartender to manage the plugin process for you.

When Bartender starts it will attempt to start all the plugins in its configured plugins directory. Since Bartender is the one starting the process you don’t need to worry about providing Beer Garden connection information - Bartender already knows how to talk to Brew View and will pass that information to the plugin by setting the correct environment variables. Bartender will read a special file named beer.conf and use it to pass additional parameter to the plugin as well.

The actual implementation of starting, initialization, running, and stopping is exactly the same for local plugins as it is for remote plugins. The difference is how the Python process is created. With remote plugins starting the plugin process is the plugin developer’s responsiblity, but with local plugins Bartender assumes that responsibility.

Since Bartender knows how to start the plugin process it’s possible to use the start feature on the administration page. With remote plugins, once the plugin is stopped Beer Garden has no way to start it again. Bartender will also monitor the plugin process and will attempt to restart the plugin if it dies unexpectedly.

Event Notifications

Event Notifications

Sometimes, particularly if you’re integrating with Beer Garden, it can be helpful to know when certain things happen. To make this process as easy as possible Beer Garden supports the concept of event notifications.

Events are generated whenever any of the following events occur:

  • Bartender started

  • Bartender stopped

  • Brew view started

  • Brew view stopped

  • Request created

  • Request started

  • Request completed

  • Instance initialized

  • Instance started

  • Instance stopped

  • System created

  • System updated

  • System removed

  • Queue cleared

  • All queues cleared

Event Structure

Events will always be well-formed JSON. Here’s an example of an event:

event.json
{
    "name": "REQUEST_CREATED", (1)
    "timestamp": 1521126132897, (2)
    "error": None, (3)
    "metadata": { (4)
        "entity_url": "https://this.is.beergarden:443/api/v1/requests/5aaa8af45991735bf1a6c123",
        "public_url": "https://this.is.beergarden:443/"
    },
    "payload": { (5)
        "id": "5aaa8af45991735bf1a6c123",
        "command": "say",
        "system": "echo",
        "system_version": "1.0.0",
        "instance_name": "default"
    }
}
1 All events will have a name
2 All events will have a timestamp (milliseconds since the epoch)
3 All events will have an error flag (boolean, None being equivalent to False)
4 Events may have a metadata field. This will contain 'extra' useful information, but will normally include at least the the public url of the Beer Garden that generated the Event. Events that relate to a specific entity will also include a url that can be used to retrieve the full entity definition.
5 Events may have a payload. The specific data included will vary based on event type.

Enabling Events

Events are disabled by default, but they can be published to a RabbitMQ topic exchange and/or persisted to MongoDB.

Mongo

To perist events to Mongo set the event_persist_mongo configuration option in Brew view to True:

config.json
{
    "event_persist_mongo": True
}
RabbitMQ

To publish events to RabbitMQ set the event_amq_virtual_host and event_amq_exchange configuration options to valid values.

config.json
{
    "event_amq_virtual_host": "/",
    "event_amq_exchange": "beergarden_event"
}
Beer Garden doesn’t create an exchange for events. While you could use the normal Beer Garden request exchange we recommend you don’t. Instead, you should create a separate exchange. Check the RabbitMQ docs for instructions on how to do this.

To consume from RabbitMQ, you’ll need to create a queue on the same virtual host as event_amq_virtual_host and then bind that queue to a routing key on the event_amq_exchange. To receive all events you can bind to the # routing key. If you want to filter which events your queue will receive there are some rules about how Beer Garden assigns routing keys to events:

  • The default routing key is 'beergarden'. Events that don’t fit any other rules will use this.

  • Request events will have a routing key of the form request.<system_name>.<mangled_system_version>.<instance_name>. Since RabbitMQ treats '.' as a delimiter the system version is mangled to replace all instances of '.' with '-'. So an example would be request.echo.1-0-0.default. If you were only interested in requests related to the 'echo' system you could bind your queue to the request.echo.# routing key.

Publishing Custom Events

This is a beta capability

It’s possible to publish your own events. Just POST a valid Event to the /api/vbeta/events endpoint. The brewtils EasyClient publish_event method can help with this.