Beer Garden Architecture
If you’d like to work on Beer Garden itself, or want to get a better understanding of what’s going on behind the scenes, this is the place for you!
Components
Beer Garden is actually a collection of several different components. In this section we’ll discuss each one individually and how each fit into the larger picture.
UI
The Beer Garden user interface frontend is an AngularJS application. It’s built with webpack and can be served by any static resource serving platform. We recommend using nginx in a production setting and webpack-dev-server for development.
Application
The Beer Garden application is responsible for several things:
-
Routing requests to plugins
-
RabbitMQ communication
-
Managing plugin status
-
Removing old requests
-
Controlling local plugins
The application receives instructions through Entry Points (more on that later). You can think of the application as the brains of Beer Garden.
Lifecycles
Beer Garden has two main lifecycles - the request lifecycle and the plugin lifecycle.
Request
Requests are created by POSTing to the /api/v1/requests/ endpoint. The HTTP request body will be used to construct the Beer Garden request. The body needs to be either application/json (preferred) or application/x-www-form-urlencoded. Both the Beer Garden frontend and the brewtils SystemClient are making a POST to this endpoint under the hood.
Beer Garden takes the request body and attempts to transform it into the Operation object to determine routing.
After the object is evaluated, if it is determined to be forwarded to a child Beer Garden the forwarding logic is invoked,
else it is passed for local processing.
Beer Garden takes the request body and attempts to parse it into a valid Beer Garden request and save it to Mongo. These steps require passing the first-level validation check. This ensures that the request is syntactically valid and meets certain basic requirements (such as the status field being a valid value). If it fails at this it will return a 400 status code, otherwise the request will continue processing.
Beer Garden then validates the request (either locally or on child Beer Garden). Here are some of the validation steps:
-
Ensure that a system (which the correct version) exists that can service the request
-
That system has an instance matching the instance the request is addressed to
-
That system has a command matching the request’s command
-
The request’s command parameters meet all the constraints placed on them by the command definition
-
There are no extra command parameters
If all of these conditions are met Beer Garden will send the request to RabbitMQ using a routing key that ensures the request will be processed by the correct plugin.
RabbitMQ will place the request in the specified plugin’s request queue.
Plugins maintain a consumer connection to RabbitMQ awaiting messages. When a new message is placed in their queue the plugin does several things. First, it attempts to parse the message into a valid Beer Garden request. If that’s successful then it checks that the request is correctly addressed. If either of those checks fail the message is discarded.
If the requests passes those checks then the plugin is able to process the request. The first step is to send an update to Beer Garden setting the status for that request to IN_PROGRESS. The plugin then invokes the actual command method and captures the return value. The plugin then sends an update to Beer Garden with the results and output of the method invocation.
If the 'final' update fails then the completed request is placed back on the RabbitMQ queue. This is to take advantage of RabbitMQ’s message durability - if the plugin goes down at this point the request completion and output will be preserved. The request will be read from the queue and placed into a periodic retry loop. The plugin will reattempt to update the request status up to a maximum of max_attempts times, waiting an increasing amount of time between attempts (up to max_timeout). Requests that fail to update before reaching max_attempts will be discarded. Note that a request in this state does not prevent processing of additional requests.
If the 'final' update succeeds the plugin will send an acknowledgement of the message to RabbitMQ. This lets RabbitMQ know the message was successfully processed, which ends the request lifecycle.
If at any time an attempt to update a request fails because Beer Garden appears to be down the plugin will enter a wait state. While in this state no new requests will be processed (since status can’t be communicated to Beer Garden). The plugin will periodically attempt to contact Beer Garden and will resume normal operation once successful.
Plugins
We’ll start by talking about remote plugins and touch on the differences with local plugins at the end.
Remote
Remote plugins are just Python processes that expect to communicate with Beer Garden. When they’re created they need to be provided with all the parameters necessary to connect to a Beer Garden. This can be as simple as a bg_host, but can be more complicated based on the Beer Garden configuration.
When a plugin is started it will immediately try to register itself with Beer Garden. This involves: - The plugin will first check to see if a system with this name and version is already registered with Beer Garden - If a system already exists the plugin will attempt to update certain fields (such as commands and metadata) for that system - The plugin will make sure an instance with its name exists on the system. If it’s unsuccessful due to a max_instance constraint the plugin will error. - The plugin will then send an initialization request to Beer Garden.
When Beer Garden receives an initialization request it: - Verifies that the plugin exists in the database - Creates the message queue for the plugin if it doesn’t already exist - Creates an admin queue for the plugin - Sets the status of the plugin to 'INITIALIZING' - Places a start message on the plugin’s admin queue
Beer Garden then returns a description of the plugin that was just initialized. This includes connection information for the RabbitMQ queues the plugin is expected to listen on. The plugin uses this to create two listeners - one on each RabbitMQ queue.
The plugin then continues listening on its queues until it receives a stop message on its admin queue, the plugin process receives a SIGINT (Ctrl-c), or it encounters a fatal exception.
Local
Local plugins use the same underlying implementation as remote plugins. The difference is that local plugins are packaged with some additional metadata that allows Beer Garden to manage the plugin process for you.
When Beer Garden starts it will attempt to start all the plugins in its configured plugins directory. Since Beer Garden is the one starting the process you don’t need to worry about providing Beer Garden connection information - Beer Garden will pass that information to the plugin by setting the correct environment variables. Beer Garden will read a special file named beer.conf and use it to pass additional parameter to the plugin as well.
The actual implementation of starting, initialization, running, and stopping is exactly the same for local plugins as it is for remote plugins. The difference is how the Python process is created. With remote plugins starting the plugin process is the plugin developer’s responsibility, but with local plugins Beer Garden assumes that responsibility.
Since Beer Garden knows how to start the plugin process it’s possible to use the start feature on the administration page. With remote plugins, once the plugin is stopped Beer Garden has no way to start it again. Beer Garden will also monitor the plugin process and will attempt to restart the plugin if it dies unexpectedly.