Setting Up RBE

This section will describe steps needed to run a Remote Build Execution backend and make Goma server use it.

Choosing the RBE Backend

There are few free RBE backends available. You can find some of them being listed in Bazel Docs. All of them should work with Goma however almost in all cases some fixes/patches will be needed in order to align them with Goma.

For purposes of this guide we’ll use BuildGrid. Personally I think it is the easiest RBE backend to setup, it is written in Python (for now this is an advantage) and more or less implements full RBE spec so it can serve as a good reference implementation.

Setting up BuildGrid

Run:

git clone https://gitlab.com/BuildGrid/buildgrid.git
cd buildgrid
git checkout 0.0.2
python3 -m venv env
env/bin/python -m pip install --upgrade setuptools pip wheel
env/bin/python -m pip install --editable .

You can check if your server works by running following command:

env/bin/bgd server start data/config/default.conf -vvv

It should start the BuildGrid server in a default setup. If everything is ok we can start preparing BuildGrid to serve requests from Goma server.

Configuring the BuildGrid for Goma

BuildGrid comes with a convenient default configuration. It’ll only need a few tweaks to make it suitable for the purposes of this guide.

BuildGrid configurations are present in the data/config directory. Go there and make a copy of default.conf. You can name it as you like, for example goma.conf. Now, open the goma.conf and patch it as follows:

@@ -8,7 +8,6 @@ description: >
     - Unauthenticated plain HTTP at :50051
     - Single instance: [unnamed]
     - In-memory data, max. 2Gio
-    - DataStore: sqlite:///./example.db
     - Hosted services:
        - ActionCache
        - Execute
@@ -22,33 +21,24 @@ monitoring:
   enabled: false

 instances:
-  - name: ''
+  - name: goma-test
     description: |
-      The unique '' instance.
+      The unique goma-test instance.

     storages:
       - !lru-storage &cas-storage
         size: 2048M


-    schedulers:
-      - !sql-scheduler &state-database
-        storage: *cas-storage
-        connection_string: sqlite:///./example.db
-        automigrate: yes
-        connection_timeout: 15
-        poll_interval: 0.5
-
     services:
       - !action-cache &build-cache
         storage: *cas-storage
         max-cached-refs: 256
-        cache-failed-actions: true
+        cache-failed-actions: false
         allow-updates: true

       - !execution
         storage: *cas-storage
         action-cache: *build-cache
-        scheduler: *state-database
         max-execution-timeout: 7200

       - !cas

We won’t go deep into the BuildGrid configuration but the above patch:

  • sets name for RBE instance - we’ll use that same name in goma server
  • removes persistent scheduler - BuildGrid can save scheduled tasks’ state into a persistent storage making the task states survive BuildGrid restart. This is not needed for our setup so we’ve removed it making BuildGrid use default non-persistent in-memory scheduler.
  • disables caching of failed actions

You can now run BuildGrid server with:

env/bin/bgd server start data/config/goma.conf -vvv

and move on to the worker setup.

Configuring BuildGrid Worker

BuildGrid provides several types of workers (also called bots) but we’ll use a host tools worker that simply runs the received command in a temporary directory created on a system actually hosting the worker. While this is not the safest method it is the most suitable for us now.

Run the worker with following command:

env/bin/bgd bot --parent=goma-test --remote=http://localhost:50051 -w OSFamily Linux -vvv host-tools

and worker should output something like this:

[DEBUG][mainthread]: Creating bot session
[INFO][mainthread]: Created bot session with name: [goma-test/4d8ba0b0-7e99-40a2-98cd-39b4d359dea0][debug][MainThread]: Updating bot session: [goma-test.kubal-Virtual-Machine.26402][debug][MainThread]: Updating bot session: [goma-test.kubal-Virtual-Machine.26402]

You just started a worker (bot) asking it to connect to the server pointed by the --remote parameter and use the RBE instance as set by the --parent parameter. The -w parameter sets the worker properties. Worker properties are simple key/value pairs that describe worker capabilities. Goma server will file a request with certain properties set and BuildGrid will schedule such a job to one of its workers only if it finds one with the same exact set of properties set. Therefore, we must configure the worker to meet the Goma server expectations.