Welcome to the IIB documentation!¶

Index Image Build (IIB) Service¶

A REST API to manage operator index container images

External Documentation¶

Coding Standards¶

The codebase conforms to the style enforced by flake8 with the following exceptions:

The maximum line length allowed is 100 characters instead of 80 characters

In addition to flake8, docstrings are also enforced by the plugin flake8-docstrings with the following exemptions:

D100: Missing docstring in public module
D104: Missing docstring in public package
D105: Missing docstring in magic method

The format of the docstrings should be in the reStructuredText style such as:

"""
Get the IIB build request from the REST API.

:param int request_id: the ID of the IIB request
:return: the request
:rtype: dict
:raises IIBError: if the HTTP request fails
"""

Additionally, black is used to enforce other coding standards with the following exceptions:

Single quotes are used instead of double quotes

To verify that your code meets these standards, you may run tox -e black,flake8.

Running the Unit Tests¶

The testing environment is managed by tox. Simply run tox and all the linting and unit tests will run.

If you’d like to run a specific unit test, you can do the following:

tox -e py37 tests/test_web/test_api_v1.py::test_add_bundle_invalid_param

Development Environment¶

docker-compose is the supported mechanism for setting up a development environment. This will automatically run the following containers:

iib-api - the IIB REST API. This is accessible at http://localhost:8080.
iib-worker - the IIB Celery worker.
rabbitmq - the RabbitMQ instance for communicating between the API and the worker. The management UI is accessible at http://localhost:8081. The username is iib and the password is iib.
db - the Postgresql database used by the IIB REST API.
registry - the Docker Registry where the worker pushes its build index images to. This is accessible at docker://localhost:8443.
message-broker - the Apache ActiveMQ instance for publishing messages for external consumers. The web console is accessible at http://localhost:8161/admin. The username is admin and the password is admin. The docker-compose environment is configured for IIB to publish AMQP 1.0 messages to the Apache ActiveMQ instance at the destinations topic://VirtualTopic.eng.iib.batch.state and topic://VirtualTopic.eng.iib.build.state.

The Flask application will automatically reload if there is a change in the codebase. If invalid syntax is added in the code, the iib-api container may shutdown. The Celery worker will automatically restart if there is a change under the iib/workers directory.

To run a built index image from the development registry, you can perform the following:

podman login --tls-verify=false -u iib -p iibpassword localhost:8443
podman pull --tls-verify=false localhost:8443/iib-build:1

If you are using Docker (a modern version is required), you can perform the following:

sudo docker login -u iib -p iibpassword localhost:8443
sudo docker run localhost:8443/iib-build:1

If your development environment requires accessing a private container registry, please read the section titled Registry Authentication.

Dependency Management¶

To manage dependencies, this project uses pip-tools so that the production dependencies are pinned and the hashes of the dependencies are verified during installation.

The unpinned dependencies are recorded in setup.py, and to generate the requirements.txt file, run pip-compile --generate-hashes --output-file=requirements.txt. This is only necessary when adding a new package. To upgrade a package, use the -P argument of the pip-compile command.

To update requirements-test.txt, run pip-compile --generate-hashes requirements-test.in -o requirements-test.txt.

When installing the dependencies in a production environment, run pip install --require-hashes -r requirements.txt. Alternatively, you may use pip-sync requirements.txt, which will make sure your virtualenv only has the packages listed in requirements.txt.

To ensure the pinned dependencies are not vulnerable, this project uses safety, which runs on every pull-request.

Registry Authentication¶

IIB does not handle authentication with container registries directly. If authentication is needed, configure the ~/.docker/config.json for the user running the IIB worker.

During development, you may choose to add a volume entry of - /root/.docker:/root/.docker:z on the workers in docker-compose.yml so that your host’s root user’s Docker configuration with authentication is used by the workers. This is only needed if you are working with private images. Please note that the containers will modify this configuration since they authenticate with the registry created by docker-compose on startup.

Configuring the REST API¶

To configure the IIB REST API, create a Python file at /etc/iib/settings.py. Any variables set in this configuration file will be applied to the Celery worker when running in production mode (default).

The custom configuration options for the REST API are listed below:

IIB_ADDITIONAL_LOGGERS - a list of Python loggers that should have the same log level that is set for IIB_LOG_LEVEL. This defaults to [].
IIB_FORCE_OVERWRITE_FROM_INDEX - a boolean that determines if privileged users should be forced to have overwrite_from_index set to True. This defaults to False.
IIB_GREENWAVE_CONFIG - the mapping, dict(<str>: dict(<str>:<str>)), of celery task queues to another dictionary of Greenwave query parameters to their values. This is useful in setting up customized gating for each queue. This defaults to {}. Use the task queue name as None to configure Greenwave config for the default Celery task queue.
IIB_LOG_FORMAT - the format of the logs. This defaults to %(asctime)s %(name)s %(levelname)s %(module)s.%(funcName)s %(message)s.
IIB_LOG_LEVEL - the Python log level of the REST API (Flask). This defaults to INFO.
IIB_MAX_PER_PAGE - the maximum number of build requests that can be shown on a single page. This defaults to 20.
IIB_PRIVILEGED_USERNAMES - the list of users that can perform privileged actions such as overwriting the input index image with the built index image. This defaults to [].
IIB_USER_TO_QUEUE - the mapping, dict(<str>: <str>), of usernames to celery task queues. This is useful in isolating the workload from certain users. The default queue is used for tasks from users not found in the mapping. This defaults to {}.
IIB_WORKER_USERNAMES - the list of case-sensitve Kerberos principals that are allowed to update build requests using the PATCH API endpoint. This defaults to [].
LOGIN_DISABLED - determines if authentication is required. This defaults to False (i.e. authentication is required).
SQLALCHEMY_DATABASE_URI - the database URI of the database the REST API connects to. See the Flask-SQLAlchemy configuration documentation.

The custom configuration options for AMQP 1.0 messaging are listed below:

IIB_MESSAGING_BATCH_STATE_DESTINATION - the AMQP 1.0 destination to send the batch state change messages. If this is not set, IIB will not send these types of messages. If this is set, IIB_MESSAGING_URLS must also be set.
IIB_MESSAGING_BUILD_STATE_DESTINATION - the AMQP 1.0 destination to send the build request state change messages. If this is not set, IIB will not send these types of messages. If this is set, IIB_MESSAGING_URLS must also be set.
IIB_MESSAGING_CA - the path to a file with the certificate authority that signed the certificate of the AMQP 1.0 message broker. This defaults to /etc/pki/tls/certs/ca-bundle.crt.
IIB_MESSAGING_CERT - the path to the identity certificate used for authentication with the AMQP 1.0 message broker. This defaults to /etc/iib/messaging.crt.
IIB_MESSAGING_DURABLE - determines if the messages are durable and cannot be lost due to an unexpected termination or restart by the AMQP 1.0 broker. If the broker is not capable of guaranteeing this, it may not accept the message. In that case, set this configuration option to False. This defaults to True.
IIB_MESSAGING_KEY - the path to the private key of the identity certificate used for authentication with the AMQP 1.0 message broker. This defaults to /etc/iib/messaging.key.
IIB_MESSAGING_TIMEOUT - the number of seconds before a messaging operation times out. Examples of messaging operations include connecting to the broker and sending a message to the broker. In this case, if the timeout is set to 30, then it could take a maximum of 60 seconds before the operation times out. This is because it can take up to 30 seconds to connect to the broker and also up to 30 seconds for the message to be sent. This defaults to 30.
IIB_MESSAGING_URLS - a list of AMQP(S) URLs to use when connecting to the AMQP 1.0 broker. This must be set if messaging is enabled.

Configuring the Worker(s)¶

To configure an IIB Celery worker, create a Python file at /etc/iib/celery.py. The location can be overridden with the IIB_CELERY_CONFIG environment variable. This is useful if the worker is running on the same host as another worker or the REST API.

Any variables set in this configuration file will be applied to the Celery worker when running in production mode (default).

The custom configuration options for the Celery workers are listed below:

broker_url - the AMQP(S) URL to connect to RabbitMQ. See the broker_url configuration documentation.
iib_api_timeout - the timeout in seconds for HTTP requests to the REST API. This defaults to 30 seconds.
iib_api_url - the URL to the IIB REST API (e.g. https://iib.domain.local/api/v1/).
iib_greenwave_url - the URL to the Greenwave REST API if gating is desired (e.g. https://greenwave.domain.local/api/v1.0/). This defaults to None.
iib_index_image_output_registry - if set, that value will replace the value from iib_registry in the output index_image pull specification. This is useful if you’d like users of IIB to pull from a proxy to a registry instead of the registry directly.
iib_image_push_template - the Python string template of the push destination for the resulting manifest list. The available variables are registry and request_id. The default value is {registry}/iib-build:{request_id}.
iib_log_level - the Python log level for iib.workers logger. This defaults to INFO.
iib_organization_customizations - this is used to customize aspects of the bundle being regenerated. The format is a dictionary where each key is an organization that requires customizations. Each value accepts the optional keys csv_annotations, package_name_suffix, and registry_replacements.
- The csv_annotations value is a dictionary where each key is the annotation to set on the ClusterServiceVersion files, and the value is a Python template string of the value to be set. IIB only substitutes {package_name} in the template string.
- The package_name_suffix value is a string of a suffix to add to the package name of the operator.
- The registry_replacements value is a dictionary where the keys are the old registries to replace and the values are the registries to replace the old registries with. This replaces the registry in all the ClusterServiceVersion files.
Here is an example that ties this all together:
```
iib_organization_customizations = {
    'company-marketplace': {
        'csv_annotations': {
            'marketplace.company.io/remote-workflow': 'https://marketplace.company.com/en-us/operators/{package_name}/pricing',
            'marketplace.company.io/support-workflow': 'https://marketplace.company.com/en-us/operators/{package_name}/support',
        },
        'package_name_suffix': '-cmp',
        'registry_replacements': {
            'registry.access.company.com': 'registry.marketplace.company.com/cm',
        },
    }
}
```
iib_registry - the container registry to push images to (e.g. quay.io).
iib_skopeo_timeout - the command timeout for skopeo commands run by IIB. This defaults to 30s (30 seconds).
iib_total_attempts - the total number of attempts to make at trying a function relating to the container registry before erroring out. This defaults to 5.

Regenerating Bundle Images¶

In addition to building operator index images, IIB can also be used to regenerate operator bundle images. This is useful for applying modifications to the manifests embedded in the bundle image. IIB uses the operator-manifest library to assist in these modifications.

Currently, IIB will not perform any modifications on a ClusterServiceVersion file if spec.relatedImages is set.

If it’s not set, IIB will pin any container image pull specification and set spec.relatedImages. See the different pull specifications to which this process applies to. There are also a variety of customizations that can be made to the bundle. See the iib_organization_customizations configuration option for more details.

Messaging¶

IIB has support to send messages to an AMQP 1.0 broker. If configured to do so, IIB will send messages when a build request state changes and when a batch state changes. Please note that if a message can’t be sent due to an infrastructure issue, the build request will continue as it is not considered a fatal error.

The build request state change message body is the JSON representation of the build request in the non-verbose format like in the /builds API endpoint. The message has the following keys set in the application properties: batch, id, state, and user.

The batch state change message body is a JSON object with the following keys: annotations, batch, requests, state, and user. The requests value is an array of JSON objects with the keys id, organization, and request_type. The message has the following keys set in the application properties: batch, state, and user.

Gating Bundle Images¶

In addition to building operator index images, IIB can also gate your bundle images before adding them to the index image. If a Greenwave configuration is setup for your queue, IIB will query Greenwave to check if your bundle image builds have passed the tests in the Greenwave policy you have defined. The IIB request submitted to that queue will succeed only if the policy is satisfied.

Read the Docs Documentation¶

Build the Docs¶

To build and serve the docs, run the following commands:

tox -e docs
google-chrome .tox/docs_out/index.html

Expanding the Docs¶

To document a new Python module, find the rst file of the corresponding Python package that contains the module. Once found, add a section under “Submodules” in alphabetical order such as:

iib.workers.tasks.build module
------------------------------

.. automodule:: iib.workers.tasks.build
   :ignore-module-all:
   :members:
   :private-members:
   :show-inheritance:

Some of the options include:

ignore-module-all - include all members regardless of the definition of __all__.
members - automatically document the members in that Python module.
private-members - include private functions and methods.
show-inheritance - show the class inheritance.

Modules Documentation¶

iib¶

Subpackages¶

iib.web¶

Submodules¶

iib.web.api_v1 module¶

iib.web.api_v1._get_user_queue()¶

Return the name of the celery task queue mapped to the current user.

Returns:	queue name to be used or None if the default queue should be used
Return type:	str or None

iib.web.api_v1._should_force_overwrite()¶

Determine if the overwrite_from_index parameter should be forced.

This is for clients that require this functionality but do not currently use the overwrite_from_index parameter already.

Returns:	the boolean that determines if the overwrite should be forced
Return type:	bool

iib.web.api_v1.add_bundles()¶

Submit a request to add operator bundles to an index image.

Return type:	flask.Response
Raises:	ValidationError – if required parameters are not supplied

iib.web.api_v1.get_build(request_id)¶

Retrieve the build request.

Parameters:	request_id (int) – the request ID that was passed in through the URL.
Return type:	flask.Response
Raises:	NotFound – if the request is not found

iib.web.api_v1.get_builds()¶

Retrieve the paginated build requests.

Return type:	flask.Response

iib.web.api_v1.patch_request(request_id)¶

Modify the given request.

Parameters:	request_id (int) – the request ID from the URL
Returns:	a Flask JSON response
Return type:	flask.Response
Raises:	Forbidden – If the user trying to patch a request is not an IIB worker NotFound – if the request is not found ValidationError – if the JSON is invalid

iib.web.api_v1.regenerate_bundle()¶

Submit a request to regenerate an operator bundle image.

Return type:	flask.Response
Raises:	ValidationError – if required parameters are not supplied

iib.web.api_v1.regenerate_bundle_batch()¶

Submit a batch of requests to regenerate operator bundle images.

Return type:	flask.Response
Raises:	ValidationError – if required parameters are not supplied

iib.web.api_v1.rm_operators()¶

Submit a request to remove operators from an index image.

Return type:	flask.Response
Raises:	ValidationError – if required parameters are not supplied

iib.web.app module¶

iib.web.app.create_app(config_obj=None)¶

Create a Flask application object.

Parameters:	config_obj (str) – the path to the configuration object to use instead of calling load_config
Returns:	a Flask application object
Return type:	flask.Flask

iib.web.app.load_config(app)¶

Determine the correct configuration to use and apply it.

Parameters:	app (flask.Flask) – a Flask application object

iib.web.app.validate_api_config(config)¶

Determine if the configuration is valid.

Parameters:	config (dict) – the dict containing the IIB REST API config
Raises:	ConfigError – if the config is invalid

iib.web.auth module¶

iib.web.auth._get_kerberos_principal(request)¶

Get the Kerberos principal from the current request.

This relies on the “REMOTE_USER” environment variable being set. This is usually set by the mod_auth_gssapi Apache authentication module.

Parameters:	request (flask.Request) – the Flask request
Returns:	the user’s Kerberos principal or None
Return type:	str

iib.web.auth.load_user_from_request(request)¶

Load the user that authenticated from the current request.

This is used by the Flask-Login library. If the user does not exist in the database, an entry will be created.

If None is returned, then Flask-Login will set flask_login.current_user to an AnonymousUserMixin object, which has the is_authenticated property set to False. Additionally, any route decorated with @login_required will raise an Unauthorized exception.

Parameters:	request (flask.Request) – the Flask request
Returns:	the User object associated with the username or None
Return type:	iib.web.models.User

iib.web.auth.user_loader(username)¶

Get the user by their username from the database.

This is used by the Flask-Login library.

Parameters:	username (str) – the username of the user
Returns:	the User object associated with the username or None
Return type:	iib.web.models.User

iib.web.config module¶

class iib.web.config.Config¶

Bases: object

The base IIB Flask configuration.

class iib.web.config.DevelopmentConfig¶

Bases: iib.web.config.Config

The development IIB Flask configuration.

class iib.web.config.ProductionConfig¶

Bases: iib.web.config.Config

The production IIB Flask configuration.

class iib.web.config.TestingConfig¶

Bases: iib.web.config.DevelopmentConfig

The testing IIB Flask configuration.

class iib.web.config.TestingConfigNoAuth¶

Bases: iib.web.config.TestingConfig

The testing IIB Flask configuration without authentication.

iib.web.docs module¶

iib.web.docs.index()¶: Return the OpenAPI documentation presented by redoc.

iib.web.errors module¶

iib.web.errors.json_error(error)¶

Convert exceptions to JSON responses.

Parameters:	error (Exception) – an Exception to convert to JSON
Returns:	a Flask JSON response
Return type:	flask.Response

iib.web.manage module¶

iib.web.models module¶

class iib.web.models.Architecture(**kwargs)¶

Bases: sqlalchemy.ext.declarative.api.Model

An architecture associated with an image.

static validate_architecture_json(arches)¶

Validate the JSON representation of architectures.

Parameters:	arches (list) – the JSON representation of architectures for a build request
Raises:	ValidationError – if the JSON does not match the required schema

class iib.web.models.BaseEnum¶

Bases: enum.Enum

A base class for IIB enums.

class iib.web.models.Batch(**kwargs)¶

Bases: sqlalchemy.ext.declarative.api.Model

A batch associated with one or more requests.

annotations¶: Return the Python representation of the JSON annotations.

request_states¶

Get the states of all the requests in the batch.

Returns:	the list of states
Return type:	list<str>

state¶

Get the state of the batch.

If one or more requests in the batch are in_progress, then the batch is in_progress. Once all the requests in the batch have completed, if one or more requests are in the failed state, then so is the batch. If all requests in the batch are in the complete state, then so is the batch.

Returns:	the state of the batch
Return type:	str

user¶

Get the User object associated with the batch.

Returns:	the `User` object associated with the batch or `None`
Return type:	User or None

static validate_batch(batch_id)¶

Validate the input batch ID.

If the input batch ID is a string, it will be converted to an integer and returned.

Parameters:	batch_id (int) – the ID of the batch
Raises:	ValidationError – if the batch ID is invalid
Returns:	the validated batch ID
Return type:	int

static validate_batch_request_params(payload)¶

Validate batch specific parameters from the input JSON payload.

The requests in the “build_requests” key’s value are not validated. Those should be validated separately.

Raises:	ValidationError – if the payload is invalid

class iib.web.models.Image(**kwargs)¶

Bases: sqlalchemy.ext.declarative.api.Model

An image that has been handled by IIB.

This will typically point to a manifest list.

classmethod get_or_create(pull_specification)¶

Get the image from the database and create it if it doesn’t exist.

Parameters:	pull_specification (str) – pull_specification of the image
Returns:	an Image object based on the input pull_specification; the Image object will be added to the database session, but not committed, if it was created
Return type:	Image
Raises:	ValidationError – if pull_specification for the image is invalid

class iib.web.models.Operator(**kwargs)¶

Bases: sqlalchemy.ext.declarative.api.Model

An operator that has been handled by IIB.

classmethod get_or_create(name)¶

Get the operator from the database and create it if it doesn’t exist.

Parameters:	name (str) – the name of the operator
Returns:	an Operator object based on the input name; the Operator object will be added to the database session, but not committed, if it was created
Return type:	Operator

class iib.web.models.Request(**kwargs)¶

Bases: sqlalchemy.ext.declarative.api.Model

A generic image build request.

add_architecture(arch_name)¶

Add an architecture associated with this image.

Parameters:	arch_name (str) – the architecture to add
Raises:	ValidationError – if the architecture is invalid

add_state(state, state_reason)¶

Add a RequestState associated with the current request.

Parameters:	state (str) – the state name state_reason (str) – the reason explaining the state transition
Raises:	ValidationError – if the state is invalid

classmethod from_json(kwargs)¶

Handle JSON requests for a request API endpoint.

Child classes MUST override this method.

Parameters:	kwargs (dict) – the user provided parameters to create a Request
Returns:	an object representation of the request
Retype:	Request

get_mutable_keys()¶

Return the set of keys representing the attributes that can be modified.

Returns:	a set of key names
Return type:	set

to_json(verbose=True)¶

Provide the basic JSON representation of a build request.

Child classes are expected to enhance the JSON representation as needed.

Parameters:	verbose (bool) – determines if the JSON output should be verbose
Returns:	a dictionary representing the JSON of the build request
Return type:	dict

type_name¶

Get the request’s type as a string.

Returns:	the request’s type
Return type:	str

validate_type(key, type_num)¶

Verify the type number used is valid.

Parameters:	key (str) – the name of the database column type_num (int) – the request type number to be verified
Returns:	the request type number
Return type:	int
Raises:	ValidationError – if the request type is invalid

class iib.web.models.RequestAdd(**kwargs)¶

Bases: iib.web.models.Request, iib.web.models.RequestIndexImageMixin

An “add” index image build request.

binary_image¶: Return the relationship to the image that the opm binary comes from.

binary_image_id¶: Return the ID of the image that the opm binary comes from.

binary_image_resolved¶: Return the relationship to the resolved image that the opm binary comes from.

binary_image_resolved_id¶: Return the ID of the resolved image that the opm binary comes from.

from_index¶: Return the relationship of the index image to base the request from.

from_index_id¶: Return the ID of the index image to base the request from.

from_index_resolved¶: Return the relationship of the resolved index image to base the request from.

from_index_resolved_id¶: Return the ID of the resolved index image to base the request from.

classmethod from_json(kwargs)¶: Handle JSON requests for the Add API endpoint.

get_mutable_keys()¶

Return the set of keys representing the attributes that can be modified.

Returns:	a set of key names
Return type:	set

index_image¶: Return the relationship to the built index image.

index_image_id¶: Return the ID of the built index image.

to_json(verbose=True)¶

Provide the JSON representation of an “add” build request.

Parameters:	verbose (bool) – determines if the JSON output should be verbose
Returns:	a dictionary representing the JSON of the build request
Return type:	dict

class iib.web.models.RequestAddBundle(**kwargs)¶

Bases: sqlalchemy.ext.declarative.api.Model

An association table between add requests and the bundles they contain.

class iib.web.models.RequestArchitecture(**kwargs)¶

Bases: sqlalchemy.ext.declarative.api.Model

An association table between requests and the architectures they were built for.

class iib.web.models.RequestIndexImageMixin¶

Bases: object

A class for shared functionality between index image requests.

This class uses the Mixin pattern as defined in: https://docs.sqlalchemy.org/en/13/orm/extensions/declarative/mixins.html

static _from_json(request_kwargs, additional_required_params=None, additional_optional_params=None)¶

Validate and process request agnostic parameters.

As part of the processing, the input request_kwargs parameter is updated to reference database objects where appropriate.

Parameters:	request_kwargs (dict) – copy of args provided in API request

get_common_index_image_json()¶

Return the common set of attributes for an index image request.

For compatibility between the different types of index image requests, any index image request must provide the combination of possible attributes. For example, the “bundles” attribute is always included even though it’s only used by RequestAdd.

The specialized index image requests should modify the value of the attributes as needed.

Returns:	a partial dictionary representing the JSON of the index image build request
Return type:	dict

get_index_image_mutable_keys()¶

Return the set of keys representing the attributes that can be modified.

Returns:	a set of key names
Return type:	set

class iib.web.models.RequestRegenerateBundle(**kwargs)¶

Bases: iib.web.models.Request

A “regenerate_bundle” image build request.

classmethod from_json(kwargs, batch=None)¶

Handle JSON requests for the Regenerate Bundle API endpoint.

Parameters:	kwargs (dict) – the JSON payload of the request. batch (Batch) – the batch to specify with the request. If one is not specified, one will be created automatically.

get_mutable_keys()¶

Return the set of keys representing the attributes that can be modified.

Returns:	a set of key names
Return type:	set

to_json(verbose=True)¶

Provide the JSON representation of a “regenerate-bundle” build request.

Parameters:	verbose (bool) – determines if the JSON output should be verbose
Returns:	a dictionary representing the JSON of the build request
Return type:	dict

class iib.web.models.RequestRm(**kwargs)¶

Bases: iib.web.models.Request, iib.web.models.RequestIndexImageMixin

A “rm” index image build request.

binary_image¶: Return the relationship to the image that the opm binary comes from.

binary_image_id¶: Return the ID of the image that the opm binary comes from.

binary_image_resolved¶: Return the relationship to the resolved image that the opm binary comes from.

binary_image_resolved_id¶: Return the ID of the resolved image that the opm binary comes from.

from_index¶: Return the relationship of the index image to base the request from.

from_index_id¶: Return the ID of the index image to base the request from.

from_index_resolved¶: Return the relationship of the resolved index image to base the request from.

from_index_resolved_id¶: Return the ID of the resolved index image to base the request from.

classmethod from_json(kwargs)¶: Handle JSON requests for the Remove API endpoint.

get_mutable_keys()¶

Return the set of keys representing the attributes that can be modified.

Returns:	a set of key names
Return type:	set

index_image¶: Return the relationship to the built index image.

index_image_id¶: Return the ID of the built index image.

to_json(verbose=True)¶

Provide the JSON representation of an “rm” build request.

Parameters:	verbose (bool) – determines if the JSON output should be verbose
Returns:	a dictionary representing the JSON of the build request
Return type:	dict

class iib.web.models.RequestRmOperator(**kwargs)¶

Bases: sqlalchemy.ext.declarative.api.Model

An association table between rm requests and the operators they contain.

class iib.web.models.RequestState(**kwargs)¶

Bases: sqlalchemy.ext.declarative.api.Model

Represents a state (historical or present) of a request.

state_name¶: Get the state’s display name.

class iib.web.models.RequestStateMapping¶

Bases: iib.web.models.BaseEnum

An Enum that represents the request states.

class iib.web.models.RequestTypeMapping¶

Bases: iib.web.models.BaseEnum

An Enum that represents the request types.

class iib.web.models.User(**kwargs)¶

Bases: sqlalchemy.ext.declarative.api.Model, flask_login.mixins.UserMixin

Represents an external user that owns an IIB request.

classmethod get_or_create(username)¶

Get the user from the database and create it if it doesn’t exist.

Parameters:	username (str) – the username of the user
Returns:	a User object based on the input username; the User object will be added to the database session, but not committed, if it was created
Return type:	User

iib.web.models.get_request_query_options(verbose=False)¶

Get the query options for a SQLAlchemy query for one or more requests to output as JSON.

This will add the joins ahead of time on relationships that are accessed in the to_json methods to avoid individual select statements when the relationships are accessed.

Parameters:	verbose (bool) – if the request relationships should be loaded for verbose JSON output
Returns:	a list of SQLAlchemy query options
Return type:	list

iib.web.models.validate_request_params(request_params, required_params, optional_params)¶

Validate parameters for a build request.

All required parameters must be set in the request_params and unknown parameters are not allowed.

Parameters:	request_params (dict) – the request parameters provided by the user required_params (set) – the set of required parameters optional_params (set) – the set of optional parameters
Raises:	iib.exceptions.ValidationError – if validation of parameters fails

iib.web.utils module¶

iib.web.utils.pagination_metadata(pagination_query, **kwargs)¶

Return a dictionary containing metadata about the paginated query.

This must be run as part of a Flask request.

Parameters:	pagination_query (flask_sqlalchemy.Pagination) – the paginated query kwargs (dict) – the query parameters to add to the URLs
Returns:	a dictionary containing metadata about the paginated query

iib.web.utils.str_to_bool(item)¶

Convert a string to a boolean.

Parameters:	item (str) – string to parse
Returns:	a boolean equivalent
Return type:	bool

iib.web.wsgi module¶

Module contents¶

iib.workers¶

Subpackages¶

iib.workers.tasks¶

Submodules¶

iib.workers.tasks.build module¶

iib.workers.tasks.build._adjust_csv_annotations(operator_csvs, package_name, organization)¶

Annotate ClusterServiceVersion objects based on an organization configuration.

Parameters:	operator_csvs (list) – the list of `OperatorCSV` objects to examine. package_name (str) – the operator package name. organization (str) – the organization this bundle is for. This determines what annotations to make.

iib.workers.tasks.build._adjust_operator_bundle(manifests_path, metadata_path, organization=None)¶

Apply modifications to the operator manifests at the given location.

For any container image pull spec found in the Operator CSV files, replace floating tags with pinned digests, e.g. image:latest becomes image@sha256:….

If spec.relatedImages is not set, it will be set with the pinned digests. If it is set but there are also RELATED_IMAGE_* environment variables set, an exception will be raised.

This method relies on the OperatorManifest class to properly identify and apply the modifications as needed.

Parameters:	manifests_path (str) – the full path to the directory containing the operator manifests. metadata_path (str) – the full path to the directory containing the bundle metadata files. organization (str) – the organization this bundle is for. If no organization is provided, no custom behavior will be applied.
Raises:	IIBError – if the operator manifest has invalid entries
Returns:	a dictionary of labels to set on the bundle
Return type:	dict

iib.workers.tasks.build._apply_package_name_suffix(metadata_path, organization=None)¶

Add the package name suffix if configured for this organization.

This adds the suffix to the value of annotations['operators.operatorframework.io.bundle.package.v1'] in metadata/annotations.yaml.

The final package name value is returned as part of the tuple.

Parameters:	metadata_path (str) – the path to the bundle’s metadata directory. organization (str) – the organization this customization is for.
Raises:	IIBError – if the `metadata/annotations.yaml` file is in an unexpected format.
Returns:	a tuple with the package name and a dictionary of labels to set on the bundle.
Return type:	tuple(str, dict)

iib.workers.tasks.build._build_image(dockerfile_dir, dockerfile_name, request_id, arch)¶

Build the index image for the specified architecture.

Parameters:	dockerfile_dir (str) – the path to the directory containing the data used for building the container image dockerfile_name (str) – the name of the Dockerfile in the dockerfile_dir to be used when building the container image request_id (int) – the ID of the IIB build request arch (str) – the architecture to build this image for
Raises:	IIBError – if the build fails

iib.workers.tasks.build._cleanup()¶

Remove all existing container images on the host.

This will ensure that the host will not run out of disk space due to stale data, and that all images referenced using floating tags will be up to date on the host.

Raises:	IIBError – if the command to remove the container images fails

iib.workers.tasks.build._copy_files_from_image(image, src_path, dest_path)¶

Copy a file from the container image into the given destination path.

The file may be a directory.

Parameters:	image (str) – the pull specification of the container image. src_path (str) – the full path within the container image to copy from. dest_path (str) – the full path on the local host to copy into.

iib.workers.tasks.build._create_and_push_manifest_list(request_id, arches)¶

Create and push the manifest list to the configured registry.

Parameters:	request_id (int) – the ID of the IIB build request arches (iter) – an iterable of arches to create the manifest list for
Returns:	the pull specification of the manifest list
Return type:	str
Raises:	IIBError – if creating or pushing the manifest list fails

iib.workers.tasks.build._finish_request_post_build(output_pull_spec, request_id, arches, from_index=None, overwrite_from_index=False, overwrite_from_index_token=None)¶

Finish the request after the manifest list has been pushed.

This function was created so that code didn’t need to be duplicated for the add and rm request types.

Parameters:

output_pull_spec (str) – pull spec of the index image generated by IIB
request_id (int) – the ID of the IIB build request
arches (set) – the set of arches that were built as part of this request
from_index (str) – the pull specification of the container image containing the index that the index image build was based from.
overwrite_from_index (bool) – if True, overwrite the input from_index with the built index image.
overwrite_from_index_token (str) – the token used for overwriting the input from_index image.

Raises:

IIBError – if the manifest list couldn’t be created and pushed

iib.workers.tasks.build._get_external_arch_pull_spec(request_id, arch, include_transport=False)¶

Get the pull specification of the single arch image in the external registry.

Parameters:	request_id (int) – the ID of the IIB build request arch (str) – the specific architecture of the container image include_transport (bool) – if true, docker:// will be prefixed in the returned pull specification
Returns:	the pull specification of the single arch image in the external registry
Return type:	str

iib.workers.tasks.build._get_image_arches(pull_spec)¶

Get the architectures this image was built for.

Parameters:	pull_spec (str) – the pull specification to a v2 manifest list
Returns:	a set of architectures of the container images contained in the manifest list
Return type:	set
Raises:	IIBError – if the pull specification is not a v2 manifest list

iib.workers.tasks.build._get_local_pull_spec(request_id, arch)¶

Get the local pull specification of the architecture specfic index image for this request.

Parameters:	request_id (int) – the ID of the IIB build request arch (str) – the specific architecture of the container image.
Returns:	the pull specification of the index image for this request.
Return type:	str

iib.workers.tasks.build._get_resolved_bundles(bundles)¶

Get the pull specification of the bundle images using their digests.

Determine if the pull spec refers to a manifest list. If so, simply use the digest of the first item in the manifest list. If not a manifest list, it must be a v2s2 image manifest and should be used as it is.

Parameters:	bundles (list) – the list of bundle images to be resolved.
Returns:	the list of bundle images resolved to their digests.
Return type:	list
Raises:	IIBError – if unable to resolve a bundle image.

iib.workers.tasks.build._get_resolved_image(pull_spec)¶

Get the pull specification of the container image using its digest.

Parameters:	pull_spec (str) – the pull specification of the container image to resolve
Returns:	the resolved pull specification
Return type:	str

iib.workers.tasks.build._opm_index_add(base_dir, bundles, binary_image, from_index=None)¶

Add the input bundles to an operator index.

This only produces the index.Dockerfile file and does not build the container image.

Parameters:

base_dir (str) – the base directory to generate the database and index.Dockerfile in.
bundles (list) – a list of strings representing the pull specifications of the bundles to add to the index image being built.
binary_image (str) – the pull specification of the container image where the opm binary gets copied from. This should point to a digest or stable tag.
from_index (str) – the pull specification of the container image containing the index that the index image build will be based from.

Raises:

IIBError – if the opm index add command fails.

iib.workers.tasks.build._opm_index_rm(base_dir, operators, binary_image, from_index)¶

Remove the input operators from the operator index.

This only produces the index.Dockerfile file and does not build the container image.

Parameters:

base_dir (str) – the base directory to generate the database and index.Dockerfile in.
operators (list) – a list of strings representing the names of the operators to remove from the index image.
binary_image (str) – the pull specification of the container image where the opm binary gets copied from.
from_index (str) – the pull specification of the container image containing the index that the index image build will be based from.

Raises:

IIBError – if the opm index rm command fails.

iib.workers.tasks.build._prepare_request_for_build(binary_image, request_id, from_index=None, add_arches=None, bundles=None)¶

Prepare the request for the index image build.

All information that was retrieved and/or calculated for the next steps in the build are returned as a dictionary.

This function was created so that code didn’t need to be duplicated for the add and rm request types.

Parameters:	binary_image (str) – the pull specification of the container image where the opm binary gets copied from. request_id (int) – the ID of the IIB build request from_index (str) – the pull specification of the container image containing the index that the index image build will be based from. add_arches (list) – the list of arches to build in addition to the arches `from_index` is currently built for; if `from_index` is `None`, then this is used as the list of arches to build the index image for bundles (list) – the list of bundles to create the bundle mapping on the request
Returns:	a dictionary with the keys: arches, binary_image_resolved, and from_index_resolved.
Raises:	IIBError – if the container image resolution fails or the architectures couldn’t be detected.

iib.workers.tasks.build._push_image(request_id, arch)¶

Push the single arch container image to the configured registry.

Parameters:	request_id (int) – the ID of the IIB build request arch (str) – the architecture of the container image to push
Raises:	IIBError – if the push fails

iib.workers.tasks.build._skopeo_copy(source, destination, copy_all=False, dest_token=None, exc_msg=None)¶

Wrap the skopeo copy command.

Parameters:	source (str) – the source to copy destination (str) – the destination to copy the source to copy_all (bool) – if True, it passes `--all` to the command dest_token (str) – the token to pass to the --dest-token` parameter of the command. If not provided, ``--dest-token parameter is also not provided. exc_msg (str) – a custom exception message to provide
Raises:	IIBError – if the copy fails

iib.workers.tasks.build._verify_index_image(resolved_prebuild_from_index, unresolved_from_index)¶

Verify if the index image has changed since the IIB build request started.

Parameters:	resolved_prebuild_from_index (str) – resolved index image before starting the build unresolved_from_index (str) – unresolved index image provided as API input
Raises:	IIBError – if the index image has changed since IIB build started.

iib.workers.tasks.build._verify_labels(bundles)¶

Verify that the required labels are set on the input bundles.

Parameters:	bundles (list) – a list of strings representing the pull specifications of the bundles to add to the index image being built.
Raises:	IIBError – if one of the bundles does not have the correct label value.

iib.workers.tasks.build.get_image_label(pull_spec, label)¶

Get a specific label from the container image.

Parameters:	label (str) – the label to get
Returns:	the label on the container image or None
Return type:	str

iib.workers.tasks.build.get_rebuilt_image_pull_spec(request_id)¶

Generate the pull specification of the container image rebuilt by IIB.

Parameters:	request_id (int) – the ID of the IIB build request
Returns:	pull specification of the rebuilt container image
Return type:	str

(task)iib.workers.tasks.build.handle_add_request(bundles, binary_image, request_id, from_index=None, add_arches=None, cnr_token=None, organization=None, overwrite_from_index=False, overwrite_from_index_token=None, greenwave_config=None)¶

Coordinate the the work needed to build the index image with the input bundles.

Parameters:

bundles (list) – a list of strings representing the pull specifications of the bundles to add to the index image being built.
binary_image (str) – the pull specification of the container image where the opm binary gets copied from.
request_id (int) – the ID of the IIB build request
from_index (str) – the pull specification of the container image containing the index that the index image build will be based from.
add_arches (list) – the list of arches to build in addition to the arches from_index is currently built for; if from_index is None, then this is used as the list of arches to build the index image for
cnr_token (str) – the token required to push backported packages to the legacy app registry via OMPS.
organization (str) – organization name in the legacy app registry to which the backported packages should be pushed to.
overwrite_from_index (bool) – if True, overwrite the input from_index with the built index image.
overwrite_from_index_token (str) – the token used for overwriting the input from_index image. This is required for non-privileged users to use overwrite_from_index. The format of the token must be in the format “user:password”.
greenwave_config (dict) – the dict of config required to query Greenwave to gate bundles.

Raises:

IIBError – if the index image build fails or legacy support is required and one of cnr_token or organization is not specified.

(task)iib.workers.tasks.build.handle_regenerate_bundle_request(from_bundle_image, organization, request_id)¶

Coordinate the work needed to regenerate the operator bundle image.

Parameters:	from_bundle_image (str) – the pull specification of the bundle image to be regenerated. organization (str) – the name of the organization the bundle should be regenerated for. request_id (int) – the ID of the IIB build request
Raises:	IIBError – if the regenerate bundle image build fails.

(task)iib.workers.tasks.build.handle_rm_request(operators, binary_image, request_id, from_index, add_arches=None, overwrite_from_index=False, overwrite_from_index_token=None)¶

Coordinate the work needed to remove the input operators and rebuild the index image.

Parameters:

operators (list) – a list of strings representing the name of the operators to remove from the index image.
binary_image (str) – the pull specification of the container image where the opm binary gets copied from.
request_id (int) – the ID of the IIB build request
from_index (str) – the pull specification of the container image containing the index that the index image build will be based from.
add_arches (list) – the list of arches to build in addition to the arches from_index is currently built for.
overwrite_from_index (bool) – if True, overwrite the input from_index with the built index image.
overwrite_from_index_token (str) – the token used for overwriting the input from_index image. This is required for non-privileged users to use overwrite_from_index. The format of the token must be in the format “user:password”.

Raises:

IIBError – if the index image build fails.

iib.workers.tasks.celery module¶

iib.workers.tasks.general module¶

(task)iib.workers.tasks.general.failed_request_callback(context, exc, traceback, request_id)¶

Wrap set_request_state for task error callbacks.

Parameters:	context (celery.app.task.Context) – the context of the task failure exc (Exception) – the exception that caused the task failure request_id (int) – the ID of the IIB request

iib.workers.tasks.legacy module¶

iib.workers.tasks.legacy._get_base_dir_and_pkg_name(package_dir)¶

Get the base directory and the package name from package directory.

Parameters:	package_dir (str) – path to the exported package directory
Returns:	base_dir, package name
Return type:	str, str

iib.workers.tasks.legacy._opm_index_export(rebuilt_index_image, package, temp_dir)¶

Export the package that needs to be backported.

Parameters:	rebuilt_index_image (str) – the pull specification of the index image rebuilt by IIB. package (set) – a string representing the name of the package to be exported. temp_dir (str) – path to the temporary directory where the package will be exported to.
Raises:	IIBError – if the export of packages fails.

iib.workers.tasks.legacy._push_package_manifest(package_dir, cnr_token, organization)¶

Push manifests.zip file created for an exported package to OMPS.

Parameters:	package_dir (str) – path to the exported package directory. cnr_token (str) – the token required to push backported packages to the legacy app registry via OMPS. organization (str) – the organization name in the legacy app registry to which the backported packages should be pushed to.
Raises:	IIBError – if the push is unsucessful

iib.workers.tasks.legacy._verify_package_info(package_dir, from_index)¶

Verify if the exported package info is generated correctly.

Parameters:	package_dir (str) – path to the exported package directory from_index (str) – the pull specification of the image image
Raises:	IIBError – if the generated package info is missing

iib.workers.tasks.legacy._zip_package(package_dir)¶

Zip content of exported package to a manifests.zip file.

Parameters:	package_dir (str) – path to the exported package directory
Raises:	IIBError – if unable to zip the exported package

iib.workers.tasks.legacy.export_legacy_packages(packages, request_id, rebuilt_index_image, cnr_token, organization)¶

Export packages to be backported and push them via OMPS.

Parameters:

packages (set) – a set of strings representing the names of the packages to be exported.
request_id (int) – the ID of the IIB build request.
rebuilt_index_image (str) – the pull specification of the index image rebuilt by IIB.
cnr_token (str) – the token required to push backported packages to the legacy app registry via OMPS.
organization (str) – the organization name in the legacy app registry to which the backported packages should be pushed to.

Raises:

IIBError – if the export of packages fails.

iib.workers.tasks.legacy.get_legacy_support_packages(bundles)¶

Get the packages that must be pushed to the legacy application registry.

Parameters:	bundles (list<str>) – a list of strings representing the pull specifications of the bundles to add to the index image being built.
Returns:	a set of packages that require legacy support
Return type:	set

iib.workers.tasks.legacy.validate_legacy_params_and_config(packages, bundles, cnr_token, organization)¶

Valiate parameters and config variables required for legacy support.

Parameters:

packages (set) – a set of strings representing the names of the packages to be exported.
bundles (list) – a list of strings representing the bundles to be added to the index image.
cnr_token (str) – the token required to push backported packages to the legacy app registry via OMPS.
organization (str) – organization name in the legacy app registry to which the backported packages should be pushed to.

Raises:

IIBError – if legacy support is required and necessary params are missing.

iib.workers.tasks.utils module¶

iib.workers.tasks.utils.get_image_labels(pull_spec)¶

Get the labels from the image.

Parameters:	labels (list<str>) – the labels to get
Returns:	the dictionary of the labels on the image
Return type:	dict

iib.workers.tasks.utils.podman_pull(*args)¶

Wrap the podman pull command.

Parameters:	args – any arguments to pass to `podman pull`
Raises:	IIBError – if the command fails

iib.workers.tasks.utils.retry(attempts=5, wait_on=<class 'Exception'>, logger=None)¶

Retry a section of code until success or max attempts are reached.

Parameters:	attempts (int) – the total number of attempts to make before erroring out wait_on (Exception) – the exception on encountering which the function will be retried logger (logging) – the logger to log the messages on
Raises:	IIBError – if the maximum attempts are reached

iib.workers.tasks.utils.run_cmd(cmd, params=None, exc_msg=None, cmd_repr=None)¶

Run the given command with the provided parameters.

Parameters:	cmd (iter) – iterable representing the command to be executed params (dict) – keyword parameters for command execution exc_msg (str) – an optional exception message when the command fails cmd_repr (iter) – an optional representation of the command to be executed. If not provided, this is derived from the `cmd` parameter. This is useful when the command contains sensitive information that must not be logged.
Returns:	the command output
Return type:	str
Raises:	IIBError – if the command fails

iib.workers.tasks.utils.skopeo_inspect(*args)¶

Wrap the skopeo inspect command.

Parameters:	args – any arguments to pass to `skopeo inspect`
Returns:	a dictionary of the JSON output from the skopeo inspect command
Return type:	dict
Raises:	IIBError – if the command fails

Module contents¶

Submodules¶

iib.workers.api_utils module¶

iib.workers.api_utils.get_request(request_id)¶

Get the IIB build request from the REST API.

Parameters:	request_id (int) – the ID of the IIB request
Returns:	the request
Return type:	dict
Raises:	IIBError – if the HTTP request fails

iib.workers.api_utils.get_requests_session(auth=False)¶

Create a requests session with authentication (when enabled).

Parameters:	auth (bool) – configure authentication on the session
Returns:	the configured requests session
Return type:	requests.Session

iib.workers.api_utils.set_request_state(request_id, state, state_reason)¶

Set the state of the request using the IIB API.

Parameters:	request_id (int) – the ID of the IIB request state (str) – the state to set the IIB request to state_reason (str) – the state reason to set the IIB request to
Returns:	the updated request
Return type:	dict
Raises:	IIBError – if the request to the IIB API fails

iib.workers.api_utils.update_request(request_id, payload, exc_msg=None)¶

Update the IIB build request.

Parameters:	request_id (int) – the ID of the IIB request payload (dict) – the payload to send to the PATCH API endpoint exc_msg (str) – an optional custom exception that can be a template
Returns:	the updated request
Return type:	dict
Raises:	IIBError – if the request to the IIB API fails

iib.workers.config module¶

class iib.workers.config.Config¶

Bases: object

The base IIB Celery configuration.

class iib.workers.config.DevelopmentConfig¶

Bases: iib.workers.config.Config

The development IIB Celery configuration.

class iib.workers.config.ProductionConfig¶

Bases: iib.workers.config.Config

The production IIB Celery configuration.

class iib.workers.config.TestingConfig¶

Bases: iib.workers.config.DevelopmentConfig

The testing IIB Celery configuration.

iib.workers.config.configure_celery(celery_app)¶

Configure the Celery application instance.

Parameters:	celery (celery.Celery) – the Celery application instance to configure

iib.workers.config.get_worker_config()¶: Return the Celery configuration.

iib.workers.config.validate_celery_config(conf, **kwargs)¶

Perform basic validatation on the Celery configuration when the worker is initialized.

Parameters:	conf (celery.app.utils.Settings) – the Celery application configuration to validate
Raises:	iib.exceptions.ConfigError – if the configuration is invalid

Module contents¶

Submodules¶

iib.exceptions module¶

exception iib.exceptions.BaseException¶

Bases: Exception

The base class for all IIB exceptions.

exception iib.exceptions.ConfigError¶

Bases: iib.exceptions.BaseException

The configuration is invalid.

exception iib.exceptions.IIBError¶

Bases: iib.exceptions.BaseException

An error was encountered in IIB.

exception iib.exceptions.ValidationError¶

Bases: iib.exceptions.BaseException

Denote invalid input.

Welcome to the IIB documentation!¶

Index Image Build (IIB) Service¶

External Documentation¶

Coding Standards¶

Running the Unit Tests¶

Development Environment¶

Dependency Management¶

Registry Authentication¶

Configuring the REST API¶

Configuring the Worker(s)¶

Regenerating Bundle Images¶

Messaging¶

Gating Bundle Images¶

Read the Docs Documentation¶

Build the Docs¶

Expanding the Docs¶

Modules Documentation¶

iib¶

Subpackages¶

iib.web¶

Submodules¶

iib.web.api_v1 module¶

iib.web.app module¶

iib.web.auth module¶

iib.web.config module¶

iib.web.docs module¶

iib.web.errors module¶

iib.web.manage module¶

iib.web.models module¶

iib.web.utils module¶

iib.web.wsgi module¶

Module contents¶

iib.workers¶

Subpackages¶

Submodules¶

iib.workers.api_utils module¶

iib.workers.config module¶

Module contents¶

Submodules¶

iib.exceptions module¶

Module contents¶

Indices and tables¶