All Posts By

Open Container Initiative

Bringing OCI images to the desktop with Flatpak

By | Blog

By Alex Larsson and Owen Taylor

Over the last five years, containers have taken the server world by storm. Many of the same things that make containers well-suited for server-side computing — the ability to test code an environment that is very similar to the deployment environment, the ability to upgrade application software independently from the host operating system, the ability to deploy applications across multiple host operating systems — also make a lot of sense for desktop applications.

At the same time that container technologies were emerging on the server, the Flatpak project was being developed by a community of contributors, including engineers from Red Hat, Collabora, and Endless Mobile, as a way to improve application deployment on desktop Linux, and allow application authors to make their applications available directly to users. During the evolution of Flatpak, leading up to a 1.0 release in August 2018, it’s been possible to share technologies with server-side containers, from namespaces, to seccomp, to the OCI Image format.

Containers for the Desktop

You might wonder if it would have been possible to go one step further and use a existing server-side runtime, such as runc, to run containers on the desktop. While it is possible to use server containers for desktop applications and get basic functionality working, the desktop world is pretty different from the server world. Instead of integrating with storage-area-networks, network routing, and orchestration, a desktop application deals with USB input devices, geolocation, and desktop environment application menus. Server-side technology might have the ability to provide access to a device, or block it off, but can’t meaningfully handle interactively working with the user to establish fine-grained access control.

For this reason, Flatpak doesn’t use runc, but instead has its own runtime that runs within the user’s desktop session and provides services (called portals) that allow applications to access the desktop under the user’s control. A portal is a service, exposed via the D-Bus IPC protocol, that sits between the application and the resource the application wants to access (local files, printing, geolocation, etc.) and provides a user interface to let the user decide whether to allow access or not. Typically this is not done as a Yes/No question about permissions but instead as a natural part of the operation. For example, instead of asking a user “Allow application X to access your files?”, the user is shown a file selection dialog, and they can either pick a file to pass back to the application, or they can cancel the operation.

In addition to a unique desktop-focused security model, Flatpak has an approach to combining operating system and application content into a single container that was inspired by the requirements of the desktop. The typical model for a server container is that a base image is arbitrarily modified to create the application container. Each application container is its own mini-operating system, and in order to fix a bug or security hole in the base operating system, the application has to be rebuilt and deployed. The OCI layer system potentially optimizes the deployment step, but the rebuilds are still necessary, and production images are often “squashed” for maximum efficiency.

The downsides of having every application independent are minimized in the server environment: we usually have a small number of applications running on a node with abundant disk space and network bandwidth, and hopefully have automation to automatically rebuild applications as necessary, as well as paid sysadmins. On a desktop, we might instead have dozens or hundreds of applications installed on a much more modest device, maintained by individual users. We also don’t want software vendors to have to rebuild their application in order to pick up a fix to the base operating system.

For this reason, when a Flatpak application is executed, two separate filesystems are mounted in its environment – the runtime filesystem is mounted at the path /usr, and the application filesystem is mounted at /app. Library and other search paths in the applications execution environment are set up to search both directories so that code and resources can be bundled with the runtime or with the application. This way a single runtime can be reused by many applications, and can be updated without having to modify applications. Different applications can use different runtimes, so some applications might use a runtime that is maintained for long-term stability with few changes, and other applications might use a runtime that gets more rapid releases to pick up new library versions.

Flatpaks as OCI Images

The native image format for Flatpaks is OSTree. It is a local storage format that automatically supports deduplication and versioning. It also naturally comes with a distribution framework, which most Flatpak repositories use. However, that is not the only distribution mechanism Flatpak supports.

Organizations seldom deploy only servers or only desktops. Having a unified way to distribute desktop applications and server applications can be highly desirable for sysadmins: they don’t really want to maintain both an OSTree repository, and a OCI registry (for servers). Luckily, the OCI format is sufficiently flexible that it is also suitable for storing desktop applications. Flatpak supports installing applications and runtimes from OCI images and a Flatpak remote can either be an ostree repository or a tentative version of the OCI registry (distribution spec). At the end-user level the difference is invisible and all available sources of applications are integrated together and displayed to the user.

One of the basic advantages that the OCI Image format has over older image formats is the concept of annotations – in addition to a compressed filesystem and standard metadata such as the operating system, architecture, and author of the image, an OCI Image provides a set of arbitrary key/value pairs. When we’re storing a Flatpak as an OCI image, these annotations are used to store information like the permissions that the Flatpak requires and its installed size.

One thing that is still actively under development is browsing available Flatpaks. Typing ‘docker pull postgresql:latest’ may be a good user interface for the server command line, but  a desktop user typically wants a nice user interface with icons, human readable names, and a user-friendly description. These things can be stored in the OCI annotations of the individual images, but it’s also necessary to be able to efficiently download the information for all the images in the registry, without having to download each individual image. Currently, Flatpak supports a draft metadata format and protocol for this. The Flagstate server allows adding this capability to an existing registry. Browsing and searching available images in a registry is useful beyond the desktop, so perhaps this is something that future versions of the OCI distribution specification can address.

As Flatpaks become more commonly used as a way to distribute desktop applications, users can benefit from an expanded set of available applications, with more robust upgrades and enhanced security. Using OCI Images and the OCI distribution mechanism as a deployment technology enables sysadmins to have a unified way of managing and distributing server side and desktop applications within their organization.

Hundreds of applications from Inkscape and Blender, to LibreOffice, to SuperTuxKart, are already available as Flatpaks. Flatpak is installed by default on current versions of some Linux distributions and can be easy to install on most others. Instructions for getting started can be found on flatpak.org.


Alex Larsson is a Senior Principal Software Engineer at Red Hat and the creator of Flatpak.

Owen Taylor is a Principal Software Engineer at Red Hat, and architect for Red Hat’s desktop and workstation engineering team.

OCI Image Support Comes to Open Source Docker Registry

By | Blog

By Phil Estes & Mike Brown

The Open Container Initiative (OCI) was formed in 2015 as a place to collaborate on the definition of a standard container runtime and image format. By that time, Docker had effectively become the de facto standard for container images, and DockerHub and many other public and private registries were filled with tens of thousands of available container images using the Docker image format.

Given this state of the world in late 2015, the OCI image specification work began in earnest with a strong group of collaborating independent and vendor-associated participants, using the Docker v2.2 image format as a starting point. Fast forward eighteen months, and by summer of 2017, both the runtime and image specifications reached their intended 1.0 milestone release.

That release was a great milestone for the OCI community and container ecosystem at large, but the next step beyond declaring victory on any specification is always: adoption! The runtime specification had a head start here. The default reference implementation within the OCI, runc, was already in use by several implementers including the Docker engine by the time the specification was released. For the image specification, tools, and most notably, container image registries would have to adopt the OCI v1.0 image specification specifically for it to be useful to developers and implementers. Many tools existed to operate on Docker’s v2.2 image manifest format already, and these tools would now need to adopt the OCI format.

This work—enabling OCI images in the core registry used on a daily basis for millions of images—started in 2016 on the Docker distribution project long before the specification even reached 1.0. This open source project is probably better known as the registry that backs  DockerHub as well as many other public and private registries. Now that this pull request has been merged, we’ll take a few minutes to describe a bit more about the OCI image spec and how the open source registry was modified to support OCI in addition to its native Docker image formats.

Background

Container registries today are usually the combination of an HTTP API backed by some form of content store, with the content itself delineated by various media types. In the Docker image specification, you have metadata types (like the image’s Docker runtime configuration) as JSON content, combined with references to image layers, usually tarred and compressed binary blobs which are then stored in the backing filesystem along with their requisite media types. For example, in the Docker v2.2 image world, each container image will have a manifest with the media type application/vnd.docker.distribution.manifest.v2+json. A list of image references (to support multi-platform images) is known as a manifest list with media type: application/vnd.docker.distribution.manifest.list.v2+json.  An image layer commonly has the media type of application/vnd.docker.image.rootfs.diff.tar.gzip.

The OCI specification took this v2.2 image specification from Docker as a starting point, but as definitions were changed in small ways during the specification process, each one of the Docker image metadata or layer media types developed into an OCI counterpart. Given these efforts, the resulting official OCI media types include application/vnd.oci.image.manifest.v1+json for the image manifest. Manifest lists have been renamed to “indexes” in OCI v1, giving us the media type: application/vnd.oci.image.index.v1+json. A layer type becomes application/vnd.oci.image.layer.v1.tar+gzip in OCI parlance, and so on.

Implementation

To add OCI v1 support into the open source distribution project meant handling all the new media types from the OCI specification, and appropriately handling the HTTP API interactions when a client of the registry wants to “speak” in OCI media-types versus the already supported Docker types. If you think of image manifests being the top-most objects in the registry’s world, then effectively the registry was currently supporting three: the original “schema 1” Docker image format; the “schema 2” single image Docker format, and the manifest list multi-platform Docker image format (part of the schema 2.2 definition, but treated as a separate object type). Adding the OCI v1 types would mean supporting OCI v1 manifests and indexes—two additional formats, which would cascade into all the OCI media type references from these high level manifests and indexes.

The first attempt looked at simply expanding the registry’s handlers for the schema 2 and manifest list types to support OCI v1 manifests and OCI v1 indexes. These two types are quite similar between Docker v2.2 and OCI v1, so it seemed like it might be a quick path to getting support for OCI into the registry codebase.  However, over time it became clear that this would not be an optimal path for long-term OCI image support.

Success

Finally, the approach presented in GitHub pull request #2076 –to add specific new handlers to the open source  registry for the OCI main image types (v1 manifests and v1 indexes)—was agreed on and in July of 2018 this PR was merged into the docker/distribution project. This PR adds new handlers for the OCI types as well as new tests, validation code, and integration that crosses various core sections of the registry codebase. As with most PRs that add significant capability, it was not an easy task, and Mike Brown from IBM persisted on the project for over a year before working out all the requested issues, reviews, and bugs found during testing and validation of the OCI support!

Summary

Interested parties can try out the recently available v2.7.0 release candidate with OCI v1 image support. We expect after final release that registries based on the open source distribution, including DockerHub, will update in the near future to adopt these new features. At that point, client tools which today already support the OCI image formats will have interoperability with these registries thanks to the hard work of many involved from Docker, to the OCI, to participating vendors and contributors.

In addition to the growing adoption of the OCI v1 image formats—used natively in other projects like the CNCF containerd project as well as the Moby project’s LinuxKit implementation—we’re also excited about the standardization of the registry API itself coming to OCI this year. The proposal to take the Docker HTTP registry API has already been accepted into the OCI. This specification will add industry standardization around the protocol to talk to registries in addition to the existing interoperability brought in with the OCI v1 image specification.

It’s great to see growing adoption of the OCI specifications, and with the added support in the open source Docker registry for OCI images, we see this as just the beginning of a whole host of tools, vendor products, and software that will be enabled to utilize the OCI specifications now and in the future.

PouchContainer: How OCI Specifications Power Alibaba

By | Blog

By Allen Sun, Alibaba Group

PouchContainer is an open source container project created by Alibaba Group to be enterprise ready and promote OCI container standards. The project is a fundamental piece of software in Alibaba’s infrastructure, it helps process transactions smoothly on millions of containers.

To become a general container engine for every scenario in production, PouchContainer seeks ways to support several OCI-compatible container runtimes. This action makes container service totally out of box:

* runc: container runtime based on Linux cgroups and namespaces;
* katacontainers: container runtime based on hypervisor; and
* runlxc: container runtime based on LXC especially on legacy kernels.

Architecture Based on OCI and Open Source Components


Three OCI-compatible runtimes are listed in the middle right part of architecture.

Features

PouchContainer’s most important features are:

  • Rich container: Besides the common ways of running container, PouchContainer includes a rich container mode, which integrates more services, hooks, and many others container internals to guarantee containers running like usual.
  • Strong isolation: PouchContainer is designed to be secure by default. It includes lots of security features, like hypervisor-based container technology, lxcfs, directory disk quota, patched Linux kernel and so on.
  • P2P distribution: PouchContainer utilizes Dragonfly, a P2P-based distribution system, to achieve lightning-fast container image distribution.
  • Kernel compatibility: Enables OCI-compatible runtimes to work on old kernel versions, like linux kernel 2.6.32+.
  • Standard compatibility: PouchContainer keeps embracing container ecosystem to support industry specifications such as CNI, CSI and so on.
  • Kubernetes Native: PouchContainer has natively implemented Kubernetes Container Runtime Interface (CRI).

Learn more about PouchContainer

PouchContainer brings many additional features to end-users. Want to learn more? Please visit the PouchContainer GitHub, where the PouchContainer community is currently busy preparing the 1.0.0 GA release.

OCI Member Spotlight: Kontena

By | Blog

The OCI community is comprised of a diverse set of member organizations that are committed to creating open industry standards around a container image format and runtime. This blog series highlights OCI members and their contributions to building an open, portable and vendor neutral specification.

Name: Miska Kaipiainen
Title: Founder & CEO
Company: Kontena, Inc.

Why did you join OCI?
We see OCI as the most important organization in the container ecosystem driving vendor neutrality, standardization and making this amazing technology accessible globally. Just like many other companies working with containers, we are strong believers in open source, open APIs and open ecosystems in general. We joined OCI to contribute to its mission which helps us and everyone else in this industry to be successful.

How is your organization involved in OCI?
We plan to take an active role in collaboratively on improving OCI specifications, tooling and look forward to being more engaged with other members.

What are the aspects of the runtime spec and/or image format spec that you are looking forward to most for your company?
We enjoy vendor neutrality and collaboration for developing container runtimes that are based on open standards.

How do you plan to use the runtime spec and/or image format spec?
We want to incorporate and support the runtime spec in our Kontena Platform open source project, via runC. Soon, we also plan to support the image spec in our own hosted image registry service.

How will these specifications help your business?
These specifications will provide our users with the confidence to get started with containers while having the promise of no vendor lock-in. In addition, these specifications have stabilized some of the core technology components we use.

How do you anticipate OCI changing the container technology landscape?
OCI has already shown its ability to create solid standards and specifications for an industry that’s moving at the speed of light. This community’s work has already produced amazing results through increased interoperability, recognition and adoption. However, there are still so many things we can accomplish with collaboration to help ensure open standards, interoperability and vendor neutrality.

What do you believe the benefits of using a runtime and image spec based on the OCI standard are for hosting providers? For small ISVs, application developers? For end users?
The benefit of using a runtime and image spec based on OCI is the ability to develop and package once, distribute, deploy and run anywhere. We predict that more vendors will be bringing out runtimes that do their own magic to benefit their specific end users. We’ve always been believers in this market not being a “one solution for all” type of situation, as there are a multitude of users, use cases and needs that no one solution can single handedly meet. Open APIs and standards will enable more vendors to build meaningful solutions that will then benefit the end user with more choice and less lock-in through easier interchangeability of components.

What advice would you give to someone considering joining OCI?
Get on-board and help shape the industry from the inside rather than staying out!

OCI Member Spotlight: InfoSiftr

By | Blog

The OCI community is comprised of a diverse set of member companies that are committed to creating open industry standards around a container image format and runtime. This blog series highlights OCI members and their contributions to building an open, portable and vendor neutral specification.  

Name: Tianon Gravi
Title: SVP of Operations
Company: InfoSiftr

Why did you join OCI and how is your organization involved?
The Open Container Initiative (OCI) is a critical open source organization helping ensure compatibility and interoperability for the basic components of containers. We believe in the mission of the OCI and, as contributors and maintainers in the container ecosystem, will continue to lend expertise and effort to further the group’s goals. On an individual level, I have been involved since the foundation of OCI and am a member of the OCI’s Technical Developer Community (TDC), having worked on the container runtime specifications since the foundation of the OCI.

What are the aspects of the the runtime spec and/or image format spec that you are looking forward to most for your company?
By encouraging standardization and interoperability, the specs help enable differentiation further up the stack, where it benefits all users.  Additionally, companies can feel safer in adopting containers knowing that at a foundational level, they aren’t locked into any one vendor’s solution (which is all thanks to these low-level standards).

How do you plan to use the runtime spec and/or image format spec?
Any development work we do in the container space will adhere to OCI specifications, and encourage their use and furtherance within the ecosystem.

How will these specifications help your business?
The specifications will help all businesses with an interest in containers, whether they are contributing to the container ecosystem or just consuming container-based technology.  This flexibility helps us in our own day-to-day operations, but also helps our clients as they evaluate and commit to container-based solutions and deployments.

How do you anticipate OCI changing the container technology landscape?  
The OCI’s standards create a “baseline for competition.”  Now that we have an agreed upon foundation, companies can proceed to innovate and compete in more interesting ways.

What do you believe the benefits of using a runtime and image spec based on the OCI standard are for hosting providers?  For small ISVs, application developers? For end users?
If you’re doing anything in technology, it’s useful to adhere to standards. Much of the container ecosystem is still new and evolving, and we’re bound to see changes to technology stacks going forward. Adhering to the specifications set forth by the OCI means we all agree the underlying details of our tooling and products will be compatible and interoperable, whatever higher-level changes may come.

What advice would you give to someone considering joining OCI?
Contributing to the underlying specifications of the container ecosystem is sometimes difficult and involved work, but it can be very rewarding. Joining the OCI is a great way to start doing exactly that: https://www.opencontainers.org/join

 

 

OCI Member Spotlight: EasyStack

By | Blog

The OCI community is comprised of a diverse set of member companies that are committed to creating open industry standards around a container image format and runtime. This blog series highlights OCI members and their contributions to building an open, portable and vendor neutral specification.  

Name: Guohui Liu
Title: Co-founder and CTO
Company: EasyStack

Why did you join OCI?
Trends in cloud native computing are still on the rise. Businesses are evolving faster than ever before as applications require more agile and scalable infrastructure, and container technology is right at the center of it all. Unified standards are imperative for container technology to evolve and develop, with input and participation from the entire ecosystem. We believe the Open Container Initiative (OCI), with its runtime and image format specifications, plays a leading role in actualizing the ability to “package once, run anywhere” for applications.

Openness is a part of EasyStack’s DNA. We believe container technology will fill the gap between traditional IT and cloud native IT. We are among the first to release a converged infrastructure cloud platform featuring both containers and OpenStack cloud, and are looking forward to sharing our rich experiences with enterprise cloud native infrastructure offerings with the global OCI community.

How is your organization involved in OCI?
EasyStack is an active participant in the broader open source community, and we believe it is equally important to contribute to upstream open source projects to help address enterprise customer needs. Today — based on OpenStack, Kubernetes, Docker, Ceph and other open source technologies– EasyStack provides an open, secure, stable, reliable, and high-performance cloud computing capability for 200+ enterprise customers for their cloud infrastructure. As part of OCI, we are sharing our enterprise cloud and application model experience with the upstream community.  

What are the aspects of the runtime spec and/or image format spec that you are looking forward to most for your company?
The runtime spec and image format spec provide container standards, which is very important to both vendors and users. Trends in cloud native are encouraging more and more enterprise users to deploy containers, and many of them are already in production. This means reliability and consistent upgrades are extremely important. Additionally, API compatibility and scalability is imperative, especially in customized enterprise scenarios.

How do you plan to use the runtime spec and/or image format spec?
We leverage OCI v1.0 the runtime spec in our ESContainer Linux already, providing a reliable and stable container runtime. This allows us to focus on the development of true application-oriented orchestration, leveraging additional technologies such as Kubernetes.

How will these specifications help your business?
These specifications provide standardized and reliable image format and runtime specifications, allowing us to better and more efficiently develop ESContainer, shortening the development cycle. With these standards, the infrastructure provided by different vendors is of the same API so this avoids vendor lock-in, which helps our business to reduce risk (as well as costs, which in turn yields higher profits).  

What do you believe the benefits of using a runtime and image spec based on the OCI standard are for hosting providers?  For small ISVs, application developers? For end users?
OCI’s runtime and image format specs help all types of customers. The use of one standardized spec enables container applications to be deployed, run, and updated in a multi-cloud environment (e.g. public cloud, private cloud or community cloud) more quickly and efficiently. This is extremely helpful in today’s hybrid cloud environment and both the hosting providers and the end users will benefit from it.

ISVs and application developers now have standards to follow, which can help reduce the risk of going in the wrong direction while avoiding vendor lock-in as they build container applications.

What advice would you give to someone considering joining OCI?
If you and/or your organization’s vision align with current cloud native trends, just join OCI. Today, it is almost impossible for one company to really dominate one hot technology and at the same time keep it active and advanced. Open technologies have proven successful for rapid development, and OCI is a great example of this. New members to the OCI community can get started quickly with shared resources and support from across the community, in addition to having fun by contributing back.