Category

Blog

Bringing OCI images to the desktop with Flatpak

By | Blog

By Alex Larsson and Owen Taylor

Over the last five years, containers have taken the server world by storm. Many of the same things that make containers well-suited for server-side computing — the ability to test code an environment that is very similar to the deployment environment, the ability to upgrade application software independently from the host operating system, the ability to deploy applications across multiple host operating systems — also make a lot of sense for desktop applications.

At the same time that container technologies were emerging on the server, the Flatpak project was being developed by a community of contributors, including engineers from Red Hat, Collabora, and Endless Mobile, as a way to improve application deployment on desktop Linux, and allow application authors to make their applications available directly to users. During the evolution of Flatpak, leading up to a 1.0 release in August 2018, it’s been possible to share technologies with server-side containers, from namespaces, to seccomp, to the OCI Image format.

Containers for the Desktop

You might wonder if it would have been possible to go one step further and use a existing server-side runtime, such as runc, to run containers on the desktop. While it is possible to use server containers for desktop applications and get basic functionality working, the desktop world is pretty different from the server world. Instead of integrating with storage-area-networks, network routing, and orchestration, a desktop application deals with USB input devices, geolocation, and desktop environment application menus. Server-side technology might have the ability to provide access to a device, or block it off, but can’t meaningfully handle interactively working with the user to establish fine-grained access control.

For this reason, Flatpak doesn’t use runc, but instead has its own runtime that runs within the user’s desktop session and provides services (called portals) that allow applications to access the desktop under the user’s control. A portal is a service, exposed via the D-Bus IPC protocol, that sits between the application and the resource the application wants to access (local files, printing, geolocation, etc.) and provides a user interface to let the user decide whether to allow access or not. Typically this is not done as a Yes/No question about permissions but instead as a natural part of the operation. For example, instead of asking a user “Allow application X to access your files?”, the user is shown a file selection dialog, and they can either pick a file to pass back to the application, or they can cancel the operation.

In addition to a unique desktop-focused security model, Flatpak has an approach to combining operating system and application content into a single container that was inspired by the requirements of the desktop. The typical model for a server container is that a base image is arbitrarily modified to create the application container. Each application container is its own mini-operating system, and in order to fix a bug or security hole in the base operating system, the application has to be rebuilt and deployed. The OCI layer system potentially optimizes the deployment step, but the rebuilds are still necessary, and production images are often “squashed” for maximum efficiency.

The downsides of having every application independent are minimized in the server environment: we usually have a small number of applications running on a node with abundant disk space and network bandwidth, and hopefully have automation to automatically rebuild applications as necessary, as well as paid sysadmins. On a desktop, we might instead have dozens or hundreds of applications installed on a much more modest device, maintained by individual users. We also don’t want software vendors to have to rebuild their application in order to pick up a fix to the base operating system.

For this reason, when a Flatpak application is executed, two separate filesystems are mounted in its environment – the runtime filesystem is mounted at the path /usr, and the application filesystem is mounted at /app. Library and other search paths in the applications execution environment are set up to search both directories so that code and resources can be bundled with the runtime or with the application. This way a single runtime can be reused by many applications, and can be updated without having to modify applications. Different applications can use different runtimes, so some applications might use a runtime that is maintained for long-term stability with few changes, and other applications might use a runtime that gets more rapid releases to pick up new library versions.

Flatpaks as OCI Images

The native image format for Flatpaks is OSTree. It is a local storage format that automatically supports deduplication and versioning. It also naturally comes with a distribution framework, which most Flatpak repositories use. However, that is not the only distribution mechanism Flatpak supports.

Organizations seldom deploy only servers or only desktops. Having a unified way to distribute desktop applications and server applications can be highly desirable for sysadmins: they don’t really want to maintain both an OSTree repository, and a OCI registry (for servers). Luckily, the OCI format is sufficiently flexible that it is also suitable for storing desktop applications. Flatpak supports installing applications and runtimes from OCI images and a Flatpak remote can either be an ostree repository or a tentative version of the OCI registry (distribution spec). At the end-user level the difference is invisible and all available sources of applications are integrated together and displayed to the user.

One of the basic advantages that the OCI Image format has over older image formats is the concept of annotations – in addition to a compressed filesystem and standard metadata such as the operating system, architecture, and author of the image, an OCI Image provides a set of arbitrary key/value pairs. When we’re storing a Flatpak as an OCI image, these annotations are used to store information like the permissions that the Flatpak requires and its installed size.

One thing that is still actively under development is browsing available Flatpaks. Typing ‘docker pull postgresql:latest’ may be a good user interface for the server command line, but  a desktop user typically wants a nice user interface with icons, human readable names, and a user-friendly description. These things can be stored in the OCI annotations of the individual images, but it’s also necessary to be able to efficiently download the information for all the images in the registry, without having to download each individual image. Currently, Flatpak supports a draft metadata format and protocol for this. The Flagstate server allows adding this capability to an existing registry. Browsing and searching available images in a registry is useful beyond the desktop, so perhaps this is something that future versions of the OCI distribution specification can address.

As Flatpaks become more commonly used as a way to distribute desktop applications, users can benefit from an expanded set of available applications, with more robust upgrades and enhanced security. Using OCI Images and the OCI distribution mechanism as a deployment technology enables sysadmins to have a unified way of managing and distributing server side and desktop applications within their organization.

Hundreds of applications from Inkscape and Blender, to LibreOffice, to SuperTuxKart, are already available as Flatpaks. Flatpak is installed by default on current versions of some Linux distributions and can be easy to install on most others. Instructions for getting started can be found on flatpak.org.


Alex Larsson is a Senior Principal Software Engineer at Red Hat and the creator of Flatpak.

Owen Taylor is a Principal Software Engineer at Red Hat, and architect for Red Hat’s desktop and workstation engineering team.

OCI Image Support Comes to Open Source Docker Registry

By | Blog

By Phil Estes & Mike Brown

The Open Container Initiative (OCI) was formed in 2015 as a place to collaborate on the definition of a standard container runtime and image format. By that time, Docker had effectively become the de facto standard for container images, and DockerHub and many other public and private registries were filled with tens of thousands of available container images using the Docker image format.

Given this state of the world in late 2015, the OCI image specification work began in earnest with a strong group of collaborating independent and vendor-associated participants, using the Docker v2.2 image format as a starting point. Fast forward eighteen months, and by summer of 2017, both the runtime and image specifications reached their intended 1.0 milestone release.

That release was a great milestone for the OCI community and container ecosystem at large, but the next step beyond declaring victory on any specification is always: adoption! The runtime specification had a head start here. The default reference implementation within the OCI, runc, was already in use by several implementers including the Docker engine by the time the specification was released. For the image specification, tools, and most notably, container image registries would have to adopt the OCI v1.0 image specification specifically for it to be useful to developers and implementers. Many tools existed to operate on Docker’s v2.2 image manifest format already, and these tools would now need to adopt the OCI format.

This work—enabling OCI images in the core registry used on a daily basis for millions of images—started in 2016 on the Docker distribution project long before the specification even reached 1.0. This open source project is probably better known as the registry that backs  DockerHub as well as many other public and private registries. Now that this pull request has been merged, we’ll take a few minutes to describe a bit more about the OCI image spec and how the open source registry was modified to support OCI in addition to its native Docker image formats.

Background

Container registries today are usually the combination of an HTTP API backed by some form of content store, with the content itself delineated by various media types. In the Docker image specification, you have metadata types (like the image’s Docker runtime configuration) as JSON content, combined with references to image layers, usually tarred and compressed binary blobs which are then stored in the backing filesystem along with their requisite media types. For example, in the Docker v2.2 image world, each container image will have a manifest with the media type application/vnd.docker.distribution.manifest.v2+json. A list of image references (to support multi-platform images) is known as a manifest list with media type: application/vnd.docker.distribution.manifest.list.v2+json.  An image layer commonly has the media type of application/vnd.docker.image.rootfs.diff.tar.gzip.

The OCI specification took this v2.2 image specification from Docker as a starting point, but as definitions were changed in small ways during the specification process, each one of the Docker image metadata or layer media types developed into an OCI counterpart. Given these efforts, the resulting official OCI media types include application/vnd.oci.image.manifest.v1+json for the image manifest. Manifest lists have been renamed to “indexes” in OCI v1, giving us the media type: application/vnd.oci.image.index.v1+json. A layer type becomes application/vnd.oci.image.layer.v1.tar+gzip in OCI parlance, and so on.

Implementation

To add OCI v1 support into the open source distribution project meant handling all the new media types from the OCI specification, and appropriately handling the HTTP API interactions when a client of the registry wants to “speak” in OCI media-types versus the already supported Docker types. If you think of image manifests being the top-most objects in the registry’s world, then effectively the registry was currently supporting three: the original “schema 1” Docker image format; the “schema 2” single image Docker format, and the manifest list multi-platform Docker image format (part of the schema 2.2 definition, but treated as a separate object type). Adding the OCI v1 types would mean supporting OCI v1 manifests and indexes—two additional formats, which would cascade into all the OCI media type references from these high level manifests and indexes.

The first attempt looked at simply expanding the registry’s handlers for the schema 2 and manifest list types to support OCI v1 manifests and OCI v1 indexes. These two types are quite similar between Docker v2.2 and OCI v1, so it seemed like it might be a quick path to getting support for OCI into the registry codebase.  However, over time it became clear that this would not be an optimal path for long-term OCI image support.

Success

Finally, the approach presented in GitHub pull request #2076 –to add specific new handlers to the open source  registry for the OCI main image types (v1 manifests and v1 indexes)—was agreed on and in July of 2018 this PR was merged into the docker/distribution project. This PR adds new handlers for the OCI types as well as new tests, validation code, and integration that crosses various core sections of the registry codebase. As with most PRs that add significant capability, it was not an easy task, and Mike Brown from IBM persisted on the project for over a year before working out all the requested issues, reviews, and bugs found during testing and validation of the OCI support!

Summary

Interested parties can try out the recently available v2.7.0 release candidate with OCI v1 image support. We expect after final release that registries based on the open source distribution, including DockerHub, will update in the near future to adopt these new features. At that point, client tools which today already support the OCI image formats will have interoperability with these registries thanks to the hard work of many involved from Docker, to the OCI, to participating vendors and contributors.

In addition to the growing adoption of the OCI v1 image formats—used natively in other projects like the CNCF containerd project as well as the Moby project’s LinuxKit implementation—we’re also excited about the standardization of the registry API itself coming to OCI this year. The proposal to take the Docker HTTP registry API has already been accepted into the OCI. This specification will add industry standardization around the protocol to talk to registries in addition to the existing interoperability brought in with the OCI v1 image specification.

It’s great to see growing adoption of the OCI specifications, and with the added support in the open source Docker registry for OCI images, we see this as just the beginning of a whole host of tools, vendor products, and software that will be enabled to utilize the OCI specifications now and in the future.

PouchContainer: How OCI Specifications Power Alibaba

By | Blog

By Allen Sun, Alibaba Group

PouchContainer is an open source container project created by Alibaba Group to be enterprise ready and promote OCI container standards. The project is a fundamental piece of software in Alibaba’s infrastructure, it helps process transactions smoothly on millions of containers.

To become a general container engine for every scenario in production, PouchContainer seeks ways to support several OCI-compatible container runtimes. This action makes container service totally out of box:

* runc: container runtime based on Linux cgroups and namespaces;
* katacontainers: container runtime based on hypervisor; and
* runlxc: container runtime based on LXC especially on legacy kernels.

Architecture Based on OCI and Open Source Components


Three OCI-compatible runtimes are listed in the middle right part of architecture.

Features

PouchContainer’s most important features are:

  • Rich container: Besides the common ways of running container, PouchContainer includes a rich container mode, which integrates more services, hooks, and many others container internals to guarantee containers running like usual.
  • Strong isolation: PouchContainer is designed to be secure by default. It includes lots of security features, like hypervisor-based container technology, lxcfs, directory disk quota, patched Linux kernel and so on.
  • P2P distribution: PouchContainer utilizes Dragonfly, a P2P-based distribution system, to achieve lightning-fast container image distribution.
  • Kernel compatibility: Enables OCI-compatible runtimes to work on old kernel versions, like linux kernel 2.6.32+.
  • Standard compatibility: PouchContainer keeps embracing container ecosystem to support industry specifications such as CNI, CSI and so on.
  • Kubernetes Native: PouchContainer has natively implemented Kubernetes Container Runtime Interface (CRI).

Learn more about PouchContainer

PouchContainer brings many additional features to end-users. Want to learn more? Please visit the PouchContainer GitHub, where the PouchContainer community is currently busy preparing the 1.0.0 GA release.

CRI-O: How Standards Power a Container Runtime

By | Blog

By Joe Brockmeier, Red Hat

The CRI-O project (part of the former Kubernetes incubator) is busy working on the upcoming 1.11 release, which will be released in conjunction with the Kubernetes 1.11 release. It will have some interesting new features, but won’t lose sight of its stated No. 1 goal: to never break Kubernetes. Parallel to that goal is to run any OCI image from any registry (when the OCI distribution specification is finalized).

Historically, Kubernetes has worked with container runtimes that were designed to do many things: build container images, manage container security, manage container orchestration, inspect container images, etc. CRI-O, on the other hand, was designed just to support the functions Kubernetes needs to actually run containers.

Depending on Standards

CRI-O moves in lock-step with Kubernetes’ Container Runtime Interface (CRI), the API for container runtimes to integrate with a kubelet. CRI-O is aligned with the upstream Kubernetes releases, so any changes to the CRI in Kubernetes are supported in the matching release of CRI-O for that release. For example, the most recent CRI-O 1.10 release matches Kubernetes 1.10. CRI-O 1.11 will release with Kubernetes 1.11, and so forth.

Most users these days are using Kubernetes with a version of Docker, but some organizations with different business needs might want to use new container types that haven’t been implemented yet, or others like Kata Containers. CRI-O opens the door for this by supporting any OCI images and runtimes.

Here’s how it works:

  • Kubernetes asks the kubelet to start a pod
  • The kubelet talks to the CRI-O daemon using the CRI
  • CRI-O uses a library that implements the OCI standard to pull the image from a registry
  • CRI-O uses another standard library to unpack the container image for use
  • CRI-O then generates a JSON file that describes how the container is to be run
  • Next, CRI-O launches an OCI-compatible runtime (currently runc or the Clear Containers runtime) to run the container processes
  • A common process handles logging for the container and monitors the process

You might also be wondering about networking. Again, the idea is to have flexibility within a standard interface, so CRI-O uses the Container Networking Interface (CNI) to set up networking. Any CNI plugin can be used with CRI-O, giving users flexibility over their container networking stack as well.

CRI-O helps achieve what the OCI specifications and CRI API set out to do – make the container runtime an implementation detail that the end user doesn’t have to worry about. Worry about how Kubernetes works with your application, not how Kubernetes works with the container runtime.

Learning More about CRI-O

Want to learn more about CRI-O? Of course you do! For now the best resources on CRI-O are the README on GitHub, the accompanying tutorial, and be sure to watch the CRI-O blog.

OCI Member Spotlight: OpenStack (Kata Containers)

By | Blog

The OCI community is comprised of a diverse set of member organizations that are committed to creating open industry standards around a container image format and runtime. This blog series highlights OCI members and their contributions to building an open, portable and vendor neutral specification.

Name: Xu Wang
Title: CTO + Kata Architecture Committee member
Company: Hyper.sh + Kata Containers

What is Kata Containers?

Kata Containers is an open source project hosted by the OpenStack Foundation that provides lightweight virtual machines that feel and perform like containers, while providing the workload isolation and security advantages of traditional VMs.

Why did OpenStack (Kata Containers) join OCI?

The Kata Containers project runs containers specified by OCI runtime spec in virtual machines. We joined OCI to guarantee the compatibility between Kata Containers and OCI runtime specs, and to help to improve the OCI specifications – enabling more efficient Kata Containers. We look forward to collaborating around tooling, compatibility and interoperability testing.

How can OCI community members contribute to Kata Containers? 

Many of the Kata community members come from the OCI community, so we look forward to more collaboration on use case sharing, specification discussion, testing and toolchains. See Github for more information: https://github.com/kata-containers/community

How do you anticipate OCI changing the container technology landscape? 

The OCI specs guarantee that container technology can be open, vendor neutral and a cornerstone of future computing infrastructure.

What is the benefit of open standards like OCI for users of Kata Containers?

The open and unified container spec gives users more options and helps Kata to be adopted in more cases.

More and more applications are shipped and run under OCI specs. With OCI, Kata could enable users to launch unified container applications no matter if the runtime isolation technology is namespace or VM.

For cloud providers, if a user application has been developed for OCI specs, they could run the application with Kata Containers directly which introduces fewer layers than running a container orchestration system on a VM pool.

On the other hand, for those users who have already invested in VM technology, they could apply their existing tools to Kata Containers and move their application to microservice with OCI containers.

Teaming up with Docker to Support a Diverse Container Ecosystem

By | Blog

With a commitment to driving inclusivity in the community, OCI is proud to be an official Diversity Scholarship sponsor for DockerCon 2018.

By actively seeking ways to increase the ecosystem’s diversity, OCI + Docker’s collective goal is to make DockerCon a safe place for everyone to learn and collaborate.

The scholarship program will provide under represented members across the global container community with a scholarship to attend the annual event.

To learn more, make sure to check out the selection process, scholarship details and requirements below + don’t forget to submit an application by Wednesday, April 25, 2018 at 5:00PM PST!

Apply Now!

Selection Process:

A committee of Docker community members will review and select the scholarship recipients. Recipients will be notified by the week of May 7, 2018

What’s included:

Full Access DockerCon Conference Pass

Requirements:

Must be able to attend DockerCon US 2018

Must be 18 years old or older to apply

OCI Announces 2018 TOB Election Results

By | Blog

By Chris Aniszczyk (@cra)

It’s 2018 and after shipping the v1.0 specifications last year, we’re nearing contributions from 300+ contributors across 70+ organizations. Today the community is hard at work on future versions of OCI, including discussions around a distribution API specification. For more details about releases and specifications in development, please visit  https://github.com/opencontainers.

Also, the OCI Technical Oversight Board (TOB) – comprised of independently elected individuals who provide oversight of the technical leadership and serve as a point of appeal – just elected the following five board members to each serve a two-year term, effective immediately:

  • Taylor Brown (Microsoft)
  • Stephen Day (Docker)
  • Phil Estes (IBM)
  • Jon Johnson (Google)
  • Mrunal Patel (Red Hat)

These newest TOB members join the following existing members, who are each in the middle of a two-year term:

  • Vincent Batts (Red Hat)
  • Michael Crosby (Docker)
  • Vishnu Kannan (Google)
  • Greg Kroah-Hartman (Linux Foundation)

The TOB also voted to elect Michael Crosby (Docker) as the new Chair. This new TOB lineup is responsible for adding, removing or reorganizing OCI Projects – to learn more, you can follow the TOB on GitHub here.

I’d also like to extend a big thank you to all of our outgoing TOB members – Chris Wright, Diogo Mónica, Jason Bouzane, John Gossman and former Chair, Brandon Philips – for their service and commitment. We look forward to your continued collaboration with and participation in OCI!

Next month, new TOB member Phil Estes of IBM will represent OCI on a panel at Container World on 2/28. Visit http://bit.ly/2Bsr5Gl to learn more about his involvement in this session!

As always, we welcome any/all contributions from the container community – as our success banks on the support and collaboration of many. If you’re interested in contributing to OCI, please join the OCI developer community. For those who are building products on OCI technology, we recommend joining as a member and participating in the future certification program.

Between our exciting new TOB, upcoming releases and a number of exciting community programs we have in the works, we’re on track to make 2018 our biggest year yet 👍🏼

OCI Member Spotlight: Kontena

By | Blog

The OCI community is comprised of a diverse set of member organizations that are committed to creating open industry standards around a container image format and runtime. This blog series highlights OCI members and their contributions to building an open, portable and vendor neutral specification.

Name: Miska Kaipiainen
Title: Founder & CEO
Company: Kontena, Inc.

Why did you join OCI?
We see OCI as the most important organization in the container ecosystem driving vendor neutrality, standardization and making this amazing technology accessible globally. Just like many other companies working with containers, we are strong believers in open source, open APIs and open ecosystems in general. We joined OCI to contribute to its mission which helps us and everyone else in this industry to be successful.

How is your organization involved in OCI?
We plan to take an active role in collaboratively on improving OCI specifications, tooling and look forward to being more engaged with other members.

What are the aspects of the runtime spec and/or image format spec that you are looking forward to most for your company?
We enjoy vendor neutrality and collaboration for developing container runtimes that are based on open standards.

How do you plan to use the runtime spec and/or image format spec?
We want to incorporate and support the runtime spec in our Kontena Platform open source project, via runC. Soon, we also plan to support the image spec in our own hosted image registry service.

How will these specifications help your business?
These specifications will provide our users with the confidence to get started with containers while having the promise of no vendor lock-in. In addition, these specifications have stabilized some of the core technology components we use.

How do you anticipate OCI changing the container technology landscape?
OCI has already shown its ability to create solid standards and specifications for an industry that’s moving at the speed of light. This community’s work has already produced amazing results through increased interoperability, recognition and adoption. However, there are still so many things we can accomplish with collaboration to help ensure open standards, interoperability and vendor neutrality.

What do you believe the benefits of using a runtime and image spec based on the OCI standard are for hosting providers? For small ISVs, application developers? For end users?
The benefit of using a runtime and image spec based on OCI is the ability to develop and package once, distribute, deploy and run anywhere. We predict that more vendors will be bringing out runtimes that do their own magic to benefit their specific end users. We’ve always been believers in this market not being a “one solution for all” type of situation, as there are a multitude of users, use cases and needs that no one solution can single handedly meet. Open APIs and standards will enable more vendors to build meaningful solutions that will then benefit the end user with more choice and less lock-in through easier interchangeability of components.

What advice would you give to someone considering joining OCI?
Get on-board and help shape the industry from the inside rather than staying out!

OCI Update: v1.0.1 Release and New Maintainer

By | Blog

The OCI community continues to be hard at work, having just issued the first update to OCI v.1.0, after five months of focusing on stability. OCI 1.0.1 contains updates to both the image format and runtime specifications.

We’re still growing and expanding, with even more collaboration since the launch of v 1.0. For example, we are now up to over 5,000 commits from 184 authors across 42 different organizations. Organizations like AWS, Docker, Cloud Foundry, CoreOS, Intel, Mesosphere, Oracle, Red Hat and Kubernetes have already taken advantage of the OCI v1.0 specifications, and with v1.0.1 now available, the industry is on the precipice of true portability and standardization. We had a strong showing on site at recent industry events, at both DockerCon Europe in Copenhagen and Open Source Summit Europe in Prague.

We are also happy to welcome Ma Shimiao as a new maintainer on the image-tools project. Ma joins Aleksa Sarai and Keyang Xie of the image format spec as the newest OCI maintainers, all of whom has been hard at work.  

Concurrently, we are gearing up for the next phase in ensuring broad adoption of common container image format and runtime specs as we prepare to launch an OCI certification/conformance program. This program will allow folks to be confident that their OCI solutions meet a high set of criteria that deliver interoperable solutions. We’ll be presenting a session on Container Runtime & Image Format Standards: What it Means to be ‘OCI-certified during CloudNatvieCon + KubeCon in Austin, Texas on Wednesday, December 6 at 11:10 am. If you’ll be on site, please stop by our booth and check the schedule for additional OCI-related sessions

OCI is always welcoming contributions from across the industry, so please join us! Follow us via @OCI_ORG, and if you’re interested in contributing to the technology, please join the OCI developer community which is open to everyone. If you’re building products on OCI technology, we recommend joining as a member and participating in the upcoming certification program.