Update README.md and documentation
Signed-off-by: Arnaud Porterie <arnaud.porterie@docker.com>master
Arnaud Porterie
parent
83b810104c
commit
e1eeec3e2f
55
README.md
55
README.md
|
|
@ -1,4 +1,55 @@
|
||||||
distribution
|
Distribution
|
||||||
============
|
============
|
||||||
|
|
||||||
The Docker toolset to pack, ship, store, and deliver content
|
The Docker toolset to pack, ship, store, and deliver content.
|
||||||
|
|
||||||
|
Planned content for this repository:
|
||||||
|
|
||||||
|
* Distribution related specifications
|
||||||
|
- Image format
|
||||||
|
- JSON registry API
|
||||||
|
* Registry implementation: a Golang implementation of the JSON API
|
||||||
|
* Client libraries to consume conforming implementations of the JSON API
|
||||||
|
|
||||||
|
# Ongoing open sprint
|
||||||
|
|
||||||
|
### What is an open sprint?
|
||||||
|
|
||||||
|
The open sprint is a focused effort of a small group of people to kick-off a new project, while commiting to becoming maintainers of the resulting work.
|
||||||
|
|
||||||
|
**Having a dedicated team work on the subject doesn't mean that you, the community, cannot contribute!** We need your input to make the best use of the sprint, and focus our work on what matters for you. For this particular topic:
|
||||||
|
|
||||||
|
* Come discuss on IRC: #docker-distribution on FreeNode
|
||||||
|
* Submit your ideas, and upvote those you think matter the most on [Google Moderator](https://www.google.com/moderator/?authuser=1#16/e=2165c3)
|
||||||
|
|
||||||
|
### Goal of the distribution sprint
|
||||||
|
|
||||||
|
Design a professional grade and extensible content distribution system, that allow users to:
|
||||||
|
|
||||||
|
* Enjoy an efficient, secured and reliable way to store, manage, package and exchange content
|
||||||
|
* Hack/roll their own on top of healthy open-source components
|
||||||
|
* Implement their own home made solution through good specs, and solid extensions mechanism.
|
||||||
|
|
||||||
|
### Schedule and expected output
|
||||||
|
|
||||||
|
The Open Sprint will start on **Monday December 29th**, and end on **Friday January 16th**.
|
||||||
|
|
||||||
|
What we want to achieve as a result is:
|
||||||
|
|
||||||
|
* Tactical fixes of today's frustrations in the existing Docker codebase
|
||||||
|
- This includes a throrough review of [docker/docker#9784](https://github.com/docker/docker/pull/9784) by core maintainers
|
||||||
|
|
||||||
|
* Laying the base of a new distribution subsystem, living independently, and with a well defined group of maintainers. This is the purpose of this repository, which aims at hosting:
|
||||||
|
- A specification of the v2 image format
|
||||||
|
- A specification of the JSON/HTTP protocol
|
||||||
|
- Server-side Go implementation of the v2 registry
|
||||||
|
- Client-side Go packages to consume this new API
|
||||||
|
- Standalone binaries providing content distribution functionalities outside of Docker
|
||||||
|
|
||||||
|
### How will this integrate with Docker engine?
|
||||||
|
|
||||||
|
Building awesome, independent, and well maintained distribution tools should give Docker core maintainers enough incentive to switch to the newly develop subsystem. We make no assumptions on a given date or milestone as urgency should be fixed through [docker/docker#9784](https://github.com/docker/docker/pull/9784), and in order to maintain focus on producing a top quality alternative.
|
||||||
|
|
||||||
|
### Relevant documents
|
||||||
|
|
||||||
|
* [Analysis of current state and goals](docs/opensprint/kickoff.md)
|
||||||
|
|
|
||||||
|
|
@ -0,0 +1,158 @@
|
||||||
|
Distribution
|
||||||
|
=========================
|
||||||
|
|
||||||
|
## Project intentions
|
||||||
|
|
||||||
|
**Problem statement and requirements**
|
||||||
|
|
||||||
|
* What is the exact scope of the problem?
|
||||||
|
|
||||||
|
|
||||||
|
Design a professional grade and extensible content distribution system, that allows docker users to:
|
||||||
|
|
||||||
|
... by default enjoy:
|
||||||
|
|
||||||
|
* an efficient, secured and reliable way to store, manage, package and exchange content
|
||||||
|
|
||||||
|
... optionally:
|
||||||
|
|
||||||
|
* can hack/roll their own on top of healthy open-source components
|
||||||
|
|
||||||
|
... with the liberty to:
|
||||||
|
|
||||||
|
* implement their own home made solution through good specs, and solid extensions mechanism
|
||||||
|
|
||||||
|
|
||||||
|
* Who will the result be useful to?
|
||||||
|
|
||||||
|
* users
|
||||||
|
* ISV (who distribute images or develop image distribution solutions)
|
||||||
|
* docker
|
||||||
|
|
||||||
|
* What are the use cases (distinguish dev & ops population where applicable)?
|
||||||
|
|
||||||
|
* Everyone (... uses docker push/pull).
|
||||||
|
|
||||||
|
* Why does it matter that we build this now?
|
||||||
|
|
||||||
|
* Shortcomings of the existing codebase are the #1 pain point (by large) for users, partners and ISV, hence the most urgent thing to address (?)
|
||||||
|
* That situation is getting worse everyday and killer competitors are going/have emerged.
|
||||||
|
|
||||||
|
* Who are the competitors?
|
||||||
|
|
||||||
|
* existing artifact storage solutions (eg: artifactory).
|
||||||
|
* emerging products that aim at handling pull/push in place of docker.
|
||||||
|
* ISV that are looking for alternatives to workaround this situation
|
||||||
|
|
||||||
|
**Current state: what do we have today?**
|
||||||
|
|
||||||
|
Problems of the existing system:
|
||||||
|
|
||||||
|
1. not reliable
|
||||||
|
* registry goes down whenever the hub goes down
|
||||||
|
* failing push result in broken repositories
|
||||||
|
* concurrent push is not handled
|
||||||
|
* python boto and gevent have a terrible history
|
||||||
|
* organically grown, under-designed features are in a bad shape (search)
|
||||||
|
2. inconsistent
|
||||||
|
* discrepancies between duplicated API (and *duplicated APIs*)
|
||||||
|
* unused features
|
||||||
|
* missing essential features (proper SSL support)
|
||||||
|
3. not reusable
|
||||||
|
* tightly entangled with hub component makes it very difficult to use outside of docker
|
||||||
|
* proper access-control is almost impossible to do right
|
||||||
|
* not easily extensible
|
||||||
|
4. not efficient
|
||||||
|
* no parallel operations (by design)
|
||||||
|
* sluggish client-side processing / bad pipeline design
|
||||||
|
* poor reusability of content (random ids)
|
||||||
|
* scalability issues (tags)
|
||||||
|
* too many useless requests (protocol)
|
||||||
|
* too much local space consumed (local garbage collection: broken + not efficient)
|
||||||
|
* no squashing
|
||||||
|
5. not resilient to errors
|
||||||
|
* no resume
|
||||||
|
* error handling is obscure or inexistent
|
||||||
|
6. security
|
||||||
|
* content is not verified
|
||||||
|
* current tarsum is broken
|
||||||
|
* random ids are a headache
|
||||||
|
7. confusing
|
||||||
|
* registry vs. registry.hub?
|
||||||
|
* layer vs. image?
|
||||||
|
8. broken features
|
||||||
|
* mirroring is not done correctly (too complex, bug-laden, caching is hard)
|
||||||
|
9. poor integration with the rest of the project
|
||||||
|
* technology discrepancy (python vs. go)
|
||||||
|
* poor testability
|
||||||
|
* poor separation (API in the engine is not defined enough)
|
||||||
|
10. missing features / prevents future
|
||||||
|
* trust / image signing
|
||||||
|
* naming / transport separation
|
||||||
|
* discovery / layer federation
|
||||||
|
* architecture + os support (eg: arm/windows)
|
||||||
|
* quotas
|
||||||
|
* alternative distribution methods (transport plugins)
|
||||||
|
|
||||||
|
**Future state: where do we want to get?**
|
||||||
|
|
||||||
|
* Deliverable
|
||||||
|
* new JSON/HTTP protocol specification
|
||||||
|
* new image format specification
|
||||||
|
* (new image store in the engine)
|
||||||
|
* new transport API between the engine and the distribution client code / new library
|
||||||
|
* new registry in go
|
||||||
|
* new authentication service on top of the trust graph in go
|
||||||
|
|
||||||
|
* What are the interactions with other components of the project?
|
||||||
|
* critical interactions with docker push/pull mechanism
|
||||||
|
* critical interactions with the way docker stores images locally
|
||||||
|
|
||||||
|
* In what way will the result be customizable?
|
||||||
|
* transport plugins allowing for radically different transport methods (bittorent, direct S3 access, etc)
|
||||||
|
* extensibility design for the registry allowing for complex integrations with other systems
|
||||||
|
* backend storage drivers API
|
||||||
|
|
||||||
|
|
||||||
|
## Kick-off output
|
||||||
|
|
||||||
|
**What is the expected output of the kick-off session?**
|
||||||
|
|
||||||
|
* draft specifications
|
||||||
|
* separate binary tool for demo purpose
|
||||||
|
* a mergeable PR that fixes 90% of the listed issues
|
||||||
|
|
||||||
|
|
||||||
|
* agree on a vision that allows solving all that are deemed worthy
|
||||||
|
* propose a long term battle plan with clear milestones that encompass all these
|
||||||
|
* define a first milestone that is compatible with the future and does already deliver some of the solutions
|
||||||
|
* deliver the specifications for image manifest format and transport API
|
||||||
|
* deliver a working implementation that can be used as a drop-in replacement for the existing v1 with an equivalent feature-set
|
||||||
|
|
||||||
|
**How is the output going to be demoed?**
|
||||||
|
|
||||||
|
docker pull
|
||||||
|
docker push
|
||||||
|
|
||||||
|
**Once demoed, what will be the path to shipping?**
|
||||||
|
|
||||||
|
A minimal PR that include the first subset of features to make docker work well with the new server side components.
|
||||||
|
|
||||||
|
## Pressing matters
|
||||||
|
|
||||||
|
* need a codename (ship, distribute)
|
||||||
|
* new repository
|
||||||
|
* new domains
|
||||||
|
|
||||||
|
* architecture / OS
|
||||||
|
* persistent ids
|
||||||
|
* registries discovery
|
||||||
|
* naming (quay.io/foo/bar)
|
||||||
|
* mirroring
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
## Assorted issues
|
||||||
|
|
||||||
|
* some devops want a docker engine that cannot do push/pull
|
||||||
|
|
||||||
Loading…
Reference in New Issue