Commit Graph

149 Commits (87e34bd307ad4c5686fed6f6352e12309e4ced7f)

Author SHA1 Message Date
Aaron Lehmann 87e34bd307 Fix two misspellings in source code comments
Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2016-03-08 15:13:24 -08:00
Brian Bland 302cc39937 [driver/s3aws] Update s3aws driver parameter parsing to match s3goamz
Mirrors changes from #1414 into the newer driver

Signed-off-by: Brian Bland <brian.bland@docker.com>
2016-03-07 11:52:34 -08:00
Richard Scothern 62d46ef07d Merge pull request #1388 from aibaars/gcs-simplify-move
StorageDriver: GCS: remove support for directory Moves
2016-03-03 10:20:51 -08:00
Arthur Baars 1593c2413f StorageDriver: GCS: remove support for directory Moves
The Move operation is only used to move uploaded blobs
to their final destination. There is no point in implementing
Move on "folders". Apart from simplifying the code, this also
saves an HTTP request.

Signed-off-by: Arthur Baars <arthur@semmle.com>
2016-03-03 13:28:13 +00:00
Stefan Weil 615c6dfced Fix some typos in comments and strings
All of them were found and fixed by codespell.

Signed-off-by: Stefan Weil <sw@weilnetz.de>
2016-02-23 22:33:38 +01:00
Anton Tiurin a048a4c8d5 Fix description of StorageDriver.WriteStream
Offset can be more than CurrentSize as long as this case is checked
by DriverSuite.testContinueStreamAppend.

Signed-off-by: Anton Tiurin <noxiouz@yandex.ru>
2016-02-17 13:57:20 +03:00
Brian Bland c770f28f04 [driver/s3aws] Fix TestStorageClass
Fixes bug in TestStorageClass for s3aws driver where the "standard" file
was checked for reduced-redundnancy storage.

Signed-off-by: Brian Bland <brian.bland@docker.com>
2016-02-16 17:50:55 -08:00
Brian Bland b9907c4853 Merge pull request #1385 from BrianBland/s3UseAWSLibrary
[driver/s3] Use aws/aws-sdk-go instead of goamz for s3 driver and cloudfront
2016-02-16 11:19:15 -08:00
liuchang0812 0758653b42 fix gofmt
Signed-off-by: liuchang0812 <liuchang0812@gmail.com>
2016-02-16 11:42:09 +08:00
liuchang0812 dbba9cf299 closes #1461, enhance log message of oss driver
Signed-off-by: liuchang0812 <liuchang0812@gmail.com>
2016-02-16 11:17:09 +08:00
Brian Bland 1319be2cae Adds new s3 driver using aws-sdk-go instead of goamz
Keeps old s3 driver, renames to s3goamz, registers new s3 driver as both
"s3" and "s3aws"

Changes cloudfront middleware to use aws-sdk-go

Signed-off-by: Brian Bland <brian.bland@docker.com>
2016-02-12 15:59:26 -08:00
Aaron Lehmann aa80478b64 Typo fixes in comments
Correct spelling of words in source code comments.

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2016-02-10 16:26:29 -08:00
Brian Bland 52b4a328dd Improves flexibility of configuration handling for S3 driver
Treats nil parameters the same as unprovided parameters (fixes issues
where certain parameters are printed to "<nil>").
Accepts "true" and "false" string values for boolean parameters.

Signed-off-by: Brian Bland <brian.bland@docker.com>
2016-02-09 17:27:16 -08:00
Richard Scothern 2cc6ccbded Merge pull request #1401 from BrianBland/s3StorageClass
Adds "storageclass" configuration parameter for S3 driver.
2016-02-04 16:44:13 -08:00
yuzou 44c0b9ae87 read the actual number of bytes according to the initial size.
Signed-off-by: yuzou <zouyu7@huawei.com>
2016-02-04 16:14:35 +08:00
Brian Bland 695bac7b0f Adds test for S3 storage class configuration option
Signed-off-by: Brian Bland <brian.bland@docker.com>
2016-02-01 16:01:55 -08:00
Brian Bland 4688d1adc2 Adds "storageclass" configuration parameter for S3 driver.
Defaults to STANDARD, also supports REDUCED_REDUNDANCY.

Signed-off-by: Brian Bland <brian.bland@docker.com>
2016-02-01 16:01:19 -08:00
Aaron Lehmann 2fc586d2a4 Merge pull request #1381 from BrianBland/s3CustomUAString
Adds custom registry User-Agent header to s3 HTTP requests
2016-02-01 15:40:36 -08:00
Brian Bland 2dc1af12a1 Adds custom registry User-Agent header to s3 HTTP requests
Uses docker/goamz instead of AdRoll/goamz

Adds a registry UA string param to the storage parameters when
constructing the storage driver for the registry App.
This could be used by other storage drivers as well

Signed-off-by: Brian Bland <brian.bland@docker.com>
2016-02-01 13:43:46 -08:00
Arthur Baars 2a4345ca4b StorageDriver: GCS: retry all api calls
Signed-off-by: Arthur Baars <arthur@semmle.com>
2016-01-20 13:24:09 +00:00
Arthur Baars 6ee339464c StorageDriver: Test suite: improve cleanup
Verify that the file(s) have been deleted after calling Delete,
and retry if this is not the case. Furthermore, report the error
if a Delete operation fails.

Signed-off-by: Arthur Baars <arthur@semmle.com>
2016-01-20 13:24:09 +00:00
Richard Scothern fc7ee6d789 Merge pull request #1355 from hopkings2008/master
In testsuites.go, enlarge the size of randomBytes to 128M to fix the …
2016-01-18 12:38:26 -08:00
Arthur Baars 19d7c9e051 StorageDriver GCS: try google.DefaultTokenSource first
Signed-off-by: Arthur Baars <arthur@semmle.com>
2016-01-15 11:47:47 +00:00
yuzou cc82b0d48e In testsuites.go, enlarge the size of randomBytes to 128M to fix the crash of running TestConcurrentStreamReads
Signed-off-by: yuzou <zouyu7@huawei.com>
2016-01-15 17:22:43 +08:00
Richard Scothern 81dbebc7d3 Change the parameters to the GCS drivers to allow CircleCI testing.
Remove the requirement of file system access to run GCS unit tests.  Deconstruct
the input parameters to take the private key and email which can be specified on
the build system via environment variables.

Signed-off-by: Richard Scothern <richard.scothern@gmail.com>
2016-01-13 18:00:59 -08:00
Richard Scothern cc0b6b86ea Merge pull request #1276 from denverdino/oss-test
Support large layer for OSS driver
2016-01-08 13:26:09 -08:00
Richard Scothern 9f83bfc071 Merge pull request #1321 from aibaars/gcs-fix-retry
GCS storage driver: fix retry function
2016-01-06 12:00:12 -08:00
Arthur Baars bb1e9e0a82 GCS Storagedriver: fix test failure caused by #1187
Signed-off-by: Arthur Baars <arthur@semmle.com>
2016-01-06 18:36:28 +00:00
Arthur Baars d3d40a8807 GCS driver: fix retry function
Signed-off-by: Arthur Baars <arthur@semmle.com>
2016-01-06 17:57:22 +00:00
weiyuan.yl 75abeeca11 Replace 404 to http.StatusNotFound
Change-Id: Ia100975cb93c0a6d94ea5542b1c9ce386bc87649
Signed-off-by: weiyuan.yl <weiyuan.yl@alibaba-inc.com>
2015-12-29 12:09:04 +08:00
Brian Bland f19deba809 Relaxes filesystem driver permissions to 0777 (dirs) and 0666 (files)
Leaves any further permissions restrictions to the process umask

Signed-off-by: Brian Bland <brian.bland@docker.com>
2015-12-28 15:22:28 -08:00
weiyuan.yl 3e4b554f8d In HEAD request for missing resource, only 404 NOT FOUND is returned
Change-Id: I73caf67b59519e6f4f82f7d78f5d4fd4ad9affcd
Signed-off-by: weiyuan.yl <weiyuan.yl@alibaba-inc.com>
2015-12-28 11:28:42 +08:00
Li Yi 54da47d636 Support large layer for OSS driver
Signed-off-by: Li Yi <denverdino@gmail.com>
2015-12-11 10:56:50 +08:00
Richard Scothern 796d6e7915 Merge pull request #1187 from stevvooe/check-storage-drivers-list-path-not-found
[WIP] registry/storage/driver: checking that non-existent path returns PathNotFoundError
2015-12-08 16:32:02 -08:00
Stephen J Day 66cd2bf950 storage/driver/s3: adjust s3 driver to return unmunged path
This fixes both the s3 driver and the oss driver to return the unmunged path
when returning errors.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-12-08 15:48:55 -08:00
Stephen Day e4208ad5a7 Merge pull request #1246 from stevvooe/decrease-driver-testsuite-allocations
storage/driver: decrease memory allocation done during testsuite
2015-12-08 14:52:59 -08:00
Li Yi b6756a3d89 Fix the issue for listing root directory
Change-Id: I1c6181fa4e5666bd2e6ec69cb608c4778ae0fe48
Signed-off-by: Li Yi <denverdino@gmail.com>
2015-12-08 12:04:04 -08:00
Li Yi 7161fa0559 Fix for stevvooe:check-storage-drivers-list-path-not-found in OSS driver
Change-Id: I5e96fe761d3833c962084fd2d597f47e8a72e7c2
Signed-off-by: Li Yi <denverdino@gmail.com>
2015-12-08 12:04:04 -08:00
davidli 79d4d7f546 driver/swift: treat empty object list as a PathNotFoundError
Swift returns an empty object list when trying to read a non-existing object path, treat it as a
PathNotFoundError when trying to list a non existing virtual directory.

Signed-off-by: David li <wenquan.li@hpe.com>
2015-12-08 12:04:04 -08:00
Vincent Giersch f25ccea279 driver/rados: treat OMAP EIO as a PathNotFoundError
RADOS returns a -EIO when trying to read a non-existing OMAP, treat it as a
PathNotFoundError when trying to list a non existing virtual directory.

Signed-off-by: Vincent Giersch <vincent@giersch.fr>
2015-12-08 12:04:04 -08:00
Stephen J Day a889f46223 storage/driver/s3: correct response on list of missing directory
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-12-08 12:04:03 -08:00
Stephen J Day b45078eb44 storage/driver/base: use correct error format style
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-12-08 12:04:03 -08:00
Stephen J Day a4eae0917e driver/filesystem: address filesystem driver on behavior of List
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-12-08 12:04:03 -08:00
Stephen J Day 0322c3bf1f registry/storage/driver: checking that non-existent path returns PathNotFoundError
Issue #1186 describes a condition where a null tags response is returned when
using the s3 driver. The issue seems to be related to a missing
PathNotFoundError in s3. This change adds a test for that to get an idea of the
lack of compliance across storage drivers. If the failures are manageable,
we'll add this test condition and fix the s3 driver.

Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-12-08 12:04:03 -08:00
Stephen Day 807a80ff06 Merge pull request #1238 from noxiouz/fix_comment
Fix comment for PathRegexp
2015-12-08 10:41:16 -08:00
Stephen J Day 9dc3c6df0e storage/driver: decrease memory allocation done during testsuite
Signed-off-by: Stephen J Day <stephen.day@docker.com>
2015-12-07 19:05:13 -08:00
Anton Tiurin d4435b79d9 Fix comment for PathRegexp
Signed-off-by: Anton Tiurin <noxiouz@yandex.ru>
2015-12-04 22:14:21 +03:00
Sylvain Baubeau ef4ab7a885 Use bulk delete to remove segments in Swift driver
Signed-off-by: Sylvain Baubeau <sbaubeau@redhat.com>
2015-12-04 12:16:49 +01:00
Richard Scothern a33095af80 Merge pull request #1141 from lebauce/swift-no-missing-segment
Ensure read after write for segments
2015-11-10 10:39:43 -08:00
Aaron Lehmann 6664a1384e Fix empty delete requests with Swift fs driver
The Delete method lists objects under the given path and tries to delete
all of them with a bulk delete request. If the path has no objects
underneath it, the body of this request will be empty, which causes
HTTP-level issues. Specifically, Go's HTTP client senses the empty
request buffer and doesn't include a Content-Length, which causes the
Swift server to fail the request.

This commit fixes the problem by avoiding sending empty bulk delete
requests. This is the correct thing to do anyway, since there's no
reason to request deletion of zero objects.

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
2015-11-09 15:33:05 -08:00