I’ve been working with an Azure Storage Account to host a static website (this blog) and am interested in implementing an upload process that uses azcopy or the azure-cli.
Both tools are good for the job, and have minor efficiency improvements, depending on your scenario and scripting skills. On first impressions, azcopy is great to perform a full directory sync, whereas the azure-cli is the friend of script writers.
I’m working on a Mac at home, so luckily both tools are compatible with Mac and Linux environments.
With either azure-cli and azcopy, I can see how it would be easy to set up a deployment pipeline, triggered by a git merge a repo hosted in Azure DevOps, to automate the publishing of my content (blog posts).
Error when using azure-cli
My specific scenario is to deploy blobs to a Azure Storage Account Blob with the container name of $web. $web is the container name given by the Admin Portal when a static website is created in your Storage Account.
After installing the azure-cli on my environment, I would execute the login command:
az login
However, executing the command to perform a Blob upload:
az storage blob upload -f /local/path/to/file/index.html -c $web -n index.html --account-name nameofstorageaccount
I received the following error:
Value for one of the query parameters specified in the request URI is invalid. ErrorCode: InvalidQueryParameterValue
With a full exception trace as follows:
<?xml version="1.0" encoding="utf-8"?><Error><Code>InvalidQueryParameterValue</Code><Message>Value for one of the query parameters specified in the request URI is invalid.
RequestId:82dc7011-b01e-0012-73b2-a6a88c000000
Time:2019-01-07T18:00:30.9812209Z</Message><QueryParameterName>comp</QueryParameterName><QueryParameterValue /><Reason /></Error>
Traceback (most recent call last):
File "/usr/local/Cellar/azure-cli/2.0.52/libexec/lib/python3.7/site-packages/knack/cli.py", line 206, in invoke
cmd_result = self.invocation.execute(args)
File "/usr/local/Cellar/azure-cli/2.0.52/libexec/lib/python3.7/site-packages/azure/cli/core/commands/__init__.py", line 343, in execute
cmd.exception_handler(ex)
File "/usr/local/Cellar/azure-cli/2.0.52/libexec/lib/python3.7/site-packages/azure/cli/command_modules/storage/__init__.py", line 234, in new_handler
handler(ex)
File "/usr/local/Cellar/azure-cli/2.0.52/libexec/lib/python3.7/site-packages/azure/cli/command_modules/storage/__init__.py", line 177, in handler
raise ex
File "/usr/local/Cellar/azure-cli/2.0.52/libexec/lib/python3.7/site-packages/azure/cli/core/commands/__init__.py", line 320, in execute
result = cmd(params)
File "/usr/local/Cellar/azure-cli/2.0.52/libexec/lib/python3.7/site-packages/azure/cli/core/commands/__init__.py", line 169, in __call__
return self.handler(*args, **kwargs)
File "/usr/local/Cellar/azure-cli/2.0.52/libexec/lib/python3.7/site-packages/azure/cli/core/__init__.py", line 440, in default_command_handler
result = op(**command_args)
File "/usr/local/Cellar/azure-cli/2.0.52/libexec/lib/python3.7/site-packages/azure/cli/command_modules/storage/operations/blob.py", line 296, in upload_blob
return type_func[blob_type]()
File "/usr/local/Cellar/azure-cli/2.0.52/libexec/lib/python3.7/site-packages/azure/cli/command_modules/storage/operations/blob.py", line 289, in upload_block_blob
return client.create_blob_from_path(**create_blob_args)
File "/usr/local/Cellar/azure-cli/2.0.52/libexec/lib/python3.7/site-packages/azure/multiapi/storage/v2018_03_28/blob/blockblobservice.py", line 463, in create_blob_from_path
timeout=timeout)
File "/usr/local/Cellar/azure-cli/2.0.52/libexec/lib/python3.7/site-packages/azure/multiapi/storage/v2018_03_28/blob/blockblobservice.py", line 582, in create_blob_from_stream
timeout=timeout)
File "/usr/local/Cellar/azure-cli/2.0.52/libexec/lib/python3.7/site-packages/azure/multiapi/storage/v2018_03_28/blob/blockblobservice.py", line 971, in _put_blob
return self._perform_request(request, _parse_base_properties)
File "/usr/local/Cellar/azure-cli/2.0.52/libexec/lib/python3.7/site-packages/azure/multiapi/storage/v2018_03_28/common/storageclient.py", line 381, in _perform_request
raise ex
File "/usr/local/Cellar/azure-cli/2.0.52/libexec/lib/python3.7/site-packages/azure/multiapi/storage/v2018_03_28/common/storageclient.py", line 306, in _perform_request
raise ex
File "/usr/local/Cellar/azure-cli/2.0.52/libexec/lib/python3.7/site-packages/azure/multiapi/storage/v2018_03_28/common/storageclient.py", line 292, in _perform_request
HTTPError(response.status, response.message, response.headers, response.body))
File "/usr/local/Cellar/azure-cli/2.0.52/libexec/lib/python3.7/site-packages/azure/multiapi/storage/v2018_03_28/common/_error.py", line 115, in _http_error_handler
raise ex
azure.common.AzureHttpError: Value for one of the query parameters specified in the request URI is invalid. ErrorCode: InvalidQueryParameterValue
<?xml version="1.0" encoding="utf-8"?><Error><Code>InvalidQueryParameterValue</Code><Message>Value for one of the query parameters specified in the request URI is invalid.
RequestId:82dc7011-b01e-0012-73b2-a6a88c000000
Time:2019-01-07T18:00:30.9812209Z</Message><QueryParameterName>comp</QueryParameterName><QueryParameterValue /><Reason /></Error>
Error when using azcopy (v10)
After hitting a wall with the above error I thought I’d try the alternative option: azcopy.
First step to copying files to Azure with azcopy is to generate a Shared Access Signature, which can be done in the Azure Portal. With the SAS, you can then use azcopy without logging in:
./azcopy copy "/local/path/to/folder/for/syncing/*" "https://nameofstorageaccount.blob.core.windows.net/$web/?sastokenfromazureportal" --recursive
However, that command would return 37 failed file transfers (the number of files I was recursively trying to upload). So I tried an individual file, for example:
./azcopy copy "/local/path/to/folder/for/syncing/index.html" "https://nameofstorageaccount.blob.core.windows.net/$web/index.html?sastokenfromazureportal"
And the following error returned:
panic: runtime error: slice bounds out of range
goroutine 1 [running]:
github.com/Azure/azure-storage-azcopy/cmd.copyHandlerUtil.urlIsContainerOrShare(0xc4207b8380, 0x1a4)
/go/src/github.com/Azure/azure-storage-azcopy/cmd/copyUtil.go:69 +0x1d7
github.com/Azure/azure-storage-azcopy/cmd.(*copyUploadEnumerator).enumerate(0xc42007e380, 0xc4200da000, 0x2b, 0x23)
/go/src/github.com/Azure/azure-storage-azcopy/cmd/copyUploadEnumerator.go:48 +0x18e2
github.com/Azure/azure-storage-azcopy/cmd.(*cookedCopyCmdArgs).processCopyJobPartOrders(0xc4200da000, 0xb, 0x1)
/go/src/github.com/Azure/azure-storage-azcopy/cmd/copy.go:641 +0x46e
github.com/Azure/azure-storage-azcopy/cmd.(*cookedCopyCmdArgs).process(0xc4200da000, 0xf8, 0xb)
/go/src/github.com/Azure/azure-storage-azcopy/cmd/copy.go:393 +0x83
github.com/Azure/azure-storage-azcopy/cmd.init.1.func2(0xc4200bea00, 0xc4207a8020, 0x2, 0x2)
/go/src/github.com/Azure/azure-storage-azcopy/cmd/copy.go:941 +0x185
github.com/Azure/azure-storage-azcopy/vendor/github.com/spf13/cobra.(*Command).execute(0xc4200bea00, 0xc4207d0c20, 0x2, 0x2, 0xc4200bea00, 0xc4207d0c20)
/go/src/github.com/Azure/azure-storage-azcopy/vendor/github.com/spf13/cobra/command.go:766 +0x2c1
github.com/Azure/azure-storage-azcopy/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0x492ff60, 0xd, 0x56, 0x18)
/go/src/github.com/Azure/azure-storage-azcopy/vendor/github.com/spf13/cobra/command.go:852 +0x30a
github.com/Azure/azure-storage-azcopy/vendor/github.com/spf13/cobra.(*Command).Execute(0x492ff60, 0x13, 0xd)
/go/src/github.com/Azure/azure-storage-azcopy/vendor/github.com/spf13/cobra/command.go:800 +0x2b
github.com/Azure/azure-storage-azcopy/cmd.Execute(0xc420022340, 0x13, 0xc420022340, 0x13)
/go/src/github.com/Azure/azure-storage-azcopy/cmd/root.go:71 +0x7d
main.main()
/go/src/github.com/Azure/azure-storage-azcopy/main.go:55 +0x1c8
The solution to both azure-cli and azcopy errors
The solution to both errors? Escape the $ (dollars) character in the command parameter - so $web becomes \$web.
The correct azure-cli command is:
az storage blob upload -f /local/path/to/file/index.html -c \$web -n index.html --account-name nameofstorageaccount
And the correct azcopy command is:
./azcopy copy "/local/path/to/folder/for/syncing/index.html" "https://nameofstorageaccount.blob.core.windows.net/\$web/index.html?sastokenfromazureportal"
Both errors occur because a value prefixed with $ in bash is considered a variable. When the command is passed in to azure-cli or azcopy, bash thinks the value of $web is blank, and passes it in as blank.