ValueError Invalid endpoint https s3 amazonaws com

Question

When an EMR machine is trying to run a step that includes boto3 initialization it sometimes gets the following error: ValueError: Invalid endpoint: https://s3..amazonaws.com When I'm trying to set up a new machine it can suddenly work. Attached is the full error:

self.client = boto3.client("s3")
  File "/usr/local/lib/python3.6/site-packages/boto3/__init__.py", line 83, in client
    return _get_default_session().client(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/boto3/session.py", line 263, in client
    aws_session_token=aws_session_token, config=config)
  File "/usr/local/lib/python3.6/site-packages/botocore/session.py", line 861, in create_client
    client_config=config, api_version=api_version)
  File "/usr/local/lib/python3.6/site-packages/botocore/client.py", line 76, in create_client
    verify, credentials, scoped_config, client_config, endpoint_bridge)
  File "/usr/local/lib/python3.6/site-packages/botocore/client.py", line 285, in _get_client_args
    verify, credentials, scoped_config, client_config, endpoint_bridge)
  File "/usr/local/lib/python3.6/site-packages/botocore/args.py", line 79, in get_client_args
    timeout=(new_config.connect_timeout, new_config.read_timeout))
  File "/usr/local/lib/python3.6/site-packages/botocore/endpoint.py", line 297, in create_endpoint
    raise ValueError("Invalid endpoint: %s" % endpoint_url)
ValueError: Invalid endpoint: https://s3..amazonaws.com 

Any idea why it happens?

Answer 1

You can set the region as an environment variable. Or set the region in your ~/.aws/credentials or ~/.aws/config files.
For example:

In bash.

export AWS_REGION="eu-west-2"

or in Powershell.

$Env:AWS_REGION="eu-west-2"

I hope this helps you.