A step by step guide to install ConsoleMe on Ubuntu Linux machine
ConsoleMe is an open-source web service published by Netflix. It is designed to make life easy for end-users and cloud administrators. Using ConsoleMe, cloud administrators can manage IAM permissions/credentials for IAM roles, S3 buckets, SQS queues, and SNS topics across multiple AWS accounts from a single interface. It also provides CLI called weep for AWS credentials management. That’s a fair introduction if you are not aware of the tool. Next, let’s get into the installation part.
ConsoleMe offers docker and local installs. We will walk you through local install in this article.
Pre-requisite:
- A machine running Ubuntu 19.04+ with root access. I used Ubuntu 20.04 LTS x86.
- Active and working package manager subscription to install packages
- Storage requirement: 2GB of disk space
- An AWS user/role for consoleme service with appropriate permissions
- AWS access keys for above user if you are not using roles. I used keys (steps below)
Installation
We are installing ConsoleMe in /consoleme directory. If you want to install in another location, make the necessary changes in the commands below. Let me give you a list of commands you need to run as root –
apt-get update
apt-get install build-essential libxml2-dev libxmlsec1 libxmlsec1-dev libxmlsec1-openssl musl-dev libcurl4-nss-dev python3-dev pkg-config python3.8-venv awscli docker-compose -y
curl -sL https://deb.nodesource.com/setup_14.x | sudo bash
apt-get install -y nodejs
npm install yarn -g
cd /
git clone https://github.com/Netflix/consoleme.git
cd consoleme
docker-compose -f docker-compose-dependencies.yaml up -d
Here, the first few commands are installing all the dependencies and related software/tools. Then, we are cloning the GitHub repo of the tool in /consoleme and lastly, we are running two containers.
These are Redis and dynamodb containers that ConsoleMe leverages for caching and aggregating the AWS accounts information. You can make use of AWS Redis and dynamodb table services, but for now, we will run these containers locally so that ConsoleMe will talk to them rather than AWS services.
I am avoiding putting up console outputs for frequently used commands like package installations etc., here.
Make sure both containers are up and running before proceeding to the next step –
root@kerneltalks:/consoleme# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
5333cdee2202 cnadiminti/dynamodb-local "java -jar DynamoDBL…" 10 seconds ago Up 4 seconds 8000/tcp, 0.0.0.0:8005->8005/tcp consoleme-dynamodb
19ac354c3d70 redis:alpine "docker-entrypoint.s…" 10 seconds ago Up 4 seconds 0.0.0.0:6379->6379/tcp consoleme-redis
4cf931d38652 aaronshaf/dynamodb-admin:latest "node bin/dynamodb-a…" 10 seconds ago Up 4 seconds 0.0.0.0:8001->8001/tcp consoleme-dynamodb-admin
Now, you need to prepare the machine to talk with AWS for fetching account details in the upcoming install steps. Ensure that you have set up account and permissions perfectly in IAM (mentioned in the pre-requisite above) to avoid any issues. You can do that by configuring AWS profile –
root@kerneltalks:/consoleme# aws configure
AWS Access Key ID [None]: AKIAQX3STVKIYRO36XEC
AWS Secret Access Key [None]: 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
Default region name [None]:
Default output format [None]:
Lastly, create a new python environment and run the final install step. This will take a while to complete since at the end of make install command, it also fetches and caches the AWS account details in the local Redis cache –
python3 -m venv env
. env/bin/activate
make install
After successful installation, you should be able to start the application.
Running ConsoleMe
On a current shell, you can run the ConsoleMe with the command. If you are in another shell, activate the python environment again –
(env) root@kerneltalks:/consoleme# python consoleme/__main__.py
{"asctime": "2021-07-25T08:32:16Z+0000", "name": "consoleme", "processName": "MainProcess", "filename": "jwt.py", "funcName": "<module>", "levelname": "ERROR", "lineno": 14, "module": "jwt", "threadName": "MainThread", "message": "Configuration key `jwt.secret` is not set. Setting a random secret", "eventTime": "2021-07-25T01:32:16.286230-07:00", "hostname": "kerneltalks", "timestamp": "2021-07-25T08:32:16Z+0000"}
2021-07-25 08:32:17,322 - DEBUG - root - [constants.py:39 - <module>() ] - Leveraging the bundled IAM Definition.
2021-07-25 08:32:17,322 - INFO - root - [iam_data.py:10 - <module>() ] - Leveraging the IAM definition at /consoleme/env/lib/python3.8/site-packages/policy_sentry/shared/data/iam-definition.json
2021-07-25 08:32:17,824 - DEBUG - git.cmd - [cmd.py:817 - execute() ] - Popen(['git', 'version'], cwd=/consoleme, universal_newlines=False, shell=None, istream=None)
2021-07-25 08:32:17,859 - DEBUG - git.cmd - [cmd.py:817 - execute() ] - Popen(['git', 'version'], cwd=/consoleme, universal_newlines=False, shell=None, istream=None)
{"asctime": "2021-07-25T08:32:18Z+0000", "name": "consoleme", "processName": "MainProcess", "filename": "__main__.py", "funcName": "init", "levelname": "DEBUG", "lineno": 57, "module": "__main__", "threadName": "MainThread", "message": "Server started", "eventTime": "2021-07-25T01:32:16.286230-07:00", "hostname": "kerneltalks", "timestamp": "2021-07-25T08:32:18Z+0000"}
But, it will exit out when you terminate the command or shell. It’s safe to run it in the background or, even better, run it as a Linux service. For running ConsoleMe as a service, create below two files –
File /usr/bin/consoleme_start.sh
#!/bin/bash
. env/bin/activate
python consoleme/__main__.py
File /etc/systemd/system/consoleme.service
[Unit]
Description=Run consoleme service.
[Service]
Type=simple
User=root
WorkingDirectory=/consoleme
ExecStart=/usr/bin/consoleme_start.sh
[Install]
WantedBy=multi-user.target
Assign executable permissions to
chmod +x /usr/bin/consoleme_start.sh
Enable and start the service
root@kerneltalks:/consoleme# systemctl enable consoleme
Created symlink /etc/systemd/system/multi-user.target.wants/consoleme.service → /etc/systemd/system/consoleme.service.
root@kerneltalks:/consoleme# systemctl start consoleme
root@kerneltalks:/consoleme# systemctl status consoleme
● consoleme.service - Run consoleme service.
Loaded: loaded (/etc/systemd/system/consoleme.service; enabled; vendor preset: enabled)
Active: active (running) since Sun 2021-07-25 08:35:52 UTC; 7s ago
Main PID: 14775 (consoleme_start)
Tasks: 5 (limit: 4706)
Memory: 159.7M
CGroup: /system.slice/consoleme.service
├─14775 /bin/bash /usr/bin/consoleme_start.sh
└─14776 python consoleme/__main__.py
Jul 25 08:35:52 kerneltalks systemd[1]: Started Run consoleme service..
Jul 25 08:35:53 kerneltalks consoleme_start.sh[14776]: {"asctime": "2021-07-25T08:35:53Z+0000", "name": "consoleme", "processName": "MainProcess", "filename": "jwt.py", "funcName": "<module>", "levelname": "ERROR", "lineno": 14, "m>
Jul 25 08:35:53 kerneltalks consoleme_start.sh[14776]: 2021-07-25 08:35:53,954 - DEBUG - root - [constants.py:39 - <module>() ] - Leveraging the bundled IAM Definition.
Jul 25 08:35:53 kerneltalks consoleme_start.sh[14776]: 2021-07-25 08:35:53,955 - INFO - root - [iam_data.py:10 - <module>() ] - Leveraging the IAM definition at /consoleme/env/lib/python3.8/site-packages/policy_sentry/shared/data/i>
Jul 25 08:35:54 kerneltalks consoleme_start.sh[14776]: 2021-07-25 08:35:54,354 - DEBUG - git.cmd - [cmd.py:817 - execute() ] - Popen(['git', 'version'], cwd=/consoleme, universal_newlines=False, shell=None, istream=None)
Jul 25 08:35:54 kerneltalks consoleme_start.sh[14776]: 2021-07-25 08:35:54,361 - DEBUG - git.cmd - [cmd.py:817 - execute() ] - Popen(['git', 'version'], cwd=/consoleme, universal_newlines=False, shell=None, istream=None)
Jul 25 08:35:54 kerneltalks consoleme_start.sh[14776]: {"asctime": "2021-07-25T08:35:54Z+0000", "name": "consoleme", "processName": "MainProcess", "filename": "__main__.py", "funcName": "init", "levelname": "DEBUG", "lineno": 57, ">
ConsoleMe GUI
Now that your console service is running, you should load its GUI on a web browser. The service listens on the 8081 port, so you need to navigate the server address with port 8081. Make sure the security group is allowing 8081 traffic if you are installing on EC2.
At this point, ConsoleMe is running with the default open example configuration. It’s very well highlighted on the web app as a warning. It would be best if you were editing this configuration to make your ConsoleMe more secure. ConsoleMe recommends Application Load Balancer authentication for securing your web app GUI. Refer to our next article on how to secure the ConsoleMe web app using ALB authentication.
I am getting error while in make install:
Created persistent docker volume for dynamodb.
–> Configuring Dynamo (Make sure local dynamo is enabled on port 8000)
python scripts/initialize_dynamodb_oss.py
Traceback (most recent call last):
File “scripts/initialize_dynamodb_oss.py”, line 5, in
from consoleme.config import config
File “/consoleme/consoleme/config/config.py”, line 374, in
async_to_sync(CONFIG.load_config)()
File “/consoleme/env/lib/python3.8/site-packages/asgiref/sync.py”, line 218, in __call__
return call_result.result()
File “/usr/lib/python3.8/concurrent/futures/_base.py”, line 437, in result
return self.__get_result()
File “/usr/lib/python3.8/concurrent/futures/_base.py”, line 389, in __get_result
raise self._exception
File “/consoleme/env/lib/python3.8/site-packages/asgiref/sync.py”, line 284, in main_wrap
result = await self.awaitable(*args, **kwargs)
File “/consoleme/consoleme/config/config.py”, line 221, in load_config
self.raise_if_invalid_aws_credentials()
File “/consoleme/consoleme/config/config.py”, line 69, in raise_if_invalid_aws_credentials
boto3.client(
File “/consoleme/env/lib/python3.8/site-packages/boto3/__init__.py”, line 92, in client
return _get_default_session().client(*args, **kwargs)
File “/consoleme/env/lib/python3.8/site-packages/boto3/session.py”, line 299, in client
return self._session.create_client(
File “/consoleme/env/lib/python3.8/site-packages/botocore/session.py”, line 950, in create_client
client = client_creator.create_client(
File “/consoleme/env/lib/python3.8/site-packages/botocore/client.py”, line 123, in create_client
client_args = self._get_client_args(
File “/consoleme/env/lib/python3.8/site-packages/botocore/client.py”, line 466, in _get_client_args
return args_creator.get_client_args(
File “/consoleme/env/lib/python3.8/site-packages/botocore/args.py”, line 123, in get_client_args
endpoint = endpoint_creator.create_endpoint(
File “/consoleme/env/lib/python3.8/site-packages/botocore/endpoint.py”, line 402, in create_endpoint
raise ValueError(“Invalid endpoint: %s” % endpoint_url)
ValueError: Invalid endpoint: https://sts..amazonaws.com
make: *** [Makefile:43: dynamo] Error 1