A collection of tools for exporting various AWS data from a complex AWS environment adopting a multi-account strategy.
Those are part of our AWS Native Cross-account Observability Dashboard as The data sources of it.
| ✅Features | |
|---|---|
| 🔴AWS Organizations Exporter: | Retrieve and export your AWS Organizations structure and policies. |
| 🔴AWS Identity Center Exporter: | Retrieve and export user, group, permisionSets, relationship with AWS accounts information from AWS Identity Center. |
| 🔴AWS Multi-Account IAM Exporter: | Export IAM details(Role, Custom-Policies, Boundaries...) across multiple AWS accounts. |
| 🟢AWS Free Tier Usage Exporter: | Export Free Tier usage. (Including one of the mysterious chargeable items Global-DataScanned-Bytes too.) |
| Phase | Description |
|---|---|
| Phase 1 (Here) |
Local Development: The initial phase where the project is run and tested locally. |
| Phase 2 | Containerization: Transform the project into a containerized application for easier deployment and scalability. |
| Phase 3 | Production Ready: Finalize the project for deployment in a production environment, ensuring stability, security, and performance. |
- AWS Native Observability Exporters
- Python 3.12+
- Boto3
- Flask
- Pytz
| Target tools | Minimum permissions |
|---|---|
| 🔴AWS Organizations Exporter: (on Mgmt. account) |
organizations:Describe*organizations:List*(Explicit deny) 🚫 organizations:ListHandshakesForOrganization🚫 organizations:ListDelegatedAdministrators🚫 organizations:ListCreateAccountStatus🚫 organizations:DescribeHandshake🚫 organizations:DescribeCreateAccountStatusaccount:ListRegionsaccount:GetRegionOptStatusiam:GenerateOrganizationsAccessReportiam:GetOrganizationsAccessReport |
| 🔴AWS Identity Center Exporter: (on Mgmt. account) |
identitystore:DescribeGroupidentitystore:DescribeUseridentitystore:IsMemberInGroupsidentitystore:ListGroupMembershipsidentitystore:ListGroupMembershipsForMemberidentitystore:ListGroupsidentitystore:ListUserssso:DescribePermissionSetsso:GetInlinePolicyForPermissionSetsso:GetPermissionsBoundaryForPermissionSetsso:ListAccountAssignmentsForPrincipalsso:ListAccountsForProvisionedPermissionSetsso:ListCustomerManagedPolicyReferencesInPermissionSetsso:ListInstancessso:ListManagedPoliciesInPermissionSetsso:ListPermissionSetssso:ListPermissionSetsProvisionedToAccount |
| 🔴AWS Multi-Account IAM Exporter: (on All accounts) |
GetAccountAuthorizationDetails |
| 🟢AWS Free Tier Usage Exporter: (on Mgmt. account) |
freetier:GetFreeTierUsagece:GetCostAndUsage |
💡 Tip:
🙃 If you want to run those tools as a Python script, you'll need to modify this line: from aws_exporters.aws_utils import create_sessionto:
from aws_utils.aws_utils import create_session # For Directory Structure
Before installing the package, it's recommended to create a virtual environment. This isolates the dependencies required for this project from your global Python environment.
📌 Using venv (Python 3.3+)
- Navigate to the working directory where you want to set up the project:
# cd /path/to/your/project- Create a virtual environment:
# git clone https://github.com/Hideki-Morita/aws-native-observability-exporters.git
# cd aws-exporters
# python3 -m venv awsvenv-
Activate the virtual environment:
- On macOS/Linux:
# source awsvenv/bin/activate- On Windows:
PS1> awsvenv\Scripts\activate
Once the virtual environment is activated, install the package using pip:
# pip install build
# python3 -m build
# pip install .📖An example of output
Collecting build Downloading build-1.2.1-py3-none-any.whl (21 kB) Collecting tomli>=1.1.0 Downloading tomli-2.0.1-py3-none-any.whl (12 kB) Collecting packaging>=19.1 Downloading packaging-24.1-py3-none-any.whl (53 kB) |████████████████████████████████| 53 kB 2.6 MB/s Collecting pyproject_hooks Downloading pyproject_hooks-1.1.0-py3-none-any.whl (9.2 kB) Collecting importlib-metadata>=4.6 Downloading importlib_metadata-8.4.0-py3-none-any.whl (26 kB) Collecting zipp>=0.5 Downloading zipp-3.20.1-py3-none-any.whl (9.0 kB) Installing collected packages: zipp, tomli, pyproject-hooks, packaging, importlib-metadata, build Successfully installed build-1.2.1 importlib-metadata-8.4.0 packaging-24.1 pyproject-hooks-1.1.0 tomli-2.0.1 zipp-3.20.1 WARNING: You are using pip version 21.3.1; however, version 24.2 is available. You should consider upgrading via the '/path/to/your/project/aws-exporters/awsvenv/bin/python3 -m pip install --upgrade pip' command. * Creating isolated environment: venv+pip... * Installing packages in isolated environment: - setuptools >= 40.8.0 * Getting build dependencies for sdist... running egg_info creating aws_exporters.egg-info writing aws_exporters.egg-info/PKG-INFO : * Building sdist... running sdist running egg_info writing aws_exporters.egg-info/PKG-INFO : running check creating aws_exporters-1.0.0+ts1.coldasyou creating aws_exporters-1.0.0+ts1.coldasyou/aws_exporters.egg-info copying files to aws_exporters-1.0.0+ts1.coldasyou... copying setup.py -> aws_exporters-1.0.0+ts1.coldasyou : * Building wheel from sdist * Creating isolated environment: venv+pip... * Installing packages in isolated environment: - setuptools >= 40.8.0 * Getting build dependencies for wheel... running egg_info writing aws_exporters.egg-info/PKG-INFO : * Building wheel... running bdist_wheel running build installing to build/bdist.linux-x86_64/wheel running install running install_egg_info running egg_info writing aws_exporters.egg-info/PKG-INFO : adding 'aws_exporters-1.0.0+ts1.coldasyou.dist-info/METADATA' adding 'aws_exporters-1.0.0+ts1.coldasyou.dist-info/WHEEL' adding 'aws_exporters-1.0.0+ts1.coldasyou.dist-info/entry_points.txt' adding 'aws_exporters-1.0.0+ts1.coldasyou.dist-info/top_level.txt' adding 'aws_exporters-1.0.0+ts1.coldasyou.dist-info/RECORD' removing build/bdist.linux-x86_64/wheel Successfully built aws_exporters-1.0.0+ts1.coldasyou.tar.gz and aws_exporters-1.0.0+ts1.coldasyou-py3-none-any.whl Processing /path/to/your/project/aws-exporters Preparing metadata (setup.py) ... done Collecting boto3 Downloading boto3-1.35.6-py3-none-any.whl (139 kB) |████████████████████████████████| 139 kB 2.5 MB/s Collecting Flask Using cached flask-3.0.3-py3-none-any.whl (101 kB) Collecting pytz Using cached pytz-2024.1-py2.py3-none-any.whl (505 kB) Collecting botocore<1.36.0,>=1.35.6 Downloading botocore-1.35.6-py3-none-any.whl (12.5 MB) |████████████████████████████████| 12.5 MB 5.3 MB/s : Using legacy 'setup.py install' for aws-exporters, since package 'wheel' is not installed. Installing collected packages: six, urllib3, python-dateutil, jmespath, MarkupSafe, botocore, Werkzeug, s3transfer, Jinja2, itsdangerous, click, blinker, pytz, Flask, boto3, aws-exporters Running setup.py install for aws-exporters ... done Successfully installed Flask-3.0.3 Jinja2-3.1.4 MarkupSafe-2.1.5 Werkzeug-3.0.4 aws-exporters-1.0.0+ts1.coldasyou blinker-1.8.2 boto3-1.35.6 botocore-1.35.6 click-8.1.7 itsdangerous-2.2.0 jmespath-1.0.1 python-dateutil-2.9.0.post0 pytz-2024.1 s3transfer-0.10.2 six-1.16.0 urllib3-1.26.19 WARNING: You are using pip version 21.3.1; however, version 24.2 is available. You should consider upgrading via the '/path/to/your/project/aws-exporters/awsvenv/bin/python3 -m pip install --upgrade pip' command.
### Check the result of Install
(Option) # pip show aws-exporters ; pip list | egrep aws-exporters
(Option) # organizations-exporter📖An example of output
Name: aws-exporters Version: 1.0.0+ts1.coldasyou Summary: A set of tools for exporting AWS information Home-page: https://github.com/Hideki-Morita/aws-native-observability-exporters Author: Hideki.M Author-email: Y29udGFjdC1tZUBhd3M0Lm1lLnVrCg== License: UNKNOWN Location: /path/to/your/project/aws-exporters Requires: boto3, Flask, pytz Required-by: aws-exporters 1.0.0+ts1.coldasyou /path/to/your/project/aws-exporters WARNING: You are using pip version 21.3.1; however, version 24.2 is available. You should consider upgrading via the '/path/to/your/project/aws-exporters/awsvenv/bin/python3 -m pip install --upgrade pip' command. usage: organizations-exporter [-h] --mgmt-account-id MGMT_ACCOUNT_ID --permission-set-name PERMISSION_SET_NAME --sso-region SSO_REGION [--port PORT] [--cache-expiry CACHE_EXPIRY] [--access-token ACCESS_TOKEN] organizations-exporter: error: the following arguments are required: --mgmt-account-id, --permission-set-name, --sso-region
The finall directory structure is like this!
. ├── aws_exporters │ ├── aws_utils │ │ ├── __init__.py │ │ └── aws_utils.py │ ├── freetier_usage_exporter.py │ ├── identity_center_exporter.py │ ├── multi_acc_iam_exporter.py │ └── organizations_exporter.py ├── aws_exporters.egg-info ├── awsvenv │ ├── bin │ │ ├── activate │ │ ├── freetier-usage-exporter │ │ ├── identity-center-exporter │ │ ├── multi-acc-iam-exporter │ │ ├── organizations-exporter ├── build ├── dist └── setup.py
When you're done working on the project, deactivate the virtual environment:
# deactivate# pip uninstall aws-exporters💡 Each script can be executed as a standalone tool or as a service via Flask. Below is a brief overview of each exporter.
| options: | |
|---|---|
-h, --help |
show this help message and exit. |
--mgmt-account-id MGMT_ACCOUNT_ID |
Management account ID. |
--permission-set-name PERMISSION_SET_NAME |
Name of the permission set to assume in the management account. |
--sso-region SSO_REGION |
AWS SSO region. |
--port PORT |
Port to run the Flask app on. |
--cache-expiry CACHE_EXPIRY |
Cache expiry time in seconds. |
--access-token ACCESS_TOKEN |
Valid access token. |
# organizations_exporter --mgmt-account-id <management_account_id> --permission-set-name <permission_set_name> --sso-region <sso_region> [--port <port>] [--cache-expiry <cache_expiry>] [--access-token <access_token>]e.g.,
# nohup organizations_exporter --mgmt-account-id 121389041924 --permission-set-name AWS-OBS-ReadOnly-ALLTOOWELL --sso-region us-east-1 &[1] 4422 appending output to nohup.out ### In the nohup file looks like this * Serving Flask app 'organizations_exporter' * Debug mode: off 2024-04-19 21:34:45,631 - werkzeug - INFO - WARNING: This is a development server. Do not use it in a production deployment. Use a production WSGI server instead. * Running on all addresses (0.0.0.0) * Running on http://127.0.0.1:7723 * Running on http://10.0.1.4:7723 2024-04-1905 21:34:45,631 - werkzeug - INFO - Press CTRL+C to quit
# identity_center_exporter --mgmt-account-id <management_account_id> --permission-set-name <permission_set_name> --sso-region <sso_region> [--port <port>] [--cache-expiry <cache_expiry>] [--access-token <access_token>]# multi_acc_iam_exporter --permission-set-name <permission_set_name> --sso-region <sso_region> [--port <port>] [--cache-expiry <cache_expiry>] [--access-token <access_token>]# freetier_usage_expoter --mgmt-account-id <management_account_id> --permission-set-name <permission_set_name> --sso-region <sso_region> [--port <port>] [--cache-expiry <cache_expiry>] [--access-token <access_token>]| Base URL | http://localhost:[port]/organization |
| Sub URL | http://localhost:[port]/organization/policies |
| Sub URL | http://localhost:[port]/organization/access-report |
| Port (default: 7723) | You can specify a different port using the --port argument when running the Flask app. |
| HTTP Method | GET |
| Query Parameters | None |
| Default Cache Period | 60 minutes(3600 seconds) |
| Status Codes | - 200 OK: Request succeeded, and the Organizations structure details are returned. - 4xx Client Error: There was an error with the request. - 5xx Server Error: There was an error on the server. |
| /organization |
|---|
📖Expected Output
{
// As Root
"organizations": [
{
"Id": "string",
"Arn": "string",
"Name": "string",
"PolicyTypes": [
{
"Type": "string",
"Status": "string"
}
],
"Accounts": [
{
"Id": "string",
"Arn": "string",
"Email": "string",
"Name": "string",
"Status": "string",
"JoinedMethod": "string",
"JoinedTimestamp": "string"
}
],
// As Level 1
"OrganizationalUnits": [
{
"Id": "string",
"Arn": "string",
"Name": "string",
"Accounts": [
{
"Id": "string",
"Arn": "string",
"Email": "string",
"Name": "string",
"Status": "string",
"JoinedMethod": "string",
"JoinedTimestamp": "string"
}
],
"OrganizationalUnits": [
{
"Id": "string",
"Arn": "string",
"Name": "string",
"Accounts": [
{
"Id": "string",
"Arn": "string",
"Email": "string",
"Name": "string",
"Status": "string",
"JoinedMethod": "string",
"JoinedTimestamp": "string"
}
],
// As Level 2
"OrganizationalUnits": []
}
]
}
]
}
]
}| /organization/access-report |
|---|
📖Expected Output
{
"AccessDetails": [
{
"ServiceName": "string",
"ServiceNamespace": "string",
"TotalAuthenticatedEntities": integer,
"EntityPath": "string",
"LastAuthenticatedTime": "string",
"Region": "string"
}
],
"IsTruncated": boolean,
"JobCompletionDate": "string",
"JobCreationDate": "string",
"JobStatus": "string",
"Marker": "string",
"NumberOfServicesAccessible": integer,
"NumberOfServicesNotAccessed": integer,
"ResponseMetadata": {
"HTTPHeaders": {
"content-length": "string",
"content-type": "string",
"date": "string",
"x-amzn-requestid": "string"
},
"HTTPStatusCode": integer,
"RequestId": "string",
"RetryAttempts": integer
}
}| Base URL | http://localhost:[port]/identity-center |
| Sub URL | http://localhost:[port]/identity-center/permsets |
| Port (default: 11121) | You can specify a different port using the --port argument when running the Flask app. |
| HTTP Method | GET |
| Query Parameters | None |
| Default Cache Period | 60 minutes(3600 seconds) |
| Status Codes | - 200 OK: Request succeeded, and the workforce users/group and permissionSets details are returned. - 4xx Client Error: There was an error with the request. - 5xx Server Error: There was an error on the server. |
| /identity-center |
|---|
📖Expected Output
{
"identity_center": [
{
"CreatedDate": "string",
"IdentityStoreId": "string",
"InstanceArn": "string",
"Name": "string",
"OwnerAccountId": "string",
"Status": "string",
"Users": [
{
"UserName": "string",
"UserId": "string",
"Name": {
"FamilyName": "string",
"GivenName": "string"
},
"DisplayName": "string",
"Emails": [
{
"Value": "string",
"Type": "string",
"Primary": true
}
],
"IdentityStoreId": "string",
"JoinedGroup": [
{
"GroupId": "string",
"DisplayName": "string",
"Description": "string",
"IdentityStoreId": "string"
}
],
"AccountAssignments": [
{
"AccountId": "string",
"PrincipalId": "string",
"PrincipalType": "string",
"PermissionSetArn": "string",
"PermissionSet": {
"Name": "string",
"PermissionSetArn": "string",
"CreatedDate": "string",
"SessionDuration": "string"
}
}
]
}
// Additional user entries go here
]
}
// Additional instance entries go here
]
}| /identity-center/permsets |
|---|
📖Expected Output
{
"PermissionSets": [
{
"AttachedManagedPolicies": [
{
"Arn": "string",
"Name": "string"
},
{
"Arn": "string",
"Name": "string"
}
],
"CreatedDate": "string",
"CustomerManagedPolicyReferences": [
{
"Name": "string",
"Path": "string"
}
],
"Description": "string",
"InlinePolicy": [
{
"None": "string"
}
],
"Name": "string",
"PermissionSetArn": "string",
"PermissionsBoundary": [
{
"None": "string"
}
],
"SessionDuration": "string"
},
// And so on
]
}| Base URL | http://localhost:[port]/multi-account-auth/[Query] |
| Port (default: 1989) | You can specify a different port using the --port argument when running the Flask app. |
| HTTP Method | GET |
| Query Parameters | filter_type: (Required) One of the following values: - User, Group, Role, LocalManagedPolicy, AWSManagedPolicy |
| Default Cache Period | 5 minutes(300 seconds) |
| Status Codes | - 200 OK: Request succeeded, and the IAM details are returned. - 4xx Client Error: There was an error with the request. - 5xx Server Error: There was an error on the server. |
| /multi-account-auth/Role |
|---|
📖Expected Output
{
"AccountID": "string",
"GroupDetailList": [
{}
],
"IsTruncated": false,
"Marker": "string",
"Policies": [],
"ResponseMetadata": {
"HTTPHeaders": {
"content-length": "string",
"content-type": "string",
"date": "string",
"x-amzn-requestid": "string"
},
"HTTPStatusCode": 200,
"RequestId": "string",
"RetryAttempts": 0
},
"RoleDetailList": [
{
"AccountID": "string",
"Arn": "string",
"AssumeRolePolicyDocument": {
"Statement": [
{
"Action": [
"string"
],
"Condition": {
"StringEquals": {
"SAML:aud": "string"
}
},
"Effect": "Allow",
"Principal": {
"Federated": "string"
}
}
],
"Version": "string"
},
"AttachedManagedPolicies": [
{
"PolicyArn": "string",
"PolicyName": "string"
}
],
"CreateDate": "string",
"InstanceProfileList": [],
"Path": "string",
"PermissionsBoundary": {
"PermissionsBoundaryArn": "string",
"PermissionsBoundaryType": "string"
},
"RoleId": "string",
"RoleLastUsed": {
"LastUsedDate": "string",
"Region": "string"
},
"RoleName": "string",
"RolePolicyList": [],
"Tags": [
{
"Key": "string",
"Value": "string"
}
]
}
],
"UserDetailList": [
{}
]
}| Base URL | http://localhost:[port]/freetier |
| Sub URL | http://localhost:[port]/freetier/cost-explorer?[Query] |
| Port (default: 4921) | You can specify a different port using the --port argument when running the Flask app. |
| HTTP Method | GET |
| Query Parameters | - usage_types: (Required) Comma-separated list of usage types(e.g., USE1-DataScanned-Bytes,USW2-DataScanned-Bytes)- time_periods: (Optional) Start and end date in YYYY-MM-DD,YYYY-MM-DD format |
| Default Cache Period | 30 minutes(1800 seconds) |
| Status Codes | - 200 OK: Request succeeded, and the Free Tier usages are returned. - 4xx Client Error: There was an error with the request. - 5xx Server Error: There was an error on the server. |
| /freetier |
|---|
📖Expected Output
{
"freeTierUsages": [
{
"actualUsageAmount": "string",
"description": "string",
"forecastedUsageAmount": "string",
"freeTierType": "string",
"limit": "string",
"operation": "string",
"region": "string",
"service": "string",
"unit": "string",
"usageType": "string"
},
{
"actualUsageAmount": "string",
"description": "string",
"forecastedUsageAmount": "string",
"freeTierType": "string",
"limit": "string",
"operation": "string",
"region": "string",
"service": "string",
"unit": "string",
"usageType": "string"
}
]
}This project is licensed under the MIT License. See the LICENSE file for more information.