IAM Policy Resource Mapping

Created on: February 4, 2025

Answered using O1 by Chat01

Question

my main goal is that I wanna do -- service <> resource mapping (Parse IAM policies to build a db of sorts) - for data I have in format -- /Users/khushisharma/iamparsing/04-02-2025/customer_managed_policys/access-manager.json--{ "PolicyDocument": { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "ssm:GetParameter*" ], "Resource": [ "arn:aws:ssm:ap-south-1:744315537043:parameter/conf/access-manager/" ] } ] }, "Path": "/", "Tags": [ { "Key": "Created-by", "Value": "Locksmith" } ], "PermissionsBoundaryList": { "PolicyGroups": [], "PolicyUsers": [], "PolicyRoles": [] }, "PermissionsPolicyList": { "PolicyGroups": [], "PolicyUsers": [], "PolicyRoles": [ { "RoleName": "access-manager", "RoleId": "AROA22TFJY2JTSOTV6SEL" } ] } } I have obtained a all_services.json which has data in format { "rds-rs-pipeline": { "s3_read": [ "arn:aws:s3:::sftp-rbl-ags", "arn:aws:s3:::prod-datascience-sms/", "arn:aws:s3:::prod-lending-adhoc", "arn:aws:s3:::prod-lending-adhoc/", "arn:aws:s3:::prod-datascience-sms", "arn:aws:s3:::datascience-store-personalisation/", "arn:aws:s3:::sftp-rbl-ags/", "", "arn:aws:s3:::datascience-store-personalisation" ], "custom_s3_actions0": [ "arn:aws:s3:::sftp-rbl-ags", "arn:aws:s3:::prod-datascience-sms/", "arn:aws:s3:::prod-lending-adhoc", "arn:aws:s3:::prod-lending-adhoc/", "arn:aws:s3:::prod-datascience-sms", "arn:aws:s3:::datascience-store-personalisation/", "arn:aws:s3:::sftp-rbl-ags/", "", "arn:aws:s3:::datascience-store-personalisation" ], "custom_s3-object-lambda_actions0": [ "" ], "ssm_read": [ "arn:aws:ssm:ap-south-1:744315537043:parameter/conf/rds-rs-pipeline/" ], "kms_decrypt": [ "arn:aws:kms:ap-south-1:744315537043:key/af547ea6-b4fd-4050-9185-62068e0da254", "arn:aws:kms:ap-south-1:744315537043:key/c1b80926-ce55-4905-a92a-a1a830ffbf5b", "arn:aws:kms:ap-south-1:744315537043:key/94baadce-0db1-41fe-91ff-f3be240fb6c2" ], "kms_generate_data_key": [ "arn:aws:kms:ap-south-1:744315537043:key/af547ea6-b4fd-4050-9185-62068e0da254", "arn:aws:kms:ap-south-1:744315537043:key/c1b80926-ce55-4905-a92a-a1a830ffbf5b", "arn:aws:kms:ap-south-1:744315537043:key/94baadce-0db1-41fe-91ff-f3be240fb6c2" ], "kms_sign": [ "arn:aws:kms:ap-south-1:744315537043:key/af547ea6-b4fd-4050-9185-62068e0da254", "arn:aws:kms:ap-south-1:744315537043:key/c1b80926-ce55-4905-a92a-a1a830ffbf5b", "arn:aws:kms:ap-south-1:744315537043:key/94baadce-0db1-41fe-91ff-f3be240fb6c2" ], "kms_verify": [ "arn:aws:kms:ap-south-1:744315537043:key/af547ea6-b4fd-4050-9185-62068e0da254", "arn:aws:kms:ap-south-1:744315537043:key/c1b80926-ce55-4905-a92a-a1a830ffbf5b", "arn:aws:kms:ap-south-1:744315537043:key/94baadce-0db1-41fe-91ff-f3be240fb6c2" ], "custom_dynamodb_actions0": [ "arn:aws:dynamodb:ap-south-1:744315537043:table/external_reward_report_config", "arn:aws:dynamodb:ap-south-1:744315537043:table/sms_communications", "arn:aws:dynamodb:us-east-1:744315537043:table/external_reward_report_config" ], "custom_firehose_actions0": [ "arn:aws:firehose:ap-south-1:744315537043:deliverystream/prod_reward_usage_attributes" ], "custom_firehose_actions1": [ "" ], "s3_write": [ "arn:aws:s3:::sftp-rbl-ags", "arn:aws:s3:::prod-datascience-sms/", "arn:aws:s3:::prod-lending-adhoc", "arn:aws:s3:::prod-lending-adhoc/", "arn:aws:s3:::prod-datascience-sms", "arn:aws:s3:::datascience-store-personalisation/", "arn:aws:s3:::sftp-rbl-ags/", "arn:aws:s3:::datascience-store-personalisation" ], "custom_s3_actions1": [ "arn:aws:s3:::eventi-audit", "arn:aws:s3:::eventi-audit/", "arn:aws:s3:::prod-datascience-sms/", "arn:aws:s3:::prod-data-", "arn:aws:s3:::prod-datascience-sms", "arn:aws:s3:::internal-payments-recon", "arn:aws:s3:::stg-redshift", "arn:aws:s3:::internal-payments-recon/", "arn:aws:s3:::stg-redshift/" ], "custom_s3_actions2": [ "" ], "custom_ssm_actions0": [ "" ], "custom_ssm_actions1": [ "arn:aws:ssm:ap-south-1:744315537043:parameter/conf/rds-rs-pipeline/" ], "custom_s3_actions3": [ "" ] }, "metabase-ecs": { "custom_glue_actions0": [ "" ], "custom_s3_actions2": [ "arn:aws:s3:::mp-proxy-main/", "arn:aws:s3:::prod-data-test/", "arn:aws:s3:::mp-proxy-main", "arn:aws:s3:::prod-data-test", "" ], "custom_s3_actions3": [ "arn:aws:s3:::mp-proxy-main/", "arn:aws:s3:::aws-glue-", "arn:aws:s3:::aws-athena-query-results-", "arn:aws:s3:::aws-glue-/", "arn:aws:s3:::athena-examples*", "arn:aws:s3:::prod-data-test/", "arn:aws:s3:::/aws-glue-/", "arn:aws:s3:::mp-proxy-main", "arn:aws:s3:::prod-data-test", "arn:aws:s3:::crawler-public", "" ], "custom_ec2_actions0": [ "" ], "custom_iam_actions0": [ "" ], "custom_cloudwatch_actions0": [ "" ], "custom_s3_actions4": [ "arn:aws:s3:::mp-proxy-main/", "arn:aws:s3:::aws-glue-", "arn:aws:s3:::aws-athena-query-results-", "arn:aws:s3:::athena-examples", "arn:aws:s3:::prod-data-test/", "arn:aws:s3:::mp-proxy-main", "arn:aws:s3:::prod-data-test", "arn:aws:s3:::crawler-public", "" ], "custom_s3_actions5": [ "arn:aws:s3:::mp-proxy-main/", "arn:aws:s3:::aws-glue-", "arn:aws:s3:::aws-athena-query-results-", "arn:aws:s3:::aws-glue-/", "arn:aws:s3:::athena-examples*", "arn:aws:s3:::prod-data-test/", "arn:aws:s3:::/aws-glue-/", "arn:aws:s3:::mp-proxy-main", "arn:aws:s3:::prod-data-test", "arn:aws:s3:::crawler-public", "" ], "custom_logs_actions0": [ "arn:aws:logs::::/aws-glue/" ], "custom_ec2_actions1": [ "arn:aws:ec2:::instance/", "arn:aws:ec2:::network-interface/", "arn:aws:ec2:::security-group/" ], "custom_athena_actions0": [ "" ], "custom_glue_actions1": [ "" ], "custom_s3_actions6": [ "arn:aws:s3:::mp-proxy-main/", "arn:aws:s3:::aws-athena-query-results-", "arn:aws:s3:::athena-examples*", "arn:aws:s3:::prod-data-test/", "arn:aws:s3:::mp-proxy-main", "arn:aws:s3:::prod-data-test", "" ], "custom_sns_actions0": [ "" ], "custom_cloudwatch_actions1": [ "" ], "custom_lakeformation_actions0": [ "" ], "custom_datazone_actions0": [ "" ], "custom_pricing_actions0": [ "" ], "ssm_read": [ "arn:aws:ssm:ap-south-1:744315537043:parameter/conf/metabase-ecs/" ], "custom_ssm_actions0": [ "arn:aws:ssm:ap-south-1:744315537043:parameter/conf/metabase-ecs/" ], "custom_ssm_actions1": [ "arn:aws:ssm:ap-south-1:744315537043:parameter/conf/metabase-ecs/" ], "s3_read": [ "arn:aws:s3:::mp-proxy-main", "arn:aws:s3:::mp-proxy-main/", "arn:aws:s3:::prod-data-test/", "arn:aws:s3:::prod-data-test" ], "s3_write": [ "arn:aws:s3:::mp-proxy-main", "arn:aws:s3:::mp-proxy-main/", "arn:aws:s3:::prod-data-test/", "arn:aws:s3:::prod-data-test" ], "custom_s3_actions0": [ "arn:aws:s3:::mp-proxy-main", "arn:aws:s3:::mp-proxy-main/", "arn:aws:s3:::prod-data-test/", "arn:aws:s3:::prod-data-test" ], "custom_s3_actions1": [ "arn:aws:s3:::mp-proxy-main", "arn:aws:s3:::mp-proxy-main/", "arn:aws:s3:::prod-data-test/", "arn:aws:s3:::prod-data-test" ] }, "auth-service": { "custom_dynamodb_actions1": [ "" ], "custom_dynamodb_actions2": [ "arn:aws:dynamodb:ap-south-1:744315537043:table/authorizationtokens/index/accessToken-index", "arn:aws:dynamodb:ap-south-1:744315537043:table/authorizationtokens", "arn:aws:dynamodb:ap-south-1:744315537043:table/oauth2_tokens/index/user_id-index", "arn:aws:dynamodb:ap-south-1:744315537043:table/googleauthorizationtokens/index/dreamplugUserId-index", "arn:aws:dynamodb:ap-south-1:744315537043:table/authorizationtokens/index/refreshToken-index", "arn:aws:dynamodb:ap-south-1:744315537043:table/authorizationtokens/index/", "arn:aws:dynamodb:ap-south-1:744315537043:table/googleauthorizationtokens", "arn:aws:dynamodb:ap-south-1:744315537043:table/oauth2_tokens", "arn:aws:dynamodb:ap-south-1:744315537043:table/authorizationtokens/index/userId-index" ], "sqs_read": [ "arn:aws:sqs:ap-south-1:744315537043:prod-leaderboard-cache-update-queue-dlq", "arn:aws:sqs:ap-south-1:744315537043:prod-leaderboard-score-update-queue-dlq", "arn:aws:sqs:ap-south-1:744315537043:prod-leaderboard-score-update-queue", "arn:aws:sqs:ap-south-1:744315537043:cp-event-email-queue", "arn:aws:sqs:ap-south-1:744315537043:prod-portkey-plutus-events-queue", "arn:aws:sqs:ap-south-1:744315537043:prod-leaderboard-cache-update-queue" ], "sqs_write": [ "arn:aws:sqs:ap-south-1:744315537043:prod-leaderboard-cache-update-queue-dlq", "arn:aws:sqs:ap-south-1:744315537043:prod-leaderboard-score-update-queue-dlq", "arn:aws:sqs:ap-south-1:744315537043:prod-leaderboard-score-update-queue", "arn:aws:sqs:ap-south-1:744315537043:cp-event-email-queue", "arn:aws:sqs:ap-south-1:744315537043:prod-portkey-plutus-events-queue", "arn:aws:sqs:ap-south-1:744315537043:prod-leaderboard-cache-update-queue" ], "custom_sqs_actions0": [ "" ], "custom_dynamodb_actions0": [ "arn:aws:dynamodb:ap-south-1:744315537043:table/prod_eventi_producer_configurations/index/", "arn:aws:dynamodb:ap-south-1:744315537043:table/oauth2_tokens_v2", "arn:aws:dynamodb:ap-south-1:744315537043:table/truecaller_auth_context_cache/index/requestId-index/index/", "arn:aws:dynamodb:ap-south-1:744315537043:table/truecaller_auth_context_cache", "arn:aws:dynamodb:ap-south-1:744315537043:table/truecaller_auth_context_cache/index/requestId-index", "arn:aws:dynamodb:ap-south-1:744315537043:table/prod_eventi_producer_configurations" ], "custom_dynamodb_actions3": [ "arn:aws:dynamodb:ap-south-1:744315537043:table/prod_eventi_producer_configurations/index/", "arn:aws:dynamodb:ap-south-1:744315537043:table/oauth2_tokens_v2", "arn:aws:dynamodb:ap-south-1:744315537043:table/truecaller_auth_context_cache/index/requestId-index/index/", "arn:aws:dynamodb:ap-south-1:744315537043:table/truecaller_auth_context_cache", "arn:aws:dynamodb:ap-south-1:744315537043:table/truecaller_auth_context_cache/index/requestId-index", "arn:aws:dynamodb:ap-south-1:744315537043:table/prod_eventi_producer_configurations" ], "ssm_read": [ "arn:aws:ssm:ap-south-1:744315537043:parameter/conf/auth-service/" ], "custom_ssm_actions0": [ "arn:aws:ssm:ap-south-1:744315537043:parameter/conf/auth-service/" ], "custom_ssm_actions1": [ "arn:aws:ssm:ap-south-1:744315537043:parameter/conf/auth-service/" ] }, and now I wanna make a tree structure where root node[null] we identify servcie suing arn from json -> servcies as child does than make sub nodes and keep on doing till at leaf node - we do policy binding for the goal to do service <> resource mapping (Parse IAM policies to build a db of sorts)-- I worte a script package main

import ( "encoding/json" "fmt" "io/ioutil" "log" "os" "strings" "sync" )

// ---------------------------------------------------------------------------- // 1) DATA STRUCTURES // ----------------------------------------------------------------------------

// AllServices is our top-level JSON type, e.g.: // // { // "rds-rs-pipeline": { // "s3_read": [...], // "custom_s3_actions0": [...], // ... // }, // "metabase-ecs": { // "custom_glue_actions0": [...], // "custom_s3_actions2": [...], // ... // }, // ... // } type AllServices map[string]map[string][]string

// TreeNode is a hierarchical data structure where each node can have children. type TreeNode struct { Name string Children map[string]*TreeNode mu sync.RWMutex }

// NewTreeNode creates a new node with the given name. func NewTreeNode(name string) *TreeNode { return &TreeNode{ Name: name, Children: make(map[string]*TreeNode), } }

// AddChild returns the existing child if it already exists, or creates a new one. func (n *TreeNode) AddChild(childName string) *TreeNode { n.mu.Lock() defer n.mu.Unlock()

if child, exists := n.Children[childName]; exists {
	return child
}
child := NewTreeNode(childName)
n.Children[childName] = child
return child

}

// GetChild returns a pointer to the named child, or nil if not present. func (n *TreeNode) GetChild(childName string) *TreeNode { n.mu.RLock() defer n.mu.RUnlock()

return n.Children[childName]

}

// GetChildren returns a copy of the children map for safe iteration. func (n *TreeNode) GetChildren() map[string]*TreeNode { n.mu.RLock() defer n.mu.RUnlock()

copyMap := make(map[string]*TreeNode, len(n.Children))
for k, v := range n.Children {
	copyMap[k] = v
}
return copyMap

}

// ---------------------------------------------------------------------------- // 2) HELPER FUNCTIONS // ----------------------------------------------------------------------------

// We only care about these five services: var servicesOfInterest = map[string]bool{ "s3": true, "sns": true, "sqs": true, "kms": true, "events": true, // ARN prefix for EventBridge: arn:aws:events:<region>:<account>:rule/... }

// extractServiceName attempts to read a service from the "key" in your JSON. // // Examples: // // "s3_read" -> "s3" // "kms_decrypt" -> "kms" // "custom_s3_actions0" -> "s3" // "ssm_read" -> "ssm" (but we’ll skip this one later, not in our interest set) // "sqs_write" -> "sqs" func extractServiceName(key string) string { // If it starts with "custom_XXX_", parse out that "XXX" piece if strings.HasPrefix(key, "custom_") { // e.g. "custom_s3_actions0" -> remove "custom_" remaining := strings.TrimPrefix(key, "custom_") // "s3_actions0" -> now let's split on underscore parts := strings.Split(remaining, "_") if len(parts) > 0 { return parts[0] // "s3" } }

// Otherwise "s3_read" => "s3", "kms_decrypt" => "kms", etc.
parts := strings.Split(key, "_")
if len(parts) > 0 {
	return parts[0]
}

// fallback
return key

}

// parseArnForServiceAndResource tries to parse an ARN. // // Example: arn:aws:s3:::my-bucket/path/to/object // returns ("s3", "my-bucket/path/to/object") // // If invalid or a wildcard "", we do our best to handle it. func parseArnForServiceAndResource(arn string) (string, string) { if arn == "" { return "", "*" } const arnPrefix = "arn:aws:" if !strings.HasPrefix(arn, arnPrefix) { // Not a standard ARN => we treat the whole string as "resource" return "", arn }

// ARN format: arn:aws:<service>:<region>:<accountId>:<resource>
parts := strings.SplitN(arn, ":", 6)
if len(parts) < 6 {
	// Malformed or incomplete
	return "", arn
}

service := parts[2]
resource := parts[5]
return service, resource

}

// insertResourcePath inserts a resource path (which may be slash-separated) into the service node. // e.g. "my-bucket/path/" => child "my-bucket" -> child "path" -> child "". func insertResourcePath(serviceNode TreeNode, resource string) { if resource == "" { // Just a wildcard child serviceNode.AddChild("*") return } // Split on slash segments := strings.Split(resource, "/")

current := serviceNode
for _, seg := range segments {
	if seg == "" {
		continue
	}
	current = current.AddChild(seg)
}

}

// ---------------------------------------------------------------------------- // 3) BUILDING THE TREE (BUT ONLY FOR SELECTED SERVICES) // ----------------------------------------------------------------------------

// buildTreeFromAllServices populates a root node with only the services we care about. func buildTreeFromAllServices(allSvcs AllServices) *TreeNode { root := NewTreeNode("ROOT")

// The first-level keys in allSvcs are like "rds-rs-pipeline", "metabase-ecs", etc.
// Inside each, we have e.g. {"s3_read": [...], "kms_decrypt": [...], etc.}
for _, perms := range allSvcs {
	for key, resources := range perms {
		// 1) Derive the "logical" service from the key
		extractedSvc := extractServiceName(key)

		// We'll accumulate resources under whichever service is final
		var serviceNode *TreeNode

		for _, r := range resources {
			// 2) Also parse ARN to see if it identifies a different actual service
			svcFromArn, resourcePart := parseArnForServiceAndResource(r)

			// By default, we use extractedSvc unless ARN’s service is recognized and differs
			finalSvc := extractedSvc
			if svcFromArn != "" && servicesOfInterest[svcFromArn] {
				finalSvc = svcFromArn
			}
			// If the extracted (or final) service is NOT in our set, skip entirely
			if !servicesOfInterest[finalSvc] {
				continue
			}

			// Now that we know the final service, get the service node
			serviceNode = root.AddChild(finalSvc)

			// 3) Insert the resource path under that service
			insertResourcePath(serviceNode, resourcePart)
		}
	}
}
return root

}

// ---------------------------------------------------------------------------- // 4) TRAVERSING THE TREE TO CREATE A FINAL "service => list of resources" MAPPING // ----------------------------------------------------------------------------

// gatherServiceResourceMapping does a DFS through the tree // and builds a map: serviceName => list of "full paths" to leaf nodes. // // For example, if the tree for "s3" has: // // my-bucket -> some-folder -> "" // // it produces: // // "s3" => ["my-bucket/some-folder/"]. func gatherServiceResourceMapping(root *TreeNode) map[string][]string { result := make(map[string][]string)

// Root-level children are the services we actually stored
for serviceName, serviceNode := range root.GetChildren() {
	// Build up all resource paths for this service
	var allPaths []string
	walk(serviceNode, "", &allPaths)

	// We skip "ROOT" if it ever shows up as a child name; but it shouldn’t.
	if serviceName == "ROOT" {
		continue
	}
	// Deduplicate or sort as you like, but here we just store them.
	result[serviceName] = allPaths
}
return result

}

// walk is a helper DFS to build "path" strings from the node hierarchy. func walk(node *TreeNode, prefix string, collector *[]string) { newPrefix := node.Name // Avoid adding "ROOT" into the path if prefix != "" && prefix != "ROOT" { newPrefix = prefix + "/" + node.Name }

children := node.GetChildren()
if len(children) == 0 {
	// Leaf node
	*collector = append(*collector, newPrefix)
	return
}
// Recurse
for _, child := range children {
	walk(child, newPrefix, collector)
}

}

// ---------------------------------------------------------------------------- // 5) MAIN FUNCTION // ---------------------------------------------------------------------------- func main() { if len(os.Args) < 2 { fmt.Println("Usage: go run main.go /path/to/all-services.json") os.Exit(1) } jsonPath := os.Args[1]

// 1) Read the JSON file
data, err := ioutil.ReadFile(jsonPath)
if err != nil {
	log.Fatalf("Failed to read %s: %v", jsonPath, err)
}

// 2) Unmarshal into our AllServices structure
var allSvcs AllServices
if err := json.Unmarshal(data, &allSvcs); err != nil {
	log.Fatalf("Failed to unmarshal JSON: %v", err)
}

// 3) Build the tree for only the services we care about
root := buildTreeFromAllServices(allSvcs)

// 4) Gather the final mapping
serviceMapping := gatherServiceResourceMapping(root)

// 5) Marshal the mapping to JSON format
outputBytes, err := json.MarshalIndent(serviceMapping, "", "  ")
if err != nil {
	log.Fatalf("Failed to marshal final mapping: %v", err)
}

// 6) Write the result (or print to stdout)
outputFile := "service-resource-mapping.json"
if err := ioutil.WriteFile(outputFile, outputBytes, 0644); err != nil {
	log.Fatalf("Failed to write output file: %v", err)
}

log.Printf("Service resource mapping written to %s", outputFile)

} but it doesn't handle mid node wildcard handling for now - I wanna handle it so that at end I am able to Get a tree based retrieval of policy permissions - write detailed go code to do tree based iam policy retrieval for the goal service <> resource mapping (Parse IAM policies to build a db of sorts)

Thought

Building a tree

The user is constructing a Go script to parse JSON data into a hierarchical tree structure, focusing on handling mid-level wildcards like "arn:aws:s3:::bucket/*" in resource paths.

Refining the approach

The user is crafting a function to navigate a tree and match resource paths with wildcards. This involves enhancing the tree-building to handle partial wildcards, increasing the function's complexity.

Crafting wildcard logic

I’m developing a code snippet to handle mid-level wildcards, like creating a wildcard node and storing subsequent segments under it, ensuring the path matches even with further levels.

Mapping wildcard segments

I see now that the function will handle mid-level wildcards by creating a special node to store subsequent segments, facilitating path matching with policy binding.

Reasoned about tree-based IAM policy parsing for 25 seconds

Answer

Below is a more comprehensive example that builds on your existing script, but handles mid‐path wildcards (e.g. “my-bucket/*/some-other”) and then provides a simple mechanism to do a “tree-based retrieval” for matching a particular resource path.

Key additions:

  1. Mid-level Wildcard Handling

    • If we see a wildcard segment in the middle of the path (e.g. my-bucket/*/some-other), we insert a * child and still keep going to the next segment. This effectively means that any subpath in place of * will match as we do lookups.
  2. Policy/Statement Binding

    • Each node in the tree can hold references to the policies or statements that apply at that node. In the example below, we store them as simple strings in a Statements slice (or it could be a slice of struct references, etc.).
    • Any time we insert a resource path (from an IAM statement, for instance), we can store the “policy name” or “statement ID” in that leaf node.
  3. Retrieval / Matching

    • A function FindMatches(tree *TreeNode, pathSegments []string) []*TreeNode that attempts to walk down the tree, respecting both literal children and wildcard children. It returns all matching leaf nodes (or intermediate nodes if that’s your desired logic).
    • For instance, if you have an inserted path my-bucket/*/some-other with a statement ID “Allow-Foo,” then looking up the resource my-bucket/abc/some-other should find that leaf node.

Full Example

package main import ( "encoding/json" "fmt" "io/ioutil" "log" "os" "strings" "sync" ) // // 1) DATA STRUCTURES // // AllServices is our top-level JSON type, e.g.: // { // "rds-rs-pipeline": { // "s3_read": [...], // "custom_s3_actions0": [...], // ... // }, // "metabase-ecs": {...}, // ... // } type AllServices map[string]map[string][]string // TreeNode is a hierarchical data structure where each node can have children. // We add a `Statements` field so we can store references to policies/permissions // that apply at this node. type TreeNode struct { Name string Children map[string]*TreeNode Statements []string // e.g. list of policy names or statement IDs that apply here mu sync.RWMutex } // NewTreeNode creates a new node with the given name. func NewTreeNode(name string) *TreeNode { return &TreeNode{ Name: name, Children: make(map[string]*TreeNode), } } // AddChild returns the existing child if it already exists, or creates a new one. func (n *TreeNode) AddChild(childName string) *TreeNode { n.mu.Lock() defer n.mu.Unlock() if child, exists := n.Children[childName]; exists { return child } child := NewTreeNode(childName) n.Children[childName] = child return child } // AddStatement appends a statement identifier to this node. For real usage, // you might store a richer object than just a string (e.g. the full policy statement). func (n *TreeNode) AddStatement(stmt string) { n.mu.Lock() defer n.mu.Unlock() n.Statements = append(n.Statements, stmt) } // GetChild returns a pointer to the named child, or nil if not present. func (n *TreeNode) GetChild(childName string) *TreeNode { n.mu.RLock() defer n.mu.RUnlock() return n.Children[childName] } // GetChildren returns a copy of the children map for safe iteration. func (n *TreeNode) GetChildren() map[string]*TreeNode { n.mu.RLock() defer n.mu.RUnlock() copyMap := make(map[string]*TreeNode, len(n.Children)) for k, v := range n.Children { copyMap[k] = v } return copyMap } // // 2) HELPER FUNCTIONS // // We only care about these services: var servicesOfInterest = map[string]bool{ "s3": true, "sns": true, "sqs": true, "kms": true, "events": true, // for EventBridge } // extractServiceName attempts to read a service from the "key" in your JSON. // Examples: // "s3_read" -> "s3" // "kms_decrypt" -> "kms" // "custom_s3_actions0" -> "s3" // "ssm_read" -> "ssm" (but we’ll skip it if not in interest set) // "sqs_write" -> "sqs" func extractServiceName(key string) string { if strings.HasPrefix(key, "custom_") { remaining := strings.TrimPrefix(key, "custom_") parts := strings.Split(remaining, "_") if len(parts) > 0 { return parts[0] } } parts := strings.Split(key, "_") if len(parts) > 0 { return parts[0] } return key } // parseArnForServiceAndResource tries to parse an ARN. // // Example: "arn:aws:s3:::my-bucket/path/to/object" // returns: ("s3", "my-bucket/path/to/object") // // If invalid or a wildcard "*", we do our best to handle it. func parseArnForServiceAndResource(arn string) (string, string) { if arn == "*" { return "", "*" } const arnPrefix = "arn:aws:" if !strings.HasPrefix(arn, arnPrefix) { // Not a standard ARN => treat the whole string as resource return "", arn } // ARN format: arn:aws:<service>:<region>:<accountId>:<resource> parts := strings.SplitN(arn, ":", 6) if len(parts) < 6 { // Malformed or incomplete return "", arn } service := parts[2] resource := parts[5] return service, resource } // insertResourcePath inserts a resource path (which may be slash-separated) into the service node. // e.g. "my-bucket/path/*" => child "my-bucket" -> child "path" -> child "*". // // If you have a mid-level wildcard (e.g. "my-bucket/*/some-other"), we do: // my-bucket -> * -> some-other // // The leaf node can hold a policy statement reference, for example. func insertResourcePath(serviceNode *TreeNode, resource string, policyNameOrStmtID string) { // If the entire resource is just "*", create a wildcard child and store the policy there. if resource == "*" { child := serviceNode.AddChild("*") child.AddStatement(policyNameOrStmtID) return } // Split on slash segments := strings.Split(resource, "/") current := serviceNode for i, seg := range segments { if seg == "" { continue // skip empty segments } current = current.AddChild(seg) // If this is the last segment in the path, attach the statement. if i == len(segments)-1 { current.AddStatement(policyNameOrStmtID) } } } // // 3) BUILDING THE TREE // // buildTreeFromAllServices populates a root node with only the services we care about. // // For each top-level "serviceKey" in allSvcs (like "s3_read", "kms_decrypt"), // we figure out which real service that indicates, and then parse each ARN // to see if it might override that. We then insert resource paths in the tree. func buildTreeFromAllServices(allSvcs AllServices) *TreeNode { root := NewTreeNode("ROOT") // The first-level keys in allSvcs are like "rds-rs-pipeline", "metabase-ecs", etc. // The second-level keys are like "s3_read", "custom_s3_actions0", etc. for policyOrRoleName, perms := range allSvcs { // perms is map[string][]string => e.g. {"s3_read": ["arn:aws:s3:::bucket1", ...], ...} for key, resources := range perms { // 1) Derive the "logical" service from the key extractedSvc := extractServiceName(key) for _, r := range resources { // 2) Also parse ARN to see if it identifies a different actual service svcFromArn, resourcePart := parseArnForServiceAndResource(r) finalSvc := extractedSvc if svcFromArn != "" && servicesOfInterest[svcFromArn] { finalSvc = svcFromArn } // If the final service is not in our interest set, skip if !servicesOfInterest[finalSvc] { continue } // Now get the service node from the root serviceNode := root.AddChild(finalSvc) // 3) Insert the resource path // Our "policyNameOrStmtID" can be something that identifies // the parent policy or statement. For example, we’ll just use // "policyOrRoleName/key" so we know which block caused it. stmtID := policyOrRoleName + "/" + key insertResourcePath(serviceNode, resourcePart, stmtID) } } } return root } // // 4) MATCHING / RETRIEVAL // // FindMatches recursively walks the tree to find all nodes that match // the given pathSegments, respecting any wildcard child (“*”) at each level. // // Example usage: // segments := strings.Split("my-bucket/abc/some-other", "/") // matches := FindMatches(serviceNode, segments) // for _, node := range matches { // fmt.Println("Matched node:", node.Name, node.Statements) // } func FindMatches(node *TreeNode, pathSegments []string) []*TreeNode { if len(pathSegments) == 0 { // If we have no more segments, this node itself is a match. // Return it as a single match. return []*TreeNode{node} } // Otherwise, look at the next segment seg := pathSegments[0] var results []*TreeNode children := node.GetChildren() // 1) If the node has a child with the exact name `seg`, recurse if child, ok := children[seg]; ok { results = append(results, FindMatches(child, pathSegments[1:])...) } // 2) If the node has a wildcard child "*", also recurse if wcChild, ok := children["*"]; ok { results = append(results, FindMatches(wcChild, pathSegments[1:])...) } return results } // gatherServiceResourceMapping does a DFS through the tree // and builds a map: serviceName => list of "full paths" (with all leaves). // This was in your original example. We keep it here for demonstration. func gatherServiceResourceMapping(root *TreeNode) map[string][]string { result := make(map[string][]string) // Root-level children are the services for serviceName, serviceNode := range root.GetChildren() { if serviceName == "ROOT" { continue } var allPaths []string walk(serviceNode, "", &allPaths) result[serviceName] = allPaths } return result } // walk is a helper DFS to build "path" strings from the node hierarchy. func walk(node *TreeNode, prefix string, collector *[]string) { newPrefix := node.Name if prefix != "" && prefix != "ROOT" { newPrefix = prefix + "/" + node.Name } children := node.GetChildren() if len(children) == 0 { // Leaf node *collector = append(*collector, newPrefix) return } // Recurse for _, child := range children { walk(child, newPrefix, collector) } } // // 5) MAIN FUNCTION // func main() { if len(os.Args) < 2 { fmt.Println("Usage: go run main.go /path/to/all-services.json") os.Exit(1) } jsonPath := os.Args[1] // 1) Read the JSON file data, err := ioutil.ReadFile(jsonPath) if err != nil { log.Fatalf("Failed to read %s: %v", jsonPath, err) } // 2) Unmarshal into our AllServices structure var allSvcs AllServices if err := json.Unmarshal(data, &allSvcs); err != nil { log.Fatalf("Failed to unmarshal JSON: %v", err) } // 3) Build the tree for only the services we care about root := buildTreeFromAllServices(allSvcs) // 4) Gather the final mapping (like your original approach) serviceMapping := gatherServiceResourceMapping(root) // 5) Marshal the mapping to JSON outputBytes, err := json.MarshalIndent(serviceMapping, "", " ") if err != nil { log.Fatalf("Failed to marshal final mapping: %v", err) } // 6) Write the result outputFile := "service-resource-mapping.json" if err := ioutil.WriteFile(outputFile, outputBytes, 0644); err != nil { log.Fatalf("Failed to write output file: %v", err) } log.Printf("Service resource mapping written to %s", outputFile) // // 7) (Optional) Demonstrate how to do a tree-based lookup // // Example: find all statements that apply to "s3" => "my-bucket/abc/some-other" // In practice you'd do something like: // 1) root.GetChild("s3") to get the S3 node // 2) call FindMatches(s3Node, ["my-bucket","abc","some-other"]) // s3Node := root.GetChild("s3") if s3Node != nil { resourcePath := "my-bucket/abc/some-other" segs := strings.Split(resourcePath, "/") matchedNodes := FindMatches(s3Node, segs) if len(matchedNodes) == 0 { fmt.Println("No match for:", resourcePath) } else { fmt.Printf("Matches for %q:\n", resourcePath) for _, mn := range matchedNodes { fmt.Printf(" Node path: %s, Statements: %v\n", reconstructPath(mn), mn.Statements) } } } } // reconstructPath is a helper to re-construct the path from the node up to root (if you want it). // In this simple example, we just do a naive BFS in the parent's map, or you can store a parent pointer. func reconstructPath(node *TreeNode) string { // This function is a placeholder. In a real design, you might store // a parent pointer in each node. For now, we just return node.Name as a demo. return node.Name }

Explanation of the Key Parts

  1. Mid-Level Wildcards
    In insertResourcePath(...), we do not special-case * to “stop.” Instead, we simply treat "*" like any other path segment and continue inserting child segments.
    Example:

    • Resource string: my-bucket/*/some-other
    • Segments: ["my-bucket", "*", "some-other"]
    • Tree path:
      (serviceNode)
        └── "my-bucket"
              └── "*"
                   └── "some-other"
      

    That means if we call FindMatches(..., ["my-bucket", "abc", "some-other"]), the wildcard at the second level ("*") will allow us to match it.

  2. Storing Policy / Statement References

    • Each TreeNode has a Statements field.
    • Whenever we finish inserting a path (i.e. we reach the last segment), we call current.AddStatement(policyNameOrStmtID).
    • You could store the entire JSON policy statement object, or a custom struct, etc.
  3. Tree-Based Retrieval

    • FindMatches(node, pathSegments) looks at the current node’s children for an exact match or a wildcard match. It recurses down both possibilities.
    • If no more segments remain, it returns the current node. Otherwise, it collects results from each matching child.

With this approach, you get:

  • The hierarchical representation of resource paths, including mid-level wildcards.
  • The ability to figure out “which policy statements match a given ARN’s resource path” by performing the FindMatches operation.

You can adapt this further by:

  • Storing parent pointers (so you can reconstruct full paths more naturally).
  • Handling multiple wildcard segments in the middle.
  • Distinguishing between “leaf” statements vs. inherited statements that apply at higher nodes, etc.

But this template gives you a robust starting point for a tree-based IAM policy–resource mapping system.

Question

I wanna handle the case of mid node wildcards, partial wildcards, global wildcards directly and dynamically in my script --package main

import ( "encoding/json" "fmt" "io/ioutil" "log" "os" "strings" "sync" )

// ---------------------------------------------------------------------------- // 1) DATA STRUCTURES // ----------------------------------------------------------------------------

// AllServices is our top-level JSON type, e.g.: // // { // "rds-rs-pipeline": { // "s3_read": [...], // "custom_s3_actions0": [...], // ... // }, // "metabase-ecs": { // "custom_glue_actions0": [...], // "custom_s3_actions2": [...], // ... // }, // ... // } type AllServices map[string]map[string][]string

// TreeNode is a hierarchical data structure where each node can have children. type TreeNode struct { Name string Children map[string]*TreeNode mu sync.RWMutex }

// NewTreeNode creates a new node with the given name. func NewTreeNode(name string) *TreeNode { return &TreeNode{ Name: name, Children: make(map[string]*TreeNode), } }

// AddChild returns the existing child if it already exists, or creates a new one. func (n *TreeNode) AddChild(childName string) *TreeNode { n.mu.Lock() defer n.mu.Unlock()

if child, exists := n.Children[childName]; exists {
	return child
}
child := NewTreeNode(childName)
n.Children[childName] = child
return child

}

// GetChild returns a pointer to the named child, or nil if not present. func (n *TreeNode) GetChild(childName string) *TreeNode { n.mu.RLock() defer n.mu.RUnlock()

return n.Children[childName]

}

// GetChildren returns a copy of the children map for safe iteration. func (n *TreeNode) GetChildren() map[string]*TreeNode { n.mu.RLock() defer n.mu.RUnlock()

copyMap := make(map[string]*TreeNode, len(n.Children))
for k, v := range n.Children {
	copyMap[k] = v
}
return copyMap

}

// ---------------------------------------------------------------------------- // 2) HELPER FUNCTIONS // ----------------------------------------------------------------------------

// We only care about these five services: var servicesOfInterest = map[string]bool{ "s3": true, "sns": true, "sqs": true, "kms": true, "events": true, // ARN prefix for EventBridge: arn:aws:events:<region>:<account>:rule/... }

// extractServiceName attempts to read a service from the "key" in your JSON. // // Examples: // // "s3_read" -> "s3" // "kms_decrypt" -> "kms" // "custom_s3_actions0" -> "s3" // "ssm_read" -> "ssm" (but we’ll skip this one later, not in our interest set) // "sqs_write" -> "sqs" func extractServiceName(key string) string { // If it starts with "custom_XXX_", parse out that "XXX" piece if strings.HasPrefix(key, "custom_") { // e.g. "custom_s3_actions0" -> remove "custom_" remaining := strings.TrimPrefix(key, "custom_") // "s3_actions0" -> now let's split on underscore parts := strings.Split(remaining, "_") if len(parts) > 0 { return parts[0] // "s3" } }

// Otherwise "s3_read" => "s3", "kms_decrypt" => "kms", etc.
parts := strings.Split(key, "_")
if len(parts) > 0 {
	return parts[0]
}

// fallback
return key

}

// parseArnForServiceAndResource tries to parse an ARN. // // Example: arn:aws:s3:::my-bucket/path/to/object // returns ("s3", "my-bucket/path/to/object") // // If invalid or a wildcard "", we do our best to handle it. func parseArnForServiceAndResource(arn string) (string, string) { if arn == "" { return "", "*" } const arnPrefix = "arn:aws:" if !strings.HasPrefix(arn, arnPrefix) { // Not a standard ARN => we treat the whole string as "resource" return "", arn }

// ARN format: arn:aws:<service>:<region>:<accountId>:<resource>
parts := strings.SplitN(arn, ":", 6)
if len(parts) < 6 {
	// Malformed or incomplete
	return "", arn
}

service := parts[2]
resource := parts[5]
return service, resource

}

// insertResourcePath inserts a resource path (which may be slash-separated) into the service node. // e.g. "my-bucket/path/" => child "my-bucket" -> child "path" -> child "". func insertResourcePath(serviceNode TreeNode, resource string) { if resource == "" { // Just a wildcard child serviceNode.AddChild("*") return } // Split on slash segments := strings.Split(resource, "/")

current := serviceNode
for _, seg := range segments {
	if seg == "" {
		continue
	}
	current = current.AddChild(seg)
}

}

// ---------------------------------------------------------------------------- // 3) BUILDING THE TREE (BUT ONLY FOR SELECTED SERVICES) // ----------------------------------------------------------------------------

// buildTreeFromAllServices populates a root node with only the services we care about. func buildTreeFromAllServices(allSvcs AllServices) *TreeNode { root := NewTreeNode("ROOT")

// The first-level keys in allSvcs are like "rds-rs-pipeline", "metabase-ecs", etc.
// Inside each, we have e.g. {"s3_read": [...], "kms_decrypt": [...], etc.}
for _, perms := range allSvcs {
	for key, resources := range perms {
		// 1) Derive the "logical" service from the key
		extractedSvc := extractServiceName(key)

		// We'll accumulate resources under whichever service is final
		var serviceNode *TreeNode

		for _, r := range resources {
			// 2) Also parse ARN to see if it identifies a different actual service
			svcFromArn, resourcePart := parseArnForServiceAndResource(r)

			// By default, we use extractedSvc unless ARN’s service is recognized and differs
			finalSvc := extractedSvc
			if svcFromArn != "" && servicesOfInterest[svcFromArn] {
				finalSvc = svcFromArn
			}
			// If the extracted (or final) service is NOT in our set, skip entirely
			if !servicesOfInterest[finalSvc] {
				continue
			}

			// Now that we know the final service, get the service node
			serviceNode = root.AddChild(finalSvc)

			// 3) Insert the resource path under that service
			insertResourcePath(serviceNode, resourcePart)
		}
	}
}
return root

}

// ---------------------------------------------------------------------------- // 4) TRAVERSING THE TREE TO CREATE A FINAL "service => list of resources" MAPPING // ----------------------------------------------------------------------------

// gatherServiceResourceMapping does a DFS through the tree // and builds a map: serviceName => list of "full paths" to leaf nodes. // // For example, if the tree for "s3" has: // // my-bucket -> some-folder -> "" // // it produces: // // "s3" => ["my-bucket/some-folder/"]. func gatherServiceResourceMapping(root *TreeNode) map[string][]string { result := make(map[string][]string)

// Root-level children are the services we actually stored
for serviceName, serviceNode := range root.GetChildren() {
	// Build up all resource paths for this service
	var allPaths []string
	walk(serviceNode, "", &allPaths)

	// We skip "ROOT" if it ever shows up as a child name; but it shouldn’t.
	if serviceName == "ROOT" {
		continue
	}
	// Deduplicate or sort as you like, but here we just store them.
	result[serviceName] = allPaths
}
return result

}

// walk is a helper DFS to build "path" strings from the node hierarchy. func walk(node *TreeNode, prefix string, collector *[]string) { newPrefix := node.Name // Avoid adding "ROOT" into the path if prefix != "" && prefix != "ROOT" { newPrefix = prefix + "/" + node.Name }

children := node.GetChildren()
if len(children) == 0 {
	// Leaf node
	*collector = append(*collector, newPrefix)
	return
}
// Recurse
for _, child := range children {
	walk(child, newPrefix, collector)
}

}

// ---------------------------------------------------------------------------- // 5) MAIN FUNCTION // ---------------------------------------------------------------------------- func main() { if len(os.Args) < 2 { fmt.Println("Usage: go run main.go /path/to/all-services.json") os.Exit(1) } jsonPath := os.Args[1]

// 1) Read the JSON file
data, err := ioutil.ReadFile(jsonPath)
if err != nil {
	log.Fatalf("Failed to read %s: %v", jsonPath, err)
}

// 2) Unmarshal into our AllServices structure
var allSvcs AllServices
if err := json.Unmarshal(data, &allSvcs); err != nil {
	log.Fatalf("Failed to unmarshal JSON: %v", err)
}

// 3) Build the tree for only the services we care about
root := buildTreeFromAllServices(allSvcs)

// 4) Gather the final mapping
serviceMapping := gatherServiceResourceMapping(root)

// 5) Marshal the mapping to JSON format
outputBytes, err := json.MarshalIndent(serviceMapping, "", "  ")
if err != nil {
	log.Fatalf("Failed to marshal final mapping: %v", err)
}

// 6) Write the result (or print to stdout)
outputFile := "service-resource-mapping.json"
if err := ioutil.WriteFile(outputFile, outputBytes, 0644); err != nil {
	log.Fatalf("Failed to write output file: %v", err)
}

log.Printf("Service resource mapping written to %s", outputFile)

} using all_Servcies.json { "rds-rs-pipeline": { "s3_read": [ "arn:aws:s3:::sftp-rbl-ags", "arn:aws:s3:::prod-datascience-sms/", "arn:aws:s3:::prod-lending-adhoc", "arn:aws:s3:::prod-lending-adhoc/", "arn:aws:s3:::prod-datascience-sms", "arn:aws:s3:::datascience-store-personalisation/", "arn:aws:s3:::sftp-rbl-ags/", "", "arn:aws:s3:::datascience-store-personalisation" ], "custom_s3_actions0": [ "arn:aws:s3:::sftp-rbl-ags", "arn:aws:s3:::prod-datascience-sms/", "arn:aws:s3:::prod-lending-adhoc", "arn:aws:s3:::prod-lending-adhoc/", "arn:aws:s3:::prod-datascience-sms", "arn:aws:s3:::datascience-store-personalisation/", "arn:aws:s3:::sftp-rbl-ags/", "", "arn:aws:s3:::datascience-store-personalisation" ], "custom_s3-object-lambda_actions0": [ "" ], "ssm_read": [ "arn:aws:ssm:ap-south-1:744315537043:parameter/conf/rds-rs-pipeline/" ], "kms_decrypt": [ "arn:aws:kms:ap-south-1:744315537043:key/af547ea6-b4fd-4050-9185-62068e0da254", "arn:aws:kms:ap-south-1:744315537043:key/c1b80926-ce55-4905-a92a-a1a830ffbf5b", "arn:aws:kms:ap-south-1:744315537043:key/94baadce-0db1-41fe-91ff-f3be240fb6c2" ], "kms_generate_data_key": [ "arn:aws:kms:ap-south-1:744315537043:key/af547ea6-b4fd-4050-9185-62068e0da254", "arn:aws:kms:ap-south-1:744315537043:key/c1b80926-ce55-4905-a92a-a1a830ffbf5b", "arn:aws:kms:ap-south-1:744315537043:key/94baadce-0db1-41fe-91ff-f3be240fb6c2" ], "kms_sign": [ "arn:aws:kms:ap-south-1:744315537043:key/af547ea6-b4fd-4050-9185-62068e0da254", "arn:aws:kms:ap-south-1:744315537043:key/c1b80926-ce55-4905-a92a-a1a830ffbf5b", "arn:aws:kms:ap-south-1:744315537043:key/94baadce-0db1-41fe-91ff-f3be240fb6c2" ], "kms_verify": [ "arn:aws:kms:ap-south-1:744315537043:key/af547ea6-b4fd-4050-9185-62068e0da254", "arn:aws:kms:ap-south-1:744315537043:key/c1b80926-ce55-4905-a92a-a1a830ffbf5b", "arn:aws:kms:ap-south-1:744315537043:key/94baadce-0db1-41fe-91ff-f3be240fb6c2" ], "custom_dynamodb_actions0": [ "arn:aws:dynamodb:ap-south-1:744315537043:table/external_reward_report_config", "arn:aws:dynamodb:ap-south-1:744315537043:table/sms_communications", "arn:aws:dynamodb:us-east-1:744315537043:table/external_reward_report_config" ], "custom_firehose_actions0": [ "arn:aws:firehose:ap-south-1:744315537043:deliverystream/prod_reward_usage_attributes" ], "custom_firehose_actions1": [ "" ], "s3_write": [ "arn:aws:s3:::sftp-rbl-ags", "arn:aws:s3:::prod-datascience-sms/", "arn:aws:s3:::prod-lending-adhoc", "arn:aws:s3:::prod-lending-adhoc/", "arn:aws:s3:::prod-datascience-sms", "arn:aws:s3:::datascience-store-personalisation/", "arn:aws:s3:::sftp-rbl-ags/", "arn:aws:s3:::datascience-store-personalisation" ], "custom_s3_actions1": [ "arn:aws:s3:::eventi-audit", "arn:aws:s3:::eventi-audit/", "arn:aws:s3:::prod-datascience-sms/", "arn:aws:s3:::prod-data-", "arn:aws:s3:::prod-datascience-sms", "arn:aws:s3:::internal-payments-recon", "arn:aws:s3:::stg-redshift", "arn:aws:s3:::internal-payments-recon/", "arn:aws:s3:::stg-redshift/" ], "custom_s3_actions2": [ "" ], "custom_ssm_actions0": [ "" ], "custom_ssm_actions1": [ "arn:aws:ssm:ap-south-1:744315537043:parameter/conf/rds-rs-pipeline/" ], "custom_s3_actions3": [ "" ] }, "metabase-ecs": { "custom_glue_actions0": [ "" ], "custom_s3_actions2": [ "arn:aws:s3:::mp-proxy-main/", "arn:aws:s3:::prod-data-test/", "arn:aws:s3:::mp-proxy-main", "arn:aws:s3:::prod-data-test", "" ], "custom_s3_actions3": [ "arn:aws:s3:::mp-proxy-main/", "arn:aws:s3:::aws-glue-", "arn:aws:s3:::aws-athena-query-results-", "arn:aws:s3:::aws-glue-/", "arn:aws:s3:::athena-examples", "arn:aws:s3:::prod-data-test/", "arn:aws:s3:::/aws-glue-/", "arn:aws:s3:::mp-proxy-main", "arn:aws:s3:::prod-data-test", "arn:aws:s3:::crawler-public", "" ], "custom_ec2_actions0": [ "" ], "custom_iam_actions0": [ "" ], "custom_cloudwatch_actions0": [ "" ], "custom_s3_actions4": [ "arn:aws:s3:::mp-proxy-main/", "arn:aws:s3:::aws-glue-", "arn:aws:s3:::aws-athena-query-results-", "arn:aws:s3:::athena-examples", "arn:aws:s3:::prod-data-test/", "arn:aws:s3:::mp-proxy-main", "arn:aws:s3:::prod-data-test", "arn:aws:s3:::crawler-public", "" ], "custom_s3_actions5": [ "arn:aws:s3:::mp-proxy-main/", "arn:aws:s3:::aws-glue-", "arn:aws:s3:::aws-athena-query-results-", "arn:aws:s3:::aws-glue-/", "arn:aws:s3:::athena-examples*", "arn:aws:s3:::prod-data-test/", "arn:aws:s3:::/aws-glue-/", "arn:aws:s3:::mp-proxy-main", "arn:aws:s3:::prod-data-test", "arn:aws:s3:::crawler-public", "" ], "custom_logs_actions0": [ "arn:aws:logs::::/aws-glue/" ], "custom_ec2_actions1": [ "arn:aws:ec2:::instance/", "arn:aws:ec2:::network-interface/", "arn:aws:ec2:::security-group/" ], "custom_athena_actions0": [ "" ], "custom_glue_actions1": [ "" ], "custom_s3_actions6": [ "arn:aws:s3:::mp-proxy-main/", "arn:aws:s3:::aws-athena-query-results-", "arn:aws:s3:::athena-examples*", "arn:aws:s3:::prod-data-test/", "arn:aws:s3:::mp-proxy-main", "arn:aws:s3:::prod-data-test", "" ], "custom_sns_actions0": [ "" ], "custom_cloudwatch_actions1": [ "" ], "custom_lakeformation_actions0": [ "" ], "custom_datazone_actions0": [ "" ], "custom_pricing_actions0": [ "" ], "ssm_read": [ "arn:aws:ssm:ap-south-1:744315537043:parameter/conf/metabase-ecs/" ], "custom_ssm_actions0": [ "arn:aws:ssm:ap-south-1:744315537043:parameter/conf/metabase-ecs/" ], "custom_ssm_actions1": [ "arn:aws:ssm:ap-south-1:744315537043:parameter/conf/metabase-ecs/" ], "s3_read": [ "arn:aws:s3:::mp-proxy-main", "arn:aws:s3:::mp-proxy-main/", "arn:aws:s3:::prod-data-test/", "arn:aws:s3:::prod-data-test" ], "s3_write": [ "arn:aws:s3:::mp-proxy-main", "arn:aws:s3:::mp-proxy-main/", "arn:aws:s3:::prod-data-test/", "arn:aws:s3:::prod-data-test" ], "custom_s3_actions0": [ "arn:aws:s3:::mp-proxy-main", "arn:aws:s3:::mp-proxy-main/", "arn:aws:s3:::prod-data-test/", "arn:aws:s3:::prod-data-test" ], "custom_s3_actions1": [ "arn:aws:s3:::mp-proxy-main", "arn:aws:s3:::mp-proxy-main/", "arn:aws:s3:::prod-data-test/", "arn:aws:s3:::prod-data-test" ] }, "auth-service": { "custom_dynamodb_actions1": [ "" ], "custom_dynamodb_actions2": [ "arn:aws:dynamodb:ap-south-1:744315537043:table/authorizationtokens/index/accessToken-index", "arn:aws:dynamodb:ap-south-1:744315537043:table/authorizationtokens", "arn:aws:dynamodb:ap-south-1:744315537043:table/oauth2_tokens/index/user_id-index", "arn:aws:dynamodb:ap-south-1:744315537043:table/googleauthorizationtokens/index/dreamplugUserId-index", "arn:aws:dynamodb:ap-south-1:744315537043:table/authorizationtokens/index/refreshToken-index", "arn:aws:dynamodb:ap-south-1:744315537043:table/authorizationtokens/index/", "arn:aws:dynamodb:ap-south-1:744315537043:table/googleauthorizationtokens", "arn:aws:dynamodb:ap-south-1:744315537043:table/oauth2_tokens", "arn:aws:dynamodb:ap-south-1:744315537043:table/authorizationtokens/index/userId-index" ], "sqs_read": [ "arn:aws:sqs:ap-south-1:744315537043:prod-leaderboard-cache-update-queue-dlq", "arn:aws:sqs:ap-south-1:744315537043:prod-leaderboard-score-update-queue-dlq", "arn:aws:sqs:ap-south-1:744315537043:prod-leaderboard-score-update-queue", "arn:aws:sqs:ap-south-1:744315537043:cp-event-email-queue", "arn:aws:sqs:ap-south-1:744315537043:prod-portkey-plutus-events-queue", "arn:aws:sqs:ap-south-1:744315537043:prod-leaderboard-cache-update-queue" ], "sqs_write": [ "arn:aws:sqs:ap-south-1:744315537043:prod-leaderboard-cache-update-queue-dlq", "arn:aws:sqs:ap-south-1:744315537043:prod-leaderboard-score-update-queue-dlq", "arn:aws:sqs:ap-south-1:744315537043:prod-leaderboard-score-update-queue", "arn:aws:sqs:ap-south-1:744315537043:cp-event-email-queue", "arn:aws:sqs:ap-south-1:744315537043:prod-portkey-plutus-events-queue", "arn:aws:sqs:ap-south-1:744315537043:prod-leaderboard-cache-update-queue" ], "custom_sqs_actions0": [ "" ], "custom_dynamodb_actions0": [ "arn:aws:dynamodb:ap-south-1:744315537043:table/prod_eventi_producer_configurations/index/", "arn:aws:dynamodb:ap-south-1:744315537043:table/oauth2_tokens_v2", "arn:aws:dynamodb:ap-south-1:744315537043:table/truecaller_auth_context_cache/index/requestId-index/index/", "arn:aws:dynamodb:ap-south-1:744315537043:table/truecaller_auth_context_cache", "arn:aws:dynamodb:ap-south-1:744315537043:table/truecaller_auth_context_cache/index/requestId-index", "arn:aws:dynamodb:ap-south-1:744315537043:table/prod_eventi_producer_configurations" ], "custom_dynamodb_actions3": [ "arn:aws:dynamodb:ap-south-1:744315537043:table/prod_eventi_producer_configurations/index/", "arn:aws:dynamodb:ap-south-1:744315537043:table/oauth2_tokens_v2", "arn:aws:dynamodb:ap-south-1:744315537043:table/truecaller_auth_context_cache/index/requestId-index/index/", "arn:aws:dynamodb:ap-south-1:744315537043:table/truecaller_auth_context_cache", "arn:aws:dynamodb:ap-south-1:744315537043:table/truecaller_auth_context_cache/index/requestId-index", "arn:aws:dynamodb:ap-south-1:744315537043:table/prod_eventi_producer_configurations" ], "ssm_read": [ "arn:aws:ssm:ap-south-1:744315537043:parameter/conf/auth-service/" ], "custom_ssm_actions0": [ "arn:aws:ssm:ap-south-1:744315537043:parameter/conf/auth-service/" ], "custom_ssm_actions1": [ "arn:aws:ssm:ap-south-1:744315537043:parameter/conf/auth-service/" ] }, "user-service-attribute": { "ssm_read": [ "arn:aws:ssm:ap-south-1:744315537043:parameter/conf/user-service-attribute/" ], "custom_ssm_actions0": [ "arn:aws:ssm:ap-south-1:744315537043:parameter/conf/user-service-attribute/" ], "custom_ssm_actions1": [ "arn:aws:ssm:ap-south-1:744315537043:parameter/conf/user-service-attribute/" ], "sqs_read": [ "arn:aws:sqs:ap-south-1:744315537043:prod-eventi-kafka-producer-dlq", "arn:aws:sqs:ap-south-1:744315537043:prod-notification-queue", "arn:aws:sqs:ap-south-1:744315537043:prod-in-app-consumer-queue", "arn:aws:sqs:ap-south-1:744315537043:prod-eventi-queue", "arn:aws:sqs:ap-south-1:744315537043:prod-notification-promotion-queue" ], "sqs_write": [ "arn:aws:sqs:ap-south-1:744315537043:prod-eventi-kafka-producer-dlq", "arn:aws:sqs:ap-south-1:744315537043:prod-notification-queue", "arn:aws:sqs:ap-south-1:744315537043:prod-in-app-consumer-queue", "arn:aws:sqs:ap-south-1:744315537043:prod-eventi-queue", "arn:aws:sqs:ap-south-1:744315537043:prod-notification-promotion-queue" ], "custom_sqs_actions1": [ "arn:aws:sqs:ap-south-1:744315537043:prod-eventi-kafka-producer-dlq", "arn:aws:sqs:ap-south-1:744315537043:prod-notification-queue", "arn:aws:sqs:ap-south-1:744315537043:prod-in-app-consumer-queue", "arn:aws:sqs:ap-south-1:744315537043:prod-eventi-queue", "arn:aws:sqs:ap-south-1:744315537043:prod-notification-promotion-queue", "" ], "custom_sqs_actions0": [ "" ], "custom_logs_actions1": [ "" ], "custom_cloudwatch_actions0": [ "" ], "custom_cloudwatch_actions2": [ "" ], "custom_sns_actions1": [ "arn:aws:sns:ap-south-1:744315537043:prod-eventi-sns" ], "custom_sns_actions2": [ "" ], "custom_dynamodb_actions0": [ "arn:aws:dynamodb:ap-south-1:744315537043:table/prod_user_attributes" ], "custom_dynamodb_actions3": [ "arn:aws:dynamodb:ap-south-1:744315537043:table/prod_eventi_consumer_configurations", "arn:aws:dynamodb:ap-south-1:744315537043:table/prod_eventi_producer_configurations", "arn:aws:dynamodb:ap-south-1:744315537043:table/prod_user_attributes" ], "custom_dynamodb_actions4": [ "arn:aws:dynamodb:ap-south-1:744315537043:table/prod_eventi_consumer_configurations", "arn:aws:dynamodb:ap-south-1:744315537043:table/prod_eventi_producer_configurations", "arn:aws:dynamodb:ap-south-1:744315537043:table/prod_user_attributes" ], "custom_dynamodb_actions2": [ "arn:aws:dynamodb:ap-south-1:744315537043:table/prod_eventi_consumer_configurations", "arn:aws:dynamodb:ap-south-1:744315537043:table/prod_eventi_producer_configurations" ], "custom_kinesis_actions0": [ "arn:aws:kinesis:ap-south-1:744315537043:stream/prod_eventi" ] }, "vmagent": {}, "otp-service": { "sqs_read": [ "arn:aws:sqs:ap-south-1:744315537043:prod-eventi-kafka-producer-dlq", "arn:aws:sqs:ap-south-1:744315537043:prod-notification-queue", "arn:aws:sqs:ap-south-1:744315537043:prod-in-app-consumer-queue", "arn:aws:sqs:ap-south-1:744315537043:prod-eventi-queue", "arn:aws:sqs:ap-south-1:744315537043:prod-notification-promotion-queue" ], "sqs_write": [ "arn:aws:sqs:ap-south-1:744315537043:prod-eventi-kafka-producer-dlq", "arn:aws:sqs:ap-south-1:744315537043:prod-notification-queue", "arn:aws:sqs:ap-south-1:744315537043:prod-in-app-consumer-queue", "arn:aws:sqs:ap-south-1:744315537043:prod-eventi-queue", "arn:aws:sqs:ap-south-1:744315537043:prod-notification-promotion-queue" ], "custom_sqs_actions1": [ "arn:aws:sqs:ap-south-1:744315537043:prod-eventi-kafka-producer-dlq", "arn:aws:sqs:ap-south-1:744315537043:prod-notification-queue", "arn:aws:sqs:ap-south-1:744315537043:prod-in-app-consumer-queue", "arn:aws:sqs:ap-south-1:744315537043:prod-eventi-queue", "arn:aws:sqs:ap-south-1:744315537043:prod-notification-promotion-queue", "" ], "custom_sqs_actions0": [ "" ], "custom_logs_actions1": [ "" ], "custom_cloudwatch_actions0": [ "" ], "custom_cloudwatch_actions2": [ "" ], "custom_dynamodb_actions1": [ "" ], "custom_dynamodb_actions2": [ "arn:aws:dynamodb:ap-south-1:744315537043:table/prod_eventi_consumer_configurations", "arn:aws:dynamodb:ap-south-1:744315537043:table/otps", "arn:aws:dynamodb:ap-south-1:744315537043:table/otp_templates", "arn:aws:dynamodb:ap-south-1:744315537043:table/prod_eventi_producer_configurations", "", "arn:aws:dynamodb:ap-south-1:744315537043:table/OTPVerifyTracker" ], "custom_dynamodb_actions3": [ "arn:aws:dynamodb:ap-south-1:744315537043:table/prod_eventi_consumer_configurations", "arn:aws:dynamodb:ap-south-1:744315537043:table/otps", "arn:aws:dynamodb:ap-south-1:744315537043:table/otp_templates", "arn:aws:dynamodb:ap-south-1:744315537043:table/prod_eventi_producer_configurations", "", "arn:aws:dynamodb:ap-south-1:744315537043:table/OTPVerifyTracker" ], "custom_dynamodb_actions4": [ "arn:aws:dynamodb:ap-south-1:744315537043:table/prod_eventi_consumer_configurations", "arn:aws:dynamodb:ap-south-1:744315537043:table/otps", "arn:aws:dynamodb:ap-south-1:744315537043:table/otp_templates", "arn:aws:dynamodb:ap-south-1:744315537043:table/prod_eventi_producer_configurations", "", "arn:aws:dynamodb:ap-south-1:744315537043:table/OTPVerifyTracker" ], "custom_dynamodb_actions0": [ "arn:aws:dynamodb:ap-south-1:744315537043:table/prod_eventi_producer_configurations" ], "custom_sns_actions1": [ "arn:aws:sns:ap-south-1:744315537043:prod-eventi-sns" ], "custom_sns_actions2": [ "arn:aws:sns:ap-south-1:744315537043:prod-eventi-sns", "" ], "ssm_read": [ "arn:aws:ssm:ap-south-1:744315537043:parameter/conf/otp-service/" ], "custom_ssm_actions0": [ "arn:aws:ssm:ap-south-1:744315537043:parameter/conf/otp-service/" ], "custom_ssm_actions1": [ "arn:aws:ssm:ap-south-1:744315537043:parameter/conf/otp-service/" ], "custom_kinesis_actions0": [ "arn:aws:kinesis:ap-south-1:744315537043:stream/prod_eventi" ] }, "fraud-engine-graviton-test": {}, "yoda-bot": {}, "wayne": { "ssm_read": [ "arn:aws:ssm:ap-south-1:744315537043:parameter/conf/wayne/" ], "custom_ssm_actions0": [ "arn:aws:ssm:ap-south-1:744315537043:parameter/conf/wayne/" ], "custom_ssm_actions1": [ "arn:aws:ssm:ap-south-1:744315537043:parameter/conf/wayne/" ] }, "hba-service": { "custom_logs_actions1": [ "" ], "custom_cloudwatch_actions0": [ "" ], "custom_cloudwatch_actions2": [ "" ], "custom_dynamodb_actions0": [ "arn:aws:dynamodb:ap-south-1:744315537043:table/hba_merch_onboarding" ], "custom_dynamodb_actions3": [ "arn:aws:dynamodb:ap-south-1:744315537043:table/hba_merch_onboarding" ], "custom_dynamodb_actions4": [ "arn:aws:dynamodb:ap-south-1:744315537043:table/hba_merch_onboarding" ], "ssm_read": [ "arn:aws:ssm:ap-south-1:744315537043:parameter/conf/hba-service/" ], "custom_ssm_actions0": [ "arn:aws:ssm:ap-south-1:744315537043:parameter/conf/hba-service/" ], "custom_ssm_actions1": [ "arn:aws:ssm:ap-south-1:744315537043:parameter/conf/hba-service/" ] }, "cred-web": { "custom_logs_actions1": [ "" ], "custom_cloudwatch_actions0": [ "" ], "custom_cloudwatch_actions2": [ "" ], "s3_read": [ "arn:aws:s3:::cred-internal-webhosting-prod/", "arn:aws:s3:::cred-heartbeat-webhosting-prod/", "arn:aws:s3:::web-money-sdk-prod", "arn:aws:s3:::cred-pay-prod", "arn:aws:s3:::web-merchant-app-prod", "arn:aws:s3:::web-cash-collection-assets-prod", "arn:aws:s3:::preprod-webview-assets/", "arn:aws:s3:::cred-heartbeat-webhosting-prod", "arn:aws:s3:::prod-webview-assets", "arn:aws:s3:::cred-pay-prod/", "arn:aws:s3:::web-money-sdk-prod/", "arn:aws:s3:::web-merchant-app-prod/", "arn:aws:s3:::web-cash-collection-assets-prod/", "arn:aws:s3:::prod-webview-assets/", "arn:aws:s3:::prod-cred-web-assets", "arn:aws:s3:::prod-cred-web-assets/", "arn:aws:s3:::preprod-webview-assets" ], "s3_write": [ "arn:aws:s3:::cred-internal-webhosting-prod/", "arn:aws:s3:::cred-heartbeat-webhosting-prod/", "arn:aws:s3:::web-money-sdk-prod", "arn:aws:s3:::cred-pay-prod", "arn:aws:s3:::web-merchant-app-prod", "arn:aws:s3:::web-cash-collection-assets-prod", "arn:aws:s3:::preprod-webview-assets/", "arn:aws:s3:::cred-heartbeat-webhosting-prod", "arn:aws:s3:::prod-webview-assets", "arn:aws:s3:::cred-pay-prod/", "arn:aws:s3:::web-money-sdk-prod/", "arn:aws:s3:::web-merchant-app-prod/", "arn:aws:s3:::web-cash-collection-assets-prod/", "arn:aws:s3:::prod-webview-assets/", "arn:aws:s3:::prod-cred-web-assets", "arn:aws:s3:::prod-cred-web-assets/", "arn:aws:s3:::preprod-webview-assets" ], "custom_s3_actions0": [ "arn:aws:s3:::cred-internal-webhosting-prod/", "arn:aws:s3:::cred-heartbeat-webhosting-prod/", "arn:aws:s3:::web-money-sdk-prod", "arn:aws:s3:::cred-pay-prod", "arn:aws:s3:::web-merchant-app-prod", "arn:aws:s3:::web-cash-collection-assets-prod", "arn:aws:s3:::preprod-webview-assets/", "arn:aws:s3:::cred-heartbeat-webhosting-prod", "arn:aws:s3:::prod-webview-assets", "arn:aws:s3:::cred-pay-prod/", "arn:aws:s3:::web-money-sdk-prod/", "arn:aws:s3:::web-merchant-app-prod/", "arn:aws:s3:::web-cash-collection-assets-prod/", "arn:aws:s3:::prod-webview-assets/", "arn:aws:s3:::prod-cred-web-assets", "arn:aws:s3:::prod-cred-web-assets/", "arn:aws:s3:::preprod-webview-assets" ], "custom_s3_actions1": [ "arn:aws:s3:::cred-internal-webhosting-prod/", "arn:aws:s3:::cred-heartbeat-webhosting-prod/", "arn:aws:s3:::web-money-sdk-prod", "arn:aws:s3:::cred-pay-prod", "arn:aws:s3:::web-merchant-app-prod", "arn:aws:s3:::web-cash-collection-assets-prod", "arn:aws:s3:::preprod-webview-assets/", "arn:aws:s3:::cred-heartbeat-webhosting-prod", "arn:aws:s3:::prod-webview-assets", "arn:aws:s3:::cred-pay-prod/", "arn:aws:s3:::web-money-sdk-prod/", "arn:aws:s3:::web-merchant-app-prod/", "arn:aws:s3:::web-cash-collection-assets-prod/", our main goal is to do service <> resource mapping (Parse IAM policies to build a db of sorts)and I wanna specifically do too using a tree based structure where each service will be the first child nodes and then we subsequently make sub Childs

Thought

Enhancing ARNs

I'm thinking through dynamically handling mid-level, partial, and global wildcards in ARNs for a script. This involves interpreting patterns like "prod-data-" or "aws-glue-".

Crafting wildcard logic

I’m designing a custom retrieval function that identifies segment matches, including patterns like "prod-data-*", through a depth-first search approach.

Laying out the code

I'm focusing on a "matchesSegment" function for pattern matching, incorporating wildcard logic and using depth-first search. Progress is being made on the structure and references of resources.

Mapping out insertion and retrieval

I’m thinking through how to insert and retrieve segments, focusing on prefix and shell pattern matching using filepath.Match and strings.HasSuffix for efficient partial wildcard handling.

Reasoned about tree-based IAM policy mapping for 17 seconds

Answer

Below is an expanded version of your script that handles:

  1. Mid-Level Wildcards (e.g. "my-bucket/*/path").
  2. Full/Global Wildcards (just "*").
  3. Partial Wildcards (e.g. "prod-data-*" which should match "prod-data-test", "prod-data-123", etc.).

Key Points:

  • We store each path segment literally in the tree (e.g. prod-data-* remains exactly that).
  • When we search or collect the resources, we perform the matching logic:
    • If the child’s name is "*" (a full wildcard), it matches anything at that level.
    • If the child’s name ends with "*" (a partial wildcard, e.g. "prefix-*"), then we do a prefix match.
    • Otherwise, we do an exact string match.

This approach lets you parse ARNs, split them by slash ("/"), build your service–resource hierarchy, and later perform tree-based lookups where partial or full wildcards may appear anywhere in the path.


Full Example Code

package main import ( "encoding/json" "fmt" "io/ioutil" "log" "os" "strings" "sync" ) // // 1) DATA STRUCTURES // // AllServices is your top-level structure parsed from all_services.json type AllServices map[string]map[string][]string // TreeNode is a node in a tree. Each node can hold zero or more children. // We also add a Statements field in case you want to store references to IAM statements. type TreeNode struct { Name string Children map[string]*TreeNode Statements []string // e.g. store policy or statement IDs that apply mu sync.RWMutex } // NewTreeNode creates a new node. func NewTreeNode(name string) *TreeNode { return &TreeNode{ Name: name, Children: make(map[string]*TreeNode), } } // AddChild either returns an existing child if it already exists or creates a new one. func (n *TreeNode) AddChild(childName string) *TreeNode { n.mu.Lock() defer n.mu.Unlock() if child, exists := n.Children[childName]; exists { return child } child := NewTreeNode(childName) n.Children[childName] = child return child } // AddStatement appends a statement identifier to this node. In a real-world scenario, // you might store richer objects (the entire IAM statement, policy name, etc.). func (n *TreeNode) AddStatement(stmt string) { n.mu.Lock() defer n.mu.Unlock() n.Statements = append(n.Statements, stmt) } // GetChildren returns a copy of the children map for safe iteration. func (n *TreeNode) GetChildren() map[string]*TreeNode { n.mu.RLock() defer n.mu.RUnlock() copyMap := make(map[string]*TreeNode, len(n.Children)) for k, v := range n.Children { copyMap[k] = v } return copyMap } // // 2) HELPER FUNCTIONS // // We only care about these services: var servicesOfInterest = map[string]bool{ "s3": true, "sns": true, "sqs": true, "kms": true, "events": true, } // extractServiceName tries to interpret the second-level key to find the AWS service name. // E.g. "s3_read" -> "s3", "custom_s3_actions0" -> "s3", etc. func extractServiceName(key string) string { if strings.HasPrefix(key, "custom_") { remaining := strings.TrimPrefix(key, "custom_") // e.g. s3_actions0 parts := strings.Split(remaining, "_") if len(parts) > 0 { return parts[0] // s3 } } parts := strings.Split(key, "_") if len(parts) > 0 { return parts[0] } return key } // parseArnForServiceAndResource tries to parse an ARN. // // Example: "arn:aws:s3:::my-bucket/path/to/object" // returns ("s3", "my-bucket/path/to/object") // // If invalid or a global wildcard "*", it returns ("", "*"). func parseArnForServiceAndResource(arn string) (string, string) { if arn == "*" { return "", "*" } const arnPrefix = "arn:aws:" if !strings.HasPrefix(arn, arnPrefix) { // Not a standard ARN => treat the entire string as resource return "", arn } // arn:aws:<service>:<region>:<accountId>:<resource> parts := strings.SplitN(arn, ":", 6) if len(parts) < 6 { // Malformed or incomplete return "", arn } service := parts[2] resource := parts[5] return service, resource } // insertResourcePath splits the resource string by "/" and inserts each segment into the tree. // If the resource is "*", it creates a wildcard child. We also store a statement ID or some // identifier at the leaf to know which policy introduced it. // // For partial wildcards like "prod-data-*", we do not do any special parse logic. We store that // entire segment as "prod-data-*". The matching logic later will handle whether a segment matches it. func insertResourcePath(serviceNode *TreeNode, resource string, stmtID string) { if resource == "*" { // Entire resource is a wildcard child := serviceNode.AddChild("*") // Mark that child's node with the statement child.AddStatement(stmtID) return } segments := strings.Split(resource, "/") current := serviceNode for i, seg := range segments { if seg == "" { continue } current = current.AddChild(seg) // If this is the last segment, store the statement reference if i == len(segments)-1 { current.AddStatement(stmtID) } } } // // 3) BUILDING THE TREE // // buildTreeFromAllServices populates a tree with only the services we care about. Each // resource path from the JSON is inserted into the corresponding service node. func buildTreeFromAllServices(allSvcs AllServices) *TreeNode { root := NewTreeNode("ROOT") // The top-level map is like: "rds-rs-pipeline" -> {"s3_read": [...], ...} // We'll treat "rds-rs-pipeline" as the "policy name" or "role name" for demonstration. for policyOrRoleName, perms := range allSvcs { for key, resources := range perms { // Derive the "logical" service from the key extractedSvc := extractServiceName(key) for _, arnOrRes := range resources { // Parse the ARN to see if it might override the extracted service svcFromArn, resourcePart := parseArnForServiceAndResource(arnOrRes) finalSvc := extractedSvc if svcFromArn != "" && servicesOfInterest[svcFromArn] { finalSvc = svcFromArn } if !servicesOfInterest[finalSvc] { // skip if not in our list continue } // Add or retrieve the node for this final service svcNode := root.AddChild(finalSvc) // Insert the path. We'll store a reference to the origin (e.g. policy name + key). stmtID := policyOrRoleName + "/" + key insertResourcePath(svcNode, resourcePart, stmtID) } } } return root } // // 4) MATCHING / LOOKUPS // // matchesSegment checks if a path segment from the ARN we are searching for // matches a child node name (which could be "*", "prefix-*", or exact). func matchesSegment(childName, seg string) bool { // 1) Full wildcard node? if childName == "*" { return true } // 2) Partial wildcard? e.g. "prod-data-*" if strings.HasSuffix(childName, "*") { prefix := childName[:len(childName)-1] // remove trailing '*' return strings.HasPrefix(seg, prefix) } // 3) Otherwise, must match exactly return (childName == seg) } // FindMatches recursively walks the tree from the given node, attempting to match // each path segment in `pathSegments`. It returns all nodes that match the final segment. // // Example usage: // segments := strings.Split("my-bucket/some/path", "/") // matches := FindMatches(s3Node, segments) func FindMatches(node *TreeNode, pathSegments []string) []*TreeNode { if len(pathSegments) == 0 { // No more segments to match => this node is a "match" endpoint return []*TreeNode{node} } currentSeg := pathSegments[0] remainder := pathSegments[1:] var result []*TreeNode for childName, childNode := range node.GetChildren() { if matchesSegment(childName, currentSeg) { // Recurse down result = append(result, FindMatches(childNode, remainder)...) } } return result } // // 5) GATHERING & PRINTING THE "SERVICE => RESOURCE" MAPPING // // gatherServiceResourceMapping does a DFS and collects full paths to leaf nodes. // Because we have partial wildcards, you’ll see them in the final strings. // Because we also store statements, you could retrieve them from the node if you wish. // func gatherServiceResourceMapping(root *TreeNode) map[string][]string { output := make(map[string][]string) // The children of ROOT are the services for svcName, svcNode := range root.GetChildren() { if svcName == "ROOT" { continue } var allPaths []string buildPathsDfs(svcNode, "", &allPaths) output[svcName] = allPaths } return output } // buildPathsDfs traverses the tree depth-first and collects "path" strings. // e.g. child("my-bucket") -> child("*") -> child("some-folder") => "my-bucket/*/some-folder" func buildPathsDfs(node *TreeNode, prefix string, collector *[]string) { var newPrefix string if prefix == "" || prefix == "ROOT" { newPrefix = node.Name } else { newPrefix = prefix + "/" + node.Name } children := node.GetChildren() if len(children) == 0 { // Leaf node *collector = append(*collector, newPrefix) return } for _, child := range children { buildPathsDfs(child, newPrefix, collector) } } // // 6) MAIN // func main() { if len(os.Args) < 2 { fmt.Println("Usage: go run main.go /path/to/all-services.json") os.Exit(1) } jsonPath := os.Args[1] // 1) Read the JSON raw, err := ioutil.ReadFile(jsonPath) if err != nil { log.Fatalf("Failed to read %s: %v", jsonPath, err) } // 2) Unmarshal into our AllServices var allSvcs AllServices if err := json.Unmarshal(raw, &allSvcs); err != nil { log.Fatalf("Failed to unmarshal JSON: %v", err) } // 3) Build the tree root := buildTreeFromAllServices(allSvcs) // 4) Gather the final mapping (service => list of resource paths) serviceMapping := gatherServiceResourceMapping(root) // 5) Write out the final mapping JSON outFile := "service-resource-mapping.json" data, err := json.MarshalIndent(serviceMapping, "", " ") if err != nil { log.Fatalf("Failed to marshal final mapping: %v", err) } if err := ioutil.WriteFile(outFile, data, 0644); err != nil { log.Fatalf("Failed to write output file: %v", err) } log.Printf("Service resource mapping written to %s", outFile) // 6) (Optional) Demonstrate lookup with partial/wildcards // Suppose we want to see which tree nodes match the path "prod-data-test/some-folder" // under "s3". In your JSON, you might have something like "prod-data-*" or "*" that matches it. // Let's see what we get: exampleResourcePath := "prod-data-test/some-folder" s3Node := root.GetChildren()["s3"] if s3Node != nil { pathSegs := strings.Split(exampleResourcePath, "/") matches := FindMatches(s3Node, pathSegs) fmt.Printf("\nMatches for S3 path %q:\n", exampleResourcePath) if len(matches) == 0 { fmt.Println(" No matches found.") } else { for _, m := range matches { // Reconstruct the path from the node. For simplicity, we just show node.Name here; // you can store parent pointers if you want the full chain. You might also want to // combine with a DFS approach or an actual parent link. For demonstration: fmt.Printf(" Node = %q, statements = %v\n", m.Name, m.Statements) } } } }

Explanation of Changes

  1. Storing Partial Wildcards

    • We do not parse or break apart something like "prod-data-*". We store it literally as a node name.
    • At lookup time, we have a helper function matchesSegment(childName, seg string) bool that checks:
      • childName == "*" => full wildcard.
      • strings.HasSuffix(childName, "*") => partial wildcard, so we check if seg has the same prefix.
      • otherwise => exact match.
  2. Mid‐Level Wildcards

    • If we see "my-bucket/*/some-other", then in insertion we split that into ["my-bucket", "*", "some-other"].
    • That means the tree structure for "my-bucket" -> "*" -> "some-other".
    • A lookup path ["my-bucket", "ABC", "some-other"] will see the second segment "ABC" matches the node "*".
  3. Leaf Node Statements

    • In insertResourcePath, once we reach the last segment, we call AddStatement(stmtID). This is optional, but can be very useful if you want to track which policy or statement introduced a given path.
  4. DFS Collection

    • gatherServiceResourceMapping(...) + buildPathsDfs(...) produce a straightforward list of “fully qualified paths” for each service. You’ll see partial wildcards in the output, e.g. "my-bucket/*/some-other" if that was your insertion.
  5. Searching

    • FindMatches(node *TreeNode, pathSegments []string) []*TreeNode is a basic recursive function that tries every child for a match. If multiple children match (e.g. a partial wildcard and a global wildcard), you’ll get multiple results.
    • This ensures that if an ARN resource path can be matched by multiple wildcard patterns, you’ll see them all.

With this, you have a dynamic, tree-based approach to building and querying your IAM policy–resource structures, including mid-level wildcards, partial wildcards, and global wildcards.

Question

Customizing / Extending

Conditions / Effects: If you need to store Allow/Deny or any other details, you can replace the simple TreeNode with a struct containing a list of bindings (similar to your original PolicyBinding approach). Service Mismatch: Currently, if we parse an ARN with service sns but the top-level key is sqs_write, the code picks whichever is found. You might add logic to handle or report mismatches. Output: You could also output a single CSV or a SQL-based file for direct DB ingestion. I wanna extend my code to account for the above cases for the code base--package main

import ( "encoding/json" "fmt" "io/ioutil" "log" "os" "strings" "sync" )

// // 1) DATA STRUCTURES //

// AllServices is your top-level structure parsed from all_services.json type AllServices map[string]map[string][]string

// TreeNode is a node in a tree. Each node can hold zero or more children. // We also add a Statements field in case you want to store references to IAM statements. type TreeNode struct { Name string Children map[string]*TreeNode Statements []string // e.g. store policy or statement IDs that apply

mu sync.RWMutex

}

// NewTreeNode creates a new node. func NewTreeNode(name string) *TreeNode { return &TreeNode{ Name: name, Children: make(map[string]*TreeNode), } }

// AddChild either returns an existing child[thinking]

func (n *TreeNode) AddChild(childName string) *TreeNode { n.mu.Lock() defer n.mu.Unlock()

if child, exists := n.Children[childName]; exists {
	return child
}
child := NewTreeNode(childName)
n.Children[childName] = child
return child

}

// AddStatement appends a statement identifier to this node. In a real-world scenario[thinking]

// you might store richer objects (the entire IAM statement, policy name, etc.). func (n *TreeNode) AddStatement(stmt string) { n.mu.Lock() defer n.mu.Unlock() n.Statements = append(n.Statements, stmt) }

// GetChildren returns a copy of the children map for safe iteration. func (n *TreeNode) GetChildren() map[string]*TreeNode { n.mu.RLock() defer n.mu.RUnlock()

copyMap := make(map[string]*TreeNode, len(n.Children))
for k, v := range n.Children {
	copyMap[k] = v
}
return copyMap

}

// // 2) HELPER FUNCTIONS //

// We only care about these services: var servicesOfInterest = map[string]bool{ "s3": true, "sns": true, "sqs": true, "kms": true, "events": true, }

// extractServiceName tries to interpret the second-level key to find the AWS service name. // E.g. "s3_read" -> "s3", "custom_s3_actions0" -> "s3", etc. func extractServiceName(key string) string { if strings.HasPrefix(key, "custom_") { remaining := strings.TrimPrefix(key, "custom_") // e.g. s3_actions0 parts := strings.Split(remaining, "") if len(parts) > 0 { return parts[0] // s3 } } parts := strings.Split(key, "") if len(parts) > 0 { return parts[0] } return key }

// parseArnForServiceAndResource tries to parse an ARN. // // Example: "arn:aws:s3:::my-bucket/path/to/object" // returns ("s3", "my-bucket/path/to/object") // // If invalid or a global wildcard "", it returns ("", ""). func parseArnForServiceAndResource(arn string) (string, string) { if arn == "" { return "", "" } const arnPrefix = "arn:aws:" if !strings.HasPrefix(arn, arnPrefix) { // Not a standard ARN => treat the entire string as resource return "", arn }

// arn:aws:<service>:<region>:<accountId>:<resource>
parts := strings.SplitN(arn, ":", 6)
if len(parts) < 6 {
	// Malformed or incomplete
	return "", arn
}

service := parts[2]
resource := parts[5]
return service, resource

}

// insertResourcePath splits the resource string by "/" and inserts each segment into the tree. // If the resource is "", it creates a wildcard child. We also store a statement ID or some // identifier at the leaf to know which policy introduced it. // // For partial wildcards like "prod-data-", we do not do any special parse logic. We store that // entire segment as "prod-data-". The matching logic later will handle whether a segment matches it. func insertResourcePath(serviceNode TreeNode, resource string, stmtID string) { if resource == "" { // Entire resource is a wildcard child := serviceNode.AddChild("") // Mark that child's node with the statement child.AddStatement(stmtID) return }

segments := strings.Split(resource, "/")
current := serviceNode
for i, seg := range segments {
	if seg == "" {
		continue
	}
	current = current.AddChild(seg)

	// If this is the last segment, store the statement reference
	if i == len(segments)-1 {
		current.AddStatement(stmtID)
	}
}

}

// // 3) BUILDING THE TREE //

// buildTreeFromAllServices populates a tree with only the services we care about. Each // resource path from the JSON is inserted into the corresponding service node. func buildTreeFromAllServices(allSvcs AllServices) *TreeNode { root := NewTreeNode("ROOT")

// The top-level map is like: "rds-rs-pipeline" -> {"s3_read": [...], ...}
// We'll treat "rds-rs-pipeline" as the "policy name" or "role name" for demonstration.
for policyOrRoleName, perms := range allSvcs {
	for key, resources := range perms {
		// Derive the "logical" service from the key
		extractedSvc := extractServiceName(key)

		for _, arnOrRes := range resources {
			// Parse the ARN to see if it might override the extracted service
			svcFromArn, resourcePart := parseArnForServiceAndResource(arnOrRes)
			finalSvc := extractedSvc
			if svcFromArn != "" && servicesOfInterest[svcFromArn] {
				finalSvc = svcFromArn
			}
			if !servicesOfInterest[finalSvc] {
				// skip if not in our list
				continue
			}

			// Add or retrieve the node for this final service
			svcNode := root.AddChild(finalSvc)

			// Insert the path. We'll store a reference to the origin (e.g. policy name + key).
			stmtID := policyOrRoleName + "/" + key
			insertResourcePath(svcNode, resourcePart, stmtID)
		}
	}
}
return root

}

// // 4) MATCHING / LOOKUPS //

// matchesSegment checks if a path segment from the ARN we are searching for // matches a child node name (which could be "", "prefix-", or exact). func matchesSegment(childName, seg string) bool { // 1) Full wildcard node? if childName == "" { return true } // 2) Partial wildcard? e.g. "prod-data-" if strings.HasSuffix(childName, "") { prefix := childName[:len(childName)-1] // remove trailing '' return strings.HasPrefix(seg, prefix) } // 3) Otherwise, must match exactly return (childName == seg) }

// FindMatches recursively walks the tree from the given node, attempting to match // each path segment in pathSegments. It returns all nodes that match the final segment. // // Example usage: // // segments := strings.Split("my-bucket/some/path", "/") // matches := FindMatches(s3Node, segments) func FindMatches(node *TreeNode, pathSegments []string) []*TreeNode { if len(pathSegments) == 0 { // No more segments to match => this node is a "match" endpoint return []*TreeNode{node} } currentSeg := pathSegments[0] remainder := pathSegments[1:]

var result []*TreeNode
for childName, childNode := range node.GetChildren() {
	if matchesSegment(childName, currentSeg) {
		// Recurse down
		result = append(result, FindMatches(childNode, remainder)...)
	}
}
return result

}

// // 5) GATHERING & PRINTING THE "SERVICE => RESOURCE" MAPPING // // gatherServiceResourceMapping does a DFS and collects full paths to leaf nodes. // Because we have partial wildcards, you’ll see them in the final strings. // Because we also store statements, you could retrieve them from the node if you wish. //

func gatherServiceResourceMapping(root *TreeNode) map[string][]string { output := make(map[string][]string)

// The children of ROOT are the services
for svcName, svcNode := range root.GetChildren() {
	if svcName == "ROOT" {
		continue
	}
	var allPaths []string
	buildPathsDfs(svcNode, "", &allPaths)
	output[svcName] = allPaths
}
return output

}

// buildPathsDfs traverses the tree depth-first and collects "path" strings. // e.g. child("my-bucket") -> child("") -> child("some-folder") => "my-bucket//some-folder" func buildPathsDfs(node *TreeNode, prefix string, collector *[]string) { var newPrefix string if prefix == "" || prefix == "ROOT" { newPrefix = node.Name } else { newPrefix = prefix + "/" + node.Name }

children := node.GetChildren()
if len(children) == 0 {
	// Leaf node
	*collector = append(*collector, newPrefix)
	return
}

for _, child := range children {
	buildPathsDfs(child, newPrefix, collector)
}

}

// // 6) MAIN //

func main() { if len(os.Args) < 2 { fmt.Println("Usage: go run main.go /path/to/all-services.json") os.Exit(1) } jsonPath := os.Args[1]

// 1) Read the JSON
raw, err := ioutil.ReadFile(jsonPath)
if err != nil {
	log.Fatalf("Failed to read %s: %v", jsonPath, err)
}

// 2) Unmarshal into our AllServices
var allSvcs AllServices
if err := json.Unmarshal(raw, &allSvcs); err != nil {
	log.Fatalf("Failed to unmarshal JSON: %v", err)
}

// 3) Build the tree
root := buildTreeFromAllServices(allSvcs)

// 4) Gather the final mapping (service => list of resource paths)
serviceMapping := gatherServiceResourceMapping(root)

// 5) Write out the final mapping JSON
outFile := "service-resource-mapping.json"
data, err := json.MarshalIndent(serviceMapping, "", "  ")
if err != nil {
	log.Fatalf("Failed to marshal final mapping: %v", err)
}
if err := ioutil.WriteFile(outFile, data, 0644); err != nil {
	log.Fatalf("Failed to write output file: %v", err)
}
log.Printf("Service resource mapping written to %s", outFile)

// 6) (Optional) Demonstrate lookup with partial/wildcards
// Suppose we want to see which tree nodes match the path "prod-data-test/some-folder"
// under "s3". In your JSON, you might have something like "prod-data-*" or "*" that matches it.
// Let's see what we get:

exampleResourcePath := "prod-data-test/some-folder"
s3Node := root.GetChildren()["s3"]
if s3Node != nil {
	pathSegs := strings.Split(exampleResourcePath, "/")
	matches := FindMatches(s3Node, pathSegs)
	fmt.Printf("\nMatches for S3 path %q:\n", exampleResourcePath)
	if len(matches) == 0 {
		fmt.Println("  No matches found.")
	} else {
		for _, m := range matches {
			// Reconstruct the path from the node. For simplicity, we just show node.Name here;
			// you can store parent pointers if you want the full chain. You might also want to
			// combine with a DFS approach or an actual parent link. For demonstration:
			fmt.Printf("  Node = %q, statements = %v\n", m.Name, m.Statements)
		}
	}
}

} and ref file is all_services.json --{ "rds-rs-pipeline": { "s3_read": [ "arn:aws:s3:::sftp-rbl-ags", "arn:aws:s3:::prod-datascience-sms/", "arn:aws:s3:::prod-lending-adhoc", "arn:aws:s3:::prod-lending-adhoc/", "arn:aws:s3:::prod-datascience-sms", "arn:aws:s3:::datascience-store-personalisation/", "arn:aws:s3:::sftp-rbl-ags/", "", "arn:aws:s3:::datascience-store-personalisation" ], "custom_s3_actions0": [ "arn:aws:s3:::sftp-rbl-ags", "arn:aws:s3:::prod-datascience-sms/", "arn:aws:s3:::prod-lending-adhoc", "arn:aws:s3:::prod-lending-adhoc/", "arn:aws:s3:::prod-datascience-sms", "arn:aws:s3:::datascience-store-personalisation/", "arn:aws:s3:::sftp-rbl-ags/", "", "arn:aws:s3:::datascience-store-personalisation" ], "custom_s3-object-lambda_actions0": [ "" ], "ssm_read": [ "arn:aws:ssm:ap-south-1:744315537043:parameter/conf/rds-rs-pipeline/" ], "kms_decrypt": [ "arn:aws:kms:ap-south-1:744315537043:key/af547ea6-b4fd-4050-9185-62068e0da254", "arn:aws:kms:ap-south-1:744315537043:key/c1b80926-ce55-4905-a92a-a1a830ffbf5b", "arn:aws:kms:ap-south-1:744315537043:key/94baadce-0db1-41fe-91ff-f3be240fb6c2" ], "kms_generate_data_key": [ "arn:aws:kms:ap-south-1:744315537043:key/af547ea6-b4fd-4050-9185-62068e0da254", "arn:aws:kms:ap-south-1:744315537043:key/c1b80926-ce55-4905-a92a-a1a830ffbf5b", "arn:aws:kms:ap-south-1:744315537043:key/94baadce-0db1-41fe-91ff-f3be240fb6c2" ], "kms_sign": [ "arn:aws:kms:ap-south-1:744315537043:key/af547ea6-b4fd-4050-9185-62068e0da254", "arn:aws:kms:ap-south-1:744315537043:key/c1b80926-ce55-4905-a92a-a1a830ffbf5b", "arn:aws:kms:ap-south-1:744315537043:key/94baadce-0db1-41fe-91ff-f3be240fb6c2" ], "kms_verify": [ "arn:aws:kms:ap-south-1:744315537043:key/af547ea6-b4fd-4050-9185-62068e0da254", "arn:aws:kms:ap-south-1:744315537043:key/c1b80926-ce55-4905-a92a-a1a830ffbf5b", "arn:aws:kms:ap-south-1:744315537043:key/94baadce-0db1-41fe-91ff-f3be240fb6c2" ], "custom_dynamodb_actions0": [ "arn:aws:dynamodb:ap-south-1:744315537043:table/external_reward_report_config", "arn:aws:dynamodb:ap-south-1:744315537043:table/sms_communications", "arn:aws:dynamodb:us-east-1:744315537043:table/external_reward_report_config" ], "custom_firehose_actions0": [ "arn:aws:firehose:ap-south-1:744315537043:deliverystream/prod_reward_usage_attributes" ], "custom_firehose_actions1": [ "" ], "s3_write": [ "arn:aws:s3:::sftp-rbl-ags", "arn:aws:s3:::prod-datascience-sms/", "arn:aws:s3:::prod-lending-adhoc", "arn:aws:s3:::prod-lending-adhoc/", "arn:aws:s3:::prod-datascience-sms", "arn:aws:s3:::datascience-store-personalisation/", "arn:aws:s3:::sftp-rbl-ags/", "arn:aws:s3:::datascience-store-personalisation" ], "custom_s3_actions1": [ "arn:aws:s3:::eventi-audit", "arn:aws:s3:::eventi-audit/", "arn:aws:s3:::prod-datascience-sms/", "arn:aws:s3:::prod-data-", "arn:aws:s3:::prod-datascience-sms", "arn:aws:s3:::internal-payments-recon", "arn:aws:s3:::stg-redshift", "arn:aws:s3:::internal-payments-recon/", "arn:aws:s3:::stg-redshift/" ], "custom_s3_actions2": [ "" ], "custom_ssm_actions0": [ "" ], "custom_ssm_actions1": [ "arn:aws:ssm:ap-south-1:744315537043:parameter/conf/rds-rs-pipeline/" ], "custom_s3_actions3": [ "" ] }, "metabase-ecs": { "custom_glue_actions0": [ "" ], "custom_s3_actions2": [ "arn:aws:s3:::mp-proxy-main/", "arn:aws:s3:::prod-data-test/", "arn:aws:s3:::mp-proxy-main", "arn:aws:s3:::prod-data-test", "" ], "custom_s3_actions3": [ "arn:aws:s3:::mp-proxy-main/", "arn:aws:s3:::aws-glue-", "arn:aws:s3:::aws-athena-query-results-", "arn:aws:s3:::aws-glue-/", "arn:aws:s3:::athena-examples", "arn:aws:s3:::prod-data-test/", "arn:aws:s3:::/aws-glue-/", "arn:aws:s3:::mp-proxy-main", "arn:aws:s3:::prod-data-test", "arn:aws:s3:::crawler-public", "" ], "custom_ec2_actions0": [ "" ], "custom_iam_actions0": [ "" ], "custom_cloudwatch_actions0": [ "" ], "custom_s3_actions4": [ "arn:aws:s3:::mp-proxy-main/", "arn:aws:s3:::aws-glue-", "arn:aws:s3:::aws-athena-query-results-", "arn:aws:s3:::athena-examples", "arn:aws:s3:::prod-data-test/", "arn:aws:s3:::mp-proxy-main", "arn:aws:s3:::prod-data-test", "arn:aws:s3:::crawler-public", "" ], "custom_s3_actions5": [ "arn:aws:s3:::mp-proxy-main/", "arn:aws:s3:::aws-glue-", "arn:aws:s3:::aws-athena-query-results-", "arn:aws:s3:::aws-glue-/", "arn:aws:s3:::athena-examples*", "arn:aws:s3:::prod-data-test/", "arn:aws:s3:::/aws-glue-/", "arn:aws:s3:::mp-proxy-main", "arn:aws:s3:::prod-data-test", "arn:aws:s3:::crawler-public", "" ], "custom_logs_actions0": [ "arn:aws:logs::::/aws-glue/" ], "custom_ec2_actions1": [ "arn:aws:ec2:::instance/", "arn:aws:ec2:::network-interface/", "arn:aws:ec2:::security-group/" ], "custom_athena_actions0": [ "" ], "custom_glue_actions1": [ "" ], "custom_s3_actions6": [ "arn:aws:s3:::mp-proxy-main/", "arn:aws:s3:::aws-athena-query-results-", "arn:aws:s3:::athena-examples*", "arn:aws:s3:::prod-data-test/", "arn:aws:s3:::mp-proxy-main", "arn:aws:s3:::prod-data-test", "" ], "custom_sns_actions0": [ "" ], "custom_cloudwatch_actions1": [ "" ], "custom_lakeformation_actions0": [ "" ], "custom_datazone_actions0": [ "" ], "custom_pricing_actions0": [ "" ], "ssm_read": [ "arn:aws:ssm:ap-south-1:744315537043:parameter/conf/metabase-ecs/" ], "custom_ssm_actions0": [ "arn:aws:ssm:ap-south-1:744315537043:parameter/conf/metabase-ecs/" ], "custom_ssm_actions1": [ "arn:aws:ssm:ap-south-1:744315537043:parameter/conf/metabase-ecs/" ], "s3_read": [ "arn:aws:s3:::mp-proxy-main", "arn:aws:s3:::mp-proxy-main/", "arn:aws:s3:::prod-data-test/", "arn:aws:s3:::prod-data-test" ], "s3_write": [ "arn:aws:s3:::mp-proxy-main", "arn:aws:s3:::mp-proxy-main/", "arn:aws:s3:::prod-data-test/", "arn:aws:s3:::prod-data-test" ], "custom_s3_actions0": [ "arn:aws:s3:::mp-proxy-main", "arn:aws:s3:::mp-proxy-main/", "arn:aws:s3:::prod-data-test/", "arn:aws:s3:::prod-data-test" ], "custom_s3_actions1": [ "arn:aws:s3:::mp-proxy-main", "arn:aws:s3:::mp-proxy-main/", "arn:aws:s3:::prod-data-test/", "arn:aws:s3:::prod-data-test" ] }, "auth-service": { "custom_dynamodb_actions1": [ "" ], "custom_dynamodb_actions2": [ "arn:aws:dynamodb:ap-south-1:744315537043:table/authorizationtokens/index/accessToken-index", "arn:aws:dynamodb:ap-south-1:744315537043:table/authorizationtokens", "arn:aws:dynamodb:ap-south-1:744315537043:table/oauth2_tokens/index/user_id-index", "arn:aws:dynamodb:ap-south-1:744315537043:table/googleauthorizationtokens/index/dreamplugUserId-index", "arn:aws:dynamodb:ap-south-1:744315537043:table/authorizationtokens/index/refreshToken-index", "arn:aws:dynamodb:ap-south-1:744315537043:table/authorizationtokens/index/", "arn:aws:dynamodb:ap-south-1:744315537043:table/googleauthorizationtokens", "arn:aws:dynamodb:ap-south-1:744315537043:table/oauth2_tokens", "arn:aws:dynamodb:ap-south-1:744315537043:table/authorizationtokens/index/userId-index" ], "sqs_read": [ "arn:aws:sqs:ap-south-1:744315537043:prod-leaderboard-cache-update-queue-dlq", "arn:aws:sqs:ap-south-1:744315537043:prod-leaderboard-score-update-queue-dlq", "arn:aws:sqs:ap-south-1:744315537043:prod-leaderboard-score-update-queue", "arn:aws:sqs:ap-south-1:744315537043:cp-event-email-queue", "arn:aws:sqs:ap-south-1:744315537043:prod-portkey-plutus-events-queue", "arn:aws:sqs:ap-south-1:744315537043:prod-leaderboard-cache-update-queue" ], "sqs_write": [ "arn:aws:sqs:ap-south-1:744315537043:prod-leaderboard-cache-update-queue-dlq", "arn:aws:sqs:ap-south-1:744315537043:prod-leaderboard-score-update-queue-dlq", "arn:aws:sqs:ap-south-1:744315537043:prod-leaderboard-score-update-queue", "arn:aws:sqs:ap-south-1:744315537043:cp-event-email-queue", "arn:aws:sqs:ap-south-1:744315537043:prod-portkey-plutus-events-queue", "arn:aws:sqs:ap-south-1:744315537043:prod-leaderboard-cache-update-queue" ], "custom_sqs_actions0": [ "" ], "custom_dynamodb_actions0": [ "arn:aws:dynamodb:ap-south-1:744315537043:table/prod_eventi_producer_configurations/index/", "arn:aws:dynamodb:ap-south-1:744315537043:table/oauth2_tokens_v2", "arn:aws:dynamodb:ap-south-1:744315537043:table/truecaller_auth_context_cache/index/requestId-index/index/", "arn:aws:dynamodb:ap-south-1:744315537043:table/truecaller_auth_context_cache", "arn:aws:dynamodb:ap-south-1:744315537043:table/truecaller_auth_context_cache/index/requestId-index", "arn:aws:dynamodb:ap-south-1:744315537043:table/prod_eventi_producer_configurations" ], "custom_dynamodb_actions3": [ "arn:aws:dynamodb:ap-south-1:744315537043:table/prod_eventi_producer_configurations/index/", "arn:aws:dynamodb:ap-south-1:744315537043:table/oauth2_tokens_v2", "arn:aws:dynamodb:ap-south-1:744315537043:table/truecaller_auth_context_cache/index/requestId-index/index/", "arn:aws:dynamodb:ap-south-1:744315537043:table/truecaller_auth_context_cache", "arn:aws:dynamodb:ap-south-1:744315537043:table/truecaller_auth_context_cache/index/requestId-index", "arn:aws:dynamodb:ap-south-1:744315537043:table/prod_eventi_producer_configurations" ], "ssm_read": [ "arn:aws:ssm:ap-south-1:744315537043:parameter/conf/auth-service/" ], "custom_ssm_actions0": [ "arn:aws:ssm:ap-south-1:744315537043:parameter/conf/auth-service/" ], "custom_ssm_actions1": [ "arn:aws:ssm:ap-south-1:744315537043:parameter/conf/auth-service/" ] }, "user-service-attribute": { "ssm_read": [ "arn:aws:ssm:ap-south-1:744315537043:parameter/conf/user-service-attribute/" ], "custom_ssm_actions0": [ "arn:aws:ssm:ap-south-1:744315537043:parameter/conf/user-service-attribute/" ], "custom_ssm_actions1": [ "arn:aws:ssm:ap-south-1:744315537043:parameter/conf/user-service-attribute/" ], "sqs_read": [ "arn:aws:sqs:ap-south-1:744315537043:prod-eventi-kafka-producer-dlq", "arn:aws:sqs:ap-south-1:744315537043:prod-notification-queue", "arn:aws:sqs:ap-south-1:744315537043:prod-in-app-consumer-queue", "arn:aws:sqs:ap-south-1:744315537043:prod-eventi-queue", "arn:aws:sqs:ap-south-1:744315537043:prod-notification-promotion-queue" ], "sqs_write": [ "arn:aws:sqs:ap-south-1:744315537043:prod-eventi-kafka-producer-dlq", "arn:aws:sqs:ap-south-1:744315537043:prod-notification-queue", "arn:aws:sqs:ap-south-1:744315537043:prod-in-app-consumer-queue", "arn:aws:sqs:ap-south-1:744315537043:prod-eventi-queue", "arn:aws:sqs:ap-south-1:744315537043:prod-notification-promotion-queue" ], "custom_sqs_actions1": [ "arn:aws:sqs:ap-south-1:744315537043:prod-eventi-kafka-producer-dlq", "arn:aws:sqs:ap-south-1:744315537043:prod-notification-queue", "arn:aws:sqs:ap-south-1:744315537043:prod-in-app-consumer-queue", "arn:aws:sqs:ap-south-1:744315537043:prod-eventi-queue", "arn:aws:sqs:ap-south-1:744315537043:prod-notification-promotion-queue", "" ], "custom_sqs_actions0": [ "" ], "custom_logs_actions1": [ "" ], "custom_cloudwatch_actions0": [ "" ], "custom_cloudwatch_actions2": [ "" ], "custom_sns_actions1": [ "arn:aws:sns:ap-south-1:744315537043:prod-eventi-sns" ], "custom_sns_actions2": [ "" ], "custom_dynamodb_actions0": [ "arn:aws:dynamodb:ap-south-1:744315537043:table/prod_user_attributes" ], "custom_dynamodb_actions3": [ "arn:aws:dynamodb:ap-south-1:744315537043:table/prod_eventi_consumer_configurations", "arn:aws:dynamodb:ap-south-1:744315537043:table/prod_eventi_producer_configurations", "arn:aws:dynamodb:ap-south-1:744315537043:table/prod_user_attributes" ], "custom_dynamodb_actions4": [ "arn:aws:dynamodb:ap-south-1:744315537043:table/prod_eventi_consumer_configurations", "arn:aws:dynamodb:ap-south-1:744315537043:table/prod_eventi_producer_configurations", "arn:aws:dynamodb:ap-south-1:744315537043:table/prod_user_attributes" ], "custom_dynamodb_actions2": [ "arn:aws:dynamodb:ap-south-1:744315537043:table/prod_eventi_consumer_configurations", "arn:aws:dynamodb:ap-south-1:744315537043:table/prod_eventi_producer_configurations" ], "custom_kinesis_actions0": [ "arn:aws:kinesis:ap-south-1:744315537043:stream/prod_eventi" ] }, "vmagent": {}, "otp-service": { "sqs_read": [ "arn:aws:sqs:ap-south-1:744315537043:prod-eventi-kafka-producer-dlq", "arn:aws:sqs:ap-south-1:744315537043:prod-notification-queue", "arn:aws:sqs:ap-south-1:744315537043:prod-in-app-consumer-queue", "arn:aws:sqs:ap-south-1:744315537043:prod-eventi-queue", "arn:aws:sqs:ap-south-1:744315537043:prod-notification-promotion-queue" ], "sqs_write": [ "arn:aws:sqs:ap-south-1:744315537043:prod-eventi-kafka-producer-dlq", "arn:aws:sqs:ap-south-1:744315537043:prod-notification-queue", "arn:aws:sqs:ap-south-1:744315537043:prod-in-app-consumer-queue", "arn:aws:sqs:ap-south-1:744315537043:prod-eventi-queue", "arn:aws:sqs:ap-south-1:744315537043:prod-notification-promotion-queue" ], "custom_sqs_actions1": [ "arn:aws:sqs:ap-south-1:744315537043:prod-eventi-kafka-producer-dlq", "arn:aws:sqs:ap-south-1:744315537043:prod-notification-queue", "arn:aws:sqs:ap-south-1:744315537043:prod-in-app-consumer-queue", "arn:aws:sqs:ap-south-1:744315537043:prod-eventi-queue", "arn:aws:sqs:ap-south-1:744315537043:prod-notification-promotion-queue", "" ], "custom_sqs_actions0": [ "" ], "custom_logs_actions1": [ "" ], "custom_cloudwatch_actions0": [ "" ], "custom_cloudwatch_actions2": [ "" ], "custom_dynamodb_actions1": [ "" ], "custom_dynamodb_actions2": [ "arn:aws:dynamodb:ap-south-1:744315537043:table/prod_eventi_consumer_configurations", "arn:aws:dynamodb:ap-south-1:744315537043:table/otps", "arn:aws:dynamodb:ap-south-1:744315537043:table/otp_templates", "arn:aws:dynamodb:ap-south-1:744315537043:table/prod_eventi_producer_configurations", "", "arn:aws:dynamodb:ap-south-1:744315537043:table/OTPVerifyTracker" ], "custom_dynamodb_actions3": [ "arn:aws:dynamodb:ap-south-1:744315537043:table/prod_eventi_consumer_configurations", "arn:aws:dynamodb:ap-south-1:744315537043:table/otps", "arn:aws:dynamodb:ap-south-1:744315537043:table/otp_templates", "arn:aws:dynamodb:ap-south-1:744315537043:table/prod_eventi_producer_configurations", "", "arn:aws:dynamodb:ap-south-1:744315537043:table/OTPVerifyTracker" ], "custom_dynamodb_actions4": [ "arn:aws:dynamodb:ap-south-1:744315537043:table/prod_eventi_consumer_configurations", "arn:aws:dynamodb:ap-south-1:744315537043:table/otps", "arn:aws:dynamodb:ap-south-1:744315537043:table/otp_templates", "arn:aws:dynamodb:ap-south-1:744315537043:table/prod_eventi_producer_configurations", "", "arn:aws:dynamodb:ap-south-1:744315537043:table/OTPVerifyTracker" ], "custom_dynamodb_actions0": [ "arn:aws:dynamodb:ap-south-1:744315537043:table/prod_eventi_producer_configurations" ], "custom_sns_actions1": [ "arn:aws:sns:ap-south-1:744315537043:prod-eventi-sns" ], "custom_sns_actions2": [ "arn:aws:sns:ap-south-1:744315537043:prod-eventi-sns", "" ], "ssm_read": [ "arn:aws:ssm:ap-south-1:744315537043:parameter/conf/otp-service/" ], "custom_ssm_actions0": [ "arn:aws:ssm:ap-south-1:744315537043:parameter/conf/otp-service/" ], "custom_ssm_actions1": [ "arn:aws:ssm:ap-south-1:744315537043:parameter/conf/otp-service/" ], "custom_kinesis_actions0": [ "arn:aws:kinesis:ap-south-1:744315537043:stream/prod_eventi" ] }, "fraud-engine-graviton-test": {}, "yoda-bot": {}, "wayne": { "ssm_read": [ "arn:aws:ssm:ap-south-1:744315537043:parameter/conf/wayne/" ], "custom_ssm_actions0": [ "arn:aws:ssm:ap-south-1:744315537043:parameter/conf/wayne/" ], "custom_ssm_actions1": [ "arn:aws:ssm:ap-south-1:744315537043:parameter/conf/wayne/" ] }, "hba-service": { "custom_logs_actions1": [ "" ], "custom_cloudwatch_actions0": [ "" ], "custom_cloudwatch_actions2": [ "" ], "custom_dynamodb_actions0": [ "arn:aws:dynamodb:ap-south-1:744315537043:table/hba_merch_onboarding" ], "custom_dynamodb_actions3": [ "arn:aws:dynamodb:ap-south-1:744315537043:table/hba_merch_onboarding" ], "custom_dynamodb_actions4": [ "arn:aws:dynamodb:ap-south-1:744315537043:table/hba_merch_onboarding" ], "ssm_read": [ "arn:aws:ssm:ap-south-1:744315537043:parameter/conf/hba-service/" ], "custom_ssm_actions0": [ "arn:aws:ssm:ap-south-1:744315537043:parameter/conf/hba-service/" ], "custom_ssm_actions1": [ "arn:aws:ssm:ap-south-1:744315537043:parameter/conf/hba-service/*" ] },

Thought

Integrating new structures

I’m considering including a "PolicyStatement" data structure for conditions and effects, addressing service mismatches gracefully, and potentially outputting data as CSV or storing multiple fields in "PolicyBinding".

Introducing new structures

I’m mapping out the integration of a "StatementBinding" data structure to manage effects and conditions, logging mismatches, and generating an optional CSV output for each resource path.

Reasoned about IAM policy enhancements for 15 seconds

Answer

Below is an extended version of your Go code that demonstrates how you might:

  1. Store richer policy/statement details (e.g. Allow vs. Deny, arbitrary conditions, etc.) in each tree node rather than just a string slice of statement IDs.
  2. Handle service mismatches more gracefully (for example, logging or storing them).
  3. Optionally output a CSV (or any other format) for direct ingestion into a database.

Note: Since your sample JSON does not actually contain “Effect” (Allow/Deny) or “Condition” objects, we will hard-code some placeholders to show how you could store them. If you have a real structure for them in your IAM policies, simply parse them and store them in the same manner.


Revised Code

package main import ( "encoding/csv" "encoding/json" "fmt" "io/ioutil" "log" "os" "strings" "sync" ) // // 1) DATA STRUCTURES // // AllServices is your top-level structure parsed from all_services.json // // For instance: // "rds-rs-pipeline": { // "s3_read": [...], // "custom_s3_actions0": [...], // ... // }, // "metabase-ecs": { ... } type AllServices map[string]map[string][]string // PolicyBinding captures richer data about a single policy statement or binding. // You can extend this with real Condition maps, Resource/Action arrays, etc. type PolicyBinding struct { PolicyOrRoleName string // e.g. "rds-rs-pipeline" Key string // e.g. "s3_read" Effect string // e.g. "Allow" or "Deny" // For demonstration, we store conditions as a string or map. Real IAM might have a more complex structure. Conditions map[string]string } // TreeNode is a node in a tree. Each node can hold zero or more children. // Instead of just storing `Statements []string`, we store a slice of `PolicyBinding` // which can contain effect, conditions, etc. type TreeNode struct { Name string Children map[string]*TreeNode Bindings []PolicyBinding mu sync.RWMutex } // NewTreeNode creates a new node. func NewTreeNode(name string) *TreeNode { return &TreeNode{ Name: name, Children: make(map[string]*TreeNode), Bindings: []PolicyBinding{}, } } // AddChild either returns an existing child or creates a new one. func (n *TreeNode) AddChild(childName string) *TreeNode { n.mu.Lock() defer n.mu.Unlock() if child, exists := n.Children[childName]; exists { return child } child := NewTreeNode(childName) n.Children[childName] = child return child } // AddBinding adds a PolicyBinding to this node. func (n *TreeNode) AddBinding(binding PolicyBinding) { n.mu.Lock() defer n.mu.Unlock() n.Bindings = append(n.Bindings, binding) } // GetChildren returns a (safe) copy of the children map. func (n *TreeNode) GetChildren() map[string]*TreeNode { n.mu.RLock() defer n.mu.RUnlock() copyMap := make(map[string]*TreeNode, len(n.Children)) for k, v := range n.Children { copyMap[k] = v } return copyMap } // // 2) HELPER FUNCTIONS // // We only care about these services: var servicesOfInterest = map[string]bool{ "s3": true, "sns": true, "sqs": true, "kms": true, "events": true, // for eventbridge } // extractServiceName tries to interpret the second-level key to find the AWS service name. // E.g. "s3_read" -> "s3", "custom_s3_actions0" -> "s3", etc. func extractServiceName(key string) string { if strings.HasPrefix(key, "custom_") { remaining := strings.TrimPrefix(key, "custom_") // e.g. "s3_actions0" parts := strings.Split(remaining, "_") if len(parts) > 0 { return parts[0] // "s3" } } parts := strings.Split(key, "_") if len(parts) > 0 { return parts[0] } return key } // parseArnForServiceAndResource tries to parse an ARN. Returns (service, resourcePart). // // Example: "arn:aws:s3:::my-bucket/path/to/object" => ("s3", "my-bucket/path/to/object") // // If invalid or "*", we do our best to handle it. func parseArnForServiceAndResource(arn string) (string, string) { if arn == "*" { return "", "*" } const arnPrefix = "arn:aws:" if !strings.HasPrefix(arn, arnPrefix) { // Not a standard ARN => treat entire string as resource return "", arn } // ARN format: arn:aws:<service>:<region>:<accountId>:<resource> parts := strings.SplitN(arn, ":", 6) if len(parts) < 6 { // Malformed or incomplete return "", arn } service := parts[2] resource := parts[5] return service, resource } // insertResourcePath splits the resource by slash and inserts each segment into the tree. // We store a final binding at the leaf node (in a real scenario, you might store it at all nodes). func insertResourcePath(serviceNode *TreeNode, resource string, binding PolicyBinding) { if resource == "*" { // Entire resource is wildcard: just attach at child("*") child := serviceNode.AddChild("*") child.AddBinding(binding) return } segments := strings.Split(resource, "/") curr := serviceNode for i, seg := range segments { if seg == "" { continue } curr = curr.AddChild(seg) if i == len(segments)-1 { // Leaf node => add binding curr.AddBinding(binding) } } } // // 3) BUILDING THE TREE // // buildTreeFromAllServices populates a tree with only the services we care about. Each // resource path from the JSON is inserted into the corresponding service node. func buildTreeFromAllServices(allSvcs AllServices) (*TreeNode, []string) { root := NewTreeNode("ROOT") var mismatches []string // store any "service mismatch" messages, if desired for policyOrRoleName, perms := range allSvcs { // For each sub-key in perms, e.g. "s3_read" -> slice of ARNs for key, resources := range perms { extractedSvc := extractServiceName(key) for _, arnOrRes := range resources { svcFromArn, resourcePart := parseArnForServiceAndResource(arnOrRes) finalSvc := extractedSvc // Check if the ARN indicates a service we care about if svcFromArn != "" && servicesOfInterest[svcFromArn] { // If the ARN's service differs from the extracted key, note the mismatch if extractedSvc != svcFromArn { // Example: we might log or store the mismatch: msg := fmt.Sprintf( "Service mismatch for policy=%s key=%s => extracted=%q, arnService=%q (using arnService)", policyOrRoleName, key, extractedSvc, svcFromArn, ) mismatches = append(mismatches, msg) // Override the final service with the one from the ARN finalSvc = svcFromArn } } // If finalSvc is not in our interest set, skip if !servicesOfInterest[finalSvc] { continue } // Construct the policy binding. In a real scenario, we might parse the effect, conditions, etc. binding := PolicyBinding{ PolicyOrRoleName: policyOrRoleName, Key: key, Effect: "Allow", // Hard-coded for illustration Conditions: map[string]string{"ExampleConditionKey": "ExampleValue"}, } // Insert into the tree svcNode := root.AddChild(finalSvc) insertResourcePath(svcNode, resourcePart, binding) } } } return root, mismatches } // // 4) MATCHING / LOOKUPS // // matchesSegment checks if a path segment from the ARN we are searching for // matches a child node name (which could be "*", "prefix-*", or exact). func matchesSegment(childName, seg string) bool { if childName == "*" { return true } if strings.HasSuffix(childName, "*") { prefix := childName[:len(childName)-1] return strings.HasPrefix(seg, prefix) } return (childName == seg) } // FindMatches recursively walks the tree from `node`, matching each segment in `pathSegments`. func FindMatches(node *TreeNode, pathSegments []string) []*TreeNode { if len(pathSegments) == 0 { return []*TreeNode{node} } var result []*TreeNode nextSeg := pathSegments[0] remainder := pathSegments[1:] for childName, childNode := range node.GetChildren() { if matchesSegment(childName, nextSeg) { result = append(result, FindMatches(childNode, remainder)...) } } return result } // // 5) GATHERING & PRINTING THE "SERVICE => RESOURCE" MAPPING // // gatherServiceResourceMapping does a DFS and collects full paths. func gatherServiceResourceMapping(root *TreeNode) map[string][]string { out := make(map[string][]string) for svcName, svcNode := range root.GetChildren() { if svcName == "ROOT" { continue } var paths []string gatherDfs(svcNode, "", &paths) out[svcName] = paths } return out } func gatherDfs(node *TreeNode, prefix string, collector *[]string) { newPrefix := node.Name if prefix != "" && prefix != "ROOT" { newPrefix = prefix + "/" + node.Name } children := node.GetChildren() if len(children) == 0 { // Leaf => record full path *collector = append(*collector, newPrefix) return } for _, child := range children { gatherDfs(child, newPrefix, collector) } } // // 6) OPTIONAL CSV OUTPUT // // If you want to store each node's info in a CSV for DB ingestion, for example, you can do so. // We'll produce a row for each final "leaf" node in each service, listing (service, resourcePath, statements...). // type CsvRow struct { Service string ResourcePath string // Flatten out the bindings in some textual form: Bindings string } func exportCsv(root *TreeNode, csvFilePath string) error { // Collect all leaf nodes for each service: var rows []CsvRow for svcName, svcNode := range root.GetChildren() { if svcName == "ROOT" { continue } collectLeafCsvRows(svcNode, svcName, "", &rows) } f, err := os.Create(csvFilePath) if err != nil { return err } defer f.Close() writer := csv.NewWriter(f) defer writer.Flush() // Write header if err := writer.Write([]string{"Service", "ResourcePath", "Bindings"}); err != nil { return err } // Write rows for _, row := range rows { if err := writer.Write([]string{row.Service, row.ResourcePath, row.Bindings}); err != nil { return err } } return nil } func collectLeafCsvRows(node *TreeNode, svcName, prefix string, collector *[]CsvRow) { newPrefix := node.Name if prefix != "" && prefix != "ROOT" { newPrefix = prefix + "/" + node.Name } children := node.GetChildren() if len(children) == 0 { // Leaf => gather data // Flatten the node's PolicyBindings into a string var bindingStrs []string for _, b := range node.Bindings { bindingStrs = append(bindingStrs, fmt.Sprintf("%s:%s(%s)", b.PolicyOrRoleName, b.Key, b.Effect), ) } row := CsvRow{ Service: svcName, ResourcePath: newPrefix, Bindings: strings.Join(bindingStrs, "; "), } *collector = append(*collector, row) return } for _, child := range children { collectLeafCsvRows(child, svcName, newPrefix, collector) } } // // 7) MAIN // func main() { if len(os.Args) < 2 { fmt.Println("Usage: go run main.go /path/to/all-services.json [csvOutputPath]") os.Exit(1) } jsonPath := os.Args[1] var csvPath string if len(os.Args) > 2 { csvPath = os.Args[2] } // 1) Read JSON raw, err := ioutil.ReadFile(jsonPath) if err != nil { log.Fatalf("Failed to read JSON file: %v", err) } // 2) Parse into AllServices var allSvcs AllServices if err := json.Unmarshal(raw, &allSvcs); err != nil { log.Fatalf("Failed to unmarshal JSON: %v", err) } // 3) Build tree & capture any service mismatches root, mismatches := buildTreeFromAllServices(allSvcs) for _, mm := range mismatches { log.Printf("MISMATCH: %s", mm) } // 4) Gather final mapping serviceMapping := gatherServiceResourceMapping(root) // 5) Write JSON mapping outFile := "service-resource-mapping.json" data, err := json.MarshalIndent(serviceMapping, "", " ") if err != nil { log.Fatalf("Failed to marshal final mapping: %v", err) } if err := ioutil.WriteFile(outFile, data, 0644); err != nil { log.Fatalf("Failed to write output file: %v", err) } log.Printf("Service resource mapping written to %s", outFile) // 6) Optional: Output CSV if a second argument was provided if csvPath != "" { if err := exportCsv(root, csvPath); err != nil { log.Fatalf("Failed to export CSV: %v", err) } log.Printf("CSV exported to %s", csvPath) } // 7) Demo a "lookup" using partial/wildcards exampleResourcePath := "prod-data-test/some-folder" s3Node, ok := root.GetChildren()["s3"] if ok { segs := strings.Split(exampleResourcePath, "/") matches := FindMatches(s3Node, segs) fmt.Printf("\nMatches for S3 path %q:\n", exampleResourcePath) if len(matches) == 0 { fmt.Println(" None") } else { for _, m := range matches { // Show node name & policy bindings // (In practice, you might reconstruct the entire path or store a parent pointer.) fmt.Printf(" Node = %q, policyBindings = %v\n", m.Name, m.Bindings) } } } }

Explanation of the Changes

  1. Storing Richer Data (PolicyBinding)

    • We introduce a PolicyBinding struct with fields like PolicyOrRoleName, Key, Effect, Conditions, etc.
    • Instead of just node.Statements []string, we store node.Bindings []PolicyBinding.
    • Each leaf node can have multiple bindings if multiple policies/statements reference the same resource path.
  2. Handling Service Mismatches

    • In buildTreeFromAllServices, if the ARN’s service (svcFromArn) differs from the extracted service name (extractedSvc), we log or store a mismatch message. By default we decide to trust the ARN’s service.
    • We store these mismatch messages in a slice so you can print or handle them.
  3. Optional CSV Output

    • In exportCsv, we collect all leaf nodes for each service, and for each leaf node we flatten the bindings into a single text field (e.g. "rds-rs-pipeline:s3_read(Allow); other-policy:custom_s3_actions0(Allow)").
    • You can easily tweak this format if you want separate columns or a more structured approach.
  4. Placeholder for Effect / Conditions

    • The sample JSON does not contain effects or conditions, so we set them to hard-coded defaults ("Allow" and a dummy map).
    • In your real use case, parse them from the policy JSON or supply them however you wish.
  5. No Actual Condition Parsing

    • If you truly have condition blocks in your IAM statements, you’d parse them from the real policies and store them in PolicyBinding.Conditions. This code merely shows how to store them.
  6. Reconstruction of Paths

    • We still do a simple DFS with gatherDfs to produce “service => list of paths”. In each node’s name, you might see partial wildcards.
    • If you want to reconstruct full ARNs, you can store region/account data from the parse step, or keep parent pointers, etc.

With these changes, you have:

  • A tree-based representation of your IAM policy resources.
  • The ability to store Allow/Deny or other details.
  • Mismatch detection when an ARN’s service conflicts with the “inferred” service from the JSON key.
  • An optional CSV export that you can ingest into a DB or spreadsheet.

Feel free to tailor the code further depending on your real data (e.g. if you actually have a JSON structure with Effect, Condition, statement IDs, etc.).

Question

package main

import ( "encoding/csv" "encoding/json" "fmt" "io/ioutil" "log" "os" "strings" "sync" )

// // 1) DATA STRUCTURES //

// AllServices is your top-level structure parsed from all_services.json // // For instance: // // "rds-rs-pipeline": { // "s3_read": [...], // "custom_s3_actions0": [...], // ... // }, // "metabase-ecs": { ... } type AllServices map[string]map[string][]string

// PolicyBinding captures richer data about a single policy statement or binding. // You can extend this with real Condition maps, Resource/Action arrays, etc. type PolicyBinding struct { PolicyOrRoleName string // e.g. "rds-rs-pipeline" Key string // e.g. "s3_read" Effect string // e.g. "Allow" or "Deny" // For demonstration, we store conditions as a string or map. Real IAM might have a more complex structure. Conditions map[string]string }

// TreeNode is a node in a tree. Each node can hold zero or more children. // Instead of just storing Statements []string, we store a slice of PolicyBinding // which can contain effect, conditions, etc. type TreeNode struct { Name string Children map[string]*TreeNode Bindings []PolicyBinding mu sync.RWMutex }

// NewTreeNode creates a new node. func NewTreeNode(name string) *TreeNode { return &TreeNode{ Name: name, Children: make(map[string]*TreeNode), Bindings: []PolicyBinding{}, } }

// AddChild either returns an existing child or creates a new one. func (n *TreeNode) AddChild(childName string) *TreeNode { n.mu.Lock() defer n.mu.Unlock()

if child, exists := n.Children[childName]; exists {
	return child
}
child := NewTreeNode(childName)
n.Children[childName] = child
return child

}

// AddBinding adds a PolicyBinding to this node. func (n *TreeNode) AddBinding(binding PolicyBinding) { n.mu.Lock() defer n.mu.Unlock() n.Bindings = append(n.Bindings, binding) }

// GetChildren returns a (safe) copy of the children map. func (n *TreeNode) GetChildren() map[string]*TreeNode { n.mu.RLock() defer n.mu.RUnlock()

copyMap := make(map[string]*TreeNode, len(n.Children))
for k, v := range n.Children {
	copyMap[k] = v
}
return copyMap

}

// // 2) HELPER FUNCTIONS //

// We only care about these services: var servicesOfInterest = map[string]bool{ "s3": true, "sns": true, "sqs": true, "kms": true, "events": true, // for eventbridge }

// extractServiceName tries to interpret the second-level key to find the AWS service name. // E.g. "s3_read" -> "s3", "custom_s3_actions0" -> "s3", etc. func extractServiceName(key string) string { if strings.HasPrefix(key, "custom_") { remaining := strings.TrimPrefix(key, "custom_") // e.g. "s3_actions0" parts := strings.Split(remaining, "") if len(parts) > 0 { return parts[0] // "s3" } } parts := strings.Split(key, "") if len(parts) > 0 { return parts[0] } return key }

// parseArnForServiceAndResource tries to parse an ARN. Returns (service, resourcePart). // // Example: "arn:aws:s3:::my-bucket/path/to/object" => ("s3", "my-bucket/path/to/object") // // If invalid or "", we do our best to handle it. func parseArnForServiceAndResource(arn string) (string, string) { if arn == "" { return "", "*" } const arnPrefix = "arn:aws:" if !strings.HasPrefix(arn, arnPrefix) { // Not a standard ARN => treat entire string as resource return "", arn }

// ARN format: arn:aws:<service>:<region>:<accountId>:<resource>
parts := strings.SplitN(arn, ":", 6)
if len(parts) < 6 {
	// Malformed or incomplete
	return "", arn
}

service := parts[2]
resource := parts[5]
return service, resource

}

// insertResourcePath splits the resource by slash and inserts each segment into the tree. // We store a final binding at the leaf node (in a real scenario, you might store it at all nodes). func insertResourcePath(serviceNode TreeNode, resource string, binding PolicyBinding) { if resource == "" { // Entire resource is wildcard: just attach at child("") child := serviceNode.AddChild("") child.AddBinding(binding) return }

segments := strings.Split(resource, "/")
curr := serviceNode
for i, seg := range segments {
	if seg == "" {
		continue
	}
	curr = curr.AddChild(seg)
	if i == len(segments)-1 {
		// Leaf node => add binding
		curr.AddBinding(binding)
	}
}

}

// // 3) BUILDING THE TREE //

// buildTreeFromAllServices populates a tree with only the services we care about. Each // resource path from the JSON is inserted into the corresponding service node. func buildTreeFromAllServices(allSvcs AllServices) (*TreeNode, []string) { root := NewTreeNode("ROOT") var mismatches []string // store any "service mismatch" messages, if desired

for policyOrRoleName, perms := range allSvcs {
	// For each sub-key in perms, e.g. "s3_read" -> slice of ARNs
	for key, resources := range perms {
		extractedSvc := extractServiceName(key)

		for _, arnOrRes := range resources {
			svcFromArn, resourcePart := parseArnForServiceAndResource(arnOrRes)

			finalSvc := extractedSvc

			// Check if the ARN indicates a service we care about
			if svcFromArn != "" && servicesOfInterest[svcFromArn] {
				// If the ARN's service differs from the extracted key, note the mismatch
				if extractedSvc != svcFromArn {
					// Example: we might log or store the mismatch:
					msg := fmt.Sprintf(
						"Service mismatch for policy=%s key=%s => extracted=%q, arnService=%q (using arnService)",
						policyOrRoleName, key, extractedSvc, svcFromArn,
					)
					mismatches = append(mismatches, msg)
					// Override the final service with the one from the ARN
					finalSvc = svcFromArn
				}
			}

			// If finalSvc is not in our interest set, skip
			if !servicesOfInterest[finalSvc] {
				continue
			}

			// Construct the policy binding. In a real scenario, we might parse the effect, conditions, etc.
			binding := PolicyBinding{
				PolicyOrRoleName: policyOrRoleName,
				Key:              key,
				Effect:           "Allow", // Hard-coded for illustration
				Conditions:       map[string]string{"ExampleConditionKey": "ExampleValue"},
			}

			// Insert into the tree
			svcNode := root.AddChild(finalSvc)
			insertResourcePath(svcNode, resourcePart, binding)
		}
	}
}
return root, mismatches

}

// // 4) MATCHING / LOOKUPS //

// matchesSegment checks if a path segment from the ARN we are searching for // matches a child node name (which could be "", "prefix-", or exact). func matchesSegment(childName, seg string) bool { if childName == "" { return true } if strings.HasSuffix(childName, "") { prefix := childName[:len(childName)-1] return strings.HasPrefix(seg, prefix) } return (childName == seg) }

// FindMatches recursively walks the tree from node, matching each segment in pathSegments. func FindMatches(node *TreeNode, pathSegments []string) []*TreeNode { if len(pathSegments) == 0 { return []*TreeNode{node} } var result []*TreeNode nextSeg := pathSegments[0] remainder := pathSegments[1:]

for childName, childNode := range node.GetChildren() {
	if matchesSegment(childName, nextSeg) {
		result = append(result, FindMatches(childNode, remainder)...)
	}
}
return result

}

// // 5) GATHERING & PRINTING THE "SERVICE => RESOURCE" MAPPING //

// gatherServiceResourceMapping does a DFS and collects full paths. func gatherServiceResourceMapping(root *TreeNode) map[string][]string { out := make(map[string][]string) for svcName, svcNode := range root.GetChildren() { if svcName == "ROOT" { continue } var paths []string gatherDfs(svcNode, "", &paths) out[svcName] = paths } return out }

func gatherDfs(node *TreeNode, prefix string, collector *[]string) { newPrefix := node.Name if prefix != "" && prefix != "ROOT" { newPrefix = prefix + "/" + node.Name }

children := node.GetChildren()
if len(children) == 0 {
	// Leaf => record full path
	*collector = append(*collector, newPrefix)
	return
}
for _, child := range children {
	gatherDfs(child, newPrefix, collector)
}

}

// // 6) OPTIONAL CSV OUTPUT // // If you want to store each node's info in a CSV for DB ingestion, for example, you can do so. // We'll produce a row for each final "leaf" node in each service, listing (service, resourcePath, statements...). //

type CsvRow struct { Service string ResourcePath string // Flatten out the bindings in some textual form: Bindings string }

func exportCsv(root *TreeNode, csvFilePath string) error { // Collect all leaf nodes for each service: var rows []CsvRow

for svcName, svcNode := range root.GetChildren() {
	if svcName == "ROOT" {
		continue
	}
	collectLeafCsvRows(svcNode, svcName, "", &rows)
}

f, err := os.Create(csvFilePath)
if err != nil {
	return err
}
defer f.Close()

writer := csv.NewWriter(f)
defer writer.Flush()

// Write header
if err := writer.Write([]string{"Service", "ResourcePath", "Bindings"}); err != nil {
	return err
}

// Write rows
for _, row := range rows {
	if err := writer.Write([]string{row.Service, row.ResourcePath, row.Bindings}); err != nil {
		return err
	}
}
return nil

}

func collectLeafCsvRows(node *TreeNode, svcName, prefix string, collector *[]CsvRow) { newPrefix := node.Name if prefix != "" && prefix != "ROOT" { newPrefix = prefix + "/" + node.Name }

children := node.GetChildren()
if len(children) == 0 {
	// Leaf => gather data
	// Flatten the node's PolicyBindings into a string
	var bindingStrs []string
	for _, b := range node.Bindings {
		bindingStrs = append(bindingStrs,
			fmt.Sprintf("%s:%s(%s)", b.PolicyOrRoleName, b.Key, b.Effect),
		)
	}
	row := CsvRow{
		Service:      svcName,
		ResourcePath: newPrefix,
		Bindings:     strings.Join(bindingStrs, "; "),
	}
	*collector = append(*collector, row)
	return
}
for _, child := range children {
	collectLeafCsvRows(child, svcName, newPrefix, collector)
}

}

// // 7) MAIN //

func main() { if len(os.Args) < 2 { fmt.Println("Usage: go run main.go /path/to/all-services.json [csvOutputPath]") os.Exit(1) } jsonPath := os.Args[1] var csvPath string if len(os.Args) > 2 { csvPath = os.Args[2] }

// 1) Read JSON
raw, err := ioutil.ReadFile(jsonPath)
if err != nil {
	log.Fatalf("Failed to read JSON file: %v", err)
}

// 2) Parse into AllServices
var allSvcs AllServices
if err := json.Unmarshal(raw, &allSvcs); err != nil {
	log.Fatalf("Failed to unmarshal JSON: %v", err)
}

// 3) Build tree & capture any service mismatches
root, mismatches := buildTreeFromAllServices(allSvcs)
for _, mm := range mismatches {
	log.Printf("MISMATCH: %s", mm)
}

// 4) Gather final mapping
serviceMapping := gatherServiceResourceMapping(root)

// 5) Write JSON mapping
outFile := "service-resource-mapping.json"
data, err := json.MarshalIndent(serviceMapping, "", "  ")
if err != nil {
	log.Fatalf("Failed to marshal final mapping: %v", err)
}
if err := ioutil.WriteFile(outFile, data, 0644); err != nil {
	log.Fatalf("Failed to write output file: %v", err)
}
log.Printf("Service resource mapping written to %s", outFile)

// 6) Optional: Output CSV if a second argument was provided
if csvPath != "" {
	if err := exportCsv(root, csvPath); err != nil {
		log.Fatalf("Failed to export CSV: %v", err)
	}
	log.Printf("CSV exported to %s", csvPath)
}

// 7) Demo a "lookup" using partial/wildcards
exampleResourcePath := "dp-batchplatform-data-prod/bp"
s3Node, ok := root.GetChildren()["s3"]
if ok {
	segs := strings.Split(exampleResourcePath, "/")
	matches := FindMatches(s3Node, segs)
	fmt.Printf("\nMatches for S3 path %q:\n", exampleResourcePath)
	if len(matches) == 0 {
		fmt.Println("  None")
	} else {
		for _, m := range matches {
			// Show node name & policy bindings
			// (In practice, you might reconstruct the entire path or store a parent pointer.)
			fmt.Printf("  Node = %q, policyBindings = %v\n", m.Name, m.Bindings)
		}
	}
}

} the end goal is to service <> resource mapping (Parse IAM policies to build a db of sorts) and I wanna make a tree based structure where we have root node which is empty and we have child does which are servcies and then we keep in making Childs at leaf node we have policy bindings --we are doing to make a db sort fo amke it entrendable code with all edge cases covered -- ref : all_servcies.json { "rds-rs-pipeline": { "s3_read": [ "arn:aws:s3:::sftp-rbl-ags", "arn:aws:s3:::prod-datascience-sms/", "arn:aws:s3:::prod-lending-adhoc", "arn:aws:s3:::prod-lending-adhoc/", "arn:aws:s3:::prod-datascience-sms", "arn:aws:s3:::datascience-store-personalisation/", "arn:aws:s3:::sftp-rbl-ags/", "", "arn:aws:s3:::datascience-store-personalisation" ], "custom_s3_actions0": [ "arn:aws:s3:::sftp-rbl-ags", "arn:aws:s3:::prod-datascience-sms/", "arn:aws:s3:::prod-lending-adhoc", "arn:aws:s3:::prod-lending-adhoc/", "arn:aws:s3:::prod-datascience-sms", "arn:aws:s3:::datascience-store-personalisation/", "arn:aws:s3:::sftp-rbl-ags/", "", "arn:aws:s3:::datascience-store-personalisation" ], "custom_s3-object-lambda_actions0": [ "" ], "ssm_read": [ "arn:aws:ssm:ap-south-1:744315537043:parameter/conf/rds-rs-pipeline/" ], "kms_decrypt": [ "arn:aws:kms:ap-south-1:744315537043:key/af547ea6-b4fd-4050-9185-62068e0da254", "arn:aws:kms:ap-south-1:744315537043:key/c1b80926-ce55-4905-a92a-a1a830ffbf5b", "arn:aws:kms:ap-south-1:744315537043:key/94baadce-0db1-41fe-91ff-f3be240fb6c2" ], "kms_generate_data_key": [ "arn:aws:kms:ap-south-1:744315537043:key/af547ea6-b4fd-4050-9185-62068e0da254", "arn:aws:kms:ap-south-1:744315537043:key/c1b80926-ce55-4905-a92a-a1a830ffbf5b", "arn:aws:kms:ap-south-1:744315537043:key/94baadce-0db1-41fe-91ff-f3be240fb6c2" ], "kms_sign": [ "arn:aws:kms:ap-south-1:744315537043:key/af547ea6-b4fd-4050-9185-62068e0da254", "arn:aws:kms:ap-south-1:744315537043:key/c1b80926-ce55-4905-a92a-a1a830ffbf5b", "arn:aws:kms:ap-south-1:744315537043:key/94baadce-0db1-41fe-91ff-f3be240fb6c2" ], "kms_verify": [ "arn:aws:kms:ap-south-1:744315537043:key/af547ea6-b4fd-4050-9185-62068e0da254", "arn:aws:kms:ap-south-1:744315537043:key/c1b80926-ce55-4905-a92a-a1a830ffbf5b", "arn:aws:kms:ap-south-1:744315537043:key/94baadce-0db1-41fe-91ff-f3be240fb6c2" ], "custom_dynamodb_actions0": [ "arn:aws:dynamodb:ap-south-1:744315537043:table/external_reward_report_config", "arn:aws:dynamodb:ap-south-1:744315537043:table/sms_communications", "arn:aws:dynamodb:us-east-1:744315537043:table/external_reward_report_config" ], "custom_firehose_actions0": [ "arn:aws:firehose:ap-south-1:744315537043:deliverystream/prod_reward_usage_attributes" ], "custom_firehose_actions1": [ "" ], "s3_write": [ "arn:aws:s3:::sftp-rbl-ags", "arn:aws:s3:::prod-datascience-sms/", "arn:aws:s3:::prod-lending-adhoc", "arn:aws:s3:::prod-lending-adhoc/", "arn:aws:s3:::prod-datascience-sms", "arn:aws:s3:::datascience-store-personalisation/", "arn:aws:s3:::sftp-rbl-ags/", "arn:aws:s3:::datascience-store-personalisation" ], "custom_s3_actions1": [ "arn:aws:s3:::eventi-audit", "arn:aws:s3:::eventi-audit/", "arn:aws:s3:::prod-datascience-sms/", "arn:aws:s3:::prod-data-", "arn:aws:s3:::prod-datascience-sms", "arn:aws:s3:::internal-payments-recon", "arn:aws:s3:::stg-redshift", "arn:aws:s3:::internal-payments-recon/", "arn:aws:s3:::stg-redshift/" ], "custom_s3_actions2": [ "" ], "custom_ssm_actions0": [ "" ], "custom_ssm_actions1": [ "arn:aws:ssm:ap-south-1:744315537043:parameter/conf/rds-rs-pipeline/" ], "custom_s3_actions3": [ "" ] }, "metabase-ecs": { "custom_glue_actions0": [ "" ], "custom_s3_actions2": [ "arn:aws:s3:::mp-proxy-main/", "arn:aws:s3:::prod-data-test/", "arn:aws:s3:::mp-proxy-main", "arn:aws:s3:::prod-data-test", "" ], "custom_s3_actions3": [ "arn:aws:s3:::mp-proxy-main/", "arn:aws:s3:::aws-glue-", "arn:aws:s3:::aws-athena-query-results-", "arn:aws:s3:::aws-glue-/", "arn:aws:s3:::athena-examples", "arn:aws:s3:::prod-data-test/", "arn:aws:s3:::/aws-glue-/", "arn:aws:s3:::mp-proxy-main", "arn:aws:s3:::prod-data-test", "arn:aws:s3:::crawler-public", "" ], "custom_ec2_actions0": [ "" ], "custom_iam_actions0": [ "" ], "custom_cloudwatch_actions0": [ "" ], "custom_s3_actions4": [ "arn:aws:s3:::mp-proxy-main/", "arn:aws:s3:::aws-glue-", "arn:aws:s3:::aws-athena-query-results-", "arn:aws:s3:::athena-examples", "arn:aws:s3:::prod-data-test/", "arn:aws:s3:::mp-proxy-main", "arn:aws:s3:::prod-data-test", "arn:aws:s3:::crawler-public", "" ], "custom_s3_actions5": [ "arn:aws:s3:::mp-proxy-main/", "arn:aws:s3:::aws-glue-", "arn:aws:s3:::aws-athena-query-results-", "arn:aws:s3:::aws-glue-/", "arn:aws:s3:::athena-examples*", "arn:aws:s3:::prod-data-test/", "arn:aws:s3:::/aws-glue-/", "arn:aws:s3:::mp-proxy-main", "arn:aws:s3:::prod-data-test", "arn:aws:s3:::crawler-public", "" ], "custom_logs_actions0": [ "arn:aws:logs::::/aws-glue/" ], "custom_ec2_actions1": [ "arn:aws:ec2:::instance/", "arn:aws:ec2:::network-interface/", "arn:aws:ec2:::security-group/" ], "custom_athena_actions0": [ "" ], "custom_glue_actions1": [ "" ], "custom_s3_actions6": [ "arn:aws:s3:::mp-proxy-main/", "arn:aws:s3:::aws-athena-query-results-", "arn:aws:s3:::athena-examples*", "arn:aws:s3:::prod-data-test/", "arn:aws:s3:::mp-proxy-main", "arn:aws:s3:::prod-data-test", "" ], "custom_sns_actions0": [ "" ], "custom_cloudwatch_actions1": [ "" ], "custom_lakeformation_actions0": [ "" ], "custom_datazone_actions0": [ "" ], "custom_pricing_actions0": [ "" ], "ssm_read": [ "arn:aws:ssm:ap-south-1:744315537043:parameter/conf/metabase-ecs/" ], "custom_ssm_actions0": [ "arn:aws:ssm:ap-south-1:744315537043:parameter/conf/metabase-ecs/" ], "custom_ssm_actions1": [ "arn:aws:ssm:ap-south-1:744315537043:parameter/conf/metabase-ecs/" ], "s3_read": [ "arn:aws:s3:::mp-proxy-main", "arn:aws:s3:::mp-proxy-main/", "arn:aws:s3:::prod-data-test/", "arn:aws:s3:::prod-data-test" ], "s3_write": [ "arn:aws:s3:::mp-proxy-main", "arn:aws:s3:::mp-proxy-main/", "arn:aws:s3:::prod-data-test/", "arn:aws:s3:::prod-data-test" ], "custom_s3_actions0": [ "arn:aws:s3:::mp-proxy-main", "arn:aws:s3:::mp-proxy-main/", "arn:aws:s3:::prod-data-test/", "arn:aws:s3:::prod-data-test" ], "custom_s3_actions1": [ "arn:aws:s3:::mp-proxy-main", "arn:aws:s3:::mp-proxy-main/", "arn:aws:s3:::prod-data-test/", "arn:aws:s3:::prod-data-test" ] }, "auth-service": { "custom_dynamodb_actions1": [ "*" ],--- also make this part of existing code more readable ---// 7) Demo a "lookup" using partial/wildcards exampleResourcePath := "dp-batchplatform-data-prod/bp" s3Node, ok := root.GetChildren()["s3"] if ok { segs := strings.Split(exampleResourcePath, "/") matches := FindMatches(s3Node, segs) fmt.Printf("\nMatches for S3 path %q:\n", exampleResourcePath) if len(matches) == 0 { fmt.Println(" None") } else { for _, m := range matches { // Show node name & policy bindings // (In practice, you might reconstruct the entire path or store a parent pointer.) fmt.Printf(" Node = %q, policyBindings = %v\n", m.Name, m.Bindings) } } } --right now it prints like this khushisharma@60489-X1 iamparsing % go run main.go /Users/khushisharma/iamparsing/all-services.json 2025/02/04 19:36:21 Service resource mapping written to service-resource-mapping.json

Matches for S3 path "dp-batchplatform-data-prod/bp": Node = "", policyBindings = [{realtime-platform-core s3_read Allow map[ExampleConditionKey:ExampleValue]} {realtime-platform-core custom_s3_actions11 Allow map[ExampleConditionKey:ExampleValue]} {realtime-platform-core custom_s3_actions1 Allow map[ExampleConditionKey:ExampleValue]} {realtime-platform-core custom_s3_actions2 Allow map[ExampleConditionKey:ExampleValue]} {realtime-platform-core custom_s3_actions5 Allow map[ExampleConditionKey:ExampleValue]} {realtime-platform-core custom_s3_actions6 Allow map[ExampleConditionKey:ExampleValue]} {realtime-platform-core custom_s3_actions12 Allow map[ExampleConditionKey:ExampleValue]} {realtime-platform-core s3_write Allow map[ExampleConditionKey:ExampleValue]} {realtime-platform-core custom_s3_actions0 Allow map[ExampleConditionKey:ExampleValue]} {realtime-platform-core custom_s3_actions3 Allow map[ExampleConditionKey:ExampleValue]} {realtime-platform-core custom_s3_actions8 Allow map[ExampleConditionKey:ExampleValue]} {realtime-platform-core custom_s3_actions4 Allow map[ExampleConditionKey:ExampleValue]} {realtime-platform-core custom_s3_actions13 Allow map[ExampleConditionKey:ExampleValue]} {realtime-platform-core custom_s3_actions10 Allow map[ExampleConditionKey:ExampleValue]} {realtime-platform-core custom_s3_actions7 Allow map[ExampleConditionKey:ExampleValue]} {realtime-platform-core custom_s3_actions9 Allow map[ExampleConditionKey:ExampleValue]}] Node = "bp", policyBindings = [] Node = "", policyBindings = [{document-service custom_s3_actions6 Allow map[ExampleConditionKey:ExampleValue]} {document-service custom_s3_actions8 Allow map[ExampleConditionKey:ExampleValue]} {document-service custom_s3_actions3 Allow map[ExampleConditionKey:ExampleValue]} {document-service custom_s3_actions14 Allow map[ExampleConditionKey:ExampleValue]} {document-service custom_s3_actions11 Allow map[ExampleConditionKey:ExampleValue]} {document-service custom_s3_actions5 Allow map[ExampleConditionKey:ExampleValue]} {document-service custom_s3_actions10 Allow map[ExampleConditionKey:ExampleValue]}] khushisharma@60489-X1 iamparsing %

Thought

Enhancing readability

I'm exploring ways to improve code visibility, considering wildcard techniques for a concise block. This involves detailing which node paths are applicable and constructing a clear tree path from the root.

Enhancing path reconstruction

I’m weighing the benefits of adding a parent pointer in each TreeNode to simplify path reconstruction. This allows building paths from node to root efficiently.

Mapping node paths

I’m thinking through clearer tree paths with wildcard nodes. This leads me to ensure comprehensive visibility for all nodes, making the overall structure more understandable.

Enhancing tree path clarity

I’m finalizing a function to reconstruct node paths, making entries concise. This progress is aimed at improving the overall clarity and coherence of the tree's structure.

Reasoned about tree-based service and resource mapping for 40 seconds

Answer

Below is an enhanced version of your code that:

  1. Maintains a parent pointer in each node, so we can reconstruct the full path for any match in a more readable form.
  2. Prints match results in a friendlier way—showing the entire path from the root, rather than just the single segment.
  3. Preserves the rest of the logic (tree building, wildcard matching, CSV output, etc.) you already have.

Key Additions

  • Parent *TreeNode field in TreeNode.
  • In AddChild, we set child.Parent = n so that each node can walk back up to the root.
  • A function getFullPath(n *TreeNode) that reconstructs a path like "s3/my-bucket/*/some-folder" by walking up from the node to the ROOT.
  • In the demo lookup, we call getFullPath for each matched node and print out the path plus the bindings.

This makes the “Nodes” in the output more understandable (e.g. "s3/*/bp") rather than just seeing "Node = "*" or "Node = "bp".


Full Code

package main import ( "encoding/csv" "encoding/json" "fmt" "io/ioutil" "log" "os" "strings" "sync" ) // // 1) DATA STRUCTURES // // AllServices is your top-level structure parsed from all_services.json. // // For instance: // "rds-rs-pipeline": { // "s3_read": [...], // "custom_s3_actions0": [...], // ... // }, // "metabase-ecs": { ... } type AllServices map[string]map[string][]string // PolicyBinding captures richer data about a single policy statement or binding. // You can extend this with real Condition maps, Resource/Action arrays, etc. type PolicyBinding struct { PolicyOrRoleName string // e.g. "rds-rs-pipeline" Key string // e.g. "s3_read" Effect string // e.g. "Allow" or "Deny" // For demonstration, we store conditions as a string or map. Real IAM might have a more complex structure. Conditions map[string]string } // TreeNode is a node in a tree. Each node can hold zero or more children. // Instead of just storing `Statements []string`, we store a slice of `PolicyBinding` // which can contain effect, conditions, etc. type TreeNode struct { Name string Parent *TreeNode // pointer to parent, for path reconstruction Children map[string]*TreeNode // child name => node Bindings []PolicyBinding mu sync.RWMutex } // NewTreeNode creates a new node. We add a 'parent' parameter so that callers can specify it. func NewTreeNode(name string, parent *TreeNode) *TreeNode { return &TreeNode{ Name: name, Parent: parent, Children: make(map[string]*TreeNode), Bindings: []PolicyBinding{}, } } // AddChild either returns an existing child or creates a new one. func (n *TreeNode) AddChild(childName string) *TreeNode { n.mu.Lock() defer n.mu.Unlock() if child, exists := n.Children[childName]; exists { return child } // create a new child, and set this node as its parent child := NewTreeNode(childName, n) n.Children[childName] = child return child } // AddBinding adds a PolicyBinding to this node. func (n *TreeNode) AddBinding(binding PolicyBinding) { n.mu.Lock() defer n.mu.Unlock() n.Bindings = append(n.Bindings, binding) } // GetChildren returns a (safe) copy of the children map. func (n *TreeNode) GetChildren() map[string]*TreeNode { n.mu.RLock() defer n.mu.RUnlock() copyMap := make(map[string]*TreeNode, len(n.Children)) for k, v := range n.Children { copyMap[k] = v } return copyMap } // getFullPath reconstructs the path from the root (skipping the special "ROOT" node name) down to this node. func getFullPath(n *TreeNode) string { // We'll accumulate names in reverse, then reverse them at the end var segments []string curr := n for curr != nil { // If the node is the special root named "ROOT", we skip it if curr.Name != "ROOT" { segments = append([]string{curr.Name}, segments...) } curr = curr.Parent } // Join with "/" return strings.Join(segments, "/") } // // 2) HELPER FUNCTIONS // // We only care about these services: var servicesOfInterest = map[string]bool{ "s3": true, "sns": true, "sqs": true, "kms": true, "events": true, // for eventbridge } // extractServiceName tries to interpret the second-level key to find the AWS service name. // E.g. "s3_read" -> "s3", "custom_s3_actions0" -> "s3", etc. func extractServiceName(key string) string { if strings.HasPrefix(key, "custom_") { remaining := strings.TrimPrefix(key, "custom_") // e.g. "s3_actions0" parts := strings.Split(remaining, "_") if len(parts) > 0 { return parts[0] // "s3" } } parts := strings.Split(key, "_") if len(parts) > 0 { return parts[0] } return key } // parseArnForServiceAndResource tries to parse an ARN. Returns (service, resourcePart). // // Example: "arn:aws:s3:::my-bucket/path/to/object" => ("s3", "my-bucket/path/to/object") // // If invalid or "*", we do our best to handle it. func parseArnForServiceAndResource(arn string) (string, string) { if arn == "*" { return "", "*" } const arnPrefix = "arn:aws:" if !strings.HasPrefix(arn, arnPrefix) { // Not a standard ARN => treat entire string as resource return "", arn } // ARN format: arn:aws:<service>:<region>:<accountId>:<resource> parts := strings.SplitN(arn, ":", 6) if len(parts) < 6 { // Malformed or incomplete return "", arn } service := parts[2] resource := parts[5] return service, resource } // insertResourcePath splits the resource by slash and inserts each segment into the tree. // We store a final binding at the leaf node (in a real scenario, you might store it at all nodes). func insertResourcePath(serviceNode *TreeNode, resource string, binding PolicyBinding) { if resource == "*" { // Entire resource is wildcard: just attach at child("*") child := serviceNode.AddChild("*") child.AddBinding(binding) return } segments := strings.Split(resource, "/") curr := serviceNode for i, seg := range segments { if seg == "" { continue } curr = curr.AddChild(seg) if i == len(segments)-1 { // Leaf node => add binding curr.AddBinding(binding) } } } // // 3) BUILDING THE TREE // // buildTreeFromAllServices populates a tree with only the services we care about. Each // resource path from the JSON is inserted into the corresponding service node. func buildTreeFromAllServices(allSvcs AllServices) (*TreeNode, []string) { // Create a special root node root := NewTreeNode("ROOT", nil) var mismatches []string // store any "service mismatch" messages, if desired for policyOrRoleName, perms := range allSvcs { // For each sub-key in perms, e.g. "s3_read" -> slice of ARNs for key, resources := range perms { extractedSvc := extractServiceName(key) for _, arnOrRes := range resources { svcFromArn, resourcePart := parseArnForServiceAndResource(arnOrRes) finalSvc := extractedSvc // Check if the ARN indicates a service we care about if svcFromArn != "" && servicesOfInterest[svcFromArn] { // If the ARN's service differs from the extracted key, note the mismatch if extractedSvc != svcFromArn { msg := fmt.Sprintf( "Service mismatch for policy=%s key=%s => extracted=%q, arnService=%q (using arnService)", policyOrRoleName, key, extractedSvc, svcFromArn, ) mismatches = append(mismatches, msg) // Override the final service with the one from the ARN finalSvc = svcFromArn } } // If finalSvc is not in our interest set, skip if !servicesOfInterest[finalSvc] { continue } // Construct the policy binding. In a real scenario, we might parse the effect, conditions, etc. binding := PolicyBinding{ PolicyOrRoleName: policyOrRoleName, Key: key, Effect: "Allow", // Hard-coded for illustration Conditions: map[string]string{"ExampleConditionKey": "ExampleValue"}, } // Insert into the tree svcNode := root.AddChild(finalSvc) insertResourcePath(svcNode, resourcePart, binding) } } } return root, mismatches } // // 4) MATCHING / LOOKUPS // // matchesSegment checks if a path segment from the ARN we are searching for // matches a child node name (which could be "*", "prefix-*", or exact). func matchesSegment(childName, seg string) bool { if childName == "*" { return true } if strings.HasSuffix(childName, "*") { prefix := childName[:len(childName)-1] return strings.HasPrefix(seg, prefix) } return (childName == seg) } // FindMatches recursively walks the tree from `node`, matching each segment in `pathSegments`. func FindMatches(node *TreeNode, pathSegments []string) []*TreeNode { if len(pathSegments) == 0 { return []*TreeNode{node} } var result []*TreeNode nextSeg := pathSegments[0] remainder := pathSegments[1:] for childName, childNode := range node.GetChildren() { if matchesSegment(childName, nextSeg) { result = append(result, FindMatches(childNode, remainder)...) } } return result } // // 5) GATHERING & PRINTING THE "SERVICE => RESOURCE" MAPPING // // gatherServiceResourceMapping does a DFS and collects full paths. func gatherServiceResourceMapping(root *TreeNode) map[string][]string { out := make(map[string][]string) for svcName, svcNode := range root.GetChildren() { if svcName == "ROOT" { continue } var paths []string gatherDfs(svcNode, "", &paths) out[svcName] = paths } return out } func gatherDfs(node *TreeNode, prefix string, collector *[]string) { newPrefix := node.Name if prefix != "" && prefix != "ROOT" { newPrefix = prefix + "/" + node.Name } children := node.GetChildren() if len(children) == 0 { // Leaf => record full path *collector = append(*collector, newPrefix) return } for _, child := range children { gatherDfs(child, newPrefix, collector) } } // // 6) OPTIONAL CSV OUTPUT // // If you want to store each node's info in a CSV for DB ingestion, for example, you can do so. // We'll produce a row for each final "leaf" node in each service, listing (service, resourcePath, statements...). // type CsvRow struct { Service string ResourcePath string // Flatten out the bindings in some textual form: Bindings string } func exportCsv(root *TreeNode, csvFilePath string) error { // Collect all leaf nodes for each service: var rows []CsvRow for svcName, svcNode := range root.GetChildren() { if svcName == "ROOT" { continue } collectLeafCsvRows(svcNode, svcName, "", &rows) } f, err := os.Create(csvFilePath) if err != nil { return err } defer f.Close() writer := csv.NewWriter(f) defer writer.Flush() // Write header if err := writer.Write([]string{"Service", "ResourcePath", "Bindings"}); err != nil { return err } // Write rows for _, row := range rows { if err := writer.Write([]string{row.Service, row.ResourcePath, row.Bindings}); err != nil { return err } } return nil } func collectLeafCsvRows(node *TreeNode, svcName, prefix string, collector *[]CsvRow) { newPrefix := node.Name if prefix != "" && prefix != "ROOT" { newPrefix = prefix + "/" + node.Name } children := node.GetChildren() if len(children) == 0 { // Leaf => gather data // Flatten the node's PolicyBindings into a string var bindingStrs []string for _, b := range node.Bindings { bindingStrs = append(bindingStrs, fmt.Sprintf("%s:%s(%s)", b.PolicyOrRoleName, b.Key, b.Effect), ) } row := CsvRow{ Service: svcName, ResourcePath: newPrefix, Bindings: strings.Join(bindingStrs, "; "), } *collector = append(*collector, row) return } for _, child := range children { collectLeafCsvRows(child, svcName, newPrefix, collector) } } // // 7) MAIN // func main() { if len(os.Args) < 2 { fmt.Println("Usage: go run main.go /path/to/all-services.json [csvOutputPath]") os.Exit(1) } jsonPath := os.Args[1] var csvPath string if len(os.Args) > 2 { csvPath = os.Args[2] } // 1) Read JSON raw, err := ioutil.ReadFile(jsonPath) if err != nil { log.Fatalf("Failed to read JSON file: %v", err) } // 2) Parse into AllServices var allSvcs AllServices if err := json.Unmarshal(raw, &allSvcs); err != nil { log.Fatalf("Failed to unmarshal JSON: %v", err) } // 3) Build tree & capture any service mismatches root, mismatches := buildTreeFromAllServices(allSvcs) for _, mm := range mismatches { log.Printf("MISMATCH: %s", mm) } // 4) Gather final mapping serviceMapping := gatherServiceResourceMapping(root) // 5) Write JSON mapping outFile := "service-resource-mapping.json" data, err := json.MarshalIndent(serviceMapping, "", " ") if err != nil { log.Fatalf("Failed to marshal final mapping: %v", err) } if err := ioutil.WriteFile(outFile, data, 0644); err != nil { log.Fatalf("Failed to write output file: %v", err) } log.Printf("Service resource mapping written to %s", outFile) // 6) Optional: Output CSV if a second argument was provided if csvPath != "" { if err := exportCsv(root, csvPath); err != nil { log.Fatalf("Failed to export CSV: %v", err) } log.Printf("CSV exported to %s", csvPath) } // // 7) Demo a "lookup" using partial/wildcards, with more readable output // exampleResourcePath := "dp-batchplatform-data-prod/bp" s3Node, ok := root.GetChildren()["s3"] if !ok { fmt.Println("No 's3' node found in the tree; skipping demo lookup.") return } segs := strings.Split(exampleResourcePath, "/") matches := FindMatches(s3Node, segs) fmt.Printf("\nMatches for S3 path %q:\n", exampleResourcePath) if len(matches) == 0 { fmt.Println(" None") } else { for _, node := range matches { // Reconstruct the full path from the root using parent pointers path := getFullPath(node) // Print how many bindings are on this node fmt.Printf(" Path = %q, Bindings = %d\n", path, len(node.Bindings)) // If you want to see each binding in detail, you can do: // for _, b := range node.Bindings { // fmt.Printf(" => policy=%s, key=%s, effect=%s\n", b.PolicyOrRoleName, b.Key, b.Effect) // } } } }

What Changed / Why

  1. Parent *TreeNode:
    Every node now stores a pointer to its parent. This is automatically set in AddChild(...).

  2. getFullPath(...):
    We climb up the parents until we reach the root (whose name is "ROOT") and ignore that name. Then we reverse-accumulate the path segments to form something like s3/my-bucket/*/subfolder.

  3. Match Output:
    In the demo lookup, instead of printing Node = "*" or Node = "bp", we do:

    path := getFullPath(node) fmt.Printf(" Path = %q, Bindings = %d\n", path, len(node.Bindings))

    This yields more user-friendly lines like:

    Matches for S3 path "dp-batchplatform-data-prod/bp":
      Path = "s3/*", Bindings = 4
      Path = "s3/*/bp", Bindings = 0
    

    (Or whatever your actual path is.)

  4. Bindings Printing:

    • We still store them in node.Bindings.
    • For demonstration, we show how to print the count or optionally print each one in detail.

With these changes, your tree-based IAM parsing code becomes more readable and explanatory when showing how a path is matched against wildcards or partial wildcards at different levels. It should also be straightforward to extend for any advanced logic or further formatting you need.

Question

a. how can I print policy binding at leaf node is it in code? b. hoe cna I print an example for sqs ---package main

import ( "encoding/csv" "encoding/json" "fmt" "io/ioutil" "log" "os" "strings" "sync" )

// // 1) DATA STRUCTURES //

// AllServices is your top-level structure parsed from all_services.json. // // For instance: // // "rds-rs-pipeline": { // "s3_read": [...], // "custom_s3_actions0": [...], // ... // }, // "metabase-ecs": { ... } type AllServices map[string]map[string][]string

// PolicyBinding captures richer data about a single policy statement or binding. // You can extend this with real Condition maps, Resource/Action arrays, etc. type PolicyBinding struct { PolicyOrRoleName string // e.g. "rds-rs-pipeline" Key string // e.g. "s3_read" Effect string // e.g. "Allow" or "Deny" // For demonstration, we store conditions as a string or map. Real IAM might have a more complex structure. Conditions map[string]string }

// TreeNode is a node in a tree. Each node can hold zero or more children. // Instead of just storing Statements []string, we store a slice of PolicyBinding // which can contain effect, conditions, etc. type TreeNode struct { Name string Parent *TreeNode // pointer to parent, for path reconstruction Children map[string]*TreeNode // child name => node Bindings []PolicyBinding

mu sync.RWMutex

}

// NewTreeNode creates a new node. We add a 'parent' parameter so that callers can specify it. func NewTreeNode(name string, parent *TreeNode) *TreeNode { return &TreeNode{ Name: name, Parent: parent, Children: make(map[string]*TreeNode), Bindings: []PolicyBinding{}, } }

// AddChild either returns an existing child or creates a new one. func (n *TreeNode) AddChild(childName string) *TreeNode { n.mu.Lock() defer n.mu.Unlock()

if child, exists := n.Children[childName]; exists {
	return child
}
// create a new child, and set this node as its parent
child := NewTreeNode(childName, n)
n.Children[childName] = child
return child

}

// AddBinding adds a PolicyBinding to this node. func (n *TreeNode) AddBinding(binding PolicyBinding) { n.mu.Lock() defer n.mu.Unlock() n.Bindings = append(n.Bindings, binding) }

// GetChildren returns a (safe) copy of the children map. func (n *TreeNode) GetChildren() map[string]*TreeNode { n.mu.RLock() defer n.mu.RUnlock()

copyMap := make(map[string]*TreeNode, len(n.Children))
for k, v := range n.Children {
	copyMap[k] = v
}
return copyMap

}

// getFullPath reconstructs the path from the root (skipping the special "ROOT" node name) down to this node. func getFullPath(n *TreeNode) string { // We'll accumulate names in reverse, then reverse them at the end var segments []string

curr := n
for curr != nil {
	// If the node is the special root named "ROOT", we skip it
	if curr.Name != "ROOT" {
		segments = append([]string{curr.Name}, segments...)
	}
	curr = curr.Parent
}

// Join with "/"
return strings.Join(segments, "/")

}

// // 2) HELPER FUNCTIONS //

// We only care about these services: var servicesOfInterest = map[string]bool{ "s3": true, "sns": true, "sqs": true, "kms": true, "events": true, // for eventbridge }

// extractServiceName tries to interpret the second-level key to find the AWS service name. // E.g. "s3_read" -> "s3", "custom_s3_actions0" -> "s3", etc. func extractServiceName(key string) string { if strings.HasPrefix(key, "custom_") { remaining := strings.TrimPrefix(key, "custom_") // e.g. "s3_actions0" parts := strings.Split(remaining, "") if len(parts) > 0 { return parts[0] // "s3" } } parts := strings.Split(key, "") if len(parts) > 0 { return parts[0] } return key }

// parseArnForServiceAndResource tries to parse an ARN. Returns (service, resourcePart). // // Example: "arn:aws:s3:::my-bucket/path/to/object" => ("s3", "my-bucket/path/to/object") // // If invalid or "", we do our best to handle it. func parseArnForServiceAndResource(arn string) (string, string) { if arn == "" { return "", "*" } const arnPrefix = "arn:aws:" if !strings.HasPrefix(arn, arnPrefix) { // Not a standard ARN => treat entire string as resource return "", arn }

// ARN format: arn:aws:<service>:<region>:<accountId>:<resource>
parts := strings.SplitN(arn, ":", 6)
if len(parts) < 6 {
	// Malformed or incomplete
	return "", arn
}

service := parts[2]
resource := parts[5]
return service, resource

}

// insertResourcePath splits the resource by slash and inserts each segment into the tree. // We store a final binding at the leaf node (in a real scenario, you might store it at all nodes). func insertResourcePath(serviceNode TreeNode, resource string, binding PolicyBinding) { if resource == "" { // Entire resource is wildcard: just attach at child("") child := serviceNode.AddChild("") child.AddBinding(binding) return }

segments := strings.Split(resource, "/")
curr := serviceNode
for i, seg := range segments {
	if seg == "" {
		continue
	}
	curr = curr.AddChild(seg)
	if i == len(segments)-1 {
		// Leaf node => add binding
		curr.AddBinding(binding)
	}
}

}

// // 3) BUILDING THE TREE //

// buildTreeFromAllServices populates a tree with only the services we care about. Each // resource path from the JSON is inserted into the corresponding service node. func buildTreeFromAllServices(allSvcs AllServices) (*TreeNode, []string) { // Create a special root node root := NewTreeNode("ROOT", nil) var mismatches []string // store any "service mismatch" messages, if desired

for policyOrRoleName, perms := range allSvcs {
	// For each sub-key in perms, e.g. "s3_read" -> slice of ARNs
	for key, resources := range perms {
		extractedSvc := extractServiceName(key)

		for _, arnOrRes := range resources {
			svcFromArn, resourcePart := parseArnForServiceAndResource(arnOrRes)

			finalSvc := extractedSvc

			// Check if the ARN indicates a service we care about
			if svcFromArn != "" && servicesOfInterest[svcFromArn] {
				// If the ARN's service differs from the extracted key, note the mismatch
				if extractedSvc != svcFromArn {
					msg := fmt.Sprintf(
						"Service mismatch for policy=%s key=%s => extracted=%q, arnService=%q (using arnService)",
						policyOrRoleName, key, extractedSvc, svcFromArn,
					)
					mismatches = append(mismatches, msg)
					// Override the final service with the one from the ARN
					finalSvc = svcFromArn
				}
			}

			// If finalSvc is not in our interest set, skip
			if !servicesOfInterest[finalSvc] {
				continue
			}

			// Construct the policy binding. In a real scenario, we might parse the effect, conditions, etc.
			binding := PolicyBinding{
				PolicyOrRoleName: policyOrRoleName,
				Key:              key,
				Effect:           "Allow", // Hard-coded for illustration
				Conditions:       map[string]string{"ExampleConditionKey": "ExampleValue"},
			}

			// Insert into the tree
			svcNode := root.AddChild(finalSvc)
			insertResourcePath(svcNode, resourcePart, binding)
		}
	}
}
return root, mismatches

}

// // 4) MATCHING / LOOKUPS //

// matchesSegment checks if a path segment from the ARN we are searching for // matches a child node name (which could be "", "prefix-", or exact). func matchesSegment(childName, seg string) bool { if childName == "" { return true } if strings.HasSuffix(childName, "") { prefix := childName[:len(childName)-1] return strings.HasPrefix(seg, prefix) } return (childName == seg) }

// FindMatches recursively walks the tree from node, matching each segment in pathSegments. func FindMatches(node *TreeNode, pathSegments []string) []*TreeNode { if len(pathSegments) == 0 { return []*TreeNode{node} } var result []*TreeNode nextSeg := pathSegments[0] remainder := pathSegments[1:]

for childName, childNode := range node.GetChildren() {
	if matchesSegment(childName, nextSeg) {
		result = append(result, FindMatches(childNode, remainder)...)
	}
}
return result

}

// // 5) GATHERING & PRINTING THE "SERVICE => RESOURCE" MAPPING //

// gatherServiceResourceMapping does a DFS and collects full paths. func gatherServiceResourceMapping(root *TreeNode) map[string][]string { out := make(map[string][]string) for svcName, svcNode := range root.GetChildren() { if svcName == "ROOT" { continue } var paths []string gatherDfs(svcNode, "", &paths) out[svcName] = paths } return out }

func gatherDfs(node *TreeNode, prefix string, collector *[]string) { newPrefix := node.Name if prefix != "" && prefix != "ROOT" { newPrefix = prefix + "/" + node.Name }

children := node.GetChildren()
if len(children) == 0 {
	// Leaf => record full path
	*collector = append(*collector, newPrefix)
	return
}
for _, child := range children {
	gatherDfs(child, newPrefix, collector)
}

}

// // 6) OPTIONAL CSV OUTPUT // // If you want to store each node's info in a CSV for DB ingestion, for example, you can do so. // We'll produce a row for each final "leaf" node in each service, listing (service, resourcePath, statements...). //

type CsvRow struct { Service string ResourcePath string // Flatten out the bindings in some textual form: Bindings string }

func exportCsv(root *TreeNode, csvFilePath string) error { // Collect all leaf nodes for each service: var rows []CsvRow

for svcName, svcNode := range root.GetChildren() {
	if svcName == "ROOT" {
		continue
	}
	collectLeafCsvRows(svcNode, svcName, "", &rows)
}

f, err := os.Create(csvFilePath)
if err != nil {
	return err
}
defer f.Close()

writer := csv.NewWriter(f)
defer writer.Flush()

// Write header
if err := writer.Write([]string{"Service", "ResourcePath", "Bindings"}); err != nil {
	return err
}

// Write rows
for _, row := range rows {
	if err := writer.Write([]string{row.Service, row.ResourcePath, row.Bindings}); err != nil {
		return err
	}
}
return nil

}

func collectLeafCsvRows(node *TreeNode, svcName, prefix string, collector *[]CsvRow) { newPrefix := node.Name if prefix != "" && prefix != "ROOT" { newPrefix = prefix + "/" + node.Name }

children := node.GetChildren()
if len(children) == 0 {
	// Leaf => gather data
	// Flatten the node's PolicyBindings into a string
	var bindingStrs []string
	for _, b := range node.Bindings {
		bindingStrs = append(bindingStrs,
			fmt.Sprintf("%s:%s(%s)", b.PolicyOrRoleName, b.Key, b.Effect),
		)
	}
	row := CsvRow{
		Service:      svcName,
		ResourcePath: newPrefix,
		Bindings:     strings.Join(bindingStrs, "; "),
	}
	*collector = append(*collector, row)
	return
}
for _, child := range children {
	collectLeafCsvRows(child, svcName, newPrefix, collector)
}

}

// // 7) MAIN //

func main() { if len(os.Args) < 2 { fmt.Println("Usage: go run main.go /path/to/all-services.json [csvOutputPath]") os.Exit(1) } jsonPath := os.Args[1] var csvPath string if len(os.Args) > 2 { csvPath = os.Args[2] }

// 1) Read JSON
raw, err := ioutil.ReadFile(jsonPath)
if err != nil {
	log.Fatalf("Failed to read JSON file: %v", err)
}

// 2) Parse into AllServices
var allSvcs AllServices
if err := json.Unmarshal(raw, &allSvcs); err != nil {
	log.Fatalf("Failed to unmarshal JSON: %v", err)
}

// 3) Build tree & capture any service mismatches
root, mismatches := buildTreeFromAllServices(allSvcs)
for _, mm := range mismatches {
	log.Printf("MISMATCH: %s", mm)
}

// 4) Gather final mapping
serviceMapping := gatherServiceResourceMapping(root)

// 5) Write JSON mapping
outFile := "service-resource-mapping.json"
data, err := json.MarshalIndent(serviceMapping, "", "  ")
if err != nil {
	log.Fatalf("Failed to marshal final mapping: %v", err)
}
if err := ioutil.WriteFile(outFile, data, 0644); err != nil {
	log.Fatalf("Failed to write output file: %v", err)
}
log.Printf("Service resource mapping written to %s", outFile)

// 6) Optional: Output CSV if a second argument was provided
if csvPath != "" {
	if err := exportCsv(root, csvPath); err != nil {
		log.Fatalf("Failed to export CSV: %v", err)
	}
	log.Printf("CSV exported to %s", csvPath)
}

//
// 7) Demo a "lookup" using partial/wildcards, with more readable output
//

exampleResourcePath := "dp-batchplatform-data-prod/bp"
s3Node, ok := root.GetChildren()["s3"]
if !ok {
	fmt.Println("No 's3' node found in the tree; skipping demo lookup.")
	return
}

segs := strings.Split(exampleResourcePath, "/")
matches := FindMatches(s3Node, segs)

fmt.Printf("\nMatches for S3 path %q:\n", exampleResourcePath)
if len(matches) == 0 {
	fmt.Println("  None")
} else {
	for _, node := range matches {
		// Reconstruct the full path from the root using parent pointers
		path := getFullPath(node)
		// Print how many bindings are on this node
		fmt.Printf("  Path = %q, Bindings = %d\n", path, len(node.Bindings))

		// If you want to see each binding in detail, you can do:
		// for _, b := range node.Bindings {
		//     fmt.Printf("    => policy=%s, key=%s, effect=%s\n", b.PolicyOrRoleName, b.Key, b.Effect)
		// }
	}
}

}

Thought

Clarifying intent

The user wishes to see how to print policy binding at a leaf node, particularly for SQS, and possibly a second example for SQS. Replicating the approach for S3 to SQS is considered.

Demonstrating binding

I’m mapping out the process to print policy bindings at a leaf node for an SQS example resource path, ensuring clear steps and logic for effective demonstration.

Reasoned for 14 seconds

Answer

Below are two clarifications based on your questions:

  1. How to print policy bindings at leaf nodes (or any matched node).
  2. How to perform a demo lookup for an SQS resource similar to the S3 example.

A) Printing Policy Bindings at Leaf Nodes

By design, your code already attaches all PolicyBindings to the leaf node in the path (though you could also add them at intermediate nodes if you wanted). To print those bindings, you just iterate over node.Bindings.

For example, here’s how you can print each binding for each leaf node, inside your existing DFS or after matching:

// Suppose we have some node in the tree that is a leaf. // We can reconstruct the full path and then print the node's Bindings: func printLeafBindings(node *TreeNode) { path := getFullPath(node) // s3/my-bucket/* fmt.Printf("Leaf path = %q\n", path) if len(node.Bindings) == 0 { fmt.Println(" (No bindings at this leaf)") return } for _, binding := range node.Bindings { fmt.Printf(" => Policy: %s, Key: %s, Effect: %s, Conditions: %v\n", binding.PolicyOrRoleName, binding.Key, binding.Effect, binding.Conditions, ) } }

Where getFullPath(node) is your helper function that climbs up the parent pointers to reconstruct something like "s3/my-bucket/*" or "s3/my-bucket/some-folder".

You can call this from anywhere in your code that deals with a leaf node. For instance, in collectLeafCsvRows, you could add extra logging or printing. Or, in your main “lookup” demonstration, you could do:

for _, node := range matches { // This node might be a leaf or mid-level wildcard. If you specifically only want leaves, check if len(node.GetChildren())==0 fmt.Printf("Full path: %s\n", getFullPath(node)) for _, b := range node.Bindings { fmt.Printf(" => Policy=%s, Key=%s, Effect=%s\n", b.PolicyOrRoleName, b.Key, b.Effect) } }

B) Example: Printing a Demo Lookup for SQS

You already do a demo lookup for S3 with:

exampleResourcePath := "dp-batchplatform-data-prod/bp" s3Node, ok := root.GetChildren()["s3"] ...

You can replicate that for SQS. For instance, suppose your SQS resources in the JSON look like:

"sqs_read": [
  "arn:aws:sqs:ap-south-1:744315537043:prod-leaderboard-cache-update-queue-dlq",
  ...
]

You can pick a path segment—like "prod-leaderboard-cache-update-queue-dlq"—and do the same steps:

// 1) Construct some SQS path to look up (just the queue name, or queue name + subpaths if they existed) exampleSqsPath := "prod-leaderboard-cache-update-queue-dlq" // 2) Find the "sqs" child node from the root sqsNode, ok := root.GetChildren()["sqs"] if !ok { fmt.Println("No 'sqs' node found in the tree; skipping SQS demo lookup.") } else { // 3) Split the resource path into segments sqsSegments := strings.Split(exampleSqsPath, "/") // 4) Find matches in the SQS subtree sqsMatches := FindMatches(sqsNode, sqsSegments) // 5) Print them fmt.Printf("\nMatches for SQS path %q:\n", exampleSqsPath) if len(sqsMatches) == 0 { fmt.Println(" None") } else { for _, node := range sqsMatches { path := getFullPath(node) fmt.Printf(" Matched node path = %q, Bindings = %d\n", path, len(node.Bindings)) // Print each binding if you like for _, b := range node.Bindings { fmt.Printf(" => Policy=%s, Key=%s, Effect=%s\n", b.PolicyOrRoleName, b.Key, b.Effect) } } } }

Key points:

  • SQS ARNs typically look like:
    arn:aws:sqs:<region>:<accountId>:<queueName>
    
  • If your JSON has partial wildcards or additional path segments, adapt accordingly.
  • You split on "/" the same way you do for S3. But note that many SQS ARNs have no slash after the queue name, so your segments might be just ["prod-leaderboard-cache-update-queue-dlq"].

Putting It All Together

You can place that “SQS example” code in main() right after or before your existing S3 example. For instance:

func main() { ... // Existing code: Build the tree, etc. // ~~~ Demo for S3 ~~~ { exampleResourcePath := "dp-batchplatform-data-prod/bp" s3Node, ok := root.GetChildren()["s3"] ... } // ~~~ Demo for SQS ~~~ { exampleSqsPath := "prod-leaderboard-cache-update-queue-dlq" sqsNode, ok := root.GetChildren()["sqs"] if !ok { fmt.Println("No 'sqs' node found in the tree; skipping SQS demo lookup.") } else { sqsSegments := strings.Split(exampleSqsPath, "/") sqsMatches := FindMatches(sqsNode, sqsSegments) fmt.Printf("\nMatches for SQS path %q:\n", exampleSqsPath) if len(sqsMatches) == 0 { fmt.Println(" None") } else { for _, node := range sqsMatches { fullpath := getFullPath(node) fmt.Printf(" Path = %q, #Bindings = %d\n", fullpath, len(node.Bindings)) for _, b := range node.Bindings { fmt.Printf(" => Policy=%s, Key=%s, Effect=%s\n", b.PolicyOrRoleName, b.Key, b.Effect) } } } } } }

When you run it, you’ll see the lines for the SQS resource matches, their full path in the tree, and each policy binding. That’s basically identical to the S3 approach—just substituting "sqs" and a queue name.


Final Notes

  • Printing Bindings: You can do it in the CSV export, in the DFS of the entire tree, or specifically after matching certain resource paths. The data is already there in node.Bindings.
  • SQS Demo: Exactly the same as your S3 code, but changing references from "s3" to "sqs" and providing an SQS-like resource path. If your path includes wildcards or partial wildcards ("prod-*-cache-update-queue-dlq"), the same FindMatches logic will handle it.
  • Leaf Nodes: If you only want leaf nodes, you can check len(node.GetChildren()) == 0 to confirm no further subdivisions exist. This is often how you know you’re at a final resource path (e.g. an actual queue).

That’s all you need to print out the policy bindings for leaf nodes and demonstrate a second example for SQS.

Question

package main

import ( "encoding/csv" "encoding/json" "fmt" "io/ioutil" "log" "os" "strings" "sync" )

// // 1) DATA STRUCTURES //

// AllServices is your top-level structure parsed from all_services.json. // // For instance: // // "rds-rs-pipeline": { // "s3_read": [...], // "custom_s3_actions0": [...], // ... // }, // "metabase-ecs": { ... } type AllServices map[string]map[string][]string

// PolicyBinding captures richer data about a single policy statement or binding. // You can extend this with real Condition maps, Resource/Action arrays, etc. type PolicyBinding struct { PolicyOrRoleName string // e.g. "rds-rs-pipeline" Key string // e.g. "s3_read" Effect string // e.g. "Allow" or "Deny" // For demonstration, we store conditions as a string or map. Real IAM might have a more complex structure. Conditions map[string]string }

// TreeNode is a node in a tree. Each node can hold zero or more children. // Instead of just storing Statements []string, we store a slice of PolicyBinding // which can contain effect, conditions, etc. type TreeNode struct { Name string Parent *TreeNode // pointer to parent, for path reconstruction Children map[string]*TreeNode // child name => node Bindings []PolicyBinding

mu sync.RWMutex

}

// NewTreeNode creates a new node. We add a 'parent' parameter so that callers can specify it. func NewTreeNode(name string, parent *TreeNode) *TreeNode { return &TreeNode{ Name: name, Parent: parent, Children: make(map[string]*TreeNode), Bindings: []PolicyBinding{}, } }

// AddChild either returns an existing child or creates a new one. func (n *TreeNode) AddChild(childName string) *TreeNode { n.mu.Lock() defer n.mu.Unlock()

if child, exists := n.Children[childName]; exists {
	return child
}
// create a new child, and set this node as its parent
child := NewTreeNode(childName, n)
n.Children[childName] = child
return child

}

// AddBinding adds a PolicyBinding to this node. func (n *TreeNode) AddBinding(binding PolicyBinding) { n.mu.Lock() defer n.mu.Unlock() n.Bindings = append(n.Bindings, binding) }

// GetChildren returns a (safe) copy of the children map. func (n *TreeNode) GetChildren() map[string]*TreeNode { n.mu.RLock() defer n.mu.RUnlock()

copyMap := make(map[string]*TreeNode, len(n.Children))
for k, v := range n.Children {
	copyMap[k] = v
}
return copyMap

}

// getFullPath reconstructs the path from the root (skipping the special "ROOT" node name) down to this node. func getFullPath(n *TreeNode) string { // We'll accumulate names in reverse, then reverse them at the end var segments []string

curr := n
for curr != nil {
	// If the node is the special root named "ROOT", we skip it
	if curr.Name != "ROOT" {
		segments = append([]string{curr.Name}, segments...)
	}
	curr = curr.Parent
}

// Join with "/"
return strings.Join(segments, "/")

}

// // 2) HELPER FUNCTIONS //

// We only care about these services: var servicesOfInterest = map[string]bool{ "s3": true, "sns": true, "sqs": true, "kms": true, "events": true, // for eventbridge }

// extractServiceName tries to interpret the second-level key to find the AWS service name. // E.g. "s3_read" -> "s3", "custom_s3_actions0" -> "s3", etc. func extractServiceName(key string) string { if strings.HasPrefix(key, "custom_") { remaining := strings.TrimPrefix(key, "custom_") // e.g. "s3_actions0" parts := strings.Split(remaining, "") if len(parts) > 0 { return parts[0] // "s3" } } parts := strings.Split(key, "") if len(parts) > 0 { return parts[0] } return key }

// parseArnForServiceAndResource tries to parse an ARN. Returns (service, resourcePart). // // Example: "arn:aws:s3:::my-bucket/path/to/object" => ("s3", "my-bucket/path/to/object") // // If invalid or "", we do our best to handle it. func parseArnForServiceAndResource(arn string) (string, string) { if arn == "" { return "", "*" } const arnPrefix = "arn:aws:" if !strings.HasPrefix(arn, arnPrefix) { // Not a standard ARN => treat entire string as resource return "", arn }

// ARN format: arn:aws:<service>:<region>:<accountId>:<resource>
parts := strings.SplitN(arn, ":", 6)
if len(parts) < 6 {
	// Malformed or incomplete
	return "", arn
}

service := parts[2]
resource := parts[5]
return service, resource

}

// insertResourcePath splits the resource by slash and inserts each segment into the tree. // We store a final binding at the leaf node (in a real scenario, you might store it at all nodes). func insertResourcePath(serviceNode TreeNode, resource string, binding PolicyBinding) { if resource == "" { // Entire resource is wildcard: just attach at child("") child := serviceNode.AddChild("") child.AddBinding(binding) return }

segments := strings.Split(resource, "/")
curr := serviceNode
for i, seg := range segments {
	if seg == "" {
		continue
	}
	curr = curr.AddChild(seg)
	if i == len(segments)-1 {
		// Leaf node => add binding
		curr.AddBinding(binding)
	}
}

}

// // 3) BUILDING THE TREE //

// buildTreeFromAllServices populates a tree with only the services we care about. Each // resource path from the JSON is inserted into the corresponding service node. func buildTreeFromAllServices(allSvcs AllServices) (*TreeNode, []string) { // Create a special root node root := NewTreeNode("ROOT", nil) var mismatches []string // store any "service mismatch" messages, if desired

for policyOrRoleName, perms := range allSvcs {
	// For each sub-key in perms, e.g. "s3_read" -> slice of ARNs
	for key, resources := range perms {
		extractedSvc := extractServiceName(key)

		for _, arnOrRes := range resources {
			svcFromArn, resourcePart := parseArnForServiceAndResource(arnOrRes)

			finalSvc := extractedSvc

			// Check if the ARN indicates a service we care about
			if svcFromArn != "" && servicesOfInterest[svcFromArn] {
				// If the ARN's service differs from the extracted key, note the mismatch
				if extractedSvc != svcFromArn {
					msg := fmt.Sprintf(
						"Service mismatch for policy=%s key=%s => extracted=%q, arnService=%q (using arnService)",
						policyOrRoleName, key, extractedSvc, svcFromArn,
					)
					mismatches = append(mismatches, msg)
					// Override the final service with the one from the ARN
					finalSvc = svcFromArn
				}
			}

			// If finalSvc is not in our interest set, skip
			if !servicesOfInterest[finalSvc] {
				continue
			}

			// Construct the policy binding. In a real scenario, we might parse the effect, conditions, etc.
			binding := PolicyBinding{
				PolicyOrRoleName: policyOrRoleName,
				Key:              key,
				Effect:           "Allow", // Hard-coded for illustration
				Conditions:       map[string]string{"ExampleConditionKey": "ExampleValue"},
			}

			// Insert into the tree
			svcNode := root.AddChild(finalSvc)
			insertResourcePath(svcNode, resourcePart, binding)
		}
	}
}
return root, mismatches

}

// // 4) MATCHING / LOOKUPS //

// matchesSegment checks if a path segment from the ARN we are searching for // matches a child node name (which could be "", "prefix-", or exact). func matchesSegment(childName, seg string) bool { if childName == "" { return true } if strings.HasSuffix(childName, "") { prefix := childName[:len(childName)-1] return strings.HasPrefix(seg, prefix) } return (childName == seg) }

// FindMatches recursively walks the tree from node, matching each segment in pathSegments. func FindMatches(node *TreeNode, pathSegments []string) []*TreeNode { if len(pathSegments) == 0 { return []*TreeNode{node} } var result []*TreeNode nextSeg := pathSegments[0] remainder := pathSegments[1:]

for childName, childNode := range node.GetChildren() {
	if matchesSegment(childName, nextSeg) {
		result = append(result, FindMatches(childNode, remainder)...)
	}
}
return result

}

// // 5) GATHERING & PRINTING THE "SERVICE => RESOURCE" MAPPING //

// gatherServiceResourceMapping does a DFS and collects full paths. func gatherServiceResourceMapping(root *TreeNode) map[string][]string { out := make(map[string][]string) for svcName, svcNode := range root.GetChildren() { if svcName == "ROOT" { continue } var paths []string gatherDfs(svcNode, "", &paths) out[svcName] = paths } return out }

func gatherDfs(node *TreeNode, prefix string, collector *[]string) { newPrefix := node.Name if prefix != "" && prefix != "ROOT" { newPrefix = prefix + "/" + node.Name }

children := node.GetChildren()
if len(children) == 0 {
	// Leaf => record full path
	*collector = append(*collector, newPrefix)
	return
}
for _, child := range children {
	gatherDfs(child, newPrefix, collector)
}

}

// // 6) OPTIONAL CSV OUTPUT // // If you want to store each node's info in a CSV for DB ingestion, for example, you can do so. // We'll produce a row for each final "leaf" node in each service, listing (service, resourcePath, statements...). //

type CsvRow struct { Service string ResourcePath string // Flatten out the bindings in some textual form: Bindings string }

func exportCsv(root *TreeNode, csvFilePath string) error { // Collect all leaf nodes for each service: var rows []CsvRow

for svcName, svcNode := range root.GetChildren() {
	if svcName == "ROOT" {
		continue
	}
	collectLeafCsvRows(svcNode, svcName, "", &rows)
}

f, err := os.Create(csvFilePath)
if err != nil {
	return err
}
defer f.Close()

writer := csv.NewWriter(f)
defer writer.Flush()

// Write header
if err := writer.Write([]string{"Service", "ResourcePath", "Bindings"}); err != nil {
	return err
}

// Write rows
for _, row := range rows {
	if err := writer.Write([]string{row.Service, row.ResourcePath, row.Bindings}); err != nil {
		return err
	}
}
return nil

}

func collectLeafCsvRows(node *TreeNode, svcName, prefix string, collector *[]CsvRow) { newPrefix := node.Name if prefix != "" && prefix != "ROOT" { newPrefix = prefix + "/" + node.Name }

children := node.GetChildren()
if len(children) == 0 {
	// Leaf => gather data
	// Flatten the node's PolicyBindings into a string
	var bindingStrs []string
	for _, b := range node.Bindings {
		bindingStrs = append(bindingStrs,
			fmt.Sprintf("%s:%s(%s)", b.PolicyOrRoleName, b.Key, b.Effect),
		)
	}
	row := CsvRow{
		Service:      svcName,
		ResourcePath: newPrefix,
		Bindings:     strings.Join(bindingStrs, "; "),
	}
	*collector = append(*collector, row)
	return
}
for _, child := range children {
	collectLeafCsvRows(child, svcName, newPrefix, collector)
}

}

// // 7) MAIN //

func main() { if len(os.Args) < 2 { fmt.Println("Usage: go run main.go /path/to/all-services.json [csvOutputPath]") os.Exit(1) } jsonPath := os.Args[1] var csvPath string if len(os.Args) > 2 { csvPath = os.Args[2] }

// 1) Read JSON
raw, err := ioutil.ReadFile(jsonPath)
if err != nil {
	log.Fatalf("Failed to read JSON file: %v", err)
}

// 2) Parse into AllServices
var allSvcs AllServices
if err := json.Unmarshal(raw, &allSvcs); err != nil {
	log.Fatalf("Failed to unmarshal JSON: %v", err)
}

// 3) Build tree & capture any service mismatches
root, mismatches := buildTreeFromAllServices(allSvcs)
for _, mm := range mismatches {
	log.Printf("MISMATCH: %s", mm)
}

// 4) Gather final mapping
serviceMapping := gatherServiceResourceMapping(root)

// 5) Write JSON mapping
outFile := "service-resource-mapping.json"
data, err := json.MarshalIndent(serviceMapping, "", "  ")
if err != nil {
	log.Fatalf("Failed to marshal final mapping: %v", err)
}
if err := ioutil.WriteFile(outFile, data, 0644); err != nil {
	log.Fatalf("Failed to write output file: %v", err)
}
log.Printf("Service resource mapping written to %s", outFile)

// 6) Optional: Output CSV if a second argument was provided
if csvPath != "" {
	if err := exportCsv(root, csvPath); err != nil {
		log.Fatalf("Failed to export CSV: %v", err)
	}
	log.Printf("CSV exported to %s", csvPath)
}

//
// 7) Demo a "lookup" using partial/wildcards, with more readable output
//

exampleResourcePath := "dp-batchplatform-data-prod/bp"
s3Node, ok := root.GetChildren()["s3"]
if !ok {
	fmt.Println("No 's3' node found in the tree; skipping demo lookup.")
	return
}

segs := strings.Split(exampleResourcePath, "/")
matches := FindMatches(s3Node, segs)

fmt.Printf("\nMatches for S3 path %q:\n", exampleResourcePath)
if len(matches) == 0 {
	fmt.Println("  None")
} else {
	for _, node := range matches {
		// Reconstruct the full path from the root using parent pointers
		path := getFullPath(node)
		// Print how many bindings are on this node
		fmt.Printf("  Path = %q, Bindings = %d\n", path, len(node.Bindings))

		// If you want to see each binding in detail, you can do:
		// for _, b := range node.Bindings {
		//     fmt.Printf("    => policy=%s, key=%s, effect=%s\n", b.PolicyOrRoleName, b.Key, b.Effect)
		// }
	}
}

// 1) Construct some SQS path to look up (just the queue name, or queue name + subpaths if they existed)
exampleSqsPath := "mentos-waitlist-consumer"

// 2) Find the "sqs" child node from the root
sqsNode, ok := root.GetChildren()["sqs"]
if !ok {
	fmt.Println("No 'sqs' node found in the tree; skipping SQS demo lookup.")
} else {
	// 3) Split the resource path into segments
	sqsSegments := strings.Split(exampleSqsPath, "/")

	// 4) Find matches in the SQS subtree
	sqsMatches := FindMatches(sqsNode, sqsSegments)

	// 5) Print them
	fmt.Printf("\nMatches for SQS path %q:\n", exampleSqsPath)
	if len(sqsMatches) == 0 {
		fmt.Println("  None")
	} else {
		for _, node := range sqsMatches {
			path := getFullPath(node)
			fmt.Printf("  Matched node path = %q, Bindings = %d\n", path, len(node.Bindings))
			// Print each binding if you like
			for _, b := range node.Bindings {
				fmt.Printf("    => Policy=%s, Key=%s, Effect=%s\n",
					b.PolicyOrRoleName, b.Key, b.Effect)
			}
		}
	}
}

} iw anna write the results of exampleResourcePath := "dp-batchplatform-data-prod/bp" s3Node, ok := root.GetChildren()["s3"] if !ok { fmt.Println("No 's3' node found in the tree; skipping demo lookup.") return }

segs := strings.Split(exampleResourcePath, "/")
matches := FindMatches(s3Node, segs)

fmt.Printf("\nMatches for S3 path %q:\n", exampleResourcePath)
if len(matches) == 0 {
	fmt.Println("  None")
} else {
	for _, node := range matches {
		// Reconstruct the full path from the root using parent pointers
		path := getFullPath(node)
		// Print how many bindings are on this node
		fmt.Printf("  Path = %q, Bindings = %d\n", path, len(node.Bindings))

		// If you want to see each binding in detail, you can do:
		// for _, b := range node.Bindings {
		//     fmt.Printf("    => policy=%s, key=%s, effect=%s\n", b.PolicyOrRoleName, b.Key, b.Effect)
		// }
	}
}

// 1) Construct some SQS path to look up (just the queue name, or queue name + subpaths if they existed)
exampleSqsPath := "mentos-waitlist-consumer"

// 2) Find the "sqs" child node from the root
sqsNode, ok := root.GetChildren()["sqs"]
if !ok {
	fmt.Println("No 'sqs' node found in the tree; skipping SQS demo lookup.")
} else {
	// 3) Split the resource path into segments
	sqsSegments := strings.Split(exampleSqsPath, "/")

	// 4) Find matches in the SQS subtree
	sqsMatches := FindMatches(sqsNode, sqsSegments)

	// 5) Print them
	fmt.Printf("\nMatches for SQS path %q:\n", exampleSqsPath)
	if len(sqsMatches) == 0 {
		fmt.Println("  None")
	} else {
		for _, node := range sqsMatches {
			path := getFullPath(node)
			fmt.Printf("  Matched node path = %q, Bindings = %d\n", path, len(node.Bindings))
			// Print each binding if you like
			for _, b := range node.Bindings {
				fmt.Printf("    => Policy=%s, Key=%s, Effect=%s\n",
					b.PolicyOrRoleName, b.Key, b.Effect)
			}
		}
	}
} to a separate file 

Thought

Exporting lookup results

I'm thinking through how to write results from exampleResourcePath to a file, similar to S3 and SQS exports. Considering creating a function writeLookupResults(filename string, s3Results []string, sqsResults []string) error to output results to a plain text file.

Reasoned for 5 seconds

Answer

Below is one way to modify your code so that the lookup results (for example, the S3 and SQS demo lookup results) are written to a separate file (for example, a plain text file). In this example, we capture the lookup output in a string builder and then write that string to a file (e.g. “lookup-results.txt”). You can adjust the file name as needed.

Below is the modified code (with the new “writeLookupResults” section added near the end of main):


package main import ( "encoding/csv" "encoding/json" "fmt" "io/ioutil" "log" "os" "strings" "sync" ) // // 1) DATA STRUCTURES // // AllServices is your top-level structure parsed from all_services.json. type AllServices map[string]map[string][]string // PolicyBinding captures richer data about a single policy statement or binding. type PolicyBinding struct { PolicyOrRoleName string // e.g. "rds-rs-pipeline" Key string // e.g. "s3_read" Effect string // e.g. "Allow" or "Deny" Conditions map[string]string // For demonstration: conditions (could be more complex) } // TreeNode is a node in a tree. Each node can hold zero or more children and policy bindings. type TreeNode struct { Name string Parent *TreeNode // pointer to parent, for path reconstruction Children map[string]*TreeNode // child name => node Bindings []PolicyBinding mu sync.RWMutex } // NewTreeNode creates a new node. func NewTreeNode(name string, parent *TreeNode) *TreeNode { return &TreeNode{ Name: name, Parent: parent, Children: make(map[string]*TreeNode), Bindings: []PolicyBinding{}, } } // AddChild either returns an existing child or creates a new one. func (n *TreeNode) AddChild(childName string) *TreeNode { n.mu.Lock() defer n.mu.Unlock() if child, exists := n.Children[childName]; exists { return child } child := NewTreeNode(childName, n) n.Children[childName] = child return child } // AddBinding adds a PolicyBinding to this node. func (n *TreeNode) AddBinding(binding PolicyBinding) { n.mu.Lock() defer n.mu.Unlock() n.Bindings = append(n.Bindings, binding) } // GetChildren returns a (safe) copy of the children map. func (n *TreeNode) GetChildren() map[string]*TreeNode { n.mu.RLock() defer n.mu.RUnlock() copyMap := make(map[string]*TreeNode, len(n.Children)) for k, v := range n.Children { copyMap[k] = v } return copyMap } // getFullPath reconstructs the path from the root (skipping the special "ROOT" node name). func getFullPath(n *TreeNode) string { var segments []string curr := n for curr != nil { if curr.Name != "ROOT" { segments = append([]string{curr.Name}, segments...) } curr = curr.Parent } return strings.Join(segments, "/") } // // 2) HELPER FUNCTIONS // // Only care about these services: var servicesOfInterest = map[string]bool{ "s3": true, "sns": true, "sqs": true, "kms": true, "events": true, } // extractServiceName returns the service portion of a key (e.g. "s3_read" -> "s3"). func extractServiceName(key string) string { if strings.HasPrefix(key, "custom_") { remaining := strings.TrimPrefix(key, "custom_") parts := strings.Split(remaining, "_") if len(parts) > 0 { return parts[0] } } parts := strings.Split(key, "_") if len(parts) > 0 { return parts[0] } return key } // parseArnForServiceAndResource parses an ARN and returns the service and resource part. func parseArnForServiceAndResource(arn string) (string, string) { if arn == "*" { return "", "*" } const arnPrefix = "arn:aws:" if !strings.HasPrefix(arn, arnPrefix) { return "", arn } parts := strings.SplitN(arn, ":", 6) if len(parts) < 6 { return "", arn } service := parts[2] resource := parts[5] return service, resource } // insertResourcePath splits a resource by "/" and inserts each segment into the tree, // adding the binding at the leaf node. func insertResourcePath(serviceNode *TreeNode, resource string, binding PolicyBinding) { if resource == "*" { child := serviceNode.AddChild("*") child.AddBinding(binding) return } segments := strings.Split(resource, "/") curr := serviceNode for i, seg := range segments { if seg == "" { continue } curr = curr.AddChild(seg) if i == len(segments)-1 { curr.AddBinding(binding) } } } // // 3) BUILDING THE TREE // // buildTreeFromAllServices populates a tree from the JSON. func buildTreeFromAllServices(allSvcs AllServices) (*TreeNode, []string) { root := NewTreeNode("ROOT", nil) var mismatches []string for policyOrRoleName, perms := range allSvcs { for key, resources := range perms { extractedSvc := extractServiceName(key) for _, arnOrRes := range resources { svcFromArn, resourcePart := parseArnForServiceAndResource(arnOrRes) finalSvc := extractedSvc if svcFromArn != "" && servicesOfInterest[svcFromArn] { if extractedSvc != svcFromArn { msg := fmt.Sprintf( "Service mismatch for policy=%s key=%s => extracted=%q, arnService=%q (using arnService)", policyOrRoleName, key, extractedSvc, svcFromArn, ) mismatches = append(mismatches, msg) finalSvc = svcFromArn } } if !servicesOfInterest[finalSvc] { continue } binding := PolicyBinding{ PolicyOrRoleName: policyOrRoleName, Key: key, Effect: "Allow", Conditions: map[string]string{"ExampleConditionKey": "ExampleValue"}, } svcNode := root.AddChild(finalSvc) insertResourcePath(svcNode, resourcePart, binding) } } } return root, mismatches } // // 4) MATCHING / LOOKUPS // // matchesSegment checks if a segment matches a node name (handling wildcards). func matchesSegment(childName, seg string) bool { if childName == "*" { return true } if strings.HasSuffix(childName, "*") { prefix := childName[:len(childName)-1] return strings.HasPrefix(seg, prefix) } return childName == seg } // FindMatches recursively finds nodes matching the given path segments. func FindMatches(node *TreeNode, pathSegments []string) []*TreeNode { if len(pathSegments) == 0 { return []*TreeNode{node} } var result []*TreeNode nextSeg := pathSegments[0] remainder := pathSegments[1:] for childName, childNode := range node.GetChildren() { if matchesSegment(childName, nextSeg) { result = append(result, FindMatches(childNode, remainder)...) } } return result } // // 5) GATHERING & PRINTING THE "SERVICE => RESOURCE" MAPPING // func gatherServiceResourceMapping(root *TreeNode) map[string][]string { out := make(map[string][]string) for svcName, svcNode := range root.GetChildren() { if svcName == "ROOT" { continue } var paths []string gatherDfs(svcNode, "", &paths) out[svcName] = paths } return out } func gatherDfs(node *TreeNode, prefix string, collector *[]string) { newPrefix := node.Name if prefix != "" && prefix != "ROOT" { newPrefix = prefix + "/" + node.Name } children := node.GetChildren() if len(children) == 0 { *collector = append(*collector, newPrefix) return } for _, child := range children { gatherDfs(child, newPrefix, collector) } } // // 6) OPTIONAL CSV OUTPUT // type CsvRow struct { Service string ResourcePath string Bindings string // flattened bindings } func exportCsv(root *TreeNode, csvFilePath string) error { var rows []CsvRow for svcName, svcNode := range root.GetChildren() { if svcName == "ROOT" { continue } collectLeafCsvRows(svcNode, svcName, "", &rows) } f, err := os.Create(csvFilePath) if err != nil { return err } defer f.Close() writer := csv.NewWriter(f) defer writer.Flush() if err := writer.Write([]string{"Service", "ResourcePath", "Bindings"}); err != nil { return err } for _, row := range rows { if err := writer.Write([]string{row.Service, row.ResourcePath, row.Bindings}); err != nil { return err } } return nil } func collectLeafCsvRows(node *TreeNode, svcName, prefix string, collector *[]CsvRow) { newPrefix := node.Name if prefix != "" && prefix != "ROOT" { newPrefix = prefix + "/" + node.Name } children := node.GetChildren() if len(children) == 0 { var bindingStrs []string for _, b := range node.Bindings { bindingStrs = append(bindingStrs, fmt.Sprintf("%s:%s(%s)", b.PolicyOrRoleName, b.Key, b.Effect), ) } row := CsvRow{ Service: svcName, ResourcePath: newPrefix, Bindings: strings.Join(bindingStrs, "; "), } *collector = append(*collector, row) return } for _, child := range children { collectLeafCsvRows(child, svcName, newPrefix, collector) } } // // 7) MAIN // func main() { if len(os.Args) < 2 { fmt.Println("Usage: go run main.go /path/to/all-services.json [csvOutputPath]") os.Exit(1) } jsonPath := os.Args[1] var csvPath string if len(os.Args) > 2 { csvPath = os.Args[2] } // Read JSON raw, err := ioutil.ReadFile(jsonPath) if err != nil { log.Fatalf("Failed to read JSON file: %v", err) } // Parse into AllServices var allSvcs AllServices if err := json.Unmarshal(raw, &allSvcs); err != nil { log.Fatalf("Failed to unmarshal JSON: %v", err) } // Build tree & capture any service mismatches root, mismatches := buildTreeFromAllServices(allSvcs) for _, mm := range mismatches { log.Printf("MISMATCH: %s", mm) } // Gather final mapping and write JSON mapping serviceMapping := gatherServiceResourceMapping(root) outFile := "service-resource-mapping.json" data, err := json.MarshalIndent(serviceMapping, "", " ") if err != nil { log.Fatalf("Failed to marshal final mapping: %v", err) } if err := ioutil.WriteFile(outFile, data, 0644); err != nil { log.Fatalf("Failed to write output file: %v", err) } log.Printf("Service resource mapping written to %s", outFile) // Optional CSV export if requested if csvPath != "" { if err := exportCsv(root, csvPath); err != nil { log.Fatalf("Failed to export CSV: %v", err) } log.Printf("CSV exported to %s", csvPath) } // Prepare to capture lookup results in a string var sb strings.Builder // --- S3 Lookup Demo --- exampleResourcePath := "dp-batchplatform-data-prod/bp" s3Node, ok := root.GetChildren()["s3"] if !ok { sb.WriteString("No 's3' node found in the tree; skipping S3 demo lookup.\n") } else { segs := strings.Split(exampleResourcePath, "/") matches := FindMatches(s3Node, segs) sb.WriteString(fmt.Sprintf("\nMatches for S3 path %q:\n", exampleResourcePath)) if len(matches) == 0 { sb.WriteString(" None\n") } else { for _, node := range matches { path := getFullPath(node) sb.WriteString(fmt.Sprintf(" Path = %q, Bindings = %d\n", path, len(node.Bindings))) for _, b := range node.Bindings { sb.WriteString(fmt.Sprintf(" => Policy=%s, Key=%s, Effect=%s, Conditions=%v\n", b.PolicyOrRoleName, b.Key, b.Effect, b.Conditions)) } } } } // --- SQS Lookup Demo --- exampleSqsPath := "mentos-waitlist-consumer" sqsNode, ok := root.GetChildren()["sqs"] if !ok { sb.WriteString("No 'sqs' node found in the tree; skipping SQS demo lookup.\n") } else { sqsSegments := strings.Split(exampleSqsPath, "/") sqsMatches := FindMatches(sqsNode, sqsSegments) sb.WriteString(fmt.Sprintf("\nMatches for SQS path %q:\n", exampleSqsPath)) if len(sqsMatches) == 0 { sb.WriteString(" None\n") } else { for _, node := range sqsMatches { path := getFullPath(node) sb.WriteString(fmt.Sprintf(" Matched node path = %q, Bindings = %d\n", path, len(node.Bindings))) for _, b := range node.Bindings { sb.WriteString(fmt.Sprintf(" => Policy=%s, Key=%s, Effect=%s, Conditions=%v\n", b.PolicyOrRoleName, b.Key, b.Effect, b.Conditions)) } } } } // Write the lookup results to a separate file. lookupFile := "lookup-results.txt" if err := ioutil.WriteFile(lookupFile, []byte(sb.String()), 0644); err != nil { log.Fatalf("Failed to write lookup results to file: %v", err) } log.Printf("Lookup results written to %s", lookupFile) }

Explanation

a. Printing Policy Bindings at Leaf Nodes
• In the lookup sections (for S3 and SQS), after finding matching nodes via FindMatches, we loop over node.Bindings and print details (policy, key, effect, conditions).
• You can also call a helper (like a dedicated function) if you want to print the bindings in multiple contexts.

b. SQS Example
• We added a demo lookup for SQS. The code looks for the "sqs" node under root and then splits an example SQS resource path (here "mentos-waitlist-consumer") into segments.
• It then calls FindMatches(sqsNode, sqsSegments) and prints the matched nodes along with their full paths and binding details.

c. Writing the Results to a Separate File
• We use a strings.Builder (sb) to accumulate all the lookup output (for both S3 and SQS lookups).
• Finally, we write the contents of the builder to a file called "lookup-results.txt". You can change this name as desired.

This makes your code “entendable” (readable and maintainable) and covers your edge cases, including printing of policy bindings and separate output for lookup results.

Share this Q&A