Security Automation with Hashicorp Vault

Our DevOps resources impact

• Created single source of truth secret store for multiple customer tenants avoiding vendor lock-in

• Fully automated with gitops declarative management

• Designed access and usage patterns to fit in with existing use cases and expand them.

• Lowered development cost of new solutions by providing a managed platform for secrets storage and security automation.

The Challenge

Mobile and fixed network operators in Europe, managing thousands of projects in Google Cloud has multiple independent divisions working on cloud solutions. Each of those units heavily used the Hashicorp Vault solution to store and manage their environment secrets. They needed to unify, simplify and improve the security of the solution while lowering the maintenance and operations costs.

The Implementation

Part of this solution was open sourced by the customer and is available here:

https://github.com/Vodafone/pulumi-vault-isolated-tenants

The Solution

Solid Potential DevOps Engineers reviewed existing usage patterns and developed new solution architecture for Hashicorp Vault leveraging Kubernetes for a host environment to automate deployment and operation, improve scaling capability and provide world class containerised management for the secret engine.

Deployed solution is encrypted at rest allows seamless scaling of workloads and enabled automated backups and disaster recovery to be built on top.

Customer work environment required secrets to be managed for different department units in a unified way and allow tenants to use their own secret namespaces while using off the shelf open-source components.

We approached this challenge building multi-tenancy configuration management by leveraging pulumi to orchestrate Hashicorp Vault API. We designed yaml based tenant definition configuration that allowed easy expansion and reach feature set providing multiple supported integration patterns of the target solution.

Provided solution allowed us to offer whitelist-based access scheme for Vault secret spaces while keeping wide range of backends available to be integrated with existing solutions. Some of the more interesting integrations that was made possible for us to build on top of this flexible orchestration was automatic token rotation, short lived access tokens and - integrating with Google Cloud service account identity management - zero-knowledge secret injection for CI jobs and kubernetes workloads.

To address low maintenance requirement we have embedded the Vault configuration management solution into customer CD systems. We have built fully automated gitops solution for Vault tenants configuration and thanks to the 2 layer access controls managed by the group access we avoided additional Vault administrative burden separating configuration and secret lifecycle management, while keeping customers in control of their secret lifecycle.

The solution was consulted with Hashicorp Solution Architects receiving stellar reviews and the rollout was so well received that the Configuration management part of it was released as company contribution to open source community.