# Enterprise GCP Networking in Practice: Shared VPC for Governance, Private Service Connect for Isolation

Enterprise cloud networking can be easy to describe and difficult to implement securely, so I came up with a hands on practice for myself.

This is a banking network, built on GCP, utilizing a shared VPC network architecture while offering a private partner some direct service accessibility.

The networking team wants consistent control over address space, firewall policy, logging, and connectivity. Application teams want independent projects, clear ownership, and the freedom to deploy without waiting for a network change every time. External partners create a third requirement: they need private access to a service, but they should not gain access to the network that hosts it.

I built this reference architecture to explore how Google Cloud handles those boundaries when the design is centered on two services:

*   **Shared VPC** for centrally governed connectivity between internal projects
    
*   **Private Service Connect (PSC)** for publishing one private service to a separate consumer VPC
    

The result is a four-project, banking-style build provisioned with Terraform. It is intentionally small, with minimal application code , but through the IaC, it demonstrates patterns that matter in larger environments: subnet-level delegation, IAM identity in firewall policy, service-oriented partner access, private administration through Identity-Aware Proxy, and centralized network telemetry.

[Project Repository](https://github.com/Josephdara/bank-shared-vpc-psc)

NB: This is a reference implementation, not a production banking platform.

## The design question

![Terraform implementation of the four-project Shared VPC and Private Service Connect architecture](https://cdn.hashnode.com/uploads/covers/6a4a80484f84ec862bd5ad26/3cf9a1ec-a63b-475c-a5f9-e75041a5e0d0.png align="center")

The first question as it is with IAM is that:

> What is the smallest amount of connectivity each project actually needs?
> 
> Least privillege should apply to network access, not just identity.

Within the organization. The web and database tiers need communication, but the network team should retain ownership of the VPC, subnets, routes, firewall rules, and logging configuration. For centralized control, Shared VPC is Ideal.

VPC Network Peering is a valid tool when two networks need private IP connectivity, but the partner in this lab does not need a route to the bank's Shared VPC which is a characteristic of Peering. It needs one API. For this reason, I decided to go with Private Service Connect, it fits the partner boundary better than peering.

I think of the two products this way:

> Shared VPC lets multiple internal projects use one governed network. Private Service Connect lets another network consume one published service.

They solve different problems, and the architecture becomes much cleaner when each is used at the boundary it was designed for.

## Architecture at a glance

The lab separates responsibilities across four Google Cloud projects.

| Project | Responsibility | Main resources |
| --- | --- | --- |
| Host project | Central network and security control plane | Custom-mode `bank-shared-vpc`, `prod-subnet`, `analytics-subnet`, `psc-nat-subnet`, firewall rules, VPC Flow Logs, BigQuery dataset |
| Service project A | Retail web and private API tier | `web-app-sa`, regional managed instance group, internal passthrough Network Load Balancer, PSC service attachment |
| Service project B | Analytics and database tier | `database-sa`, private database test VM in `analytics-subnet` |
| Partner project | Standalone service consumer | `partner-vpc`, private client VM, PSC endpoint, IAP-only SSH access |

The default address plan is deliberately simple:

| Subnet | CIDR | Region | Purpose |
| --- | --- | --- | --- |
| `prod-subnet` | `10.0.1.0/24` | `us-central1` | Web and API workload interfaces |
| `analytics-subnet` | `10.0.2.0/24` | `us-east1` | Database-tier workload interfaces |
| `psc-nat-subnet` | `10.0.3.0/28` | `us-central1` | Producer-side source NAT for PSC connections |
| `partner-subnet` | `192.168.1.0/24` | `us-central1` | Partner client and consumer endpoint |

The Shared VPC is global, while its subnets are regional. This allows the web tier in `us-central1` and the analytics tier in `us-east1` to communicate over internal addresses, subject to the central firewall policy. The PSC endpoint and producer service attachment are both placed in `us-central1`, which satisfies the regional requirement for this endpoint pattern.

## Two traffic paths, two different trust models

There are two important data paths in the design.

### Internal east-west path

```text
web managed instance group
  → prod-subnet
  → Shared VPC firewall evaluation
  → analytics-subnet
  → database test VM on TCP 5432
```

Both workloads live in service projects, but their network interfaces use subnets owned by the host project. The allow rule is also owned by the host project because that project owns the Shared VPC.

### Partner-to-service path

```text
partner client VM
  → private PSC endpoint in partner-subnet
  → PSC service attachment
  → producer-side PSC NAT
  → internal passthrough Network Load Balancer
  → web managed instance group
```

The partner sends traffic to an IP address from its own subnet. It does not call the producer load balancer address directly, and it receives no general route into the Shared VPC. On the producer side, PSC source NAT translates the original consumer source to an address from `psc-nat-subnet` before the traffic reaches the service.

That path is the main architectural point of the project. The partner can reach the published service, but not the producer network as a whole.

## Security pillar 1: delegate subnets, not the entire network

Attaching a service project to a Shared VPC does not mean every principal in that project should be able to use every subnet. The `roles/compute.networkUser` role can be granted on an individual subnetwork, which gives the design a more precise control point.

In this implementation, the web workload identity receives Network User on `prod-subnet`, while the database workload identity receives it on `analytics-subnet`.

```hcl
resource "google_compute_subnetwork_iam_member" "web_prod" {
  project    = var.host_project_id
  region     = var.prod_subnet.region
  subnetwork = var.prod_subnet.name
  role       = "roles/compute.networkUser"
  member     = "serviceAccount:${google_service_account.web_app.email}"
}

resource "google_compute_subnetwork_iam_member" "db_analytics" {
  project    = var.host_project_id
  region     = var.analytics_subnet.region
  subnetwork = var.analytics_subnet.name
  role       = "roles/compute.networkUser"
  member     = "serviceAccount:${google_service_account.db.email}"
}
```

The managed instance group also needs Google Cloud's service project agent to be able to attach new instance interfaces to the production subnet. The Terraform configuration grants that service account Network User on `prod-subnet`, not across the host project.

This is an important operational detail. Least privilege has to include both the human or deployment principal and the Google-managed identities that create resources on its behalf.

Subnet-level IAM separates two kinds of authority:

*   The host project controls which subnet an identity can use.
    
*   Each service project controls the lifecycle of its own workloads.
    

This reduces the chance that a workload or automation process from one service project is placed in another team's subnet. In a larger organization, the same pattern can be applied to developer groups, deployment service accounts, or platform automation identities.

## Security pillar 2: express east-west policy with workload identity

An IP-based firewall rule answers, "Which addresses may connect?" For stable infrastructure that can be enough. In a dynamic environment, I often care more about, "Which workload is making the connection?"

The central firewall rule allows TCP `5432` only when the source VM uses `web-app-sa` and the target VM uses `database-sa`:

```hcl
resource "google_compute_firewall" "allow_web_to_db" {
  project   = var.host_project_id
  name      = "allow-web-to-db"
  network   = var.network_self_link
  direction = "INGRESS"

  allow {
    protocol = "tcp"
    ports    = [tostring(var.db_port)]
  }

  source_service_accounts = [var.web_app_sa_email]
  target_service_accounts = [var.db_sa_email]
}
```

For this rule, Google Cloud identifies the source through the service account associated with the VM that emitted traffic from its primary internal interface. The target filter limits the rule to VM interfaces whose instance uses the database service account. Shared VPC supports service accounts from its host and attached service projects as firewall criteria.

This makes the policy resilient to routine address changes. A recreated web instance can receive a different internal IP and still match the intended rule because the workload identity remains the same.

There is an equally important warning here: service account attachment becomes a security-sensitive permission. If a user can attach `web-app-sa` to an arbitrary VM, that user can create a workload that matches the database firewall rule. For that reason, `roles/iam.serviceAccountUser`, and specifically the `iam.serviceAccounts.actAs` permission it contains, must be granted narrowly.

Identity-based firewalling is still a network control. It does not replace database credentials, TLS, authorization inside the API, or application-level audit logs. It should be one layer in the design, not the only layer.

The lab also includes two infrastructure-specific allow rules:

*   Google Cloud health-check ranges can reach the web workload on the application port.
    
*   `psc-nat-subnet` can reach the web workload on the application port.
    

This is why I describe the database path as identity-based instead of claiming the entire VPC has no CIDR-based rules. Managed load balancers and PSC still need narrowly scoped infrastructure rules.

## Security pillar 3: publish the API, not the VPC

The producer side places an internal passthrough Network Load Balancer in front of the web managed instance group. A service attachment publishes that load balancer through PSC.

```hcl
resource "google_compute_service_attachment" "api" {
  project = var.service_a_project_id
  name    = "retail-api-psc"
  region  = var.region

  connection_preference = "ACCEPT_MANUAL"
  enable_proxy_protocol = false
  nat_subnets           = [var.psc_nat_subnet_self_link]
  target_service        = google_compute_forwarding_rule.ilb.self_link

  consumer_accept_lists {
    project_id_or_num = var.partner_project_id
    connection_limit  = 1
  }
}
```

I chose manual acceptance and an explicit consumer allowlist so that publishing the service does not make it available to arbitrary projects. The connection limit is set to one because the lab has one partner endpoint.

The partner creates a regional forwarding rule that targets the service attachment and assigns it a private address from `partner-subnet`. From the partner's point of view, the API is simply a private IP in its own VPC.

This has several useful properties:

*   **No VPC Peering:** There is no peering relationship between `partner-vpc` and `bank-shared-vpc`.
    
*   **No general producer route:** The consumer is given a path to the service attachment, not a path to the producer subnets.
    
*   **Explicit admission:** The producer decides which consumer project may connect.
    
*   **Port-level publication:** The forwarding rule publishes the configured application port, which is TCP `80` in the lab.
    
*   **Address-space independence:** PSC uses producer-side NAT, so producer and consumer address plans can overlap without the peering overlap constraint.
    

PSC isolation should not be confused with application authentication. A client that can reach the endpoint can send traffic to the published port. A production API should still authenticate the caller, authorize operations, encrypt traffic, validate requests, apply rate limits, and record application-level audit events.

The current lab also disables the PROXY protocol. That keeps the demonstration simple, but it means the backend sees a source address from the PSC NAT subnet rather than the partner client's original address. If the application requires source attribution, that requirement needs to be addressed deliberately at the load-balancing or application layer.

## The evidence layer: VPC Flow Logs and BigQuery

A network control is more valuable when I can verify how traffic actually moved.

Both workload subnets enable VPC Flow Logs with five-second aggregation, `0.5` flow sampling, and all available metadata. The Terraform configuration creates a partitioned BigQuery destination in the host project and a folder-level aggregated sink filtered to VPC Flow Logs.

For Shared VPC resources, the network belongs to the host project and its VPC Flow Logs are reported there. The records can still identify service-project workloads through instance metadata, which makes it possible to analyze traffic across project boundaries from a central dataset.

For example, this query looks for the web-to-database flow on TCP `5432`:

```sql
SELECT
  jsonPayload.src_instance.vm_name  AS src_vm,
  jsonPayload.dest_instance.vm_name AS dest_vm,
  jsonPayload.connection.src_ip     AS src_ip,
  jsonPayload.connection.dest_ip    AS dest_ip,
  COUNT(*) AS flow_records,
  SUM(CAST(jsonPayload.bytes_sent AS INT64)) AS total_bytes
FROM `<HOST_PROJECT>.vpc_flow_logs.compute_googleapis_com_vpc_flows`
WHERE timestamp >= TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 1 DAY)
  AND jsonPayload.reporter = 'SRC'
  AND CAST(jsonPayload.connection.protocol AS INT64) = 6
  AND CAST(jsonPayload.connection.dest_port AS INT64) = 5432
GROUP BY src_vm, dest_vm, src_ip, dest_ip
ORDER BY total_bytes DESC;
```

The repository includes additional queries for recent flows, top talkers, API ingress, and subnet-to-subnet volume. Flow Logs are sampled telemetry, so I use them for visibility and investigation rather than as the only source of security audit evidence.

## Building the environment in stages

Terraform wires the deployment into a dependency graph, but I still find it useful to think about the build in stages:

1.  **Bootstrap the resource hierarchy.** Create or select a folder, create the four projects, link billing, and create the Terraform state bucket.
    
2.  **Build the host network.** Promote the host project, create the custom-mode Shared VPC, and create the three producer subnets.
    
3.  **Attach service projects and delegate access.** Attach service projects A and B, create workload service accounts, and apply subnet-level Network User bindings.
    
4.  **Deploy workloads and central policy.** Create the web managed instance group, the database test VM, and the firewall rules in the host project.
    
5.  **Publish the private service.** Create the health check, internal load balancer, service attachment, partner VPC, partner endpoint, and private client VM.
    
6.  **Turn traffic into evidence.** Route the relevant VPC Flow Logs into BigQuery and run the positive and isolation tests.
    

That order is useful during troubleshooting. A failed endpoint test can be separated into control-plane acceptance, load-balancer health, producer firewalling, and backend application behavior instead of being treated as one opaque PSC problem.

## Deploying the lab

After installing `gcloud` and Terraform, authenticate the CLI and Application Default Credentials:

```bash
gcloud auth login
gcloud auth application-default login
```

From the repository root, prepare the bootstrap environment and create the projects:

```bash
cp .env.example .env
# Add ORG_ID or FOLDER_ID, plus BILLING_ACCOUNT_ID, to .env
./bootstrap.sh
```

Then configure and apply Terraform:

```bash
cd terraform
cp backend.hcl.example backend.hcl
cp terraform.tfvars.example terraform.tfvars
# Add the generated project IDs, folder ID, and state bucket configuration
terraform init -backend-config=backend.hcl
terraform plan
terraform apply
```

The most useful outputs are:

```bash
terraform output psc_endpoint_ip
terraform output ilb_ip
terraform output web_app_sa_email
terraform output db_sa_email
```

## Verifying what the architecture actually guarantees

I split verification into control-plane checks and data-plane tests. That prevents a successful HTTP response from being treated as proof that every isolation control is correct.

### 1\. Confirm the partner reaches the API through PSC

The partner VM has no external IP. I connect to it through IAP and call the endpoint address produced by Terraform:

```bash
gcloud compute ssh "$(terraform output -raw partner_client_vm)" \
  --project="$(terraform output -raw partner_project_id)" \
  --zone="$(terraform output -raw partner_client_zone)" \
  --tunnel-through-iap \
  --command="curl -fsS http://$(terraform output -raw psc_endpoint_ip)/"
```

Expected response:

```text
Connection successful
```

### 2\. Confirm the partner cannot bypass PSC

The test suite also checks that the partner cannot call the producer's internal load-balancer IP directly, cannot reach the database VM on TCP `5432`, and cannot use an adjacent unpublished port on the PSC endpoint.

These negative path checks matter as much as the successful request. They demonstrate that the endpoint exposes the intended service path instead of accidentally providing broader network reachability.

### 3\. Confirm the Shared VPC control plane

The tests verify that both service projects are attached to the expected host project and that the workload interfaces use subnets owned by that host. This proves the resources are not quietly using standalone VPCs in their service projects.

### 4\. Confirm the identity-based database path

The database VM is a network-policy test target. The lab does not install PostgreSQL. During verification, the test suite starts a temporary listener on TCP `5432`, confirms that the web VM has `web-app-sa`, confirms that the target VM has `database-sa`, and then tests the allowed connection.

The current suite proves the positive identity path. It does not yet provision a third VM with an unrelated service account for an automated negative identity test. Adding that fixture would complete the explicit proof that an untrusted workload identity is denied when no broader allow rule applies.

The full commands and expected results are documented in `[test-cases.md](test-cases.md)`.

## What I would add before calling this production-ready

This lab focuses on network structure, so several controls are intentionally outside its scope. For a production financial workload, I would extend it with:

*   TLS on the published service and authenticated, authorized API requests
    
*   A real database service with its own credentials, encryption, backup, and recovery controls
    
*   More than one web instance, tested autoscaling, and a documented regional or multi-region availability strategy
    
*   Private DNS for the partner endpoint instead of requiring clients to use a raw IP address
    
*   Tight `iam.serviceAccounts.actAs` permissions and a dedicated Terraform deployment identity
    
*   Organization policies that restrict Shared VPC hosts, allowed subnets, external IPs, and approved PSC producers or consumers
    
*   Egress policy, DNS policy, and a deliberate software update path for private workloads
    
*   Alerting for unhealthy load-balancer backends, rejected PSC connections, and PSC NAT subnet consumption
    
*   Log retention, access controls, cost controls, and detections built on the centralized telemetry
    
*   A dedicated unauthorized-identity test fixture so the database deny path is continuously verified
    

The `/28` PSC NAT subnet is generous for one endpoint, but NAT capacity still needs to be treated as a resource. PSC consumes NAT addresses per connected endpoint or backend, and a service attachment can fail to accept additional connections when its NAT subnets run out of usable addresses.

## Lessons I took from the build

The most useful lesson was that project separation and network separation are not the same thing.

Shared VPC gives internal teams separate projects without forcing every project to own a separate network. PSC does the reverse at the partner boundary: it keeps the networks separate while making one service privately consumable.

Three other details stood out to me:

1.  **The ability to attach a service account is part of the network security model.** An identity-based firewall rule is only as strong as the IAM policy around that identity.
    
2.  **A successful connection is not enough evidence.** I also need tests that prove the partner cannot reach the producer ILB, database, or unpublished ports.
    
3.  **Private connectivity does not equal application security.** PSC removes the need to expose the service publicly and avoids broad network sharing, but authentication, encryption, authorization, and auditing still belong in the application design.
    

For this use case, the cleanest boundary is also the easiest one to explain:

> Internal teams share a governed VPC. External consumers receive a private endpoint to one approved service.

That is the design this repository sets out to demonstrate.

## Further reading

*   [Shared VPC overview and subnet-level IAM](https://cloud.google.com/vpc/docs/shared-vpc)
    
*   [VPC firewall rules and service-account filtering](https://cloud.google.com/firewall/docs/firewalls)
    
*   [Private Service Connect published services](https://cloud.google.com/vpc/docs/about-vpc-hosted-services)
    
*   [VPC Network Peering limitations](https://cloud.google.com/vpc/docs/vpc-peering)
    
*   [VPC Flow Logs for Shared VPC](https://cloud.google.com/vpc/docs/about-traffic-flows)
    
*   [IAM roles used by this repository](https://github.com/Josephdara/bank-shared-vpc-psc/blob/main/IAM.md)