This month, we greatly expanded cost reporting and telemetry functionality with the addition of GCP cost reporting and a full telemetry story with logs, traces, and metrics able to ship to Better Stack, Uptrace, or Datadog on AWS and GCP. We also redesigned the Environment Variables screen and added GPU node pools on GKE. See more features and modules below.
# Product
- GCP Cost Reporting
- Environment Variables Redesign
- IaC Sync on Tags and Releases
- Invoke Lambda Functions from the CLI
- Cloud Composer Apps
- Clearer `nullstone iac test` Diffs
- Integrations Docs
## GCP Cost Reporting
Nullstone has GCP Cost Reporting now. A GCP organization sees the same cost dashboard an AWS organization has had, sourced from BigQuery billing export. [Connect cost reporting to GCP](https://docs.nullstone.io/integrations/cloud-providers/gcp.html#cost-provider)
The provider model mirrors AWS: one cost provider per Cloud Billing account, covering every project that bills to it.
Reseller subaccounts roll up into the parent account, and an org with multiple billing accounts gets one provider each — projects link to exactly one account, so nothing double-counts.
What's new:
- **GCP cost provider** — connect a billing account's BigQuery export and Nullstone reconciles the project list automatically
- **Reworked "Add Cost Provider"** — configure a brand-new cost provider inline instead of only picking from existing providers. This matters on GCP, where billing typically lives in a standalone project that isn't one of your infra providers
- **In-product setup instructions** for both clouds, with links out to the full docs when an access test fails
- **Setup docs for AWS and GCP** — AWS cost reporting had shipped without docs; both clouds are now documented end to end
One caveat worth knowing up front on GCP: BigQuery billing export backfills at most a month, so cost history starts near zero and fills in from there.
## Environment Variables Redesign
The Environment Variables screen was rebuilt around a single principle: **rows never move**.
Every change lives on the variable's own row, staged in place until you apply it.
What's new:
- Variables group by shared prefix (`NULLSTONE_`, `OTEL_`) into collapsible sections with counts and pending-change badges
- Updated rows show `old → new` inline with the old value struck through, and a **Revert** action
- Removed rows dim red in place with **Undo** — the row doesn't disappear out from under you
- A fixed-width action slot means remove, undo, and revert never shift the value column
- Secrets are masked with a lock icon; Nullstone-injected variables (`NULLSTONE_STACK`, `NULLSTONE_COMMIT_SHA`, and friends) are marked read-only rather than silently uneditable
- An **N pending changes** chip in the header, with **Apply changes** / **Discard** for the whole staged set
## IaC Sync on Tags and Releases
You can now configure GitOps to trigger IaC sync on three kinds of events: a **push to a branch**, a **tag creation**, or a **release creation**.
Promoting infrastructure config can follow the same versioning flow as your application releases.
Tag triggers accept an optional filter using GitHub Actions' `on.push.tags` syntax, so you can configure a sync on `v*` without firing on every tag you push.
## Invoke Lambda Functions from the CLI
`nullstone run` supports AWS Lambda, joining ECS/Fargate tasks, Kubernetes jobs, Cloud Run jobs, ACA, and VMs:
```
nullstone run --app=<lambda-app> --env=<env> --payload='{"key":"value"}'
```
Lambda has no command and no per-invocation env vars, so the CLI adapts rather than pretending otherwise:
- `--payload` takes inline JSON, `@path/to/file.json`, or `-` for stdin
- Trailing `-- command args` and `--env-var` fail fast with an explanation, instead of being silently ignored
- Logs for exactly that invocation stream back to stdout, and the response payload prints on completion
- A `FunctionError` exits non-zero with the error payload — so a failed invocation fails your CI step
- Retries are disabled, because a retried `RequestResponse` invoke means running a possibly non-idempotent job twice
Works for both `zip` and `container` packaging.
## Cloud Composer Apps
Nullstone now supports GCP Cloud Composer (Airflow) as a first-class app pattern.
`nullstone push` syncs DAG files from your source directory to the Composer-managed GCS bucket, `nullstone deploy` updates the environment's version and commit metadata, and logs stream through the standard log surface — the same workflow you already use for services and jobs.
## Clearer `nullstone iac test` Diffs
Three improvements to diff rendering, aimed at CI output you can actually read:
- **No more phantom float-vs-int changes.** Effective config decoded from the API carried `float64` where your IaC carried `int`, so identical configs rendered as diffs full of `%!s(float64=60) => %!s(int=60)`. Numeric comparison is now value-based
- **Capability diffs are expanded.** Previously the output told you *something* in a capability changed; it now shows the module, version, vars, connections, and namespace changes
- **Terraform-style structured diffs.** Map and list values expand to per-item changes instead of dumping the whole object on both sides of an arrow
## Integrations Docs
Docs gained a top-level **Integrations** section at `/integrations/`, organized by category — cloud providers, source control, CI/CD, telemetry, and notifications — so each grows independently as we add third parties.
The cloud provider connection guides were rewritten with one section per auth method, matching what the UI actually offers:
- **AWS** — the automated CloudFormation setup flow is now documented (it was the UI default and undocumented), alongside real Access Keys instructions in place of the old "coming soon" placeholder
- **GCP** — Cloud Shell only, and a proper Service Account Impersonation section. The UI had been linking to that anchor for a while; the page it pointed at didn't exist
- **Azure** — a new page covering Workload Identity Federation, fixing another in-app link that led nowhere
Old URLs redirect to the new ones.
- [Integrations overview](https://docs.nullstone.io/integrations/overview.html)
- [Connect to AWS](https://docs.nullstone.io/integrations/cloud-providers/aws.html)
- [Connect to GCP](https://docs.nullstone.io/integrations/cloud-providers/gcp.html)
- [Connect to Azure](https://docs.nullstone.io/integrations/cloud-providers/azure.html)
# Catalog
### Telemetry Destinations: Better Stack, Uptrace, and Datadog
Three complete module families landed this month, each delivering **logs, traces, and metrics** to a third-party observability vendor on both AWS and GCP.
They share one shape: a datastore block holds the credentials in your cloud's secret manager, and you pick a delivery path.
- **Collector extender** — drop the extender alongside an existing OTel collector and it contributes an exporter plus per-signal pipelines. Your existing destinations (X-Ray, CloudWatch, Cloud Trace) keep receiving everything; the vendor gets a copy. No collector forks
- **Direct from the app** — a per-app capability that emits standard OTel env vars, no collector required
- **ECS/Fargate sidecar** (Datadog only) — an OTLP-receiving sidecar for apps that aren't on Kubernetes, running non-essential so a crashed collector degrades telemetry instead of taking down the task
Signal selection is a variable throughout, so you can send traces to one vendor and logs to another.
**Better Stack:**
- [aws-betterstack](https://app.nullstone.io/registry/modules/nullstone/aws-betterstack) / [gcp-betterstack](https://app.nullstone.io/registry/modules/nullstone/gcp-betterstack) — datastore holding the ingesting host and source token
- [k8s-otel-betterstack-extender](https://app.nullstone.io/registry/modules/nullstone/k8s-otel-betterstack-extender) — collector extender, works against both EKS and GKE
- [aws-otel-direct-betterstack](https://app.nullstone.io/registry/modules/nullstone/aws-otel-direct-betterstack) / [gcp-otel-direct-betterstack](https://app.nullstone.io/registry/modules/nullstone/gcp-otel-direct-betterstack) — direct-from-app capability
**Uptrace:**
- [aws-uptrace](https://app.nullstone.io/registry/modules/nullstone/aws-uptrace) / [gcp-uptrace](https://app.nullstone.io/registry/modules/nullstone/gcp-uptrace) — datastore holding the DSN and OTLP endpoint
- [k8s-otel-uptrace-extender](https://app.nullstone.io/registry/modules/nullstone/k8s-otel-uptrace-extender) — collector extender
- [aws-otel-direct-uptrace](https://app.nullstone.io/registry/modules/nullstone/aws-otel-direct-uptrace) / [gcp-otel-direct-uptrace](https://app.nullstone.io/registry/modules/nullstone/gcp-otel-direct-uptrace) — direct-from-app capability, with Uptrace's delta temporality and exponential histogram defaults
**Datadog** — every Datadog module now lives in one repo, and a single tag publishes the family:
- [aws-datadog](https://app.nullstone.io/registry/modules/nullstone/aws-datadog) / [gcp-datadog](https://app.nullstone.io/registry/modules/nullstone/gcp-datadog) — datastore holding the API and app keys
- [aws-k8s-otel-datadog-extender](https://app.nullstone.io/registry/modules/nullstone/aws-k8s-otel-datadog-extender) / [gcp-k8s-otel-datadog-extender](https://app.nullstone.io/registry/modules/nullstone/gcp-k8s-otel-datadog-extender) — collector extenders
- [aws-ecs-otel-datadog-agent](https://app.nullstone.io/registry/modules/nullstone/aws-ecs-otel-datadog-agent) — the ECS/Fargate OTLP sidecar
- [aws-otel-direct-datadog](https://app.nullstone.io/registry/modules/nullstone/aws-otel-direct-datadog) / [gcp-otel-direct-datadog](https://app.nullstone.io/registry/modules/nullstone/gcp-otel-direct-datadog) — direct-from-app capability
- [aws-ecs-datadog](https://app.nullstone.io/registry/modules/nullstone/aws-ecs-datadog) — the existing Firehose log and metric path
Datadog users should note a breaking change: `aws-ecs-datadog` is now logs-and-metrics only.
Its OTLP receiver moved into the new `aws-ecs-otel-datadog-agent` capability — attach both to keep traces and custom metrics flowing.
### OpenTelemetry Collectors for EKS and GKE
EKS now has an OTel collector module to match GKE's, built on the AWS Distro for OpenTelemetry operator.
Apps point their OTLP exporters at the in-cluster endpoint and the collector relays traces to X-Ray, metrics to CloudWatch (EMF), and logs to CloudWatch Logs — with the same optional extender connection the GKE module has, so the telemetry modules above plug straight in.
Outputs match the GKE collector exactly, making it a drop-in on either cloud.
On GKE, `gcp-gke` now enables **managed OpenTelemetry** through the provider instead of requiring a manual `gcloud container clusters update --enable-managed-otel` step. Requires GKE 1.34.1-gke.2178000 or later.
- [GKE Cluster](https://app.nullstone.io/registry/modules/nullstone/gcp-gke)
### GPU Workloads on GKE
GKE clusters can now run GPU workloads, with time-slicing so several models share one physical GPU.
- **`gcp-gke-gpu-node-pool`** — a standalone GPU node pool with time-sharing config, Hyperdisk defaults, explicit zones, and spot support
- **`gcp-gke-gpu-cores`** — a capability that requests GPU slots, and handles the node selector, taint toleration, and topology spread so replicas land on different nodes
- **`gcp-gke-service`** gained pod disruption budgets, configurable termination grace, and rollout strategy controls — so a `max_surge = 0` image roll against a full GPU node completes with zero failed requests
`gcp-gke` also learned to run with both blue and green node pools disabled, which is what makes an all-GPU cluster possible.
- [GKE GPU Node Pool](https://app.nullstone.io/registry/modules/nullstone/gcp-gke-gpu-node-pool)
- [GPU Cores](https://app.nullstone.io/registry/modules/nullstone/gcp-gke-gpu-cores)
### Private Service Connect
Two new modules publish and consume internal endpoints across GCP projects and networks, one-way and allowlisted.
- **`gcp-gke-psc-gateway`** — publishes an internal load balancer from your GKE cluster as a PSC service attachment, with manual connection acceptance driven by an explicit consumer project allowlist. Paired with a route capability that exposes each app at `<app>.<internal-domain>`
- **`gcp-psc-endpoint`** — creates the consumer-side endpoint in another VPC and registers a friendly private DNS name for it. Cross-region access is a single variable, and `psc_connection_status` surfaces `ACCEPTED` / `PENDING` / `CLOSED` so a missing allowlist entry is obvious rather than mysterious
- [PSC Gateway](https://app.nullstone.io/registry/modules/nullstone/gcp-gke-psc-gateway)
- [PSC Endpoint](https://app.nullstone.io/registry/modules/nullstone/gcp-psc-endpoint)
### ECR Lifecycle Policies
ECR and Inspector scan every image a repository holds, so stale images inflate vulnerability counts — the same CVE reported across dozens of dead tags.
The new **ECR Lifecycle** capability attaches a lifecycle policy to the repository behind any container-image app, on ECS, Fargate, EKS, or container-image Lambda.
Rules are deliberately simplified — `keep_last` and `expire_after_days`, no raw ECR policy JSON, no user-facing rule priorities:
```hcl
rules = [
{ type = "keep_last", value = 10, tag_status = "any" },
{ type = "expire_after_days", value = 14, tag_status = "untagged" },
]
```
Keep `keep_last` comfortably above your deploy-plus-rollback window: it counts by push recency, not by what's in use, and expiring a digest a live workload still references breaks the next pull.
- [ECR Lifecycle](https://app.nullstone.io/registry/modules/nullstone/aws-ecr-lifecycle)
### EKS and GKE Module Parity
`aws-eks-app` and `gcp-gke-service` had each grown features the other lacked. This month they traded.
EKS gained from GKE:
- `rolling_update_strategy`, replacing a hardcoded strategy
- `termination_grace_seconds` with a `preStop` hook
- PodDisruptionBudgets when `replicas >= 2`
- Capability outputs for `resource_limits`, `node_selectors`, `tolerations`, and `topology_spread_constraints`
GKE gained from EKS:
- `max_cpu` / `max_memory`, splitting resource limits from requests
- Structured env var references — `{{ k8s.field(...) }}`, `{{ k8s.configMap(...) }}`, `{{ k8s.resourceField(...) }}`, `{{ k8s.fileKey(...) }}` — previously GKE handled only plain values and secrets. Also added to `gcp-gke-job` and `aws-eks-job`, where the refs were declared but silently dropped
- `private_hosts` / `public_hosts` outputs
**Two behavior changes on next apply.**
EKS apps move from a 25%/25% rolling update to `max_surge = 1` / `max_unavailable = 0` — rollouts get slower at high replica counts but no longer dip below full serving capacity.
GKE apps become burstable: `resources.limits` previously always equalled `resources.requests`, so leaving `max_cpu` / `max_memory` unset now means no limit at all, and workloads stop being CPU-throttled or OOMKilled at their request values.
- [EKS App](https://app.nullstone.io/registry/modules/nullstone/aws-eks-app)
- [GKE Service](https://app.nullstone.io/registry/modules/nullstone/gcp-gke-service)
### Scoped db-admin IAM Grants
`gcp-pg-db-admin` — which every GCP Postgres datastore consumes — granted its invoker service account `roles/run.invoker` at **project** scope.
Because one invoker account is created per datastore, that meant any datastore's invoker could reach any other datastore's db-admin function, and every other Cloud Run service in the project besides.
The endpoint was never anonymous; the problem was the breadth of what a valid token could reach.
The binding is now scoped to the function's own Cloud Run service, and the builder's project-wide `storage.objectViewer` grant is scoped to the binaries bucket.
Upgrading is safe — every consumer reads the invoker from its own connected datastore — though a concurrent access-capability apply may see a transient, retriable permission failure.
`mysql-db-admin` carries the same pattern and is tracked for the same fix.
### Datastore Retention and Lifecycle Controls
- Configurable log retention on the Postgres and MySQL modules for both AWS and GCP
- Lifecycle policy support on `gcp-gcs-bucket`
- [GCS Bucket](https://app.nullstone.io/registry/modules/nullstone/gcp-gcs-bucket)
