Skip to main content

Advanced Data Expressions

Project Drift provides a powerful expression language to help you inject and manipulate data within your test cases. This allows you to create dynamic tests that avoid data collisions without writing complex setup scripts.

1. Basic Data Injection​

You can inject values from a registered dataset using the ${sourceName:path.to.data} syntax.

# yaml
getProductByID:
  target: product-oas:getProductByID
  dataset: "product"
  parameters:
    path:
      id: ${product:products.product10.id} # Injects the value '10' from the dataset

2. Using Built-in Expressions​

Drift includes built-in functions to help you generate values that are guaranteed to interact with your dataset correctly.

The notIn Expression​

The notIn expression is used to generate a value that is explicitly not present in a specific dataset path. This is ideal for testing 404 Not Found scenarios.

# yaml
getProductByID_DoesNotExist:
  target: product-oas:getProductByID
  dataset: "product"
  parameters:
    path:
      # Generates an ID that does NOT exist in the products list
      id: ${ product:notIn(products.*.id) } 
  expected:
    response:
      statusCode: 404
  • products.*.id: The wildcard * tells Drift to look at the id field across all objects in the products collection.

3. Expression Types​

Drift supports three types of expressions that can be embedded in your test files:

Dataset Expressions​

Reference data from loaded datasets:

${dataset-name:path.to.value}

Environment Variable Expressions​

Access environment variables for configuration and secrets:

${env:VARIABLE_NAME}

Lua Function Expressions​

Call exported Lua functions for dynamic values:

${functions:function_name}

4. Using Environment Variable Expressions​

Environment variable expressions allow you to externalize configuration and keep secrets out of your test files.

Basic Environment Variable Usage​

# yaml-language-server: $schema=https://download.pactflow.io/drift/schemas/drift.testcases.v1.schema.json
drift-testcase-file: v1
title: "Product API Tests"

sources:
  - name: product-oas
    path: openapi.yaml
  - name: product-data
    # Load different datasets based on environment
    path: ${env:ENVIRONMENT}-dataset.yaml
  - name: functions
    path: product.lua

global:
  auth:
    apply: true
    parameters:
      authentication:
        scheme: bearer
        # Load token from environment variable
        token: ${env:API_TOKEN}

Common Use Cases for Environment Variables​

1. Multi-Environment Testing

sources:
  - name: api-spec
    # Dynamically load environment-specific spec
    uri: https://${env:ENVIRONMENT}.api.example.com/openapi.yaml
  - name: test-data
    # Load environment-specific dataset
    path: ${env:ENVIRONMENT}-dataset.yaml

2. Secret Management

global:
  auth:
    apply: true
    parameters:
      authentication:
        scheme: bearer
        token: ${env:CI_API_TOKEN}

sources:
  - name: remote-spec
    uri: https://internal.api.example.com/spec.yaml
    auth:
      username: ${env:SPEC_USERNAME}
      secret: ${env:SPEC_PASSWORD}

3. CI/CD Integration

sources:
  - name: api-spec
    # Use build-specific version
    path: openapi-${env:BUILD_NUMBER}.yaml

5. Using Lua Function Expressions​

Call exported Lua functions to generate dynamic values at runtime.

Basic Function Call​

drift.yaml:

operations:
  createProduct:
    target: api-spec:createProduct
    parameters:
      headers:
        x-request-id: ${functions:generate_request_id}
      request:
        body:
          name: ${functions:random_product_name}

product.lua:

local function generate_request_id()
  return string.format("req-%d", os.time())
end

local function random_product_name()
  return "Product-" .. os.time()
end

local exports = {
  exported_functions = {
    generate_request_id = generate_request_id,
    random_product_name = random_product_name
  }
}

return exports

Dynamic Source Names​

You can even use Lua functions to determine source names:

sources:
  - name: functions
    path: config.lua
  - name: ${functions:get_env_prefix}-oas
    path: openapi.yaml

config.lua:

local function get_env_prefix()
  return os.getenv("ENVIRONMENT") or "dev"
end

local exports = {
  exported_functions = {
    get_env_prefix = get_env_prefix
  }
}

return exports

6. Where Expressions Can Be Used​

Expressions can be embedded in most simple string values throughout your Drift files, but with important limitations based on execution order.

✅ Supported Locations​

Expressions work in:

  • Source paths and URIs (with ordering rules - see below)
  • Source names (with ordering rules)
  • Source authentication credentials
  • Global parameters, expected values, and ignore rules
  • Operation descriptions
  • Operation dataset references
  • Operation parameters (path, query, headers, body values)
  • Operation expected values
  • Operation ignore rules
  • Plugin configurations

❌ Unsupported Locations​

Expressions cannot be used in:

  • File header: drift-testcase-file: v1
  • Operation keys: The operation names in the operations map
  • Operation tags: The tags field
  • Global apply field: The apply: true/false boolean

Example: Valid Expression Usage​

# yaml-language-server: $schema=https://download.pactflow.io/drift/schemas/drift.testcases.v1.schema.json
drift-testcase-file: v1  # ❌ No expressions allowed here
title: "${env:TEAM} API Tests"  # ✅ Allowed

sources:
  - name: ${env:ENVIRONMENT}-oas  # ✅ Allowed
    path: specs/${env:ENVIRONMENT}/openapi.yaml  # ✅ Allowed
    uri: https://${env:API_HOST}/spec.yaml  # ✅ Allowed
    auth:
      username: ${env:SPEC_USER}  # ✅ Allowed
      secret: ${env:SPEC_PASS}  # ✅ Allowed

global:
  auth:
    apply: true  # ❌ No expressions allowed here
    parameters:
      authentication:
        scheme: bearer
        token: ${env:API_TOKEN}  # ✅ Allowed

operations:
  getProduct_Success:  # ❌ No expressions in operation keys
    description: "Test product ${env:PRODUCT_ID}"  # ✅ Allowed
    dataset: ${env:DATASET_NAME}  # ✅ Allowed
    parameters:
      path:
        id: ${env:TEST_PRODUCT_ID}  # ✅ Allowed

7. Execution Order and Dependencies​

Drift resolves expressions in a specific order. Understanding this order is critical for avoiding errors.

Resolution Order​

  1. Sources section - Sources are loaded in the order they appear
  2. Operation description - Resolved first (for error messages)
  3. Operation dataset - Resolved second
  4. Operation parameters and expected values - Resolved third

Dependency Rules​

Rule 1: Can't reference later fields in earlier fields

operations:
  createProduct:
    # ❌ WRONG: Can't use dataset in description
    description: "Create ${product:products.product10.name}"
    dataset: product  # Dataset is resolved AFTER description
    
    # ❌ WRONG: Can't use parameters in dataset
    dataset: ${parameters:dataset_name}  # Parameters haven't been resolved yet

Rule 2: Environment variables and Lua functions work everywhere

operations:
  createProduct:
    # ✅ CORRECT: env and functions work in all fields
    description: "Create product in ${env:ENVIRONMENT}"
    dataset: ${env:DATASET_NAME}
    parameters:
      path:
        id: ${functions:generate_id}

Rule 3: Lua files must be loaded before use

# ❌ WRONG: Using functions before loading the Lua file
sources:
  - name: ${functions:name}-oas  # Error: functions not loaded yet!
    path: openapi.yaml
  - name: functions
    path: product.lua

# ✅ CORRECT: Load Lua file first
sources:
  - name: functions
    path: product.lua
  - name: ${functions:name}-oas  # Now functions are available
    path: openapi.yaml

8. Practical Use Cases​

Use Case 1: Environment-Specific Testing​

Run the same tests against multiple environments:

drift.yaml:

# yaml-language-server: $schema=https://download.pactflow.io/drift/schemas/drift.testcases.v1.schema.json
drift-testcase-file: v1
title: "API Tests - ${env:ENVIRONMENT}"

sources:
  - name: api-spec
    uri: https://${env:ENVIRONMENT}.api.example.com/openapi.yaml
  - name: test-data
    path: data/${env:ENVIRONMENT}.dataset.yaml
  - name: functions
    path: ${env:ENVIRONMENT}.lua

operations:
  getProduct:
    target: api-spec:getProduct
    dataset: test-data
    parameters:
      path:
        id: ${test-data:testProducts.validProduct.id}

Run against different environments:

# Development
ENVIRONMENT=dev drift verify --test-files drift.yaml

# Staging
ENVIRONMENT=staging drift verify --test-files drift.yaml

# Production
ENVIRONMENT=prod drift verify --test-files drift.yaml

Use Case 2: Secure CI/CD Integration​

Keep secrets out of source control:

drift.yaml:

# yaml-language-server: $schema=https://download.pactflow.io/drift/schemas/drift.testcases.v1.schema.json
drift-testcase-file: v1

sources:
  - name: api-spec
    uri: ${env:SPEC_URL}
    auth:
      username: ${env:SPEC_USERNAME}
      secret: ${env:SPEC_PASSWORD}

global:
  auth:
    apply: true
    parameters:
      authentication:
        scheme: bearer
        token: ${env:API_TOKEN}

GitHub Actions workflow:

- name: Run Drift Tests
  env:
    SPEC_URL: ${{ secrets.SPEC_URL }}
    SPEC_USERNAME: ${{ secrets.SPEC_USERNAME }}
    SPEC_PASSWORD: ${{ secrets.SPEC_PASSWORD }}
    API_TOKEN: ${{ secrets.API_TOKEN }}
  run: drift verify --test-files drift.yaml

Use Case 3: Dynamic Test Data Generation​

Generate unique values to avoid test conflicts:

drift.yaml:

operations:
  createUser:
    target: api-spec:createUser
    parameters:
      request:
        body:
          email: ${functions:unique_email}
          username: ${functions:unique_username}
          timestamp: ${functions:current_timestamp}

functions.lua:

local function unique_email()
  return string.format("test-%d@example.com", os.time())
end

local function unique_username()
  return string.format("user_%d", os.time())
end

local function current_timestamp()
  return os.date("!%Y-%m-%dT%H:%M:%SZ")
end

local exports = {
  exported_functions = {
    unique_email = unique_email,
    unique_username = unique_username,
    current_timestamp = current_timestamp
  }
}

return exports

Use Case 4: Team-Specific Configurations​

Different teams can use the same test suite with their own configs:

drift.yaml:

# yaml-language-server: $schema=https://download.pactflow.io/drift/schemas/drift.testcases.v1.schema.json
drift-testcase-file: v1
title: "${env:TEAM_NAME} API Tests"

sources:
  - name: api-spec
    path: openapi.yaml
  - name: team-data
    path: teams/${env:TEAM_NAME}/data.yaml
  - name: functions
    path: teams/${env:TEAM_NAME}/functions.lua

Usage:

# Team A
TEAM_NAME=team-a drift verify --test-files drift.yaml

# Team B
TEAM_NAME=team-b drift verify --test-files drift.yaml

Use Case 5: Feature Flag Testing​

Test different API behaviors based on feature flags:

drift.yaml:

operations:
  getProduct_WithNewFeature:
    target: api-spec:getProduct
    parameters:
      headers:
        x-feature-flag: ${env:FEATURE_BETA_ENABLED}
      path:
        id: 123
    expected:
      response:
        statusCode: 200

9. Troubleshooting​

Error: "Cannot resolve expression"​

Problem: Expression references an undefined variable or function.

Solutions:

  • Check environment variable is set: echo $ENVIRONMENT
  • Verify Lua function is exported in the exported_functions table
  • Ensure dataset name matches exactly (case-sensitive)

Error: "Source not found" when using Lua function in source name​

Problem: Lua file loaded after the source that uses it.

Solution: Move the Lua source file above any sources that reference its functions:

# ✅ CORRECT order
sources:
  - name: functions
    path: config.lua
  - name: ${functions:get_env}-oas
    path: openapi.yaml

Error: Expression in operation key​

Problem: Trying to use expression in operation name.

# ❌ WRONG
operations:
  ${env:OPERATION_NAME}:
    target: api-spec:operation

Solution: Use static operation names, but dynamic descriptions:

# ✅ CORRECT
operations:
  standardOperation:
    description: "Testing ${env:SCENARIO}"
    target: api-spec:operation

Expression not resolving in description​

Problem: Using dataset or parameter references in description.

# ❌ WRONG: dataset not yet loaded when description is resolved
operations:
  test:
    description: "Test product ${product:name}"
    dataset: product

Solution: Use only env vars and functions in descriptions:

# ✅ CORRECT
operations:
  test:
    description: "Test product in ${env:ENVIRONMENT}"
    dataset: product

10. Best Practices​

  1. Use environment variables for secrets - Never commit tokens or passwords
  2. Prefix environment variables - Use namespacing like DRIFT_, API_, TEST_ to avoid conflicts
  3. Document required environment variables - Add a README with all required env vars
  4. Provide defaults in Lua - Lua functions can provide fallback values: 
    local function get_api_url()
      return os.getenv("API_URL") or "http://localhost:8080"
    end
  5. Load Lua files first - Always place Lua sources at the top of the sources list
  6. Use expressions sparingly - Only use them when you need dynamic behavior
  7. Test locally first - Verify expressions resolve correctly before using in CI/CD

How Expressions DRY Up Your Tests​

Expressions help you follow the DRY (Don't Repeat Yourself) principle by centralizing configuration and reusing values across multiple test operations.

Without Expressions: Repetition and Maintenance Burden​

┌─────────────────────────────────────┐
│   drift.yaml (Hard-coded values)    │
├─────────────────────────────────────┤
│                                     │
│  operations:                        │
│    test1:                           │
│      parameters:                    │
│        headers:                     │
│          authorization: "token123"  │ ← Repeated
│          x-api-version: "v2"        │ ← Repeated
│                                     │
│    test2:                           │
│      parameters:                    │
│        headers:                     │
│          authorization: "token123"  │ ← Repeated
│          x-api-version: "v2"        │ ← Repeated
│                                     │
│    test3:                           │
│      parameters:                    │
│        headers:                     │
│          authorization: "token123"  │ ← Repeated
│          x-api-version: "v2"        │ ← Repeated
└─────────────────────────────────────┘

Problems:

  • Multiple updates needed to change API token
  • Copy-paste errors
  • No environment-specific configuration
  • Secrets in source control

With Expressions: Single Source of Truth​

┌──────────────────────┐     ┌──────────────────────┐     ┌──────────────────────┐
│  Environment Vars    │     │   Lua Functions      │     │     Datasets         │
│ ──────────────────── │     │ ──────────────────── │     │ ──────────────────── │
│                      │     │                      │     │                      │
│ API_TOKEN=xyz789     │     │ function bearer()    │     │ testUsers:           │
│ ENVIRONMENT=staging  │     │   return token()     │     │   adminUser:         │
│ API_VERSION=v2       │     │ end                  │     │     id: 42           │
│                      │     │                      │     │     role: admin      │
└──────────┬───────────┘     └──────────┬───────────┘     └──────────┬───────────┘
           │                            │                            │
           │                            │                            │
           └────────────────┬───────────┴────────────────────────────┘
                            │
                            ▼
           ┌────────────────────────────────────────┐
           │         drift.yaml (DRY)               │
           ├────────────────────────────────────────┤
           │                                        │
           │ global:                                │
           │   auth:                                │
           │     apply: true                        │
           │     parameters:                        │
           │       headers:                         │
           │         authorization:                 │
           │           ${env:API_TOKEN}         ◄───┼─── ONE reference
           │         x-api-version:                 │
           │           ${env:API_VERSION}       ◄───┼─── ONE reference
           │                                        │
           │ operations:                            │
           │   getUserById:                         │
           │     parameters:                        │
           │       path:                            │
           │         id: ${testUsers:adminUser.id}◄─┼─── ONE reference
           │                                        │
           │   updateUser:                          │
           │     # Inherits global auth             │
           │     parameters:                        │
           │       path:                            │
           │         id: ${testUsers:adminUser.id}◄─┼─── Reused
           │                                        │
           │   deleteUser:                          │
           │     # Inherits global auth             │
           │     parameters:                        │
           │       path:                            │
           │         id: ${testUsers:adminUser.id}◄─┼─── Reused
           └────────────────────────────────────────┘

Benefits:

  • One place to change API token
  • Environment-specific via ${env:ENVIRONMENT}
  • Secrets stay out of source control
  • Type-safe data from datasets
  • Dynamic values from Lua functions

Real-World Impact​

Before Expressions:

operations:
  test1:
    parameters:
      headers:
        authorization: "Bearer prod-token-12345"
      path:
        userId: "100"
  test2:
    parameters:
      headers:
        authorization: "Bearer prod-token-12345"  # Copy-paste
      path:
        userId: "100"  # Hardcoded
  test3:
    parameters:
      headers:
        authorization: "Bearer prod-token-12345"  # Copy-paste
      path:
        userId: "100"  # Hardcoded

To test in staging, you'd need to update 50+ places manually.

After Expressions:

global:
  auth:
    apply: true
    parameters:
      authentication:
        scheme: bearer
        token: ${env:API_TOKEN}

operations:
  test1:
    dataset: users
    parameters:
      path:
        userId: ${users:testUser.id}
  test2:
    dataset: users
    parameters:
      path:
        userId: ${users:testUser.id}
  test3:
    dataset: users
    parameters:
      path:
        userId: ${users:testUser.id}

Change environment with ONE variable: ENVIRONMENT=staging drift verify ...