Khader Vali 
Building Feature Flag Systems for Continuous Delivery
In the fast-paced world of software development, the ability to release new features rapidly and reliably is no longer a luxury—it’s a necessity. Continuous Delivery (CD) promises exactly this: a pipeline that automates the journey of code from commit to production, ready for release at any time. Yet, a common dilemma arises: how do you deploy frequently without immediately exposing incomplete or potentially risky features to all your users?
This is where feature flag systems come into play. They are a powerful paradigm shift that decouples code deployments from feature releases, giving developers and product managers unprecedented control over who sees what, and when. As a senior engineer at Khadervali.com, I’ve seen firsthand the transformative impact of well-designed feature flag systems on team agility, product experimentation, and operational safety.
In this comprehensive guide, we’ll dive deep into the world of feature flags. We’ll explore their core concepts, design a robust architecture, walk through the steps of building your own system with practical code examples, discuss advanced strategies, and share real-world scenarios. By the end, you’ll have a solid understanding of how to leverage feature flags to supercharge your continuous delivery pipeline.
What Are Feature Flags? The Foundation of Controlled Releases
At its heart, a feature flag (also known as a feature toggle, feature switch, or conditional feature) is a technique that allows you to turn specific functionalities on or off during runtime without deploying new code. Think of it like a light switch for your features. Instead of deploying a completely new version of your application to enable a feature, you simply flip a switch in a configuration system.
This simple concept unlocks a myriad of powerful capabilities:
- Decoupling Deployment from Release: You can deploy new code containing incomplete or experimental features to production without exposing them to users. Once ready, you activate the feature through the flag.
- Gradual Rollouts: Introduce new features to a small percentage of users first, monitor their behavior and system performance, and then gradually expand the rollout. This minimizes risk.
- A/B Testing and Experimentation: Easily define different user groups (variants A and B) and expose them to different versions of a feature or UI element to gather data and make data-driven product decisions.
- Kill Switches: In case a new feature introduces a critical bug or performance degradation in production, you can immediately disable it with a flick of a switch, preventing further impact without needing a hotfix deployment.
- Dark Launches: Deploy and test backend services or infrastructure changes in production under real load, but without exposing any user-facing functionality. This helps validate stability and performance before a full user rollout.
- Personalization and Entitlements: Control access to features based on user attributes like subscription level, geographic location, or specific user IDs.
- Operational Control: Temporarily disable non-critical features during peak load or maintenance windows to preserve system stability.
Without feature flags, continuous delivery often feels like walking a tightrope without a safety net. Every deployment is a full-stakes gamble. With flags, you gain the agility to deploy fearlessly, experiment confidently, and recover instantly.
The Architecture of a Robust Feature Flag System
A well-designed feature flag system isn’t just a collection of if-statements. It’s a cohesive ecosystem of components working together to provide reliable, scalable, and secure control over your application’s behavior. Let’s break down the typical architecture.
Core Components
Imagine a central brain that manages all your feature switches, and client-side agents that listen to this brain and act accordingly. Here are the key components:
-
Flag Management UI/API
This is the control panel where product managers, developers, and QA engineers define, configure, and toggle feature flags. It’s typically a web application or a set of APIs that allows for:
- Creating, updating, and deleting flags.
- Defining flag types (boolean, string, JSON, multivariate).
- Setting targeting rules (e.g., enable for 10% of users, specific user IDs, users in a certain region).
- Viewing the current state of all flags.
- Auditing changes made to flags.
-
Flag Store (Database/Cache)
This is the persistent storage for all your feature flag configurations and their associated rules. It needs to be highly available and performant, as it will be queried frequently. Common choices include:
- Relational databases (PostgreSQL, MySQL): Good for structured data, audit trails, and complex queries.
- NoSQL databases (MongoDB, DynamoDB): Flexible schema, scalable for large volumes of flags.
- Key-Value stores (Redis, Memcached): Excellent for caching and high-speed lookups of flag states.
- Cloud storage (S3, GCS): Can store flag configurations as static files for simpler, read-heavy systems.
-
Flag Evaluation Engine (SDKs/Libraries)
This is the intelligence embedded within your applications (backend services, frontend web apps, mobile apps). Its primary responsibility is to determine the state of a given flag for a specific user or context. It:
- Fetches flag configurations from the Flag Store (or a local cache).
- Takes contextual information (user ID, subscription plan, browser type, location, etc.).
- Applies the defined targeting rules to evaluate whether the flag is “on” or “off” (or what variant to show).
- Returns the evaluated flag value to the application.
These are often provided as language-specific SDKs (e.g., Python SDK, Java SDK, JavaScript SDK).
-
Flag Distribution/Sync Mechanism
This component ensures that the Flag Evaluation Engine in your applications has the most up-to-date flag configurations. Options include:
- Polling: Applications periodically fetch flag configurations from the Flag Store or a dedicated API endpoint. Simple but can have latency.
- Webhooks/Push Notifications: When a flag is updated via the Management UI, a webhook triggers a notification to interested applications, prompting them to refresh their configurations. More real-time, but adds complexity.
- Streaming (SSE/WebSockets): Applications maintain a persistent connection to a flag service that streams updates in real-time. Highest real-time capability, but most complex.
- Sidecar/Proxy: A dedicated process running alongside your application fetches and caches flags, acting as a local source of truth for the application.
-
Telemetry and Observability
To understand the impact and usage of your flags, it’s crucial to collect data. This component involves:
- Logging flag evaluations: Which flag was evaluated, for whom, and what was the result.
- Metrics: Tracking how often flags are requested, latency of evaluation, and error rates.
- Integration with analytics platforms: Sending flag exposure data to tools like Amplitude, Mixpanel, or Google Analytics for deeper experimentation analysis.
Architectural Flow (in words)
Let’s visualize the interaction:
1. A product manager uses the Flag Management UI to create a new feature flag called “NewCheckoutFlow” and configures it to be enabled for 10% of users in the “EU” region and specifically for a list of internal beta testers (user IDs: beta_user_1, beta_user_2). This configuration is then saved to the Flag Store (e.g., a PostgreSQL database).
2. The Flag Distribution Mechanism (e.g., a background service that watches the database for changes or a webhook listener) detects this update. It then either pushes the updated configuration to a cache (like Redis) or triggers a refresh in all interested application instances.
3. A user visits your e-commerce website. Your backend service (e.g., a Python microservice) needs to know if the “NewCheckoutFlow” should be active for this user. It calls its embedded Flag Evaluation SDK: flag_sdk.is_enabled("NewCheckoutFlow", user_context).
4. The Flag Evaluation SDK first checks its local cache. If the configuration isn’t there or is stale, it fetches the latest configuration for “NewCheckoutFlow” from the fast Flag Store Cache (Redis). It receives the rules: 10% rollout for EU users, specific user IDs enabled.
5. The SDK then takes the provided user_context (which might contain user_id="customer_123", region="US", subscription_plan="premium"). It applies the rules to this context. For example, if customer_123 is not in the beta tester list and region="US", the 10% rollout rule won’t apply, and the flag might be evaluated as ‘off’. If the user was beta_user_1 or in the EU, it would be ‘on’.
6. The SDK returns ‘false’ (or ‘true’ if applicable) to your backend service.
7. Your backend service then renders the appropriate checkout experience (old or new) based on this evaluation.
8. The SDK also sends an event to the Telemetry/Observability system (e.g., a Kafka topic or directly to an analytics service) indicating that “NewCheckoutFlow” was evaluated as ‘off’ for customer_123. This data is crucial for monitoring and A/B test analysis.
Building Your Own Feature Flag System – A Step-by-Step Guide
Now that we understand the architecture, let’s roll up our sleeves and consider how to build a simplified, yet robust, feature flag system. We’ll focus on the core logic and components, using Python and conceptual database schemas.
1. Defining Flag States and Types
Before writing any code, we need to decide what kinds of flags we’ll support.
-
Boolean Flags (On/Off): The simplest type. Is the feature enabled or disabled?
# Example: Is 'NewLoginPage' enabled? is_new_login_page_enabled = feature_flag_client.is_enabled("NewLoginPage", user_context) if is_new_login_page_enabled: render_new_login_page() else: render_old_login_page() -
Multivariate Flags: For A/B testing or showing different experiences. Instead of just on/off, it returns a variant (e.g., “control”, “variant_A”, “variant_B”).
# Example: Which 'PricingPageLayout' variant to show? pricing_layout = feature_flag_client.get_variant("PricingPageLayout", user_context) if pricing_layout == "variant_A": render_pricing_layout_A() elif pricing_layout == "variant_B": render_pricing_layout_B() else: # default/control render_default_pricing_layout() -
Configuration Flags: To deliver dynamic configuration values (strings, numbers, JSON objects) without code deployments.
# Example: Get 'MaxConcurrentUsers' value max_users = feature_flag_client.get_config("MaxConcurrentUsers", user_context, default=100) # Example: Get 'WelcomeMessage' welcome_msg = feature_flag_client.get_config("WelcomeMessage", user_context, default="Hello there!")
2. Designing the Flag Store
Our database schema needs to store all flag configurations, including their rules. A simple approach could use a table like feature_flags and a related flag_rules table.
-- feature_flags table
CREATE TABLE feature_flags (
id SERIAL PRIMARY KEY,
name VARCHAR(255) UNIQUE NOT NULL, -- e.g., 'NewCheckoutFlow'
description TEXT,
flag_type VARCHAR(50) NOT NULL, -- 'boolean', 'multivariate', 'config'
default_value JSONB NOT NULL, -- Default value if no rules match or flag is off
is_enabled BOOLEAN DEFAULT TRUE, -- Master switch for the flag
created_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP,
updated_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP
);
-- flag_rules table (for targeting)
CREATE TABLE flag_rules (
id SERIAL PRIMARY KEY,
flag_id INTEGER NOT NULL REFERENCES feature_flags(id) ON DELETE CASCADE,
priority INTEGER NOT NULL DEFAULT 0, -- Higher priority rules evaluated first
rule_type VARCHAR(50) NOT NULL, -- 'percentage', 'user_list', 'attribute_match', etc.
rule_config JSONB NOT NULL, -- Details of the rule (e.g., {"percentage": 10}, {"user_ids": ["user1", "user2"]})
evaluated_value JSONB NOT NULL, -- The value to return if this rule matches (e.g., true, "variant_A", "{"key": "value"}")
created_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP,
updated_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP
);
-- Add index for efficient flag lookup
CREATE INDEX idx_feature_flags_name ON feature_flags (name);
default_value and evaluated_value are JSONB to accommodate different flag types (boolean, string, object for multivariate/config flags).
3. Implementing the Flag Management API/UI (Conceptual)
For a production system, you’d build a web application (e.g., using Flask, Django, Node.js with Express, or Go with Gin) with authenticated endpoints. Here’s a conceptual API for creating and updating flags:
# Conceptual Python/Flask API endpoint
from flask import Flask, request, jsonify
from datetime import datetime
import json
import psycopg2 # Assuming PostgreSQL
app = Flask(__name__)
# --- Database connection (simplified) ---
def get_db_connection():
conn = psycopg2.connect(
host="localhost",
database="feature_flags_db",
user="your_user",
password="your_password"
)
return conn
@app.route('/flags', methods=['POST'])
# @require_admin_auth # Placeholder for actual auth decorator
def create_flag():
data = request.json
name = data.get('name')
description = data.get('description', '')
flag_type = data.get('flag_type')
default_value = json.dumps(data.get('default_value'))
is_enabled = data.get('is_enabled', True)
if not all([name, flag_type, default_value is not None]):
return jsonify({"error": "Missing required fields"}), 400
conn = get_db_connection()
cur = conn.cursor()
try:
cur.execute(
"""
INSERT INTO feature_flags (name, description, flag_type, default_value, is_enabled)
VALUES (%s, %s, %s, %s, %s) RETURNING id;
""",
(name, description, flag_type, default_value, is_enabled)
)
flag_id = cur.fetchone()[0]
conn.commit()
return jsonify({"message": "Flag created successfully", "id": flag_id}), 201
except psycopg2.IntegrityError:
conn.rollback()
return jsonify({"error": f"Flag with name '{name}' already exists"}), 409
except Exception as e:
conn.rollback()
return jsonify({"error": str(e)}), 500
finally:
cur.close()
conn.close()
@app.route('/flags//rules', methods=['POST'])
# @require_admin_auth
def add_flag_rule(flag_id):
data = request.json
priority = data.get('priority', 0)
rule_type = data.get('rule_type')
rule_config = json.dumps(data.get('rule_config'))
evaluated_value = json.dumps(data.get('evaluated_value'))
if not all([rule_type, rule_config, evaluated_value is not None]):
return jsonify({"error": "Missing required rule fields"}), 400
conn = get_db_connection()
cur = conn.cursor()
try:
cur.execute(
"""
INSERT INTO flag_rules (flag_id, priority, rule_type, rule_config, evaluated_value)
VALUES (%s, %s, %s, %s, %s);
""",
(flag_id, priority, rule_type, rule_config, evaluated_value)
)
conn.commit()
return jsonify({"message": "Rule added successfully"}), 201
except Exception as e:
conn.rollback()
return jsonify({"error": str(e)}), 500
finally:
cur.close()
conn.close()
# Other endpoints for GET /flags, PUT /flags/, DELETE /flags/, etc.
# And endpoints for updating/deleting rules.
This API would be consumed by a frontend UI (e.g., React, Vue, Angular) to provide a user-friendly interface for managing flags.
4. Developing the Flag Evaluation Engine (SDK)
This is the core logic. Our SDK needs to fetch flag configurations and evaluate them against a given user context. For performance, it should cache flag data.
import json
import hashlib
import threading
import time
from collections import defaultdict
# --- Mock Database/API for demonstration ---
# In a real system, this would fetch from your Flag Store via API or direct DB connection
_MOCK_FLAG_STORE = {
"NewCheckoutFlow": {
"id": 1,
"name": "NewCheckoutFlow",
"description": "Enables the new checkout experience",
"flag_type": "boolean",
"default_value": False,
"is_enabled": True,
"rules": [
{
"priority": 100,
"rule_type": "user_list",
"rule_config": {"user_ids": ["khadervali-dev", "test-user-123"]},
"evaluated_value": True
},
{
"priority": 50,
"rule_type": "percentage",
"rule_config": {"percentage": 20},
"evaluated_value": True
},
{
"priority": 10,
"rule_type": "attribute_match",
"rule_config": {"attribute": "region", "operator": "equals", "value": "EU"},
"evaluated_value": True
}
]
},
"PricingPageLayout": {
"id": 2,
"name": "PricingPageLayout",
"description": "Different layouts for pricing page A/B test",
"flag_type": "multivariate",
"default_value": "control",
"is_enabled": True,
"rules": [
{
"priority": 100,
"rule_type": "percentage",
"rule_config": {"percentage": 50},
"evaluated_value": "variant_A"
},
{
"priority": 90,
"rule_type": "percentage", # Remaining 50%
"rule_config": {"percentage": 100}, # This effectively means the remaining 50%
"evaluated_value": "variant_B"
}
]
},
"MaxConcurrentUsers": {
"id": 3,
"name": "MaxConcurrentUsers",
"description": "Configures maximum concurrent users",
"flag_type": "config",
"default_value": 100,
"is_enabled": True,
"rules": [
{
"priority": 10,
"rule_type": "attribute_match",
"rule_config": {"attribute": "env", "operator": "equals", "value": "production"},
"evaluated_value": 500
}
]
}
}
class FeatureFlagClient:
def __init__(self, refresh_interval_seconds=60):
self._flags_cache = {}
self._lock = threading.RLock()
self._last_refresh_time = 0
self._refresh_interval = refresh_interval_seconds
self._start_background_refresher()
def _fetch_flags_from_source(self):
# In a real system, this would be an HTTP call to your Flag Management API
# Or direct database query if running in a tightly coupled environment.
# For this example, we'll simulate fetching from our mock store.
print(f"[{datetime.now().isoformat()}] Refreshing flags from source...")
return _MOCK_FLAG_STORE # Simulate fetching all flags
def _refresh_cache(self):
with self._lock:
try:
fetched_flags = self._fetch_flags_from_source()
self._flags_cache = fetched_flags
self._last_refresh_time = time.time()
print(f"[{datetime.now().isoformat()}] Flags cache refreshed.")
except Exception as e:
print(f"[{datetime.now().isoformat()}] Error refreshing flags cache: {e}")
def _start_background_refresher(self):
def refresher():
while True:
self._refresh_cache()
time.sleep(self._refresh_interval)
# Start the refresher in a daemon thread so it doesn't prevent program exit
thread = threading.Thread(target=refresher, daemon=True)
thread.start()
def _get_flag_config(self, flag_name):
with self._lock:
if not self._flags_cache or (time.time() - self._last_refresh_time > self._refresh_interval):
self._refresh_cache() # Force refresh if cache is empty or stale
return self._flags_cache.get(flag_name)
def _evaluate_rules(self, flag_config, user_context):
if not flag_config.get('is_enabled', False):
return flag_config['default_value']
rules = sorted(flag_config.get('rules', []), key=lambda r: r['priority'], reverse=True)
for rule in rules:
rule_type = rule['rule_type']
rule_config = rule['rule_config']
evaluated_value = rule['evaluated_value']
if rule_type == "user_list":
user_ids = rule_config.get("user_ids", [])
if user_context.get("user_id") in user_ids:
return evaluated_value
elif rule_type == "attribute_match":
attribute = rule_config.get("attribute")
operator = rule_config.get("operator")
value = rule_config.get("value")
user_attribute_value = user_context.get(attribute)
if user_attribute_value is not None:
if operator == "equals" and user_attribute_value == value:
return evaluated_value
# Add other operators: contains, greater_than, less_than, etc.
elif rule_type == "percentage":
percentage = rule_config.get("percentage")
user_id = user_context.get("user_id")
if user_id:
# Deterministic hashing for percentage rollout
# Ensure the same user always gets the same bucket
hash_input = f"{flag_config['name']}-{user_id}"
hash_val = int(hashlib.md5(hash_input.encode()).hexdigest(), 16)
bucket = (hash_val % 100) + 1 # 1 to 100
if bucket <= percentage:
return evaluated_value
else:
# For anonymous users, a simple random choice, but not deterministic
# For true A/B testing, user_id is crucial.
import random
if random.randint(1, 100) <= percentage:
return evaluated_value
# Add more rule types as needed (e.g., time-based, IP range, device type)
# If no rules match, return the default value
return flag_config['default_value']
def is_enabled(self, flag_name: str, user_context: dict = None) -> bool:
"""Evaluates a boolean feature flag."""
flag_config = self._get_flag_config(flag_name)
if not flag_config or flag_config['flag_type'] != 'boolean':
print(f"Warning: Flag '{flag_name}' not found or not boolean type. Returning default False.")
return False # Or raise an error, depending on desired behavior
user_context = user_context if user_context is not None else {}
result = self._evaluate_rules(flag_config, user_context)
# Telemetry logging (conceptual)
self._log_evaluation(flag_name, user_context, result)
return bool(result)
def get_variant(self, flag_name: str, user_context: dict = None, default_variant: str = "control") -> str:
"""Evaluates a multivariate feature flag."""
flag_config = self._get_flag_config(flag_name)
if not flag_config or flag_config['flag_type'] != 'multivariate':
print(f"Warning: Flag '{flag_name}' not found or not multivariate type. Returning default '{default_variant}'.")
return default_variant
user_context = user_context if user_context is not None else {}
result = self._evaluate_rules(flag_config, user_context)
self._log_evaluation(flag_name, user_context, result)
return str(result) # Ensure it's a string variant
def get_config(self, flag_name: str, user_context: dict = None, default_value=None):
"""Evaluates a config feature flag."""
flag_config = self._get_flag_config(flag_name)
if not flag_config or flag_config['flag_type'] != 'config':
print(f"Warning: Flag '{flag_name}' not found or
Khader Vali
Senior Software Engineer specializing in cloud architecture, real-time systems, and enterprise-scale applications.
