1.28.0 (October 18, 2023)

Incompatible behavior changes

Changes that are expected to cause an incompatibility if applicable; deployment changes are likely required

  • UHV: Introduced runtime flag envoy.reloadable_features.enable_universal_header_validator for toggling Universal Header Validator (UHV) on and off. The default value is off. This option is currently functional only when the ENVOY_ENABLE_UHV build flag is enabled. See https://github.com/envoyproxy/envoy/issues/10646 for more information about UHV.

  • eds: Introduced caching of EDS assignments when used with ADS. Prior to this change, Envoy required that EDS assignments were sent after an EDS cluster was updated. If no EDS assignment was received for the cluster, it ended up with an empty assignment. Following this change, after a cluster update, Envoy waits for an EDS assignment until initial_fetch_timeout times out, and will then apply the cached assignment and finish updating the warmed cluster. This change is disabled by default, and can be enabled by setting the runtime flag envoy.restart_features.use_eds_cache_for_ads to true.

  • http: Add runtime flag http.max_requests_per_io_cycle for setting the limit on the number of HTTP requests processed from a single connection in a single I/O cycle. Requests over this limit are processed in subsequent I/O cycles. This mitigates CPU starvation by connections that simultaneously send high number of requests by allowing requests from other connections to make progress. This runtime value can be set to 1 in the presence of abusive HTTP/2 or HTTP/3 connections. By default this limit is disabled.

  • http: Introduced a new runtime flag envoy.reloadable_features.no_downgrade_to_canonical_name to disable the name downgrading in the per filter config searching. See doc Http filter route specific config or issue https://github.com/envoyproxy/envoy/issues/29461 for more specific detail and examples.

  • http: Switch from http_parser to BalsaParser for handling HTTP/1.1 traffic. See https://github.com/envoyproxy/envoy/issues/21245 for details. This behavioral change can be reverted by setting runtime flag envoy.reloadable_features.http1_use_balsa_parser to false.

  • jwt: Changed behavior of the JWT extraction, passing entire token for validation, instead cut him in the non-Base64 character. This change can be reverted temporarily by setting the runtime guard envoy.reloadable_features.token_passed_entirely to false.

  • udp_proxy: When the UDP proxy has session filters, choosing the upstream host and creating a socket only happens after iterating all onNewSession() calls for all the filters in the chain. Upstream host health check for each downstream datagram does not apply when there are session filters, and per-packet load balancing can’t be used when there are session filters.

  • zone-aware routing: Zone-aware routing is now enabled even when the originating and upstream cluster have different numbers of zones. Previously, zone-aware routing was disabled in that case and the lb_zone_number_differs stat on the cluster was incremented. This behavioral change can be reverted by setting runtime guard envoy.reloadable_features.enable_zone_routing_different_zone_counts to false. Additionally, zone-aware routing now works correctly even when the originating and upstream cluster have different zone sets. Previously, zone-aware routing would not route fairly in this case. To revert the entire change, set the runtime flag envoy.reloadable_features.locality_routing_use_new_routing_logic to false to get the old behavior and well-tested codepaths, undoing both changes.

Minor behavior changes

Changes that may cause incompatibilities for some users, but should not for most

  • access_log: When emitting grpc logs, only downstream HTTP filter state was used. Now, both downstream and upstream HTTP filter states will be tried to find the keys configured in filter_state_objects_to_log.

  • alternate_protocols_cache_filter: Changed the alternate protocols cache filter to get the cache from cluster config rather than filter config. This allows one downstream HTTP filter to be used with multiple clusters with different caches. This change can be reverted by setting runtime guard envoy.reloadable_features.use_cluster_cache_for_alt_protocols_filter to false.

  • ext_authz: Don’t append the local address to x-forwarded-for header when sending an http (not gRPC) auth request. This behavior can be reverted by setting runtime flag envoy.reloadable_features.ext_authz_http_send_original_xff to false.

  • ext_authz: Removing any query parameter in the presence of repeated query parameter keys no longer drops the repeats.

  • ext_proc: Envoy will only take mode_override when waiting for the header responses. It will be ignored if it is in other processing states.

  • http: Abort the HTTP filter chain iteration when the stream is reset by resetStream() of filter callbacks. This behavior can be reverted by setting runtime flag envoy.reloadable_features.abort_filter_chain_on_stream_reset to false.

  • http: Change the proxy status for UpstreamRequestTimeout to HttpResponseTimeout. It can be disabled by the runtime guard envoy.reloadable_features.proxy_status_upstream_request_timeout.

  • http oauth2 filter: Change HMAC cookie encoding to base64-encoded only. This change can be reverted temporarily by setting the runtime guard envoy.reloadable_features.hmac_base64_encoding_only to false.

  • local_rate_limit: Added new configuration field always_consume_default_token_bucket to allow for setting if default token bucket should be always consumed or only be consumed when there is no matching descriptor.

  • outlier detection: A node must stay in healthy state for at least one period of check interval before ejection time multiplier is decremented.

  • outlier detection: Outlier detection will always respect max_ejection_percent now. This behavioral change can be reverted by setting runtime guard envoy.reloadable_features.check_mep_on_first_eject to false.

  • quic: Enable QUICHE request and response headers validation. This behavior can be reverted by setting runtime flag envoy.reloadable_features.FLAGS_envoy_quic_reloadable_flag_quic_act_upon_invalid_header to false.

  • redis: The redis network filter connection_rate_limit_per_sec must be greater than 0. A config that sets this value to 0 will be rejected.

  • router: Enable copying response_code from the upstream stream_info onto the downstream stream_info. This behavior can be reverted by setting runtime guard envoy.reloadable_features.copy_response_code_to_downstream_stream_info to false.

  • router: Enable environment_variable in router direct response.

  • tls: changed ssl failure reason format in ssl socket for a better handling. It can be disabled by the runtime guard envoy.reloadable_features.ssl_transport_failure_reason_format.

  • tls_inspector: Updated the security posture of the TLS inspector listener filter to robust against untrusted downstreams and upstreams.

  • upstream: Deprecate code path of legacy upstream load balancer. Ideally, this is implementation detail changes and should not affect users. However, if there is any user who encounters issues, this behavior can be reverted by setting runtime flag envoy_reloadable_features_convert_legacy_lb_config to false.

  • xds: Set the lower bound of fill_rate to once per year. Values lower than once per year will automatically be set to that value.

Bug fixes

Changes expected to improve the state of the world and are unlikely to have negative effects

  • aws signer: Fixed a bug where expiration timestamp on task roles failed to validate. This causes failure of credential caching which results in constant hits to the task role metadata URL.

  • connection limit: Fixed a use-after-free bug in the connection limit filter.

  • dns: Fixed a bug where dns response was not always conforming RFC 2181 for TTL values. Previously a malicious user could add a TTL greater than 2^31 - 1, and with c-ares library using 32 bit signed int data type would overflow and send a negative TTL.

  • dns: Fixed a bug where when respect_dns_ttl was set to true, c-ares dns resolver only considered address record for TTL calculation while ignoring CNAME records TTL. Now when respect_dns_ttl is set to true minimum of all TTL records is considered.

  • docker/publishing: Update base images to resolve various glibc vulnerabilities.

  • extension_discovery_service: Fixed a bug causing crash if ECDS is used with upstream HTTP filters.

  • healthcheck: The default behavior of unejecting outlier-detection-ejected host on successful active health checking can be disabled by setting outlier_detection.successful_active_health_check_uneject_host to false. This new configuration flag is a substitute for the removed runtime option envoy.reloadable_features_successful_active_health_check_uneject_host.

  • http: Close HTTP/2 and HTTP/3 connections that prematurely reset streams. The runtime key overload.premature_reset_min_stream_lifetime_seconds determines the interval where received stream reset is considered premature (with 1 second default). The runtime key overload.premature_reset_total_stream_count, with the default value of 500, determines the number of requests received from a connection before the check for premature resets is applied. The connection is disconnected if more than 50% of resets are premature, or if the number of suspect streams is already large enough to guarantee that more than 50% of the streams will be suspect upon reaching the total stream threshold (even if all the remaining streams are considered benign). Setting the runtime key envoy.restart_features.send_goaway_for_premature_rst_streams to false completely disables this check.

  • http: Fixed a bug that could cause metadata to be decoded after a local reply has been triggered. Can be disabled by setting envoy.reloadable_features.stop_decode_metadata_on_local_reply to false.

  • http1: Fixed a bug where HTTP/1.1 requests with Connection: close header is handled differently if the requested is internally redirected. Without internal redirect, the response will also have a Connection: close header and the connection will be closed after finishing that request. Requests with internal redirect should be handled in the same way. This behavior can be reverted by setting runtime envoy.reloadable_features.http1_connection_close_header_in_redirect to false.

  • redis: Fixed a bug where redis key formatter is using the closed stream because of life time issues.

  • redis: Fixed a bug where redis key with % in the key is failing with a validation error.

  • router check tool: Fixed a bug where the route coverage is not correctly calculated when a route has weighted clusters.

  • subset load balancer: Fixed a bug where overprovisioning_factor and weighted_priority_health values were not respected when subset load balacing was enabled. The default values of 140 and false were always used.

  • tls: Fixed a bug where handshake may fail when both private key provider and cert validation are set.

  • tracing: Fixed a bug in the Datadog tracer where Datadog’s “operation name” field would contain what should be in the “resource name” field.

  • unix domain sockets: Fixed a crash on some versions of macOS when using a listener on a unix-domain socket.

  • xds: Fix a bug where the nonce was not reset after reconnecting to the xDS server, when using State-of-the-World.

Removed config or runtime

Normally occurs at the end of the deprecation period

  • dns: Removed envoy.restart_features.use_apple_api_for_dns_lookups and legacy code paths.

  • ext_authz: Removed envoy.reloadable_features.http_ext_auth_failure_mode_allow_header_add runtime option and substituted it with failure_mode_allow_header_add configuration flag.

  • healthcheck: Removed envoy.reloadable_features_successful_active_health_check_uneject_host runtime option and substituted it with outlier_detection.successful_active_health_check_uneject_host outlier detection configuration flag.

  • http filters: Removed envoy_reloadable_features_http_filter_avoid_reentrant_local_reply runtime flag and legacy code paths.

  • listener: Removed envoy.reloadable_features.enable_update_listener_socket_options runtime flag and legacy code paths.

  • quic: Removed envoy.reloadable_features.reject_require_client_certificate_with_quic and legacy code paths.

  • runtime: Removed envoy.restart_features.remove_runtime_singleton and legacy code paths.

  • runtime: Removed envoy_reloadable_features_append_query_parameters_path_rewriter and legacy code paths.

  • tcp: Removed runtime key envoy.reloadable_features.tcp_pool_idle_timeout.

  • tcp_proxy: Removed envoy_reloadable_features_finish_reading_on_decode_trailers runtime flag and legacy code paths.

  • xDS: Removed envoy.restart_features.explicit_wildcard_resource and legacy code paths.

New features

  • access_log: Added %RESPONSE_FLAGS_LONG% substitution string, that will output a pascal case string representing the response flags. The output response flags will correspond with %RESPONSE_FLAGS%, only with a long textual string representation.

  • access_log: Added a field lookup to %FILTER_STATE% for objects that have reflection enabled.

  • access_logs: Added json_format_options config option to support JSON output formatting and the sort_properties option to print the JSON output with sorted properties.

  • admin: Added a new skip_exit query parameter to /drain_listeners to skip exiting after the drain period.

  • admin_logging: added support for glob control of fine-grain loggers in admin /logging interface.

  • compression: Added remove_accept_encoding_header for per-route configuration of this value.

  • config: Added the capability to defer broadcasting of certain cluster (CDS, EDS) to worker threads from the main thread. This optimization can save significant amount of memory in cases where there are (1) a large number of workers and (2) a large amount of config, most of which is unused. This capability is guarded by enable_deferred_cluster_creation.

  • ext_proc: Added disable_immediate_response config API to ignore the immediate_response message from the external processing server.

  • extension_discovery_service: Added ECDS support for downstream network filters.

  • extension_discovery_service: Added ECDS support for upstream network filters.

  • extension_discovery_service: Added metric listener.listener_stat.network_extension_config_missing to track closed connections due to missing config.

  • filters: Added filters to update the filter state for the HTTP requests and the TCP connections.

  • geoip: Added support for Maxmind geolocation provider.

  • http: Added Json-To-Metadata filter.

  • http: Added disabled flag to disable a filter in the filter chain by default. The filter can be enabled explicitly by setting valid per filter config in the route configuration.

  • listener: Added possibility to track global downstream connection limit via downstream connections monitor in overload manager.

  • lua: Added downstreamRemoteAddress() method to the Stream info object API.

  • oauth2: Added use_refresh_token to support updating an access token via a refresh token if that is provided by authorization server.

  • original_dst: added support for the internal listener address recovery using the original destination listener filter.

  • otlp_stats_sink: Added stats prefix option to OTLP stats sink that enables adding a static prefix to all stats flushed by this sink.

  • quic: Added support for QUIC listener filters with ECDS support reusing the same config API listener_filters as TCP does.

  • redis: Added new configuration field read_command_policy to specify Envoy should route read commands to another cluster.

  • redis: Added support for lmove command.

  • redis: Added support for time command (returns a local response).

  • router: Added metadata support for virtual host and route configuration.

  • tap: Added custom_sink type to enable writing tap data out to a custom sink extension.

  • tap: Added record_downstream_connection to control writing downstream connection address info in trace output.

  • tap: Added record_headers_received_time to control writing request and response headers received time in trace output.

  • tcp: Added the support to detect and send TCP RST for raw buffer socket based connections. This is currently supported on Linux only. It can be disabled by the runtime guard envoy_reloadable_features_detect_and_raise_rst_tcp_connection.

  • tls: Added disable_stateful_session_resumption config option to disable stateful TLS session resumption.

  • tls: Added fallback fallback to support private key provider to fallback to boringssl TLS handshake. If the private key provider isn’t available (eg. the required hardware capability doesn’t existed), Envoy will fallback to the BoringSSL default implementation when the fallback is true. The default value is false.

  • tracing: Added spawn_upstream_span to control whether to create separate upstream span for upstream request.

  • tracing: Added support for exporting spans via HTTP on the OpenTelemetry tracer.

  • udp_proxy: Added http_capsule UDP session filter that can be used to encapsule or decapsulate UDP datagrams in HTTP, when used for UDP tunneling.

  • udp_proxy: Added session_filters config to support optional filters that will run for each upstream UDP session. More information can be found in the UDP proxy documentation.

  • udp_proxy: Added injectDatagramToFilterChain() callback to UDP session filters that allows session filters to inject datagrams downstream or upstream the filter chain during a filter chain iteration. This can be used, for example, by session filters that are required to buffer datagrams due to an asynchronous call.

  • udp_proxy: added dynamic_forward_proxy UDP session filter that can be used to have dynamic forward proxy UDP flows, when used in conjunction with another session filter that sets required filter state values.

  • upstream: Added allow_redundant_keys to suppport redundant keys in request metadata for subset load balancing.

  • upstream: Added the ability to specify a custom upstream local address selector using local_address_selector.

  • zookeeper: Added support for emitting per opcode request bytes metrics via enable_per_opcode_request_bytes_metrics. added support for emitting per opcode response bytes metrics via enable_per_opcode_response_bytes_metrics.

Deprecated

  • listener: deprecated runtime key overload.global_downstream_max_connections in favor of downstream connections monitor.

  • tracing: start_child_span is deprecated by spawn_upstream_span. Please use the new field to control whether to create separate upstream span for upstream request.

  • tracing: OpenTracing is deprecated and will be removed at version 1.30, since the upstream project has been abandoned.

  • tracing: Opencensus is deprecated and will be removed at version 1.30, since the upstream project has been abandoned.