[BUG] UnsupportedCharsetException When Enabling HTTP Compression on ARM in OpenSearch Output Plugin #269
Description
Describe the bug
As soon as http_compression is enabled, the following error appears in the logs:
An unknown error occurred sending a bulk request to OpenSearch (will retry indefinitely) {:message=>"ASCII-8BIT", :exception=>Java::JavaNioCharset::UnsupportedCharsetException
To Reproduce
Steps to reproduce the behaviour:
- Enable
http_compressionin Opensearch output plugin configuration. - Send any bulk requests.
- Observe the Logstash logs.
Expected behaviour
No charset-related errors should occur when http_compression is enabled.
Plugins
- logstash-output-opensearch (with AWS IAM auth)
Screenshots
If applicable, add screenshots to help explain your problem.
Host/Environment (please complete the following information):
- AWS OpenSearch: 2.5
- Logstash: tested on 8.14.3-aarch64 and 8.17.0-aarch64
- OS: Amazon Linux 2023 (EC2 instance)
Additional context
logstash.yml
pipeline.workers: 4
pipeline.batch.size: 100
pipeline.ecs_compatibility: disabled
pipeline.ordered: false
dead_letter_queue.enable: true
dead_letter_queue.storage_policy: drop_older
dead_letter_queue.retain.age: 12h
path.dead_letter_queue: /data/logstash_temp
# queue config
queue.type: memory
queue.drain: true
# Handling of non-standard field names
config.field_reference.escape_style: percent
opensearch-output-plugin config
input { pipeline { address => opensearch } }
output {
opensearch {
id => "opensearch-<REDACTED>"
hosts => ["<REDACTED>:443"]
auth_type => {
type => 'aws_iam'
region => "<REDACTED>"
}
ssl => true
ssl_certificate_verification => false
manage_template => false
legacy_template => true
action => "create"
index => "%{[@metadata][index_name]}"
ecs_compatibility => "disabled"
pipeline => "logstash_pipeline"
http_compression => true
}
}
input config
input {
beats {
id => "beats-input"
port => 5044
add_field => { "ingestion.input" => "input-otp-beat" }
type => "log"
}
}
output {
pipeline { send_to => filters_pipeline }
}
filter config
input { pipeline { address => filters_pipeline } }
filter {
mutate {
remove_field => [ "[host]" ]
remove_field => [ "[_hash]" ]
}
# add service.hostname if source is filebeat
if ![service][hostname] {
if [agent][name] {
mutate {
add_field => {
"service.hostname" => "%{[agent][name]}"
}
}
}
}
mutate {
add_field => {
"ingestion.logstash.name" => "<REDACTED>"
"[@metadata][index_name]" => "aws-logstash-log-%{+YYYY.MM.dd}"
}
}
if ![appAlias] {
mutate { add_field => { "appAlias" => "default" } }
}
if ![appId] {
mutate { add_field => { "appId" => "unknown" } }
}
}
output {
# Main output
pipeline { send_to => opensearch }
# Secondary outputs if any
}