Messages

1

Zenarmor (Sensei) / Re: Sensei on OPNsense - Application based filtering

« on: May 16, 2020, 02:32:27 am »

Hi, I have an external Elasticsearch container (7.7.0) and it is complaining a lot about invalid UTF-8 bytes from Sensei, eg :


{"type": "server", "timestamp": "2020-05-16T00:28:33,695Z", "level": "DEBUG", "component": "o.e.a.b.TransportShardBulkAction", "cluster.name": "docker-cluster", "node.name": "da8d9957dfaf", "message": "[conn-200516][0] failed to execute bulk item (index) index {[conn_write][_doc][_9_hGnIBvp4cvgKY7pYd], source[{\"transport_proto\":\"UDP\",\"policyid\":\"0\",\"interface\":\"vtnet0\",\"vlanid\":\"0\",\"conn_uuid\":\"12a6680a-5ce0-4a7c-ae38-1a27c85ff66d\",\"src_hostname\":\"librarian.local\",\"src_username\":\"\",\"ip_src_saddr\":\"10.1.1.10\",\"ip_src_port\":65062,\"src_dir\":\"EGRESS\",\"dst_hostname\":\"81.0.84.116\",\"dst_username\":\"\",\"ip_dst_saddr\":\"81.0.84.116\",\"ip_dst_port\":57997,\"dst_dir\":\"INGRESS\",\"input\":1,\"output\":1,\"src_npackets\":1,\"src_nbytes\":0,\"src_pbytes\":104,\"dst_npackets\":2,\"dst_nbytes\":345,\"dst_pbytes\":317,\"src tcp_flags\":\"\",\"dst tcp_flags\":\"\",\"start_time\":1589588789000,\"end_time\":1589588911000,\"encryption\":\"TLS\",\"app_id\":16,\"app_proto\":\"QUIC\",\"app_name\":\"Quic UDP Connection\",\"app_category\":\"Streaming\",\"tags\":\"Encrypted,SSL,QUIC\",\"src_geoip\":{\"timezone\":\"\",\"continent_code\":\"\",\"city_name\":\"\",\"country_name\":\"\",\"country_code2\":\"\",\"country_code3\":\"\",\"dma_code\":\"0\",\"region_name\":\"\",\"region_code\":\"\",\"postal_code\":\"\",\"area\":\"0\",\"metro\":\"0\",\"asn\":\"0\",\"latitude\":0.0,\"longitude\":0.0,\"location\":{\"lat\":0.0,\"lon\":0.0}},\"dst_geoip\":{\"timezone\":\"\",\"continent_code\":\"\",\"city_name\":\"Duna�jv�ros\",\"country_name\":\"HU\",\"country_code2\":\"\",\"country_code3\":\"\",\"dma_code\":\"0\",\"region_name\":\"\",\"region_code\":\"\",\"postal_code\":\"\",\"area\":\"0\",\"metro\":\"0\",\"asn\":\"0\",\"latitude\":46.983299255371097,\"longitude\":18.933300018310548,\"location\":{\"lat\":46.983299255371097,\"lon\":18.933300018310548}}}]}", "cluster.uuid": "3zoVrbvRRfmZcZZHbXwCZw", "node.id": "5MoI-6jVTFGAfVm-XSZ4TA" ,
"stacktrace": ["org.elasticsearch.index.mapper.MapperParsingException: failed to parse field [dst_geoip.city_name] of type [text] in document with id '_9_hGnIBvp4cvgKY7pYd'. Preview of field's value: ''",
"Caused by: com.fasterxml.jackson.core.JsonParseException: Invalid UTF-8 middle byte 0x72",
" at [Source: (org.elasticsearch.common.bytes.AbstractBytesReference$MarkSupportingStreamInputWrapper); line: 1, column: 1108]",
"at com.fasterxml.jackson.core.JsonParser._constructError(JsonParser.java:1840) ~[jackson-core-2.10.4.jar:2.10.4]",
"at com.fasterxml.jackson.core.base.ParserMinimalBase._reportError(ParserMinimalBase.java:712) ~[jackson-core-2.10.4.jar:2.10.4]",
"at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._reportInvalidOther(UTF8StreamJsonParser.java:3574) ~[jackson-core-2.10.4.jar:2.10.4]",
"at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._reportInvalidOther(UTF8StreamJsonParser.java:3581) ~[jackson-core-2.10.4.jar:2.10.4]",
"at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._decodeUtf8_3fast(UTF8StreamJsonParser.java:3386) ~[jackson-core-2.10.4.jar:2.10.4]",
"at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._finishString2(UTF8StreamJsonParser.java:2490) ~[jackson-core-2.10.4.jar:2.10.4]",
"at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._finishAndReturnString(UTF8StreamJsonParser.java:2438) ~[jackson-core-2.10.4.jar:2.10.4]",
"at com.fasterxml.jackson.core.json.UTF8StreamJsonParser.getText(UTF8StreamJsonParser.java:294) ~[jackson-core-2.10.4.jar:2.10.4]",
"at org.elasticsearch.common.xcontent.json.JsonXContentParser.text(JsonXContentParser.java:83) ~[elasticsearch-x-content-7.7.0.jar:7.7.0]",
"at org.elasticsearch.common.xcontent.support.AbstractXContentParser.textOrNull(AbstractXContentParser.java:253) ~[elasticsearch-x-content-7.7.0.jar:7.7.0]",
"at org.elasticsearch.index.mapper.TextFieldMapper.parseCreateField(TextFieldMapper.java:823) ~[elasticsearch-7.7.0.jar:7.7.0]",
"at org.elasticsearch.index.mapper.FieldMapper.parse(FieldMapper.java:284) ~[elasticsearch-7.7.0.jar:7.7.0]",

And so on. The Opnsense install is the DVD ISO in Proxmox 6.2, the Elasticsearch is in a Docker container on an adjacent host. Any ideas?