Kyma Application Cannot Send Traces to the Telemetry Module (RBAC: Access Denied)

[Lozan]

Hi everyone,

We wanted to perform some tracing tests on our Kyma cluster with the Telemetry module enabled.
For this purpose, we deployed the OpenTelemetry Demo application in our custom namespace, following the guide:

Kyma OpenTelemetry Demo Integration

All prerequisites were completed successfully. Our telemetry backend is an ELK stack instance — the connection to it works, and we can see logs arriving there. However, the demo application is not able to send traces and logs to the dedicated Telemetry gateways.
We see the following errors in the application logs:

[Error] File: /opentelemetry-cpp/exporters/otlp/src/otlp_grpc_log_record_exporter.cc:173
[OTLP LOG GRPC Exporter] Export() failed: RBAC: access denied

[Error] File: /opentelemetry-cpp/exporters/otlp/src/otlp_grpc_exporter.cc:159
[OTLP TRACE GRPC Exporter] Export() failed with status_code: "PERMISSION_DENIED" error_message: "RBAC: access denied"

The logs are visible in Kibana, so they are probably being collected by the log agent(s). Do we need any additional configuration (e.g., custom Roles or Istio AuthorizationPolicies), or is the demo app expected to work out of the box?

Thanks in advance for your help!

Lozan 

[a-thaler · Accepted solution]

Hi Lozan,

by default there are no AuthorizationPolicies defined for the gateways and a Istio RBAC rejection should not happen. Only by applying a custom policy such behaviour could get triggered. After talking to you directly, indeed it turned out to be the problem. Removing any authrorizationPolicy for the gateways fixed the problem.

[Lozan · Accepted solution]

Thank you for the support, the problem seemed to be on our side!