Security

Internal Network Connection Issue

SPIRE is currently implemented to rotate the certificates that secure mTLS traffic internally. An internal network connection issue can arise if SPIRE elements go down.

kubectl -n spire logs spire-server-0 -c spire-controller-manager

kubectl -n spire logs spire-server-1 -c spire-controller-manager

Oauth2-Proxy Pod in CrashLoopBackOff Error State

The container in the oauth2-proxy pod started, but crashes and repeatedly restarts incorrectly. To resolve this issue, update the DNS A record with the node IP address where the Istio Ingress gateway pod was deployed. You can get the node IPs on the Status page of the HPE Ezmeral Unified Analytics Software installer.

Authentication

If authentication is not working, check to see if pods are running and check pod logs. You may just need to kill a pod and restart it on a healthy node. For more complicated issues, you may need to collect some logs and talk to HPE support.

ldap-svc.hpe-ldap.svc.cluster.local:389

cn=readonly,dc=example,dc=com
password: mapr

ldapsearch -Y EXTERNAL -Q -H ldapi:/// -b ou=users,dc=example,dc=com

kubectl -n oauth2-proxy logs -l app=oauth2-proxy

kubectl -n keycloak logs keycloak-0

Keycloak/LDAP User Authentication Test

Use a direct grant endpoint to see if a user can authenticate to Keycloak. Run this test on the command line to verify that the AD/LDAP integration is working for a particular user and that the password is correct. This test hits a client in Keycloak that allows the direct grant on the authentication flow. This test is not doing a web-based redirect flow; it is just saying give me your credentials for a token in return.

USER=<username>
PASS=<password>
DOMAIN=<your-domain>.com
RESULT=$(curl -k --data "username=$USER&password=$PASS&grant_type=password&client_id=ua-grant" https://keycloak.$DOMAIN/realms/UA/protocol/openid-connect/token)
ACCESS_TOKEN=$(echo $RESULT | sed 's/.*access_token":"//g' | sed 's/".*//g')

You can also use the direct grant endpoint for REST API endpoints that must accept and validate username and password credentials. The direct grant endpoint can validate the incoming username and password and get an access token. The remainder of the flow inside HPE Ezmeral Unified Analytics Software is based on the access token.

Keycloak Admin Web Console

The Keycloak Admin Web Console is useful for finding and resolving issues. To access the Keycloak Admin Web Console, you will need the system master's kubectl privileges that you got when you first installed and created the HPE Ezmeral Unified Analytics Software cluster. This is required to get the password for the keycloak administrator.

To access the Keycloak Admin Web Console:

1. Go to keycloak.<your-UA-domain>.
2. Enter admin as the user.
3. Enter the password. You can get the password with kubectl and kube.config:

    kubectl -n keycloak get secret admin-pass -o jsonpath="{.data.password}" | base64 -D​

   NOTE

   For Linux, use base64 -d. For Mac, use base64 -D​.
4. In the left navigation bar of the Keycloak UI, switch over to the UA realm.

To find all users:

1. In the left navigation bar, select Users.
2. On the User list tab, enter * in the search field to see all known users.

To change how users sign in (email vs username):

1. In the left navigation bar, go to Realm Settings > Login tab.
2. On the Login tab, select the Login with email switch.

   Duplicate emails turns off automatically when you do this. This is safe to do for external AD/LDAP servers. For internal AD/LDAP servers, you can also turn this on, but there are no guard rails on the internal user management to prevent duplicate emails. If this happens, one of the users will not be able to log in.

You can change the following settings:

• How Keycloak batch requests to the remote server.
• How often Keycloak synchs users from the server; the default is hourly.
  NOTE

  You should not modify the settings you entered through the HPE Ezmeral Unified Analytics Software installer, such as the server address. Changing those settings here can have negative consequences because this is not the only place that those settings are stored. Keycloak uses the LDAP information, but so do some of the other HPE Ezmeral Unified Analytics Software applications. Currently, it is best not to change LDAP settings because it can break applications that depend on them.

Bad HTTP Request

400 Bad Request - The plain HTTP request was sent to HTTPS port.

apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: {{ include "test-app.fullname" . }}
  namespace: {{ .Release.Namespace }}
  labels:
    {{- include "test-app.labels" . | nindent 4 }}

 #The URL should point to the corresponding service. 
 #Kubernetes provides an internal DNS mapping for services using the format <ServiceName>.<ServiceNamespace>.svc.cluster.local. 
spec:
  host: {{ include "test-app.fullname" . }}.{{ .Release.Namespace }}.svc.cluster.local
  trafficPolicy:
    tls:
      mode: SIMPLE

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: {{ include "test-app.fullname" . }}
  namespace: {{ .Release.Namespace }}
  labels:
    {{- include "test-app.labels" . | nindent 4 }}
spec:
  gateways:
    - {{ .Values.ezua.VirtualService.istioGateway }}
  hosts:
    - {{ .Values.ezua.VirtualService.endpoint }}
  #The following VirtualService options are specific and depend on the application implementation.
  #This example is a simple application with single service and simple match routes.
  #The URL should point to the corresponding service. 
  #Kubernetes provides an internal DNS mapping for services using the format <ServiceName>.<ServiceNamespace>.svc.cluster.local. 
  http:
    - match:
        - uri:
            prefix: /
      rewrite:
        uri: /
      route:
        - destination:
            host: {{ include "test-app.fullname" . }}.{{ .Release.Namespace }}.svc.cluster.local
            port:
              number: {{ .Values.service.port }}