Application Configuration API

This article describes the HPE Ezmeral Runtime Enterprise API that is used for automated application configuration. An application can use this API to query various configuration parameters and then include the responses as part of the configuration process that occurs when the application starts. The API framework consists of three primary components:

  • BlueData Agent: This agent is installed in each of the container nodes in a cluster. The agent is configured when the cluster starts up and is completely transparent to the application and related configuration scripts.
  • Python API: The BD_VLIB API includes the BD_VCLI command line utility provides a friendly shell script interface.
  • Application configuration bundle: This is provided by a third-party developer who is developing a Catalog for HPE Ezmeral Runtime Enterprise.

This article describes the following topics:

Application Configuration Bundle

An application configuration bundle may be either an uncompressed .tar file (.tar) or a gzipped .tar file (.tar.gz or .tgz). The bundle must contain two entry points implemented in either Python or bash, and must be named as follows:

  • startscript: Must accept the following command line options:
    • --configure: Indicates that the application is being configured for the first time on this cluster. This option is invoked on all nodes that are deployed for the virtual cluster.
    • --addnodes: Indicates that new nodes were added to the virtual cluster. All nodes that existed before the new additions receive this notification, even if they belong to a different nodegroup. The following additional arguments are always specified to describe the new additions:
      • --nodegroup: Nodegroup ID of the added nodes.
      • --role: Role of the newly-added nodes in the above nodegroup.
      • --fqdns: Comma-separated list of FQDNs of the new nodes for quick identification.
    • --delnodes: Indicates that the nodes have been deleted from the virtual cluster. All remaining nodes receive this notification, even if they belong to a different nodegroup. The following additional arguments are always specified:
      • --nodegroup: Nodegroup ID of the deleted nodes.
      • --role: Role of the deleted nodes in the above nodegroup.
      • --fqdns: Comma-separated list of FQDNs of the deleted nodes for quick identification.
    • --reattach: Indicates that the cluster is being (re)attached to another cluster.
    • --delete: Invoked on all of the virtual nodes (containers) in a virtual cluster before cluster deletion. Also invoked when shrinking a virtual cluster on the virtual node(s) that are being deleted, as shown in the following table:
API calls for various cluster lifecycle events
Virtual Cluster Lifecycle Event All Existing or Remaining Containers New or Containers Being Deleted
Create N/A

All containers are new:

startscript --configure
Expand startscript --addnodes --nodegroup <NGID> --role <RID> --fqdns <NEW_FQDNS>

On new containers:

startscript --configure
Shrink startscript --delnodes --nodegroup <NGID> --role <RID> --fqdns <NEW_FQDNS>

On containers being deleted:

startscript --configure
Delete

startscript -delete

Only applies for config_api version 9 or higher.

N/A

The BD_VLIB/BD_VCLI API

When using the bd_vlib library, developers may opt to use either a list of tokenized keys or a custom letter-delimited key. For simplicity, this article only describes the bd_vcli command line options.

  • --get=KEY: Returns the value defined for the provided KEY. The available keys are divided into various namespaces, as described below.
    • Platform namespace: The keyword platform at the beginning of the key indicates the Platform namespace to the API. This namespace can only be accessed by a super user. It includes the following keys:
      • platform.version: Application configuration API version. Default is 8.
      • platform.plha: Whether (true) or not (false) platform High Availability protection has been enabled.
      • platform.clusterip: IP address of the primary Controller host, regardless of the platform.plha value. The platform.clustername key will be empty if platform High Availability is not enabled, or if a DNS name was not specified when platform High Availability was enabled.
      • platform.controllerip: IP address of the original Controller host.
      • platform.shadowip: IP address of the Shadow Controller host, if platform High Availability is enabled.
      • platform.arbiterip: IP address of the Arbiter host, if platform High Availability is enabled.
      • platform.controllerfqdn: FQDN of the Controller host.
      • platform.shadownfqdn: FQDN of the Shadow Controller host, if platform High Availability is enabled.
      • platform.aribterfqdn: FQDN of the Arbiterhost, if platform High Availability is enabled.
      • platform.clustername: Name of the cluster name that was defined when enabling platform High Availability; otherwise, none.
      • platform.isskudocker: Whether (true) or not (false) HPE Ezmeral Runtime Enterprise is installed on-premises or hybrid.
      • platform.isskuec2: true/false: Whether (true) or not (false) HPE Ezmeral Runtime Enterprise is installed on EC2 only (not hybrid).
      • platform.isskuk8s: true/false: - Whether (true) or not (false) HPE Ezmeral Runtime Enterprise is installed on Kubernetes.
    • Tenant namespace: The keyword tenant at the beginning of the key indicates the Tenant namespace to the API. This namespace can only be accessed by a super user. It includes the following keys:
      • tenant.id: ID of the tenant.
      • tenant.name: Name of the tenant.
      • tenant.key_visibility: Visibility of the tenant keypair. This will be either all (visible to everyone), all_admins (visible to the Tenant Administrator only), or site_admin_only (visible to the Platform Administrator only).
      • tenant.user_groups: Comma-separated list of user role names. Standard HPE Ezmeral Runtime Enterprise user roles are Admin and Member.
      • tenant.user_groups.<role_name>: Comma-separated list of indexes in to a list of DNs that can be used to log in to the role described by the <role_name>. The list is 0-indexed.
      • tenant.user_groups.<role_name>.<list_index>: An LDAP or Active Directory DN that defines a group. Users who are assigned to that group will be permitted to log in to the virtual node.
      • tenant.kdc_type: Kerberos protection configured for the tenant. This will be either none (if Kerberos is not enabled), or MIT KDC or Active Directory (if Kerberos is enabled).
    • Tenant namespace (with MIT KDC): The following APIs apply if MIT KDC is configured for the tenant:
      • tenant.kdc_host: Name or IP address of the Kerberos host(s).
      • tenant.kdc_realm: Namespace that helps define access permissions.
      • tenant.krb_enc_types: Type of Kerberos encryption specified during tenant configuration.
      • tenant.kdc_admin_user: Username of the Kerberos administrator.
      • tenant.kdc_admin_password: Password of the Kerberos administrator.
    • Tenant namespace (with Active Directory KDC): The following APIs apply if AD KDC is configured for the tenant:
      • tenant.kdc_host: Name or IP address of the Kerberos host(s).
      • tenant.kdc_realm: Namespace that helps define access permissions.
      • tenant.krb_enc_types: Type of Kerberos encryption specified during tenant configuration.
      • tenant.kdc_admin_user: Username of the Kerberos administrator.
      • tenant.kdc_admin_password: Password of the Kerberos administrator.
      • tenant.kdc_ad_prefix: Optional prefix to be added to all newly-created accounts.
      • tenant.kdc_ad_suffix: Active Directory suffix where all of the accounts used inside virtual clusters will be created.
      • tenant.kdc_ad_ldaps_port: Active Directory port for LDAPS.
    • Node namespace: The keyword node at the beginning of the key indicates the Node namespace to the API. This namespace includes the following keys:
      • node.hostname: Hostname assigned to the node.
      • node.domain: Domain name of the node.
      • node.fqdn: Fully Qualified Domain Name (FQDN) of the host.
      • node.role_id: Role identifier of the node, as specified in the Catalog.
      • node.distro_id: Distro identification of the node as specified by the Catalog. This value may be empty.
      • node.nodegroup_id: Cluster-wide unique nodegroup identifier this node belongs to.
      • node.depends_on: Cluster-wide unique service IDs that this node depends on.
    • Cluster namespace: The keyword cluster at the beginning of the key indicates the Cluster namespace to the API. This namespace includes the following keys:
      • cluster.name: Name assigned to the cluster by the user who created it.
      • cluster.created_by_user_name: The username associated with the user who created the cluster.
      • cluster.service_tokens: List of nodegroups ids where the services with auth token are running.
      • cluster.service_tokens.<ng_id>: List of service IDs in this namespace with authorization tokens.
      • cluster.service_tokens.<ng_id>.<srvc_id>.auth_token: The authorization token associated with the service.
      • cluster.config_choice_selections: List of all nodegroup IDs that are part of the virtual cluster. This information is based on what is specified in the Catalog and specific selections made by the user at cluster creation time.
      • cluster.config_choice_selections.<ng_id>: Keys that may be available for the specific node group.
      • cluster.config_choice_selections.<ng_id>.<key>: Value assigned to the specific key.
      • cluster.config_metadata: A list of all nodegroup IDs that are part of the virtual cluster.
      • cluster.config_metadata.<ng_id>: List of all configuration metadata keys available for the particular nodegroup. The available keys are gathered from the information specified in the Catalog of this distribution.
      • cluster.config_metadata.<ng_id>.<key>: Value assigned to the specific key.
    • Distros namespace: The keyword distros at the beginning of the key indicates the Distros namespace to the API. This namespace includes the following keys:
      • distros: List of Catalog IDs of the various distributions deployed in the virtual cluster.
      • distros.<d_id>: List of all nodegroup IDs that use the specific distribution.
      • distros.<d_id>.<ng_id>: List of keys available for this specific distribution deployed in the given nodegroup.
      • distros.<d_id>.<ng_id>.distro_id: The Catalog distribution ID for this distribution.
      • distros.<d_id>.<ng_id>.catalog_entry_version: Version defined in the Catalog describing the distribution.
      • distros.<d_id>.<ng_id>.config_metadata: Configuration metadata declared in the Catalog.
      • distros.<d_id>.<ng_id>.config_choice_selections: Configuration choices made by the user when creating the cluster.
      • distro.<d_id>.<ng_id>.roles: List of all the roles deployed in the virtual cluster that use this distribution ID and belong to the specified nodegroup.
    • Services namespace: The keyword services at the beginning of the key indicates the Services namespace to the API. This namespace includes the following keys:
      • services: List of all cluster-wide services. Services are only listed here once, even if a particular service will be configured on multiple nodes.
      • services.<s_id>: List of all nodegroup IDs on which this particular service is expected to be available.
      • services.<s_id>.<ng_id>: List of all role IDs in the specified <ng_id> that is/are expected to run the service specified by <s_id>.
      • services.<s_id>.<ng_id>.<r_id>: List of keys available for this particular service on the specified nodegroup and role.
      • services.<s_id>.<ng_id>.<r_id>.fqdns: FQDNs of all nodes that will be running the service.
      • services.<s_id>.<ng_id>.<r_id>.hostnames: Hostnames of all nodes that will be running the service.
      • services.<s_id>.<ng_id>.<r_id>.endpoints: List of URIs to the selected service.
      • services.<s_id>.<ng_id>.<r_id>.qualifiers: List of all qualifiers associated with this service. This information is gathered from the Catalog.
    • Attachments namespace: This namespace is available when an AI/ML cluster is attached to another cluster. Otherwise, it will be empty.
      • attachments: clusters and models are the only available sub-keys.
      • attachments.clusters: List of all cluster IDs attached to the current cluster.
      • attachments.clusters.<cluster_id>: List of all keys describing the cluster.
      • attachments.clusters.<cluster_id>.id: The attached cluster's ID.
      • attachments.clusters.<cluster_id>.name: The attached cluster's name.
      • attachments.clusters.<cluster_id>.config_metadata: The attached cluster's config metadata when it was deployed. Please refer to cluster.config_metadata for sub-keys.
      • attachments.clusters.<cluster_id>.config_choice_selections: The attached cluster's selected configuration choices when it was deployed. Please refer to cluster. config_choice_selections for sub-keys.
      • attachments.clusters.<cluster_id>.isolated: Boolean describing whether the attached cluster is in Isolated mode.
      • attachments.clusters.<cluster_id>.services: The Services namespace of the attached cluster. Please refer to the Services namespace for sub-keys.
      • attachments.clusters.<cluster_id>.distros: The Distro namespace of the attached cluster. Please refer to the Distros namespace for sub-keys.
      • attachments.models: Comma separated list of IDs of all models attached to this cluster.
      • attachments.models.<model_id>: Comma-separated list of keys available to query.
      • attachments.models.<model_id>.name: The name given to the model.
      • attachments.models.<model_id>.version: The model's versions.
      • attachments.models.<model_id>.status: The current status of the model.
      • attachments.models.<model_id>.created_by_user_id: The ID of the user who created the model.
      • attachments.models.<model_id>.create_by_user_name: The username associated with the creator of the model.
      • attachments.models.<model_id>.model_location: The model's location in the project repository.
      • attachments.models.<model_id>.scoring_script: The path to the model's scoring script in the project repository.
      • attachments.models.<model_id>. input_parameter_file: Input parameter specification file for the model.
      • attachments.models.<model_id>. inference_bootstrap_script: Bootstrap script specified when the model was created.
      • attachments.models.<model_id>. training_data: Pointer to the training data.
      • attachments.models.<model_id>. training_cluster: Training cluster used when creating the model.
  • Distributed synchronization: Application configuration on different nodes with different roles may depend on each other and may have to wait for one to come up before the other. This API provides the ability to wait for a service to be registered and running before proceeding with the configuration.
    • --wait=<KEY(S)>: Comma-separated list of key(s) specifying which services to wait for. The key(s) specified could be either the end services or a single key representing a group of services, but not both at the same time.
    • --timeout=SEC: Maximum time to wait for a response from the remote node running the service specified to --wait options. If not specified, this timeout defaults to one hour.
  • Service registration: All services defined in the Catalog must be registered with the application configuration framework at the time of initial configuration. The registration process also starts the service at both create time and after a node reboot. The registration also provides service lifecycle management facilities. The following API is available for registering catalog services:
    • --service_key=KEY: The key from the virtual cluster metadata querying API that uniquely identifies the specific service being registered.
    • --sysv=SERVICE: Node-wide unique system service name to register, such as sssd.
    • --desc=DESCRIPTION: Description for the system service being registered.
  • Miscellaneous: The API also includes the following miscellaneous functions:
    • --version: Current version of the metadata representation. Configuration scripts may use this information as deemed necessary; however, HPE Ezmeral Runtime Enterprise does not currently support upgrading guest configuration bundles on existing virtual clusters.
    • --get_local_group_fqdns: Returns a comma-separated list of FQDNs of all the nodes deployed in the same nodegroup as the requesting node.
    • --get_all_fqdns: Returns a comma-separated list of FQDNs of all the nodes deployed in the cluster.
    • --restart_all_services: Restarts all services registered with the application configuration framework. This may be used to restart all registered services when the parameters were modified after successful cluster creation. Alternatively, the application configuration scripts may choose to restart individual services as required.
    • --cp --node <fqdn> --src <local_path> --dest <remote_path> --perms <remote_permissions>: Copies a file from the node invoking this API to another specified by the FQDN. For security, the destination file will be owned by the same user that initiated the API on the source node. In this command:
      • <fqdn> is the fully qualified domain name of the source container.
      • <local_path> is the full path to the file to be copied.
      • <remote_path> is the full path to the location where the file will be copied.
      • <remote_permissions> are the permissions to assign to the file on the destination, in absolute (octal) notation. If needed, you may access a permissions calculator that will help you determine the proper permissions to assign to the copied file (link opens an external website in a new browser tab/window).
    • --execute --remote_node <fqdn> --script <remote_path>: Executes a script on a remote Docker container (virtual node), where:
      • <fqdn> is the fully qualified domain name of the source container.
      • <remote_path> is the full path to the script that will be executed. This script must already exist in the specified path before executing this command.