Certificate management in Zowe API Mediation Layer

Review details of certificate management in Zowe API Mediation Layer including running on localhost, Zowe runtime on z/OS. This topic also addressing key information about the API ML truststore and keystore, and API ML SAF Keyring.

• Running on localhost
  • How to start API ML on localhost with full HTTPS
  • Certificate management script
  • Generate certificates for localhost
  • Generate a certificate for a new service on localhost
  • Add a service with an existing certificate to API ML on localhost
  • Service registration to Discovery Service on localhost
• Zowe runtime on z/OS
  • Import the local CA certificate to your browser
  • Generate a keystore and truststore for a new service on z/OS
  • Add a service with an existing certificate to API ML on z/OS
  • Procedure if the service is not trusted
• API ML truststore and keystore
• API ML SAF Keyring

Running on localhost

How to start API ML on localhost with full HTTPS

The https://github.com/zowe/api-layer repository contains pre-generated certificates that can be used to start API ML with HTTPS on your computer. The certificates are not trusted by your browser so you can either ignore the security warning or generate your own certificates and add them to the truststore of your browser or system.

For more information about certificates, see TLS Certificates for localhost.

Note: When running on localhost, only the combination of using a keystore and truststore is supported.

Certificate management script

Zowe API Mediation Layer provides a script that can be used on Windows, Mac, Linux, and z/OS to generate a certificate and keystore for the local CA, API Mediation Layer, and services.

This script is stored in zowe/zowe-install-packaging repository bin/apiml_cm.sh. It is a UNIX shell script that can be executed by Bash or z/OS Shell.

Generate certificates for localhost

Use the following procedure to generate certificates for localhost.

Follow these steps:

1. Clone the zowe-install-packaging repository to your local machine.
2. Place the bin/apiml_cm.sh script into the scripts directory in your API Mediation Layer repository folder
3. Use the following script in the root of the api-layer repository to generate certificates for localhost:

scripts/apiml_cm.sh --action setup

This script creates the certificates and keystore for the API Mediation Layer in your current workspace.

Generate a certificate for a new service on localhost

To generate a certificate for a new service on localhost, see Generating certificate for a new service on localhost.

Add a service with an existing certificate to API ML on localhost

For more information about adding a service with an existing certificate to API ML on localhost, see Trust certificates of other services.

Service registration to Discovery Service on localhost

To register a new service to the Discovery Service using HTTPS, provide a valid client certificate that is trusted by the Discovery Service.

Zowe runtime on z/OS

Certificates for the API ML local CA and API ML service are managed by installing the Zowe runtime on z/OS. Follow the instructions in Installing the Zowe runtime on z/OS.

There are two ways to set up certificates on a z/OS machine:

• Certificates in SAF keyring
• Certificates in UNIX files (keystore and truststore)

The Configuring PKCS12 certificates and Configuring JCERACFS certificates in a key ring contain instructions about how to set up certificates during installation. Follow the procedure in the applicable section described in this article during installation.

Import the local CA certificate to your browser

Trust in the API ML server is a necessary precondition for secure communication between Browser or API Client application. Ensure this trust through the installation of a Certificate Authority (CA) public certificate. By default, API ML creates a local CA. Import the CA public certificate to the truststore for REST API clients and to your browser. You can also import the certificate to your root certificate store.

Notes:

• If a SAF keyring is being used and set up with ZWEKRING JCL, the procedure to obtain the certificate does not apply. It is recommended that you work with your security system administrator to obtain the certificate. Start the procedure at step 2.

• The public certificate in the PEM format is stored at <KEYSTORE_DIRECTORY>/local_ca/localca.cer where <KEYSTORE_DIRECTORY> is defined in a customized <RUNTIME_DIR>/bin/zowe-setup-certificates.env file during the installation step that generates Zowe certificates. The certificate is stored in UTF-8 encoding so you need to transfer it as a binary file. Since this is the certificate to be trusted by your browser, it is recommended to use a secure connection for transfer.

• Windows currently does not recognize the PEM format. For Windows, use the P12 version of the local_cer.

Follow these steps:

1. Download the local CA certificate to your computer. Use one of the following methods to download the local CA certificate to your computer:

   - **Use [Zowe CLI](https://github.com/zowe/zowe-cli#zowe-cli--) (Recommended)**  

   Issue the following command:

       `zowe zos-files download uss-file --binary $KEYSTORE_DIRECTORY/local_ca/localca.cer`
   
   - **Use `sftp`**  

   Issue the following command:

       ```
       sftp <system>
       get $KEYSTORE_DIRECTORY/local_ca/localca.cer
       ```
   
   To verify that the file has been transferred correctly, open the file. The following heading and closing should appear:
   
   ```
   -----BEGIN CERTIFICATE-----
   ...
   -----END CERTIFICATE-----
   ```

2. Import the certificate to your root certificate store and trust it.

   • For Windows, run the following command:

     certutil -enterprise -f -v -AddStore "Root" localca.cer

     Note: Ensure that you open the terminal as administrator. This will install the certificate to the Trusted Root Certification Authorities.

   • For macOS, run the following command:

     $ sudo security add-trusted-cert -d -r trustRoot -k /Library/Keychains/System.keychain localca.cer

   • For Firefox, manually import your root certificate via the Firefox settings, or force Firefox to use the Windows truststore.

     Note: Firefox uses its own certificate truststore.

     Create a new Javascript file firefox-windows-truststore.js at C:\Program Files (x86)\Mozilla Firefox\defaults\pref with the following content:

     /* Enable experimental Windows truststore support */
     pref("security.enterprise_roots.enabled", true);

Generate a keystore and truststore for a new service on z/OS

You can generate a keystore and truststore for a new service by calling the apiml_cm.sh script in the directory with API Mediation Layer.

Note: This procedure applies to UNIX file keystore and truststore only. For the SAF keyring option, it is recommended that you perform the actions manually using your security system commands.

Call the apiml_cm.sh script in the directory with the API Mediation Layer as in the following example.

Example:

cd $RUNTIME_DIR
bin/apiml_cm.sh --action new-service --service-alias <alias> --service-ext <ext> \
--service-keystore <keystore_path> --service-truststore <truststore_path> \
--service-dname <dname> --service-password <password> --service-validity <days> \
--local-ca-filename $KEYSTORE_DIRECTORY/local_ca/localca

where:

• service-alias
  is a unique string to identify the key entry. All keystore entries (key and trusted certificate entries) are accessed via unique aliases. Since the keystore has only one certificate, you can omit this parameter and use the default value localhost.

• service-keystore
  specifies repository of security certificates plus corresponding private keys. The <keystore_path> is the path excluding the extension to the keystore that is generated. It can be an absolute path or a path relative to the current working directory. The key store is generated in PKCS12 format with the .p12 extension. Ensure that the path is in an existing directory where your service expects the keystore.

  Example: /opt/myservice/keystore/service.keystore.

• service-truststore
  contains certificates from other parties that you expect to communicate with, or from Certificate Authorities that you trust to identify other parties. The <truststore_path> is the path excluding the extension to the trust store that is generated. It can be an absolute path or a path relative to the current working directory. The truststore is generated in PKCS12 format.

• service-ext
  specifies the X.509 extension that should be the Subject Alternate Name (SAN). The SAN contains host names that are used to access the service. You need to specify the same hostname that is used by the service during API Mediation Layer registration.

  Example: "SAN=dns:localhost.localdomain,dns:localhost,ip:127.0.0.1"

  Note: For more information about SAN, see SAN or SubjectAlternativeName at Java Keytool - Common Options.

• service-dname
  specifies the X.509 Distinguished Name and is used to identify entities, such as those which are named by the subject and issuer (signer) fields of X.509 certificates.

  Example: "CN=Zowe Service, OU=API Mediation Layer, O=Zowe Sample, L=Prague, S=Prague, C=CZ"

• service-validity
  specifies the number of days until the certificate expires.

• service-password
  specifies the keystore password. The purpose of the password is the integrity check. The access protection for the keystore and keystore need to be achieved by making them accessible only by the ZOVESVR user ID and the system administrator. The local-ca-filename is the path to the keystore that is used to sign your new certificate with the local CA private key. It should point to the $KEYSTORE_DIRECTORY/local_ca/localca where $KEYSTORE_DIRECTORY is defined in a customized $ZOWE_ROOT_DIR/bin/zowe-setup-certificates.env file during the installation step that generates Zowe certificates.

Add a service with an existing certificate to API ML on z/OS

The API Mediation Layer requires validation of the certificate of each service that it accessed by the API Mediation Layer. The API Mediation Layer requires validation of the full certificate chain.

Note: This procedure applies only to UNIX file keystore/truststore. For the SAF keyring option, we recommend to perform the actions manually using your security system commands.

Import the public certificate of the root CA that has signed the certificate of the service to the APIML truststore.

Note: Validation fails if a service does not provide an intermediate CA certificates to the API ML. This can be circumvented by importing the intermediate CA certificates to the API ML truststore.

The following path is an example of how to import a public certificate to the API ML truststore by calling in the directory with API Mediation Layer:

cd <RUNTIME_DIR>
bin/apiml_cm.sh --action trust --certificate <path-to-certificate-in-PEM-format> --alias <alias>

Procedure if the service is not trusted

If your service is not trusted, you may receive a response with the HTTP status code 502 Bad Gateway and a JSON response in the standardized format for error messages. The following request is an example of when this error response may occur.

Example:

http --verify=$KEYSTORE_DIRECTORY/local_ca/localca.cer GET https://<gatewayHost>:<port>/<untrustedService>/api/v1/greeting

In this example, you receive a similar response:

    HTTP/1.1 502
    Content-Type: application/json;charset=UTF-8

    {
        "messages": [
            {
                "messageContent": "The certificate of the service accessed by HTTPS using URI '/<untrustedService>/api/v1/greeting' is not trusted by the API Gateway: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target",
                "messageKey": "apiml.common.tlsError",
                "messageNumber": "AML0105",
                "messageType": "ERROR"
            }
        ]
    }

The message has the key apiml.common.tlsError, and message number AML0105. The content explains details about the message.

If you receive this message, import the certificate of your service or the CA that signed it to the truststore of the API Mediation Layer as described previously.

API ML truststore and keystore

A keystore is a repository of security certificates consisting of either authorization certificates or public key certificates with corresponding private keys (PK), used in TLS encryption. A keystore can be stored in Java specific format (JKS) or use the standard format (PKCS12). The Zowe API ML uses PKCS12 to enable the keystores to be used by other technologies in Zowe (Node.js).

API ML SAF Keyring

As an alternative to using a keystore and truststore, API ML can read certificates from a SAF keyring. The user running the API ML must have rights to access the keyring. From the java perspective, the keyring behaves as the JCERACFKS keystore. The path to the keyring is specified as safkeyring:////user_id/key_ring_id. The content of SAF keyring is equivalent to the combined contents of the keystore and the truststore.

Note: When using JCEFACFKS as the keystore type, ensure that you define the class to handle the RACF keyring using the -D options to specify the java.protocol.handler.pkgs property:

-Djava.protocol.handler.pkgs=com.ibm.crypto.provider

The elements in the following list, which apply to the API ML SAF Keyring, have these corresponding characteristics:

The API ML local certificate authority (CA)

• The API ML local CA contains a local CA certificate and a private key that needs to be securely stored.
• The API ML local certificate authority is used to sign certificates of services.
• The API ML local CA certificate is trusted by API services and clients.

The API ML keystore or API ML SAF Keyring

• Server certificate of the Gateway (with PK). This can be signed by the local CA or an external CA.
• Server certificate of the Discovery Service (with PK). This can be signed by the local CA.
• Server certificate of the Catalog (with PK). This can be signed by the local CA.
• Private asymmetric key for the JWT token, alias jwtsecret. The public key is exported to the localhost.keystore.jwtsecret.cer directory.
• The API ML keystore is used by API ML services.

The API ML truststore or API ML SAF Keyring

• Local CA public certificate.
• External CA public certificate (optional).
• Can contain self-signed certificates of API Services that are not signed by the local or external CA.
• Used by API ML services.

Zowe core services

• Services can use the same keystore and truststore or the same keyring as APIML for simpler installation and management.
• When using a keystore and truststore, services have to have rights to access and read them on the filesystem.
• When using a keyring, the user of the service must have authorization to read the keyring from the security system.
• Alternatively, services can have individual stores for higher security.

API service keystore or SAF keyring (for each service)

• The API service keystore contains a server and client certificate signed by the local CA.

API service truststore or SAF keyring (for each service)

• (Optional) The API service truststore contains a local CA and external CA certificates.

Client certificates

• A client certificate is a certificate that is used for validation of the HTTPS client. The client certificate of a Discovery Service client can be the same certificate as the server certificate of the services which the Discovery Service client uses.