WildFly 32+ and JBoss EAP 8

The following sections provide information on configuring your application server.

The application server-specific information for WildFly 32+ is also applicable for JBoss EAP 8.

Download WildFly

WildFly can either be downloaded as a zip package or customized using Galleon. Keyfactor recommends using Galleon, since it only downloads the components you need, as opposed to using the zip package, which bundles all configurations and JBoss modules, as SignServer is only using a subset of all functionality in WildFly.

WildFly is usually put in the /opt folder on Linux systems, and using a symlink to the WildFly folder makes it easy to switch to a new version of WildFly by simply overriding the symlink.

Using the Zip Package

You need to use the Jakarta EE Full & Web Distribution package of WildFly when downloading the zip package. If you have not downloaded WildFly already, you can use the following command to do so:

Download and unpack the WildFly zip package

CODE

wget https://download.jboss.org/wildfly/38.0.0.Final/wildfly-38.0.0.Final.zip -O /tmp/wildfly-38.0.0.Final.zip
unzip -q /tmp/wildfly-38.0.0.Final.zip -d /opt/
ln -snf /opt/wildfly-38.0.0.Final /opt/wildfly

Using Galleon

Instead of downloading the WildFly zip package containing everything (all default server configurations and all JBoss modules), using Galleon tooling you can choose to only install the components needed by SignServer.

Download Galleon

Galleon is provided as a zip package from GitHub which you can download and run without any installation:

CODE

wget https://github.com/wildfly/galleon/releases/download/4.2.8.Final/galleon-4.2.8.Final.zip -O /tmp/galleon-4.2.8.Final.zip
unzip -q /tmp/galleon-4.2.8.Final.zip -d /tmp/
cd /tmp/galleon-4.2.8.Final/bin

Layers Used By SignServer

The table below lists the Galleon layers used by SignServer.

The core-tools layer will include the JBoss and Elytron CLI which comes in handy if the Elytron credential store or the standalone.xml configuration file need to be tweaked later. If you do not need this, exclude the core-tools and management layers and add elytron separately. bean-validation can be excluded as well, but is good to have, at least in staging environments. Picketbox-based web security is required for SignServer to detect authentication using client certificates.

Name	Description	Dependencies
cdi	Support for Jakarta Contexts and Dependency Injection.	base-server bean-validation (optional)
core-tools	Support for jboss-cli, add-user and elytron-tool launch scripts and configuration files.	management (optional)
datasources	Support for datasources.	transactions
deployment-scanner	Support for deployment directory scanning.	base-server
discovery	Support for discovery.	base-server
ee	Support for common functionality in the Jakarta EE platform.	jsonb (optional) naming
ejb	Support for Jakarta Enterprise Beans, excluding the IIOP protocol.	ejb-lite messaging-activemq remoting resource-adapters undertow
elytron-oidc-client	Support for OIDC.	undertow elytron
io	Support for XNIO workers and buffer pools.	base-server
jaxrs	Support for JAXRS.	web-server
jpa	Support for JPA (using the latest WildFly supported Hibernate release). Alternative: jpa-distributed	bean-validation (optional) datasources
jsf	Support for Jakarta Server Faces.	bean-validation (optional) cdi (optional) web-server
logging	Support for the logging subsystem.	base-server
mail	Support for Jakarta Mail.	base-server naming
management	Support for remote access to management interfaces secured using Elytron.	elytron
webservices	Support for Jakarta XML Web Services	ejb-lite (optional) messaging-activemq (optional) web-server
legacy-security	Support for legacy Picketbox-based web security.	naming vault

Download and Configure WildFly Using Galleon

Galleon can download and extract WildFly to the directory specified using the install command. SignServer is running on a standalone server, and only this server configuration is needed. For a typical installation, use the following CLI recipe:

CODE

./galleon.sh install wildfly:current#38.0.0.Final --dir=/opt/wildfly-38.0.0.Final --default-configs=standalone/standalone.xml --layers=cdi,core-tools,datasources,deployment-scanner,discovery,ee,-jsonb,ejb,elytron-oidc-client,io,jaxrs,jpa,jsf,logging,mail,management,webservices
ln -snf /opt/wildfly-38.0.0.Final /opt/wildfly

Remove RESTEasy-Crypto

The application server can sometimes load its own version of Bouncy Castle, resulting in incompatibility and/or conflict issues.

A typical error occurring as a consequence of the wrong Bouncy Castle being loaded in SignServer is:

ClassCastException setting BagAttributes, can not set friendly name: : java.lang.ClassCastException: org.bouncycastle.jcajce.provider.asymmetric.x509.X509CertificateObject cannot be cast to org.bouncycastle.jce.interfaces.PKCS12BagAttributeCarrier

To avoid issues and prevent WildFly from loading the Bouncy Castle library included in the RESTEasy-Crypto module, rather than the library included in SignServer, run the following:

CODE

sed -i '/.*org.jboss.resteasy.resteasy-crypto.*/d' /opt/wildfly/modules/system/layers/base/org/jboss/as/jaxrs/main/module.xml
rm -rf /opt/wildfly/modules/system/layers/base/org/jboss/resteasy/resteasy-crypto/

Create a Custom Configuration

Replace /opt/wildfly/bin/standalone.conf with the following Jinja2 template.

/opt/wildfly/bin/standalone.conf

CODE

if [ "x$JBOSS_MODULES_SYSTEM_PKGS" = "x" ]; then
     JBOSS_MODULES_SYSTEM_PKGS="org.jboss.byteman"
fi
if [ "x$JAVA_OPTS" = "x" ]; then
     JAVA_OPTS="-Xms{{ HEAP_SIZE }}m -Xmx{{ HEAP_SIZE }}m -XX:MetaspaceSize=96M -XX:MaxMetaspaceSize=256m"
     JAVA_OPTS="$JAVA_OPTS -Dhttps.protocols=TLSv1.2,TLSv1.3"
     JAVA_OPTS="$JAVA_OPTS -Djdk.tls.client.protocols=TLSv1.2,TLSv1.3"
     JAVA_OPTS="$JAVA_OPTS -Djava.net.preferIPv4Stack=true"
     JAVA_OPTS="$JAVA_OPTS -Djboss.modules.system.pkgs=$JBOSS_MODULES_SYSTEM_PKGS"
     JAVA_OPTS="$JAVA_OPTS -Djava.awt.headless=true"
     JAVA_OPTS="$JAVA_OPTS -Djboss.tx.node.id={{ TX_NODE_ID }}"
     JAVA_OPTS="$JAVA_OPTS -XX:+HeapDumpOnOutOfMemoryError"
     JAVA_OPTS="$JAVA_OPTS -Djdk.tls.ephemeralDHKeySize=2048"
else
     echo "JAVA_OPTS already set in environment; overriding default settings with values: $JAVA_OPTS"
fi

Set Allowed Memory Usage

By default, 512 MB of heap (RAM) is allowed to be used by the application server. This is not sufficient to run SignServer. We recommend to allocate at least 2048 MB of RAM. To increase the default value, run the following command:

CODE

sed -i -e 's/{{ HEAP_SIZE }}/2048/g' /opt/wildfly/bin/standalone.conf

Set the Transaction Node ID

Set the transaction node ID to a unique number. The node ID is used by the transactions subsystem and ensures that the transaction manager only recovers branches which match the specified identifier. It is imperative that this identifier is unique between WildFly instances sharing either an object store or access common resource managers (i.e. when multiple SignServer instances access the same database).

CODE

sed -i -e "s/{{ TX_NODE_ID }}/$(od -A n -t d -N 1 /dev/urandom | tr -d ' ')/g" /opt/wildfly/bin/standalone.conf

Optional: Configure WildFly as a Service

Modern Linux systems use systemd to start and stop services. The WildFly zip package already contains the necessary files to run as a service, but they need to be installed manually. Once started as a service, WildFly will run as the wildfly user, and you need to add this user as well.

The following provides a suggestion for setting up WildFly as a systemd service. Note that the example may need to be adapted according to the system you are using.

CODE

cp /opt/wildfly/docs/contrib/scripts/systemd/launch.sh /opt/wildfly/bin
cp /opt/wildfly/docs/contrib/scripts/systemd/wildfly.service /etc/systemd/system
mkdir /etc/wildfly
cp /opt/wildfly/docs/contrib/scripts/systemd/wildfly.conf /etc/wildfly
systemctl daemon-reload
useradd -r -s /bin/false wildfly
chown -R wildfly:wildfly /opt/wildfly-38.0.0.Final/

The docs directory does not exist when using Galleon. For reference, this is what the missing files look like:

/opt/wildfly/docs/contrib/scripts/systemd/launch.sh

CODE

#!/bin/bash
if [ "x$WILDFLY_HOME" = "x" ]; then
    WILDFLY_HOME="/opt/wildfly"
fi
if [[ "$1" == "domain" ]]; then
    $WILDFLY_HOME/bin/domain.sh -c $2 -b $3
else
    $WILDFLY_HOME/bin/standalone.sh -c $2 -b $3
fi

/opt/wildfly/docs/contrib/scripts/systemd/wildfly.service

CODE

[Unit]
Description=The WildFly Application Server
After=syslog.target network.target
Before=httpd.service
[Service]
Environment=LAUNCH_JBOSS_IN_BACKGROUND=1
EnvironmentFile=-/etc/wildfly/wildfly.conf
User=wildfly
LimitNOFILE=102642
PIDFile=/var/run/wildfly/wildfly.pid
ExecStart=/opt/wildfly/bin/launch.sh $WILDFLY_MODE $WILDFLY_CONFIG $WILDFLY_BIND
StandardOutput=null
[Install]
WantedBy=multi-user.target

/opt/wildfly/docs/contrib/scripts/systemd/wildfly.conf

CODE

# The configuration you want to run
WILDFLY_CONFIG=standalone.xml
# The mode you want to run
WILDFLY_MODE=standalone
# The address to bind to
WILDFLY_BIND=0.0.0.0

Start WildFly

The following provides two examples for starting WildFly, one if you are using a systemd service in the background and alternatively for starting WildFly in a separate terminal for testing.

Start WildFly installed as a systemd service

To start WildFly installed as a systemd service:

CODE

systemctl start wildfly

Start WildFly in a separate terminal for testing

Alternatively, start WildFly in a separate terminal for quick testing:

CODE

/opt/wildfly/bin/standalone.sh

Create an Elytron Credential Store

You can protect passwords by storing them in a credential store. The credential is encrypted with a master password which is fetched by WildFly on startup.

Create a Master Password

Create a script which outputs the master password to stdout and ensure the script can only be executed by the wildfly user.

CODE

echo '#!/bin/sh' > /usr/bin/wildfly_pass
echo "echo '$(openssl rand -base64 24)'" >> /usr/bin/wildfly_pass
chown wildfly:wildfly /usr/bin/wildfly_pass
chmod 700 /usr/bin/wildfly_pass

Create the Credential Store

Create a credential store in /opt/wildfly/standalone/configuration encrypted with the password echoed by the wildfly_pass script.

CODE

mkdir /opt/wildfly/standalone/configuration/keystore
chown wildfly:wildfly /opt/wildfly/standalone/configuration/keystore
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=elytron/credential-store=defaultCS:add(location=keystore/credentials, relative-to=jboss.server.config.dir, credential-reference={clear-text="{EXT}/usr/bin/wildfly_pass", type="COMMAND"}, create=true)'

Add Database Driver

For most database management systems, the JDBC driver can be added by hot-deploying it into the deployment directory. This will be picked up by WildFly and deployed so we can create a data source straight away. You can use a generic name, without version number, in order to get a generic driver-name for the data source command.

MariaDB

Download and deploy MariaDB JDBC driver

CODE

wget https://dlm.mariadb.com/3852266/Connectors/java/connector-java-3.4.1/mariadb-java-client-3.4.1.jar -O /opt/wildfly/standalone/deployments/mariadb-java-client.jar

PostgreSQL

Download and deploy PostgreSQL JDBC driver

CODE

wget https://jdbc.postgresql.org/download/postgresql-42.7.4.jar -O /opt/wildfly/standalone/deployments/postgresql-jdbc4.jar

Microsoft SQL Server

Download and deploy Microsoft SQL Server driver

CODE

wget https://github.com/microsoft/mssql-jdbc/releases/download/v12.4.2/mssql-jdbc-12.4.2.jre11.jar -O /opt/wildfly/standalone/deployments/mssql-jdbc.jre11.jar

Oracle DB

Download and deploy Oracle DB driver

CODE

wget https://download.oracle.com/otn-pub/otn_software/jdbc/235/ojdbc11.jar -O /opt/wildfly/standalone/deployments/oracledb-jdbc11.jar

Add a Datasource

To add a datasource for SignServer to use, run the commands in JBoss CLI according to the examples below.

Note that --driver-name is should be the same as the filename of the JAR file you copied in the step above.

The --jndi-name is is defined by the property datasource.jndi-name in database.properties, and the default value is used in the following example.

The --user-name and --password defines the username and password of the SignServer database user.

MariaDB

For MariaDB, run the following:

CODE

/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=elytron/credential-store=defaultCS:add-alias(alias=dbPassword, secret-value="signserver")'
/opt/wildfly/bin/jboss-cli.sh --connect 'data-source add --name=signserverds --connection-url="jdbc:mysql://127.0.0.1:3306/signserver" --jndi-name="java:/SignServerDS" --use-ccm=true --driver-name="mariadb-java-client.jar" --driver-class="org.mariadb.jdbc.Driver" --user-name="signserver" --credential-reference={store=defaultCS, alias=dbPassword} --validate-on-match=true --background-validation=false --prepared-statements-cache-size=50 --share-prepared-statements=true --min-pool-size=5 --max-pool-size=150 --pool-prefill=true --transaction-isolation=TRANSACTION_READ_COMMITTED --check-valid-connection-sql="select 1;"'
/opt/wildfly/bin/jboss-cli.sh --connect ':reload'

Wait for the reload to complete by checking the server log or the result of :read-attribute(name=server-state) before continuing.

PostgreSQL

For PostgreSQL, run the following:

CODE

/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=elytron/credential-store=defaultCS:add-alias(alias=dbPassword, secret-value="signserver")'
/opt/wildfly/bin/jboss-cli.sh --connect 'data-source add --name=signserverds --connection-url="jdbc:postgresql://127.0.0.1/signserver" --jndi-name="java:/SignServerDS" --use-ccm=true --driver-name="postgresql-jdbc4.jar" --driver-class="org.postgresql.Driver" --user-name="signserver" --credential-reference={store=defaultCS, alias=dbPassword} --validate-on-match=true --background-validation=false --prepared-statements-cache-size=50 --share-prepared-statements=true --min-pool-size=5 --max-pool-size=150 --pool-prefill=true --transaction-isolation=TRANSACTION_READ_COMMITTED --check-valid-connection-sql="select 1;"'
/opt/wildfly/bin/jboss-cli.sh --connect ':reload'

Wait for the reload to complete by checking the server log or the result of :read-attribute(name=server-state) before continuing.

Microsoft SQL Server

For Microsoft SQL Server, run the following:

CODE

/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=elytron/credential-store=defaultCS:add-alias(alias=dbPassword, secret-value="signserver")'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=datasources/data-source=signserverds:add(connection-url="jdbc:sqlserver://foobar.YOUR.DOMAIN:1433;DatabaseName=signserver;encrypt=true;trustServerCertificate=false;hostNameInCertificate=*.database.windows.net;loginTimeout=30;sendStringParametersAsUnicode=false", min-pool-size=5, max-pool-size=150, jndi-name="java:/SignServerDS", driver-name=mssql-jdbc.jre11.jar, user-name="signserver", credential-reference={store=defaultCS, alias=dbPassword}, pool-prefill=false, pool-use-strict-min=false, idle-timeout-minutes=2)'
/opt/wildfly/bin/jboss-cli.sh --connect ':reload'

Wait for the reload to complete by checking the server log or the result of :read-attribute(name=server-state) before continuing.

Oracle DB

For Oracle DB, run the following:

CODE

/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=elytron/credential-store=defaultCS:add-alias(alias=dbPassword, secret-value="signserver")'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=datasources/data-source=signserverds:add(connection-url="jdbc:oracle:thin:@//YOUR.DOMAIN:1521/signserver", min-pool-size=5, max-pool-size=150, jndi-name="java:/SignServerDS", driver-name=oracle-jdbc11.jar, user-name="signserver", credential-reference={store=defaultCS, alias=dbPassword}, pool-prefill=false, pool-use-strict-min=false, idle-timeout-minutes=2)'
/opt/wildfly/bin/jboss-cli.sh --connect ':reload'

Wait for the reload to complete by checking the server log or the result of :read-attribute(name=server-state) before continuing.

Configure WildFly Remoting

SignServer needs to use JBoss Remoting for the SignServer Admin CLI to work. Configure it to use a separate port 4447 and remove any other dependency on remoting except for what SignServer needs.

CODE

/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=remoting/http-connector=http-remoting-connector:write-attribute(name=connector-ref,value=remoting)'
/opt/wildfly/bin/jboss-cli.sh --connect '/socket-binding-group=standard-sockets/socket-binding=remoting:add(port=4447,interface=management)'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=undertow/server=default-server/http-listener=remoting:add(socket-binding=remoting,enable-http2=true)'
/opt/wildfly/bin/jboss-cli.sh --connect ':reload'

Wait for the reload to complete by checking the server log or the result of :read-attribute(name=server-state) before continuing.

Configure Logging

Configure logging in WildFly to be able to dynamically change logging while the application server is running.

Logging Configurations

Choose one of the logging configurations below.

Option 1 - Recommended Logging

INFO log level for org.signserver and org.cesecore etc. is recommended for production systems.

CODE

/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=logging/logger=org.signserver:add(level=INFO)'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=logging/logger=org.cesecore:add(level=INFO)'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=logging/logger=org.ejbca:add(level=INFO)'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=logging/logger=com.keyfactor:add(level=INFO)'

Option 2 - Quiet Logging

If you prefer more quiet logging, configure WildFly to only log audit log messages, warnings and errors.

CODE

/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=logging/logger=org.cesecore.audit.impl.log4j.Log4jDevice:add(level=INFO)'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=logging/logger=org.signserver:add(level=WARN)'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=logging/logger=org.cesecore:add(level=WARN)'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=logging/logger=org.ejbca:add(level=WARN)'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=logging/logger=com.keyfactor:add(level=WARN)'

Additional Logging Configuration

You may additionally want to add the following configuration:

CODE

/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=logging/logger=org.jboss.as.config:write-attribute(name=level, value=WARN)'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=logging/logger=org.jboss.as:add(level=WARN)'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=logging/logger=org.wildfly:add(level=WARN)'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=logging/logger=org.xnio:add(level=WARN)'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=logging/logger=org.hibernate:add(level=WARN)'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=logging/logger=org.apache.cxf:add(level=WARN)'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=logging/logger=org.cesecore.config.ConfigurationHolder:add(level=WARN)'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=logging/logger=org.infinispan:add(level=WARN)'

Add Access Logging

To log all requests processed by the server, similar to the Apache access log, add the following configuration:

CODE

/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=undertow/server=default-server/host=default-host/setting=access-log:add(pattern="%h %t \"%r\" %s \"%{i,User-Agent}\"", relative-to=jboss.server.log.dir, directory=access-logs)'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=logging/logger=io.undertow.accesslog:add(level=INFO)'

Remove the Console Handler

Console logging is not used when running WildFly with systemd, and removing it can increase logging performance.

CODE

/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=logging/root-logger=ROOT:remove-handler(name=CONSOLE)'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=logging/console-handler=CONSOLE:remove()'

Remove Old Log Files

Log files are rotated on a daily basis by default, but old log files are not deleted automatically. You can delete old log files using a cronjob:

/etc/cron.daily/remove-old-wildfly-logs.sh

CODE

#!/bin/sh
# Remove log files older than 7 days
find /opt/wildfly/standalone/log/ -type f -mtime +7 -name 'server.log*' -execdir rm -- '{}' \;

Make the file executable:

CODE

chmod +x /etc/cron.daily/remove-old-wildfly-logs.sh

Enable Syslog Shipping

To additionally send the server log over UDP to a syslog server such as Graylog or Logstash, configure a syslog handler in WildFly. Set the hostname and port of the syslog server, as well as the desired log level according to the following example:

CODE

/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=logging/json-formatter=logstash:add(exception-output-type=formatted, key-overrides=[timestamp="@timestamp"],meta-data=[@version=1])'
/opt/wildfly/bin/jboss-cli.sh --connect "/subsystem=logging/syslog-handler=syslog-shipping:add(app-name=SignServer,enabled=true,facility=local-use-0,hostname=$(hostname -f),level=INFO,named-formatter=logstash,port=514,server-address=syslog.server,syslog-format=RFC5424)"
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=logging/root-logger=ROOT:add-handler(name=syslog-shipping)'

Enable Audit Logging To File

You can write the SignServer audit log to a separate file. E.g. to log to /opt/wildfly/standalone/log/cesecore-audit.log, rotate every 128 MB and keep one rotated file:

CODE

/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=logging/size-rotating-file-handler=cesecore-audit-log:add(file={path=cesecore-audit.log, relative-to=jboss.server.log.dir}, max-backup-index=1, rotate-size=128m)'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=logging/logger=org.cesecore.audit.impl.log4j.Log4jDevice:add'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=logging/logger=org.cesecore.audit.impl.log4j.Log4jDevice:add-handler(name=cesecore-audit-log)'

HTTP(S) Configuration

The following section explains how to configure HTTP(S) using Undertow.

Remove Existing TLS and HTTP Configuration

Run the following commands in JBoss CLI to remove existing TLS and HTTP configuration:

CODE

/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=undertow/server=default-server/http-listener=default:remove()'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=undertow/server=default-server/https-listener=https:remove()'
/opt/wildfly/bin/jboss-cli.sh --connect '/socket-binding-group=standard-sockets/socket-binding=http:remove()'
/opt/wildfly/bin/jboss-cli.sh --connect '/socket-binding-group=standard-sockets/socket-binding=https:remove()'
# These two lines are not needed if Galleon was used
/opt/wildfly/bin/jboss-cli.sh --connect '/core-service=management/security-realm=ApplicationRealm/server-identity=ssl:remove()'
/opt/wildfly/bin/jboss-cli.sh --connect ':reload'

Wait for the reload to complete by checking the server log or the result of :read-attribute(name=server-state) before continuing.

Use 3-Port Separation

The following section explains how to set up Undertow with 3-port separation. Port 8080 is used for HTTP (unencrypted traffic), port 8442 for HTTPS (encrypted) traffic with only server authentication and port 8443 for HTTPS (encrypted) traffic with both server and client authentication.

Add New Interfaces and Sockets

To add new interfaces and sockets, use the following:

Consider binding to a specific interface instead of binding to all interfaces using 0.0.0.0.

CODE

/opt/wildfly/bin/jboss-cli.sh --connect '/interface=http:add(inet-address="0.0.0.0")'
/opt/wildfly/bin/jboss-cli.sh --connect '/interface=httpspub:add(inet-address="0.0.0.0")'
/opt/wildfly/bin/jboss-cli.sh --connect '/interface=httpspriv:add(inet-address="0.0.0.0")'
/opt/wildfly/bin/jboss-cli.sh --connect '/socket-binding-group=standard-sockets/socket-binding=http:add(port="8080",interface="http")'
/opt/wildfly/bin/jboss-cli.sh --connect '/socket-binding-group=standard-sockets/socket-binding=httpspub:add(port="8442",interface="httpspub")'
/opt/wildfly/bin/jboss-cli.sh --connect '/socket-binding-group=standard-sockets/socket-binding=httpspriv:add(port="8443",interface="httpspriv")'

Configure TLS

Configure TLS according to the following instructions.

Make sure the password of the keystore and truststore in this section are correct, in order for the deployment not to fail. In production the passwords should be changed to "real" passwords.

Note that TLS v1.3 is only available when running JDK 11 or greater.

CODE

/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=elytron/credential-store=defaultCS:add-alias(alias=httpsKeystorePassword, secret-value="serverpwd")'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=elytron/credential-store=defaultCS:add-alias(alias=httpsTruststorePassword, secret-value="changeit")'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=elytron/key-store=httpsKS:add(path="keystore/keystore.jks",relative-to=jboss.server.config.dir,credential-reference={store=defaultCS, alias=httpsKeystorePassword},type=JKS)'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=elytron/key-store=httpsTS:add(path="keystore/truststore.jks",relative-to=jboss.server.config.dir,credential-reference={store=defaultCS, alias=httpsTruststorePassword},type=JKS)'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=elytron/key-manager=httpsKM:add(key-store=httpsKS,algorithm="SunX509",credential-reference={store=defaultCS, alias=httpsKeystorePassword})'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=elytron/trust-manager=httpsTM:add(key-store=httpsTS)'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=elytron/server-ssl-context=httpspub:add(key-manager=httpsKM,protocols=["TLSv1.3","TLSv1.2"],use-cipher-suites-order=false,cipher-suite-filter="TLS_DHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256",cipher-suite-names="TLS_AES_256_GCM_SHA384:TLS_AES_128_GCM_SHA256")'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=elytron/server-ssl-context=httpspriv:add(key-manager=httpsKM,protocols=["TLSv1.3","TLSv1.2"],use-cipher-suites-order=false,cipher-suite-filter="TLS_DHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256",cipher-suite-names="TLS_AES_256_GCM_SHA384:TLS_AES_128_GCM_SHA256:TLS_CHACHA20_POLY1305_SHA256",trust-manager=httpsTM,need-client-auth=true)'

Add HTTP(S) Listeners

To add HTTP(S) listeners:

CODE

/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=undertow/server=default-server/http-listener=http:add(socket-binding="http", redirect-socket="httpspriv")'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=undertow/server=default-server/https-listener=httpspub:add(socket-binding="httpspub", ssl-context="httpspub", max-parameters=2048)'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=undertow/server=default-server/https-listener=httpspriv:add(socket-binding="httpspriv", ssl-context="httpspriv", max-parameters=2048)'
/opt/wildfly/bin/jboss-cli.sh --connect ':reload'

Wait for the reload to complete by checking the server log or the result of :read-attribute(name=server-state) before continuing.

Configure the Firewall

Open port 8080, 8442 and 8443 for incoming TCP traffic.

Open ports in RHEL / CentOS

CODE

systemctl enable firewalld --now
firewall-cmd --set-default-zone=dmz
firewall-cmd --zone=dmz --permanent --add-port 8080/tcp
firewall-cmd --zone=dmz --permanent --add-port 8442/tcp
firewall-cmd --zone=dmz --permanent --add-port 8443/tcp
firewall-cmd --reload

Use 2-Port Separation

Install SignServer.

Debug SignServer

The following explains the configuration options in WildFly for troubleshooting SignServer.

Switch to Debug Logging Globally

To enable debug logging globally, if you need to troubleshoot, follow these steps.

Make a note of the current log level for the packages org.cesecore and org.signserver.

CODE

/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=logging/logger=org.cesecore:read-attribute(name=level)'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=logging/logger=org.signserver:read-attribute(name=level)'

Switch to debug logging.

CODE

/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=logging/logger=org.signserver:write-attribute(name=level, value=DEBUG)'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=logging/logger=org.cesecore:write-attribute(name=level, value=DEBUG)'

Once you are done troubelshooting, switch back to the log level you used previously. Note that having debug logging enabled globally will produce a lot of output, avoid using this configuration in production and consider enabling debugging for individual packages instead.

Log Requests and Responses for the WebService API

To log the SOAP messages received by and sent from SignServer:

CODE

/opt/wildfly/bin/jboss-cli.sh --connect '/system-property=org.apache.cxf.logging.enabled:add(value=true)'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=logging/logger=org.apache.cxf.services:add(level=INFO)'
systemctl restart wildfly

To remove the configuration when you are done:

CODE

/opt/wildfly/bin/jboss-cli.sh --connect '/system-property=org.apache.cxf.logging.enabled:remove'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=logging/logger=org.apache.cxf.services:remove'
systemctl restart wildfly

Enable Hibernate Statistics

Hibernate is the ORM library used by SignServer to create SQL queries. You can enable Hibernate statistics to get the SQL query, as well as the time it takes to execute it, written in the server log.

Hibernate statistics are quite verbose and there may be database-specific tools better suited to troubleshoot database performance, however if you want to enable it:

CODE

/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=datasources/data-source=signserverds/statistics=pool:write-attribute(name=statistics-enabled,value=true)'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=datasources/data-source=signserverds/statistics=jdbc:write-attribute(name=statistics-enabled,value=true)'
/opt/wildfly/bin/jboss-cli.sh --connect '/system-property=hibernate.generate_statistics:add(value=true)'
/opt/wildfly/bin/jboss-cli.sh --connect '/system-property=hibernate.show_sql:add(value=true)'
/opt/wildfly/bin/jboss-cli.sh --connect '/system-property=hibernate.format_sql:add(value=true)'
/opt/wildfly/bin/jboss-cli.sh --connect '/system-property=hibernate.use_sql_comments:add(value=true)'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=logging/logger=org.hibernate.stat:add(level=DEBUG)'
systemctl restart wildfly

To remove the configuration:

CODE

/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=logging/logger=org.hibernate.stat:remove'
/opt/wildfly/bin/jboss-cli.sh --connect '/system-property=hibernate.use_sql_comments:remove'
/opt/wildfly/bin/jboss-cli.sh --connect '/system-property=hibernate.format_sql:remove'
/opt/wildfly/bin/jboss-cli.sh --connect '/system-property=hibernate.show_sql:remove'
/opt/wildfly/bin/jboss-cli.sh --connect '/system-property=hibernate.generate_statistics:remove'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=datasources/data-source=signserverds/statistics=jdbc:write-attribute(name=statistics-enabled,value=false)'
/opt/wildfly/bin/jboss-cli.sh --connect '/subsystem=datasources/data-source=signserverds/statistics=pool:write-attribute(name=statistics-enabled,value=false)'
systemctl restart wildfly

Enable Profiling Using Glowroot

Download Glowroot and put it in your WildFly directory.

CODE

wget https://github.com/glowroot/glowroot/releases/download/v0.13.6/glowroot-0.13.6-dist.zip -O /tmp/glowroot.zip
unzip -q /tmp/glowroot.zip -d /opt/wildfly
chown -R wildfly:wildfly /opt/wildfly/glowroot

Register Glowroot as a Java agent and restart WildFly.

CODE

sed -i '/-Djdk.tls.ephemeralDHKeySize=2048/ a \ \ \ JAVA_OPTS=\"$JAVA_OPTS -javaagent:/opt/wildfly/glowroot/glowroot.jar"' /opt/wildfly/bin/standalone.conf
systemctl restart wildfly

Glowroot should now be available on http://localhost:4000.