Master Service Issues

The Master Service

One of the three critical services that run on your hosted server, the "Master" service is in constant communication with your BATMs. All BATM transactions & events use this channel.

Symptom:

All of your BATMs report "Server Unreachable" or similar. None are online.

If only a single BATM is experiencing this symptom - and you operate multiple BATMs, the Master service is unlikely to be the problem. Please refer to these articles instead:

Is your account locked? Unpaid invoices may trigger an automatic lock if they exceed certain thresholds. Your CAS will shut down. Please check that your GB account is current.

How to Pay your GB Partner Account Invoice

The underlying cause of this type of problem may arise in any of the four outlined areas:

Confirm the symptom.

First, identify if the Master service is running.

Login to your CAS server console via SSH.
From your server's command line interface (CLI), type:

sudo /batm/batm-manage info

The status of the Master, Admin, and Gate services (running/stopped) should be displayed.

If the Master service is stopped, then the cause is usually explained in the Master log.

Investigate the Master log:

You should still be able to access the CAS GUI/Admin. You can easily view the Master log from there:

Navigate to: Log Viewer/Master Log
Click on "GO TO END OF FILE".
Examine the log for errors which might explain the problem (“FATAL”).
The Master log contains communication notes between your server and BATMs and contains a great deal of troubleshooting information, very little of it will be relevant to this symptom.

If the CAS GUI is also unavailable, then your Admin service is down, and you most likely have a much larger issue that cannot be resolved by the suggestions in this article.

See: Admin Service Issues

Start the Master service using the “fg” option.

The Master service can be started in verbose mode using this special command:

sudo /batm/batm-manage fg master

The Master service will then attempt to start - but it will display ALL messages onscreen - successes AND failures. Not all failures are unexpected or unwanted; they are merely reporting tools and the information is used for complex debugging.
Errors reported towards the end are the most relevant.

If the Master service is started without the “fg" switch (using “start"), then the Master log will be concatenated to any previous Master log, and can be examined live by typing:

sudo tail -f /var/log/batm/master.log

Try these possible solutions:

1 Reboot!
2 Check that you have enough free space on your drive:
3 Is your CPU or memory overloaded? Run:
4 Check your buy & sell rate sources.
5 Increase the Master service heap size.
6 Do you have a swap file?
7 Other possible causes may include…

Reboot!

Ubuntu should be updated regularly, and it will need to be restarted occasionally. This is easy to overlook. If the Master service is suddenly problematic, update Ubuntu and reboot your server.

sudo apt update && sudo apt dist-upgrade
sudo reboot

Check that you have enough free space on your drive:

sudo df

Should report at least 5gb free for the Master service to run reliably.

Is your CPU or memory overloaded? Run:

ps -ef -O %cpu,%mem

If the CPU or memory of any individual process exceeds 50%, you may need to increase your system specifications.
Usage surges (of either type) may result in random intermittent server reboots.
Memory depletion may result in the unpredictable termination of processes (killed).

Check your buy & sell rate sources.

The Master service may shut down if it is unable to communicate with your rate provider for a substantial period of time. This is for your protection - to prevent sales & purchases that may be considerably askew from actual current prices. Always check that your rate provider is reliable and functioning. This shutdown may occur intermittently!

Increase the Master service heap size.

This measure may reduce Out Of Memory crashes. This unusual measure requires you to edit the batm-manage file: /batm/batm-manage

Locate the line containing:

        eval "$MASTERD" java "-Djava.library.path=./lib $DEBUG_OPTIONS" -XX:+UseG1GC -XX:+UseStringDeduplication \
                -Dbatm.home=/batm -Dcoin.cz="$COIN_CZ" -Djava.security.egd=file:/dev/./urandom \
                -Dmaster.keystore=/batm/config/master.keystore -Dbatm.log.dir=/var/log/batm \
                -cp "$CLASSPATH" com.generalbytes.batm.server.ServerMain

Add the following (as instructed by Support) “-Xmx8g" in the following location:

eval "$MASTERD" java "-Djava.library.path=./lib $DEBUG_OPTIONS" -Xmx8g -XX:+UseG1GC -XX:+UseStringDeduplication \
                -Dbatm.home=/batm -Dcoin.cz="$COIN_CZ" -Djava.security.egd=file:/dev/./urandom \
                -Dmaster.keystore=/batm/config/master.keystore -Dbatm.log.dir=/var/log/batm \
                -cp "$CLASSPATH" com.generalbytes.batm.server.ServerMain

The setting may change depending upon your specific needs, and
the example provided is for CAS hosts with 16GB+ of RAM.

Do you have a swap file?

Digital Ocean droplets do not have a swap file (by default).

A swap file is used by Linux to move memory to the file system when cramped. Linux (and more especially MySQL) uses A LOT or memory, and in fact MySQL will use all available permitted memory. Even if you increase your RAM to 256GB, MySQL may use almost all of it - resulting in continued occasional crashes. The solution is a swap file.

Check to see if you have a swap file in place with:

sudo free

Create a swap file if none exists (if you see this):

To create a 2GB swap file (any size > 1GB on a 8GB system should suffice):

sudo fallocate -l 2G /swapfile
sudo chmod 600 /swapfile
sudo mkswap /swapfile
sudo swapon /swapfile

To make the change permanent, edit the /etc/fstab file and append /swapfile swap swap sw 0 0 to it:

sudo nano /etc/fstab

Press Ctrl+X and “Y” to save and exit the editor.

The swap file should already be active, but verify with: