1. Overview

As a Bitcoin ATM operator, you want to maximize your fleet’s uptime, even when performing necessary upgrades and maintenance. This article will help you to adopt the best practices in the area of upgrading your server and terminal network with minimum downtime and the other risks that might be involved.

2. Understand release versioning and frequency

Naming

Our product comes with major releases and minor releases. Minor releases are also called patch releases.

Major releases contain predominantly new functionality, database, architectural, and security improvements and may contain changes that introduce bugs.

Minor releases contain just bug fixes and are designed to be safe and easy for the operators to install. Patch releases are numbered in incremental order.

Release frequency

GENERAL BYTES typically publishes a new major release every month. Between these major releases, GB will publish patches whenever bugs are discovered. These are called minor, point, or patch releases.

3. Stay supported by GB

GENERAL BYTES actively maintains the current release and the prior release, meaning that only these latest two releases receive patches and are considered as supported versions. Releases older than these two get marked as EOL(End Of Life) on the patch releases page and they become unsupported.

It is highly recommended to always run your production server on a supported release. Risks arising from not staying on a supported version include; but are not limited to, sudden downtime due to 3rd party wallet provider API change or 0-day vulnerability discovery. GENERAL BYTES can provide a fix in such situations via a new patch, but only for currently supported releases.

4. Right time for upgrading production

Please time your upgrades to coincide with periods of maximum available GB support

• Don’t upgrade your operation to a release that has not been publicly released. It may contain bugs.

• Don’t upgrade on a Friday or a Weekend. If you hit an issue, GB development and QA departments are not available on weekends. Any new patch release won’t be released before the next working day.

• Don’t upgrade on a Monday. GB development and QA departments are frequently busy with team meetings or are responding to issues that arose over the weekend. Limited resources must be delegated amongst several items competing for priority.

• Don’t upgrade on US and Czech business holidays.

• Don’t upgrade over Christmas. The last two weeks of the year and the first days of the new year (December 18th to January 3rd) are when most GB employees partake in family-oriented vacations. The office is a ghost town.

• Do follow the patch releases page to know when a new patch version is released.

• Do let other operators discover bugs first (and GB fix them) if you don’t need to upgrade ASAP.

5. Upgrade Pillars

GB has perfected the following pillars over the years on how operators should evaluate and install new releases. Performing upgrades on these pillars will minimize the risk of negative impacts of any upgrade to your operation. Upgrade pillars are essential to the largest operators that have fleets of 200 or more machines. They are also highly recommended for operators with smaller fleets.

Pillar I: Establish environments if not present already

1. Production environment. The good news, you don’t have to create this environment as it probably already exists. This is where your business operation is happening. This is where your revenue is generated and where all sensitive information resides.

2. Development/Sandbox environment. Optional. Install this server with a different license key than the one your production is using and connect one of the machines to it.

   1. You may skip the creation of this environment if you don’t plan on creating extensions for your server.

   2. This environment is typically used by developers testing their extensions.

3. Testing environment. Install this server with a different license key than the one your production is using, and connect one of the machines to it.

   1. This is the environment where you install the new release first.

   2. Here you get familiar with new features and make sure that the product operates correctly in your scenario with your configuration.

   3. Make sure the API keys for wallets are different than production.

4. Pre-production environment. Created this environment by cloning your production one. This server shares the licensing key with your production server. Connect one of the machines to it.

   1. Be sure that you clone the database not share it with your production environment. Otherwise, you risk destroying your production database.

   2. Make sure that you DISABLE publishing to HQ on the pre-production server. Otherwise, it will report to coinatmradar.com that all of your terminals are offline. This can be achieved by running the following statement in the pre-production database:

      UPDATE terminalpublishsettings SET publishtohq = 0;

   3. Don’t give access to this server to anyone who doesn’t already have access to your production server as it contains valuable data.

Pillar II: Upgrade Laws

• SERVER FIRST always upgrade the server first, and then the terminals. Major releases may not be compatible with previous major release. Terminals with older release installed may not respond to commands sent from newer server.

• NO DOWNGRADES Don’t rollback the server version or database version. Installing a new server version includes database table structure changes that aren’t always compatible with previous versions. Just don’t do it.

• SKIPPING OK It is usually OK to skip versions forward. You don’t need to install every release we make. Just install our latest release and the software will perform the necessary changes from previous releases such as database structure changes.

• DO TERMINALS IN BATCHES Upgrade your network in batches. After each batch evaluate that upgraded machines work correctly. Batch growth explained: If 1 machine upgrade is a success, then do 20, then 50, then 100, then 500, then 1000, then 2000…

Pillar III: The Upgrade Flow

1. Test new release on testing environment

   1. Evaluate new features.

   2. Make sure essential features work.

   3. Perform test transactions on your machine.

   4. Test that your extensions are compatible with this release and are working as expected.

   5. When something seems to be broken contact GB tech support to fix the issue with patch release.

   6. When everything works go to step 2.

2. Test upgrade on your pre-production environment.

   1. Perform test transactions on your machine.

   2. Perform upgrade and take notes of the steps you had to make and values you filled in.

   3. Make sure essential features work and perform quickly - spot possible performance issues when listing terminals or identities.

   4. When something seems to be broken contact GB tech support to fix the issue with patch release.

   5. When everything works go to step 3.

3. Upgrade production environment

   1. Choose a good date for your upgrade.

      1. When your downtime will have minimum impact on your customers.

      2. Not far from your previous pre-production upgrade so you remember well what steps you took and workarounds has been taken.

   2. Use the notes from your dry-run on pre-production environment.

   3. Follow upgrade laws

6. How new release gets manufactured

1. Any change to our software is associated with a development ticket containing description of a change and desired new behavior may it be a new feature or bug fix.

2. Every developer who implements the function tests the function himself and describes how feature should be tested.

3. Every change made by a developer undergoes peer review and automated tests before is merged into the product making sure that only quality, secure, standard-following, non-breaking code makes it into the product. At GB we develop new features into the main branch.

4. Every month we take what is in the main branch and create a new major release with patch level 1 and create Changelog draft. Every release undergoes automated testing involving multiple terminals running in emulators testing various configurations.

5. GB’s quality assurance(QA) department is testing every new change to the code ticket by ticket. Whenever QA decides that feature or bugfix needs improvement it is returned to the developer.

6. QA department is testing also not changed functionality making sure that essential functionality is not impacted.

7. Any correction to tested release results in production of new patch release that gets tested again.

8. Once testing is nearing competition (few tickets need to be tested) our Support department reviews Changelog draft and selects candidates for QA department. Candidates represent functionality that deserves to be documented as KB article. QA creates draft of the KB articles.

9. Support department reviews KB articles and improves the wording and structure making sure articles are easy to read, straight forward, unambiguous and quickly understood.

10. Once release has only a few bugs to be resolved it is deployed into GB’s beta environment representing small(15 terminals) ATM operation in Prague, Czech Republic. During this period the release is tested by real end-customers in real operation when additional issues are discovered and fixed.

11. When there are all issues resolved major release is introduced to all of the operators via Telegram channel and Changelog page.

12. Small operators tend to adopt new releases sooner than larger operators and discover additional bugs that get fixed in subsequent patch releases.

13. Publicly available supported major releases continue to receive the patch releases for their entire life until marked as and of life.