Summary
This article will help you to adopt the best practices in the area of upgrading your server and terminal network with minimum down time and the other risks that might be involved.
Release versioning system
Our product comes with major releases and minor releases. Minor releases are also called patch releases.
Naming convention: yyyymmdd.pp example 20231012.20.
Where 20231012 is a major version and .20 is a patch level.
Major releases contain predominantly new functionality, database, architectural, and security improvements and may contain changes that introduce bugs.
Minor releases contain just bug fixes and are designed to be safe and easy for the operators to install. Patch releases are numbered in incremental order.
Release frequency
GENERAL BYTES typically publishes, on average, a new major release every month. Between these major releases, GB will publish patches whenever bugs are discovered. These are called minor, point, or patch releases.
Stay on a supported version
GENERAL BYTES actively maintains the current release and previous release, meaning that only these latest two major releases ever receive patch releases. The previous major release is then marked as EOL(End Of Life) on the patch releases page and becomes unsupported.
It is highly recommended to always have your production running on supported release. Risks involving not staying on supported version include but are not limited to sudden downtime due to 3rd party wallet provider API change or 0-day vulnerability discovery. GENERAL BYTES is able to provide fix for such situations via new patch release but only on supported releases.
Timing upgrade
Don’t upgrade your operation to release that has not been released publicly or has been announced yesterday. It may contain bugs. Let other operators to discover them if you don’t need to upgrade ASAP.
Don’t upgrade on Friday or Weekends. GB development and QA departments are not available on weekends when you hit the issue and new patch release needs to be released.
Don’t upgrade on Monday. GB development and QA departments is likely to have limited resources on Monday due to team meetings and accumulated issues that has been identified over the past weekend and need to be addressed with priority.
Don’t upgrade on US and Czech business holidays.
Don’t upgrade from 18th of Dec to 3rd of January when GB has limited resources due to Christmas season.
Upgrade
GB has perfected over the years following procedure on how operators should evaluate and install new releases with minimum risk of negative impact on their operation with upgrade. Following procedure is recommended at least for operators that have 100+ machines but smaller operator may adopt this procedure too.
Establish environments if not present already
Do not share database between any of the environments.
Production environment. Good news, you don’t have to create this environment as it already probably exists. This is where you business operation is happening. This is where your revenue is generated and where all sensitive information resides.
Development/Sandbox environment. Optional. Install server with different license key that your production is using and connect one of the machines to it. You may skip creation of this environment if you don’t have developers creating extensions for your server. This environment is typically used by developers testing their extensions.
Testing environment. Install server with different license key that your production is using and connect one of the machines to it. This is environment where you install new release first. Here you get familiar with new features and make sure that product operates correctly in your scenario with your configuration. Make sure the API keys for wallets are different than production.
Pre-production environment. This environment is created by cloning your production environment. This server shares licensing key with your production server. Connect one of the machines to it.
Make sure that you clone also database and it isn’t shared with your production environment. Otherwise you can destroy your database.
Make sure that you DISABLE on pre-production server publishing to HQ as it will otherwise report to coinatmradar.com that all of your terminals are offline. This can be achieved by running following statement in the pre-production database:
UPDATE terminalpublishsettings SET publishtohq = 0;
Don’t give access to this server to anyone who doesn’t already have access to your production server as it contains valuable data.
Upgrade Laws
Always upgrade server first and then upgrade terminals. Major releases may not be compatible with previous major release. Terminals with older release installed may not respond to commands sent from newer server.
Do not rollback the server installations or database. Installing new server version includes database table structure changes that are not compatible with previous server version.
It is OK to skip versions forward. You don’t need to install every release we make. Just install our latest release and our software will perform the changes that were introduced in previous releases such as database structure changes.
Upgrade your network in batches. After each batch evaluate that upgraded machines work correctly.
First upgrade 1 machine, then 20, then 50, then 100, then 500, then 1000, then 2000…
Upgrade Flow
Test new release on testing environment
Evaluate new features.
Make sure essential features work.
Perform test transactions on your machine.
Test that your extensions are compatible with this release and are working as expected.
When something seems to be broken contact GB tech support to fix the issue with patch release.
When everything works go to step 2.
Test upgrade on your pre-production environment.
Perform test transactions on your machine.
Perform upgrade and take notes of the steps you had to make and values you filled in.
Make sure essential features work and perform quickly - spot possible performance issues when listing terminals or identities.
When something seems to be broken contact GB tech support to fix the issue with patch release.
When everything works go to step 3.
Upgrade production environment
Choose a good date for your upgrade.
When your downtime will have minimum impact on your customers.
Not far from your previous pre-production upgrade so you remember well what steps you took and workarounds has been taken.
Use the notes from your dry-run on pre-production environment.
Follow upgrade laws
How new release gets manufactured
Any change to our software is associated with a development ticket containing description of a change and desired new behavior may it be a new feature or bug fix.
Every developer who implements the function tests the function himself and describes how feature should be tested.
Every change made by a developer undergoes peer review and automated tests before is merged into the product making sure that only quality, secure, standard-following, non-breaking code makes it into the product. At GB we develop new features into the main branch.
Every month we take what is in the main branch and create a new major release with patch level 1 and create Changelog draft. Every release undergoes automated testing involving multiple terminals running in emulators testing various configurations.
GB’s quality assurance(QA) department is testing every new change to the code ticket by ticket. Whenever QA decides that feature or bugfix needs improvement it is returned to the developer.
QA department is testing also not changed functionality making sure that essential functionality is not impacted.
Any correction to tested release results in production of new patch release that gets tested again.
Once testing is nearing competition (few tickets need to be tested) our Support department reviews Changelog draft and selects candidates for QA department. Candidates represent functionality that deserves to be documented as KB article. QA creates draft of the KB articles.
Support department reviews KB articles and improves the wording and structure making sure articles are easy to read, straight forward, unambiguous and quickly understood.
Once release has only a few bugs to be resolved it is deployed into GB’s beta environment representing small(15 terminals) ATM operation in Prague, Czech Republic. During this period the release is tested by real end-customers in real operation when additional issues are discovered and fixed.
When there are all issues resolved major release is introduced to all of the operators via Telegram channel and Changelog page.
Small operators tend to adopt new releases sooner than larger operators and discover additional bugs that get fixed in subsequent patch releases.
Publicly available supported major releases continue to receive the patch releases for their entire life until marked as and of life.