Summary
This article will help you to adopt the best practices in the area of upgrading your server and terminal network with minimum downtime and the other risks that might be involved.
Release versioning system
Our product comes with major releases and minor releases. Minor releases are also called patch releases.
Naming convention: yyyymmdd.pp example 20231012.20.
Where 20231012 is a major version and .20 is a patch level.
Major releases contain predominantly new functionality, database, architectural, and security improvements and may contain changes that introduce bugs.
Minor releases contain just bug fixes and are designed to be safe and easy for the operators to install. Patch releases are numbered in incremental order.
Release frequency
GENERAL BYTES typically publishes, on average, a new major release every month. Between these major releases, GB will publish patches whenever bugs are discovered. These are called minor, point, or patch releases.
Stay on a supported version
GENERAL BYTES actively maintains the current release and previous release, meaning that only these latest two major releases ever receive patch releases. The previous major release is then marked as EOL(End Of Life) on the patch releases page and becomes unsupported.
It is highly recommended to always run your production server on a supported release. Risks arising from not staying on a supported version include; but are not limited to, sudden downtime due to 3rd party wallet provider API change or 0-day vulnerability discovery. GENERAL BYTES can provide a fix in such situations via a new patch, but only for currently supported releases.
Dos and Don’ts: timing your upgrade
Do follow the patch releases page to know when a new version is released.
Don’t upgrade your operation to a release that has not been publicly released. It may contain bugs.
Do let other operators discover them first (and us fix them) if you don’t need to upgrade ASAP.
Don’t upgrade on a Friday or a Weekend. If you hit an issue, GB development and QA departments are not available on weekends. Any new patch release won’t be released before the next working day.
Don’t upgrade on a Monday. GB development and QA departments are frequently busy with team meetings or are responding to issues that arose over the weekend. Limited resources must be delegated amongst several items competing for priority.
Don’t upgrade on US and Czech business holidays.
Don’t upgrade over Christmas. The last two weeks of the year and the first days of the new year (December 18th to January 3rd) are when most GB employees partake in family-oriented vacations. The office is a ghost town.
Upgrade
GB has perfected over the years following procedure on how operators should evaluate and install new releases with minimum risk of negative impact on their operation with upgrade. Following procedure is recommended at least for operators that have 100+ machines but smaller operator may adopt this procedure too.
Establish environments if not present already
Do not share database between any of the environments.
Production environment. Good news, you don’t have to create this environment as it already probably exists. This is where you business operation is happening. This is where your revenue is generated and where all sensitive information resides.
Development/Sandbox environment. Optional. Install server with different license key that your production is using and connect one of the machines to it. You may skip creation of this environment if you don’t have developers creating extensions for your server. This environment is typically used by developers testing their extensions.
Testing environment. Install server with different license key that your production is using and connect one of the machines to it. This is environment where you install new release first. Here you get familiar with new features and make sure that product operates correctly in your scenario with your configuration. Make sure the API keys for wallets are different than production.
Pre-production environment. This environment is created by cloning your production environment. This server shares licensing key with your production server. Connect one of the machines to it.
Make sure that you clone also database and it isn’t shared with your production environment. Otherwise you can destroy your database.
Make sure that you DISABLE on pre-production server publishing to HQ as it will otherwise report to coinatmradar.com that all of your terminals are offline. This can be achieved by running following statement in the pre-production database:
UPDATE terminalpublishsettings SET publishtohq = 0;
Don’t give access to this server to anyone who doesn’t already have access to your production server as it contains valuable data.
Upgrade Laws
Always upgrade server first and then upgrade terminals. Major releases may not be compatible with previous major release. Terminals with older release installed may not respond to commands sent from newer server.
Do not rollback the server installations or database. Installing new server version includes database table structure changes that are not compatible with previous server version.
It is OK to skip versions forward. You don’t need to install every release we make. Just install our latest release and our software will perform the changes that were introduced in previous releases such as database structure changes.
Upgrade your network in batches. After each batch evaluate that upgraded machines work correctly.
First upgrade 1 machine, then 20, then 50, then 100, then 500, then 1000, then 2000…
Upgrade Flow
Test new release on testing environment
Evaluate new features.
Make sure essential features work.
Perform test transactions on your machine.
Test that your extensions are compatible with this release and are working as expected.
When something seems to be broken contact GB tech support to fix the issue with patch release.
When everything works go to step 2.
Test upgrade on your pre-production environment.
Perform test transactions on your machine.
Perform upgrade and take notes of the steps you had to make and values you filled in.
Make sure essential features work and perform quickly - spot possible performance issues when listing terminals or identities.
When something seems to be broken contact GB tech support to fix the issue with patch release.
When everything works go to step 3.
Upgrade production environment
Choose a good date for your upgrade.
When your downtime will have minimum impact on your customers.
Not far from your previous pre-production upgrade so you remember well what steps you took and workarounds has been taken.
Use the notes from your dry-run on pre-production environment.
Follow upgrade laws
How new release gets manufactured
Any change to our software is associated with a development ticket containing description of a change and desired new behavior may it be a new feature or bug fix.
Every developer who implements the function tests the function himself and describes how feature should be tested.
Every change made by a developer undergoes peer review and automated tests before is merged into the product making sure that only quality, secure, standard-following, non-breaking code makes it into the product. At GB we develop new features into the main branch.
Every month we take what is in the main branch and create a new major release with patch level 1 and create Changelog draft. Every release undergoes automated testing involving multiple terminals running in emulators testing various configurations.
GB’s quality assurance(QA) department is testing every new change to the code ticket by ticket. Whenever QA decides that feature or bugfix needs improvement it is returned to the developer.
QA department is testing also not changed functionality making sure that essential functionality is not impacted.
Any correction to tested release results in production of new patch release that gets tested again.
Once testing is nearing competition (few tickets need to be tested) our Support department reviews Changelog draft and selects candidates for QA department. Candidates represent functionality that deserves to be documented as KB article. QA creates draft of the KB articles.
Support department reviews KB articles and improves the wording and structure making sure articles are easy to read, straight forward, unambiguous and quickly understood.
Once release has only a few bugs to be resolved it is deployed into GB’s beta environment representing small(15 terminals) ATM operation in Prague, Czech Republic. During this period the release is tested by real end-customers in real operation when additional issues are discovered and fixed.
When there are all issues resolved major release is introduced to all of the operators via Telegram channel and Changelog page.
Small operators tend to adopt new releases sooner than larger operators and discover additional bugs that get fixed in subsequent patch releases.
Publicly available supported major releases continue to receive the patch releases for their entire life until marked as and of life.