Illuvium | The Path Forward: Lessons from Minting, Bots, and Mission Challenges

The Path Forward: Lessons from Minting, Bots, and Mission Challenges

Rangers,

Over the past few weeks, many of you have shared your frustrations with the delays in minting items on Illuvidex, the impact of bots, and some of the updates surrounding Season 2 of missions. Today, I want to share an analysis of what's happened, why it happened and what we're doing about it. Before we dive into the details, I want to express our sincere apologies for the frustration caused and thank you, exalted rangers, for your continued patience and passion. My name is Stuart and I'm the new CTO here at Illuvium.

Let's start by looking at what happened with the bots over the past few weeks.

Bots

Bots come in many forms, some good and some bad. Over the past few weeks, the team & I have been constantly reviewing not only who has been using bots, but the way the bots have been interacting with our systems. We've been going through various levels of deterrence with the people behind the bots, and we're now in a position where the bot activity on our system has reduced significantly. There will never be a perfect system to prevent a genuinely motivated person. Still, we are dealing with any gaps in our current systems whilst simultaneously raising the skill level required to a point where the return on investment would no longer be worth the rewards obtained. 

While raising the bar to combat bot activity, we acknowledge that some legitimate players and VPN users were unintentionally affected. We're refining our approach to minimise disruptions and ensure legitimate players can fully enjoy the game. In cases where legitimate players have been banned accidentally, we have been reviewing the evidence from both players and our internal logs to ensure that they are resolved correctly, whether that's unbanning or keeping the ban in place - as our evidence shows. We're reviewing the protections we have in place and will make the necessary adjustments to lessen the impact on legitimate players. For users of VPNs, we don't want to have this block in place forever and plan to start releasing the block we had on data centres soon. These data centres are known to have large amounts of bot activity, so we may need to revert to a blocked state while we improve other protections.

Moving forward, we're making a substantial change to our architecture, which will prevent many of the bots without the need for strict protections and help deal with those trying hard to stay under the radar. This change simultaneously improves the development efficiency of our game developers, resulting in fast feature development over the coming months. An architectural change like this takes time to complete safely, so we will roll it out in phases. You'll see the first part of this change by being able to join your fellow rangers in Sanctum Mesa in the coming months, viewing the skins of others and more. In addition, we'll expand on our new automated cheat detection to ensure a fair playing field for all.

Minting Issues

The bot issue is important to understand as it's one of the factors behind some of the problems minting of late. The issues can broadly be categorised into two types of issue:

1. Delays in minting and/or failure to mint

2. Showing as minting on Illuvidex, with no data on Immutascan, yet appearing in-game

The delays in minting can be caused by multiple issues, some of which we are in control of and some of which we are not. One of the things outside our control is the bots mentioned above trying to speedrun their way through different missions to farm airdrop points (more on this below). When they do this, they tend to fill up a vital part of our pipeline which assigns the correct token id for the things that we want to mint. Under normal usage for our player base, this isn't a problem; everything goes through quickly. Once we pass this stage, we will send the information to IMX to mint on your behalf, from which we will be at the mercy of the blockchain processing speed. Once minted, IMX asks us for the metadata associated with the token that we've just minted (this is important to know for later). Everything should be available to view in your inventory and on Illuvidex from that point onwards. We're looking at this from the architecture and game design perspectives here to review how/what things are minted, like whether the fragments should be minted or just their cured counterparts for example, amongst other techniques to improve the speed and experience that we have today without losing too much of our ethos.

Our process that handles all of the forge requests, illuvial captures and fusions is called CraftOrders. It's a relatively generic process that we've designed that accounts for how the blockchain works and the need to burn items from the client. This burning process has caused an issue and is something we have limited control over. After many long hours of debugging and support in close collaboration with the team at Immutable, we collectively discovered an issue with Immutable's SDK that caused any burn requests we sent to disappear silently. We would see the SDK acknowledge the burn request, but our systems would never receive the updates to say the burns had been completed, leaving people with missing Illuvials and forged items stuck in processing. During a root cause analysis, Immutable managed to reproduce the issue from their side and offer mitigation, which we shipped in the last client update for Overworld. Anticipating that this issue may return as it wasn't a complete fix, our client team implemented a notification to let you know as a player when you encounter this issue and that you'll be asked to restart the game. This allows the SDK to reset appropriately and the client to process the pending list of burns for you. This issue has remained, but it should happen far less frequently and be more visible to you as a player. We have an ongoing commitment from the top level of Immutable that they will be working to resolve this problem as best they can moving forward. In addition, we've extended the timeout functionality for all CraftOrders in our backend to ensure the best possible chance of getting your items to you.

The issues with Illuvidex showing items as minting has a relatively simple cause but a fix that's turned out to be more like concentric circles than straight lines. Remember earlier, I noted that Immutable called us to ask for information about a token? Unfortunately, during the high-load, Immutable hit some of the rate limits that we had in place, which prevented the metadata from being returned. Once we realised this was the cause, we excluded these IPs from the rate limits so that they could fix that issue normally. We still needed to fix the already broken data, but it's important to emphasise that no data has been lost. There are 3 processes that we could invoke to run this repair. The first took about 72 hours to complete, and we resolved a few of the broken tokens, so we moved on to the second repair option that we had. Again, this was potentially a multi-day repair, given the volume of specific collections. We started with Illuvials, which is one of the most important collections. After dealing with some unexpected issues starting the process, the repair was completed 72 hours later. 

When investigating the results of the second repair, we noticed that we still had tokens that were unexpectedly missing their metadata. We are currently investigating where the issue lies with this repair process and have high-priority tasks to ensure it's fixed in case we need it again. This brings us to where we are today, running repair option number three. This is the slowest of the options, which is why we've left it until last, but most importantly, it's restoring missing items on Illuvidex. At the time of writing, we've only processed missing Illuvials, with the remainder coming over the next 72 hours or so, which is as fast as we can process, given our rate limits with Immutable. Once this repair has been completed, we will announce it on our Discord server to gather information about any missing things and get a resolution to them.

Overall, there are some definite improvements that we will be making in the coming months to our repair processes (including the testing of), some more architectural changes to improve resiliency/performance in critical areas and most importantly, proactive tooling to deal with these issues quicker or even preventing them from happening where we can. We will fix as many of the broken things as possible while finding solutions for anything we can't fix. We have a shared understanding of our problems with Immutable. Still, more importantly, we are working collaboratively with them to resolve issues not only for ourselves but for all games based on Immutable (for example, Immutable have enhanced bot detection at the point of sign-up based on our experiences and recommendations). Whilst the current chain has had its share of issues, migrating to another chain is a serious undertaking in terms of time and money, which could be spent on making other stuff that enhances our proposition to the gaming community as a whole. We should adequately evaluate it as a project and ensure that we continue to try and make the best decisions for the project.

Missions

The start of Season 2 has been different from what we all would have wanted. A process that we had for making the announced changes to Season 2 failed. As a direct result the milestones were not updated as we expected them to do because it happened over a weekend, we took a little longer to notice than we normally would. This meant that we needed to go and make some adjustments to our tooling to ensure we can correct the failed process, which is where we are today. At the time of writing, these changes are being tested and Season 2 will be corrected as per the original plan. There will be no mid-season changes when we release these updates, which are in line with IIP-58 and ICCP-19 (preventing mid-season updates and the survival wave points adjustment). Our reasoning behind setting milestones to 0 points (for those that have not been completed already) is to disincentivise bot makers and some genuine players from using multiple accounts to farm airdrop points. This also returns milestones to their original intention of being an achievement-like system where you can track your progress through the game. Our internal process for checking for exploitations has also concluded. Bans that have been issued will have their corresponding points removed from Season 1 & Season 2, increasing the total reward share for genuine players. Moving forward, any players that are banned will have their points removed from the season automatically between 48-72 hours after the ban was issued. The delay is to allow time for appeals, and subsequent verification, as we continue to enhance our anti-bot protections.

In addition to the above, we're looking at the improvements we can make to future seasons from a variety and challenge perspective. These improvements, including those suggested in the feedback thread, take time to implement and we'll have more updates in the coming weeks on what to expect from Season 3 and news of special events that rewards points for the current season (hint: you'll want flintcaps). We're also reviewing some of the points assigned to specific daily/weekly missions which could do with some better balancing and make them more rewarding.

The Path Forward

Our path forward together is clear. We take some of the learnings from the past month or so and we improve that little bit each day, whether that's better communication, an additional test or getting feedback from the community/partners. A lot of the changes that we want to make based on the details we've shared today are already in the planning and discovery processes, or in some cases implemented already, so that we can deliver better experiences for you. Together we are stronger, together we'll make the best game possible!