Launching CardanoKeys – A Retrospective

Before getting into the details of the issues we faced, the aftermath (which was much milder than we feared) and our future steps I wanted to thank everyone who supported us during the launch, the newly founded club “#100” and the five people waiting patiently for their refund after their payment got stuck.

I said this a bunch of times on Discord but I cannot stress this enough:

When we delayed it by two hours initially you had our backs and understood.
When we considered delaying it another day – despite the hype – you preferred this to a faulty start.
When issues started we got nothing but supportive comments from each and every one of you.
And when the server rebooted and accidentally opened up you bought every single key until the 1k supply was emptied.

We’re looking forward to give back to you and have fun with you in this project for years to come!

Issues we faced during the launch

We did have a few issues when trying to open everything up, some more while the sale was live and a tad more when it ended. After revisiting everything which happened that day we found both human errors and technical faults.

On a high level four issues hampered a smooth start:

• Metadata
• Policy (kinda)
• Stuck Transactions
• Slightly worsening the rest: Unhelpful Error Messages and lack of documentation of Cardano binaries

Metadata

Of all the issues we faced the metadata was the one causing the most problems.

A major cause of this was the very late delivery. We did have a lot of discussion about some specifics, tried some things, rejected our proposed “skeleton” which only needs the variables specific for the keys twice but had everything set a few days before the sale.
We did trust that (now “former”) team member to deliver at this point because only the links to the art were supposedly missing.

We do have to trust people with this project.
We need to trust the developer about basically everything since he needs to have the private key for minting for his system to work – otherwise we could not offer a “Vending Machine”-style purchasing model.
We have to trust the community to not get taken over by speculant and scalper mentality, we plan to develop this community and project for years.
We have trusted and we will keep trusting.

It’s situations like the one during the launch which test this mentality.
The metadata arrived 14:18 GMT, almost twenty minutes after the sale was supposed to start.
Of course we asked about the metadata repeatedly but the time estimates shifted further and further, from 10:00 to 11:00 to 12:30 until it was too late.

Since we had to move fast, already having postponed the sale to 16:00 GMT our developer integrated everything at breakneck speed and got that part of the system up and running. In that rush we managed to spot some errors which wouldn’t let the system load the metadata file as a clean JSON file – i.e. syntax errors – but missed a fairly obvious error in the metadata.

It is not obvious if the metadata is displayed in a single line with ~6 million characters all the way off the screen, but it’s blatantly obvious once the actual minting starts.

The key with the name “CardanoKeys #100” was minted a total of 18 times this night. We were worried at first that these were re-mints and that for some reason the randomization algorithm favors this key above others.

Even before we were able to verify that they are all different assets and not a single key was double minted, the keymunity founded the exclusive club “#100”, exclusive to holders with this key name.

As it turns out the metadata simply specified the name “CardanoKeys #100” for all keys with the numbers from 100 to 249. Each of these keys is strictly distinguishable from the other seventeen and raffles are not affected by this.

The club has decided that all rewards towards any one of the 18 keys shall be distributed equally among them and we think this is an amazing idea!

We still offer to re-mint any of these keys to display the correct number in its name but so far not a single holder wanted to “leave the club” by getting a clean re-mint and we respect that!

Special Diamond keys with these IDs were minted:
100, 114, 126, 128, 133, 141, 144, 154, 170, 173, 179, 180, 191, 197, 216, 219, 224, 233

We did look into the possibility that pulling 18 special diamond keys within the first 1000 pulls might be abnormally high and not “random” and gave myself a good excuse to pull out my favorite Hypergeometric Calculator and see if that would be unreasonably high. It turns out the odds for this are 24.06% and roughly one in four is close enough to the expected value (which would be 50%) to not warrant worries about a faulty randomization.

There were two more issues with the normal metadata, one causing some confusion and the other one which is so common that viewers like pool.pm or the NFT Viewer by Novellia simply fix it on their side.

Sadly all special silver have the same description as the unique silver key. This did cause some confusion and we’re still exploring how we could get this fixed quickly before the policy slot locks but it’s a cosmetic issue.
Still we do need to find a way to fix this.

Finally three of the key designs have badly formatted IPFS Links. Generally the format is “ipfs://ipfs/IMAGEHASH” but these were simply “IMAGEHASH”. Any decent NFT viewer will interpret this correctly. This affects the “Special Obsidian” (one was minted), “Unique Silver” (was not minted that night) and “Special Silver” (32 were minted).

A final challenge was implementing the novellia_1 standard. We did have some issues but they were not exclusive to that night so I’ll go into depth another time.
It was a challenge and we did get it done but it wasn’t perfect.
Luckily the people over at Rektangular Studios are absolute rockstars and will develop their NFT Viewer under the assumption that “this kind of misstake will probably happen to other people, too” and make this work.

As of now they released version 0.6.0 of their NFT Wallet viewer and they really did manage to get everything working with the imperfect Metadata we used in the novellia_1 extension. Give it a try, they really create awesome stuff and their wallet viewer is pretty amazing, too 🙂

We had to get the metadata in order before the second offering of the initial sale was started. I had been entrusted with that part and since we had no similar issues with the metadata after the first sale I personally consider that issue solved for good.

Policy (kinda)

The policy itself wasn’t a primary problem but it reflects the core issue we had when preparing the launch. One person handled the art, another one wrote the metadata, a third was in charge of the policy and a fourth was entrusted with developing the code.

We did run a lot of tests but we mainly tested either one component individually or together with a second one.

For example:
We did test minting a token manually with the skeleton of the metadata to make sure that this works properly.
It did not take into account the policy or the backend and did have some valuable results, namely that this wouldn’t be something we’d accept without changes:

All of the data above is not supposed to be interpreted by pool.pm and is exclusively meant for the novellia_1 standard. The general idea is to be able to provide multiple Data sources for NFTs and also specify the loading priority. One would be able to first load a small, static image, then a high resolution image and finally the animated gif. If all NFTs did that then NFT viewers would be much better to use.

We tried to implement the version 1.0 of that standard back then, saw how horrid this looks, talked with the excellent folks at Rektangular Studios (we were not even partnered with them at that point) and they worked out a great addition to their standard which we adopted for our final Metadata version.

Similarly we did test the backend itself with the artwork but did not test it in conjuction with the policy.

Now the policy issue is absolutely on me. I believed to have had a great idea and wanted to brute force a vanity policy ID. The idea was that instead of the random hexadecimal any policy ID has we’d start with something instantly recognizable. So I created the required files which would give us the policy ID “000000AD” and I was really excited about this.

Somehow integrating these files into the sale system did not work.

To this day we don’t know why exactly that is. We searched for all the information we could find to why that did not work, searched for people who had similar errors and just could not find it. Supposedly that issue would have been solvable with one more parameter given to the cardano-cli but despite having that already in there it simply did not work.

So this is something we’ll avoid at all cost in the future. We must test both the components individually and in conjunction with each other. In order to do that we are working on a fully separated environment just for testing purposes and will – of course – also test if switching from testing to production does not cause any issues.

Stuck Transactions

Of all the things we worried, having bugs in the code which either throw a wrench into the system or somehow get stuck were among the worst fears we had. Luckily only a total of six transactions did neither return the NFT or the ADA and we managed to track down the reasons for this.

Primary Cause: Shutting down the sale unexpectedly.

When we shut down the sale to investigate the reason for having minted multiple #100 keys (which only have tho same name but are not the same asset) a handful of transactions got stuck. As long as the system was operating in normal parameters everything worked fine but some unexpected cases did not go as planned.

For all the transactions we could look into the sale shutting down was only one of the causes but it needed “more” to cause this behavior. Usually the addresses were either double spent to because the address was copied twice or the transaction didn’t go through for over 10 minutes and it ran into a timeout.

Only in these circumstances did the transactions not get handled as they were supposed to.

We were able to un-stuck all transactions since the server did keep the files required to access these wallets and refunded every person. No ADA was lost and six stuck transactions in a sale of 1000 keys is a decent if not perfect track record. We did have a second wave of sales one week later and not a single transaction got stuck with 600 further keys sold. We’re confident that this was a one-off due to the emergency shutdown.

While investigating this we sadly also realized that one single key, the Gold Key #1042, has ended up in Exchange-Purgatory. A single transaction was sent from some exchange and sadly this means that this gold key won’t be usable by anyone.

Making difficult things more difficult – cardano-cli documentation and error messages

Anything and everything which went wrong was completely on us, we’re not trying to shift the blame. The metadata was not as it should be, the delays were disheartening and with even more tests we probably could have avoided a lot of what went wrong.

In order to help other people with their own NFT projects I would still like to bring forth some of the issues which made the situation even harder to handle, namely the lack of documentation for the cardano-cli and the less than helpful error messages.

Again: Anything and everything which went wrong was on us.

Now onto the issues! Some errors generated by the cardano-cli and cardano-node don’t help the users a lot.

ShelleyTxValidationError ShelleyBasedEraMary (ApplyTxError [UtxowFailure (ScriptWitnessNotValidatingUTXOW (fromList [ScriptHash "[...]"]))])

That, isolated in itself, is not very helpful for those who aren’t developing the actual cardano-node or cardano-cli. Error messages don’t always have to yield all information required for non-developers to fix the issue but there should be some documentation on what causes this and maybe some suggestions on how to fix it.

This is the error which was thrown when we tried to use my policy files in the backend and have not yet found the solution to.

If you try to find a solution for this you’ll mostly find two things. Either you find some code snippets from the developers with no documentation, just code changes, or some forum entries which suggest to add a parameter to the minting transaction.

But neither are proper documentation. It’s a community effort to understand what the developers try to express with their error messages but no central documentation.

There was a similar issue with the cardano-node 1.27.0 where the prebuilt binaries, they just did not work.

Aeson exception:
Error in $: key "AlonzoGenesisFile" not found

This error is similarly undocumented and searching for the error does not yield a proper documentation. When trying to get a cardano-node working the only two solutions were to find a build which does not have that issue or to compile the binaries oneself. There is nothing the users can do to fix this issue if they use the officially provided binaries and no entry in any forum or documentation stated this as an issue.

Minor stuff

Aside from these four larger factors there were some less significant things which could have gone better. This part is mostly for people looking to create their own NFT projects, you will probably have to face similar challenges.

“Dust” (minAda) for keys sent out was too high

When sending NFTs using Daedalus or Yoroi the dust required to send in conjunction with the NFTs is calculated by these wallets, it’s harder for systems using the cardano-cli directly.

When sending an NFT usually 1.45 to 1.6 ADA is sent with it. To make sure everything works we increased that to 1.7 ADA per NFT and whenever people bought three or ten tokens we sent that much times three or ten respectively. This was way too much but the community benefited from that so we don’t mind that too much. But for the NFT project it could be detremental if too much dust is sent out in addition to the NFTs, especially if the Tokens cost 5 ADA each or less.

Luckily the amount of dust sent out is very well documented and there are formulas to precisely calculate these values. For CardanoKeys we decided to ask for the dust in addition to the purchasing price and send that dust back to the buyers. Asking for 17 ADA whenever someone buys 10 NFTs was too much and so we chose values which are much closer to the actual dust required while still having enough to make sure everything works properly.

Larger asset names need more dust but these values should still be enough even with very large asset names:

• 2 ADA for 1 Token
• 3 ADA for 3 Tokens
• 4 ADA for 10 Tokens

These are just rough suggestions and add almost 0.5 ADA to each but look much nicer than asking for 1.4445 ADA as dust or something similar to that.

Project Management was too lax

Another challenge we faced and did not handle perfectly was the project management. We are only a small team but we do have to work fully remote and have not met anyone in person so far. Most of the coordination was using the Discord chat and we handled all the TODOs by reminding each other of what was still to do.

This works only for very small projects and is not something which scales.

This is why we changed to use a proper project management tool. We try to avoid overhead by adding too much management but using a tool to keep track of the larger and smaller TODOs with the appropriate responsibilities assigned to each.

A big thank you (incl. club #100)

I personally want to thank everyone involved in this. Thank you for taking the time to read this, thank you to the entire community for supporting us all the way, even if things went wrong, thank you to the team for working overtime even when things got rough and also thank you to the club #100 for turning a mistake on our end into an exclusive club of incredible supporters.

We have been met with nothing but support and understanding, even when announcing a second postpone of the sale and we do not take this for granted.

As usual: DMs are always open, if you have suggestions or questions please do join the Discord server and let us know your thoughts!