An update is going out that will resolve this issue. I'm sorry it took a bit longer than expected to track this one down - so much stuff has been going on.
We know Deviation Games was founded this year and according to their LinkedIn, they have at least 70 employees.
Dave Anthony, CEO, did a lot of work on the Call of Duty franchise. Jon Burton, CIO, created the LEGO game franchise. I can't tell who else is on their payroll that came from Blizzard since a lot of people on their employee listing have private LinkedIn profiles, but I definitely wouldn't be surprised if there's at least a couple more over there.
There were some of you that were refresh spamming to earn your treats, but really, you should have been receiving tricks!
An update has gone out that reduces the ability to do this somewhat - go browse the site naturally and engage in conversation or maybe find a deck a fellow community member has created and give it an upvote that it deserves.
I won't go into exactly how it works for obvious reasons, but you definitely can't sit here and F5 anymore.
We'll see how it goes for the rest of the day. It's really easy for us to tune the numbers on everything so if it isn't working, we'll scale it back a bit.
Also, on Halloween everyone is going to get a bucket of candy, so look forward to that!
I do see the page hanging now and think this may be related to yesterday's deploy. Going to go digging. Thanks for the report and sorry for the trouble!
We were supposed to have our Halloween event go live earlier today, as teased via "Out of Cards Surprise" on the October card back post for Hearthstone. We're behind on schedule due to the craze that was this past week (sup Hearthstone announcements and Duels deckbuilder) so I just wanted to let everyone know that it is delayed until tomorrow.
I don't have the exact time when it will go live tomorrow, but I can tell you that we'll be extending the availability of the Halloween event a couple of days to make sure we have at least a full week for it. It doesn't line up perfectly with Halloween being the final day, but we'll just say we're also celebrating Day of the Dead alongside it =D
I think people will like what we've got in store. It's a jump up from our previous site events and will also be the basis for our Holiday event in December. Super stoked!
At this point, we're definitely delayed until tomorrow on the Spooks. This week has kicked my ass and was meant to be most of our final implementations of stuff. Since I focused hard on getting the duels deckbuilder feature out, we're behind =( I've also slept for 4 hours over the last 32 hours and should probably take another couple hour nap.
All worth it though. I think people will like what we've got in store. It's a big jump from previous site events and will also be the basis for our Holiday fun in December, so I'm excited about that! Speaking of, work on our Holiday event is going to take place while the Halloween one takes place because November is going to be absolutely nuts. Oh, I'm so excited!
There was some decent thought that went into the initial deckbuilder design, because we'd have to support things like adventures and tavern brawls, but this actually required a lot more work than originally anticipated.
The original goal was to have this out on Thursday night after we learned of the changes. Most of the work went into rebuilding our API that serves cards to the deckbuilder (and other site features) to get more control over how we can filter and cache results.
Even then, with the front-end part being easier, I still want to fully rebuild our deckbuilder to make it even better. I'd like to solidify our deckbuilder tech between all the builders we support (Runeterra one has some better code over the Hearthstone since it was built later) to also make it much easier to build out a new, basic deckbuilders given very few pieces of data so we can support more card databases easily. That isn't in the cards yet though.
Typically achievement points are just a measure of comparison and not a real currency. They didn't announce anything specific, so I can only imagine that will be the case here too.
We had some unintended downtime this way-too-early morning. Because this kind of stuff always interests me, and transparency can be fun, let's see what went wrong!
Event Log
All times in Eastern.
2:43 AM - Out of Cards goes offline.
2:49 AM - A service unreachable email gets fired off.
This email did not trigger the alerts that it should have. Since we did a site migration a couple of months ago, our tech for that changed and was never tested. Pfft, downtime. That'll never happen!
3:01 AM - The messages begin to hit me on Discord and Twitter.
See, you don't need proper downtime checks!
3:03 AM - Investigation Begins
3:19 AM - Problem Discovered
Our container service decided it wanted to do an automatic update. That really should not have been an issue but because the containers running our site were not set to automatically restart, the site went down.
3:22 AM - Site comes back online
Updates for the service have been disabled - we'd rather manage these ourselves anyway.
What We're Changing
Obviously, our big issue is that our app containers didn't restart themselves. That's shitty and a huge oversight. A configuration change will go out in the morning that will resolve this.
The secondary issue is garbage monitoring of the site. There should be some serious alarms going off when stuff goes wrong. I'll make sure we have better processes in place going forward to notify of downtime.
I'm so sorry! My fault that this wasn't triggering the achievement to reward candy. That issue has been resolved.
An update is going out that will resolve this issue. I'm sorry it took a bit longer than expected to track this one down - so much stuff has been going on.
We know Deviation Games was founded this year and according to their LinkedIn, they have at least 70 employees.
Dave Anthony, CEO, did a lot of work on the Call of Duty franchise. Jon Burton, CIO, created the LEGO game franchise. I can't tell who else is on their payroll that came from Blizzard since a lot of people on their employee listing have private LinkedIn profiles, but I definitely wouldn't be surprised if there's at least a couple more over there.
Max Ma is still on the team! (Lead on UI/UX) He joined earlyish in 2014. No one from the very beginning is there though iirc
Edit: I don't know why I thought your comment had originally mentioned Dean Ayala, but he is still there too and joined around the same time Max did.
There were some of you that were refresh spamming to earn your treats, but really, you should have been receiving tricks!
An update has gone out that reduces the ability to do this somewhat - go browse the site naturally and engage in conversation or maybe find a deck a fellow community member has created and give it an upvote that it deserves.
I won't go into exactly how it works for obvious reasons, but you definitely can't sit here and F5 anymore.
We'll see how it goes for the rest of the day. It's really easy for us to tune the numbers on everything so if it isn't working, we'll scale it back a bit.
Also, on Halloween everyone is going to get a bucket of candy, so look forward to that!
No, though I would love to do something Old God themed down the line.
There's always next Halloween :D
Funny you ask. The deploy is building right now!
As the event page says, avatar borders will be available later this week. FWIW, they will be 125 Candy a piece.
Glad you're digging it! My goal with our site events is to make them better for each one, so it'll be exciting to see what the future brings :D
So far, 1066 clues have been found! I'm expecting to see that get a sharp rise today as more people see the event for the first time.
Glad everyone is enjoying it so far!
Fixes are going out for this now!
This has now been flagged as a treasure as intended!
Hey BogAl,
I do see the page hanging now and think this may be related to yesterday's deploy. Going to go digging. Thanks for the report and sorry for the trouble!
Have fun!
Making a post here for visibility.
We were supposed to have our Halloween event go live earlier today, as teased via "Out of Cards Surprise" on the October card back post for Hearthstone. We're behind on schedule due to the craze that was this past week (sup Hearthstone announcements and Duels deckbuilder) so I just wanted to let everyone know that it is delayed until tomorrow.
I don't have the exact time when it will go live tomorrow, but I can tell you that we'll be extending the availability of the Halloween event a couple of days to make sure we have at least a full week for it. It doesn't line up perfectly with Halloween being the final day, but we'll just say we're also celebrating Day of the Dead alongside it =D
I think people will like what we've got in store. It's a jump up from our previous site events and will also be the basis for our Holiday event in December. Super stoked!
At this point, we're definitely delayed until tomorrow on the Spooks. This week has kicked my ass and was meant to be most of our final implementations of stuff. Since I focused hard on getting the duels deckbuilder feature out, we're behind =( I've also slept for 4 hours over the last 32 hours and should probably take another couple hour nap.
All worth it though. I think people will like what we've got in store. It's a big jump from previous site events and will also be the basis for our Holiday fun in December, so I'm excited about that! Speaking of, work on our Holiday event is going to take place while the Halloween one takes place because November is going to be absolutely nuts. Oh, I'm so excited!
There was some decent thought that went into the initial deckbuilder design, because we'd have to support things like adventures and tavern brawls, but this actually required a lot more work than originally anticipated.
The original goal was to have this out on Thursday night after we learned of the changes. Most of the work went into rebuilding our API that serves cards to the deckbuilder (and other site features) to get more control over how we can filter and cache results.
Even then, with the front-end part being easier, I still want to fully rebuild our deckbuilder to make it even better. I'd like to solidify our deckbuilder tech between all the builders we support (Runeterra one has some better code over the Hearthstone since it was built later) to also make it much easier to build out a new, basic deckbuilders given very few pieces of data so we can support more card databases easily. That isn't in the cards yet though.
Glad you like it though!
As a member, I'm voicing my support.
As staff, I'm hiding.
At least a week which should be enough time to gather the gold if you are starting from scratch =)
Typically achievement points are just a measure of comparison and not a real currency. They didn't announce anything specific, so I can only imagine that will be the case here too.
Hey all,
We had some unintended downtime this way-too-early morning. Because this kind of stuff always interests me, and transparency can be fun, let's see what went wrong!
Event Log
All times in Eastern.
2:43 AM - Out of Cards goes offline.
2:49 AM - A service unreachable email gets fired off.
This email did not trigger the alerts that it should have. Since we did a site migration a couple of months ago, our tech for that changed and was never tested. Pfft, downtime. That'll never happen!
3:01 AM - The messages begin to hit me on Discord and Twitter.
See, you don't need proper downtime checks!
3:03 AM - Investigation Begins
3:19 AM - Problem Discovered
Our container service decided it wanted to do an automatic update. That really should not have been an issue but because the containers running our site were not set to automatically restart, the site went down.
3:22 AM - Site comes back online
Updates for the service have been disabled - we'd rather manage these ourselves anyway.
What We're Changing
Obviously, our big issue is that our app containers didn't restart themselves. That's shitty and a huge oversight. A configuration change will go out in the morning that will resolve this.
The secondary issue is garbage monitoring of the site. There should be some serious alarms going off when stuff goes wrong. I'll make sure we have better processes in place going forward to notify of downtime.
Hopefully there isn't a next time!