Blake's back at it this week, onward to something new after the Exceptionless 4.0 launch. Today we talk deduplicate event totals, setting minimum log levels, updates to Foundatio, and tracking down an event processor job issue!
We had previously reverted to earlier commits that showed the deduplicated event total separately from the event total, but since we are running on new indexes now, we are able to reliably include the deduplicated total as the total number of events. GitHub Issue #270
You can now set the min log level in configuration using:
The Foundatio updates we made include performance and handling improvements to the message bus. Learn more about these improvements on GitHub. We also added IHybridCacheClient interface marker, giving you granular control over dependency injection. Now, you can inject either a cache client or a hybrid cache client much easier. Learn more.
Lastly for this week, we are working on tracking down the root cause of an issue with the event processor job that is causing it to stop processing events. More on that soon!
During the Exceptionless 4.0 Release, we covered a lot of enhancements and changes. In this week's Live Code Demo, Blake reviews the 4.0 release notes and talks through other tweaks and post-launch cleanup.
Exceptionless 4.0 has been released for a few weeks now, with ElasticSearch 5 running in production for even longer. Let's take another look at the release notes and talk through some notable topics.
We look at the self hosting & upgrading documentation, which has been updated for Exceptionless 4.0.
In the process of the latest release, we upgraded Foundatio to new csproject format, logging many bugs on the .NET CLI along the way. Blake reviews this in the above video as well.
In the end, we obviously had to merge in pull requests and clean up issues, which he also reviews.
Lastly, we cleaned up the new Azure infrastructure!
In this live code/feature/functionality review video, Blake talks about:
Deploying Elasticsearch 5.x to production with completely new Azure resources.
Fixing bugs and doing extensive testing on the new Elasticsearch 5 and Azure implementation.
Massive performance boosts due to daily indexes, SSD-backed storage, and bug fixes. These boosts should result in near0instant page loads and huge improvements to session dashboards.
Automating cluster setup using Azure CLI and Elasticsearch ARM Templates.
A new custom SSH script that allows us to quickly and securely connect to machines, as well as forward ports for things like Kibana.
New Azure resources and dashboards.
Setting up WebJobs to run as cron jobs.
We also found PM2, a new way to manage Node.js services for our StatsD service.
That's right folks, Exceptionless 4.0 is here and we've got the full low-down on all the improvements, upgrades, and enhancements in this week's blog article, along with how to upgrade instructions for self hosters.
The primary focus of version 4.0 was to add support for Elasticsearch 5. By migrating from Elasticsearch 1.7 to 5, we removed growing technical debt and benefited greatly in the process. Here is an overview of the benefits gained by migrating to Elasticsearch 5 in 4.0.0.
Since we first migrated to Elasticsearch 1.x from MongoDB, we've learned a lot of hard lessons. With 4.0 and the upgrade to Elasticsearch 5, we took all those lessons and applied them to the new release and infrastructure setup. Here are some performance improvements and notes that came out of those implementations.
We've moved from monthly indexes to daily indexes. This change means that when you are filtering by the last 24 hours or last week, you are greatly reducing the amount of work Elasticsearch has to do to return the requested data.
Because of this, the average use case will see between 25% and 80% improvement to indexing throughput. This means that, with the same hardware, we can index the same documents much faster!
We moved from Groovy scripts to Painless scripts, which are four times faster. By moving to Painless scripts, we also greatly simplified the burden of having to modify configuration files to enable scripting!
Elasticsearch has added a lot of new data types since 1.7, and we are now taking advantage of them to reduce memory, storage and query costs. This means we can query the same data faster, using less memory doing it. We've also set up more sensible defaults to ensure we don't index very long strings.
We've also made several performance and reliability improvements to snapshots (backups).
Our goal is to have everyone be able to setup and use Exceptionless in a matter of minutes. One of the areas that had hendered a lot of self hosters in the past was forgetting to update the elasticsearch.yml file, which is no longer an issue with the migration to Painless scripts. If you're a self hoster, check out the
The move to using Painless scripts as part of bulk updates and ingest pipelines reduced the burden of having to modify configuration files to enable scripting! Long gone are the times where you would have to reset your setup because you missed a configuration step.
We've also added various maintenance jobs that handle backups and restores automatically, so you don't have to worry about losing your data!
Future upgrades should be seamless as Elasticsearch now handles reindexing out of the box! Once you have migrated your data to Elasticsearch 5, we think future major upgrades will just be handled by the Exceptionless app itself!
You can now also use Docker or Azure arm templates to quickly set up a cluster. We really like this direction and will continue moving down this path, hopefully getting to the point where you can click a single button and have an Exceptionless production instance running locally!
With Exceptionless 4.0.0, we continued to focus on finishing up backend improvements to both our repositories and parsers that we made in the 3.5 releases. We feel that all the pieces are finally in place to allow us to do custom dashboards in the near future, something we talk about in our 2017 Roadmap blog post.
Foundatio.Parsers now gives us the ability to define and validate custom aggregations using Lucene-style syntax. For example, lets say I want a date histogram that shows the min, max and average event value. I'd just need to pass date:(date min:value max:value avg:value) to the following endpoint: /api/v2/events/count?aggregations=date:(date min:value max:value avg:value) to return aggregations across a time series!
The only users that need to worry about upgrading anything for this new release are self hosters. If you are self hosting Exceptionless, please review the Self Hosting Documentation, which contains information about upgrading your existing install.
Also, please take a look at the change log for a full list of the changes.
We’re always striving to improve the efficiency of Exceptionless and all of our projects. If you see any room for improvement or have any comments when using anything from us, please send us an in-app message, submit a GitHub issue or contact us on the website.
In this live code review, we go over some of the recent changes and processes to get ready for Exceptionless 4.0, which include implementing a new Elasticsearch cluster, reindexing into it using the Elastic Arm templates, and updating existing events mapping.
The Elastic Arm templates automate the setup of an Elasticsearch with a few clicks, which is awesome.
Can you believe it's almost February already! Us either. No rest for the whicked, though, as we sit down to get cranking on our list of Exceptionless changes for 2017.
We have a a great list of features, functionality changes, and enhancements planned, and no doubt we'll work in a few other popular feature requests and, naturally, bug fixes along the way.
Let's take a look at the primary roadmap, shall we?
Upgrading to Elasticsearch 5 has been in the works for some time now, with early testing starting in late 2016. As of January 23rd, we have officially pushed the upgrade to production and are looking for feedback, missed bugs, and performance improvement reports from anyone using it!
Elasticsearch 5 is the latest version of ElasticSearch and brings in massive improvements to event indexing speed, reduced memory sizes, and much more.
The Elasticsearch libraries support .NET Core, bringing us one step closer to fulfilling a vision that includes cross-platform support.
Our new implementation is built on a new, faster Microsoft Azure and SSD storage infrastructure. This will greatly increase the indexing and searching performance, reducing dashboard latencies, among other things.
With the power of Foundatio.Parsers project, we have gained the ability to consume generic aggregations and pivot/report on data any way our users like it.
This upgrade also makes it easier to self host through the use of containers. Our goal is to move the entire app to use containers, which will allow you to self host in seconds!
We will now be doing daily indexes for more performance, fine grained backups, and the ability to more quickly change our indexes.
We want to port the Exceptionless app to .NET Core so we can run it anywhere (macOS, Linux or Windows).
Running on .NET Core also brings massive performance improvements and lower overhead, which will have positive trickle down affects throughout the app.
Once we support .NET Core, we can containerize the entire app, allowing us to scale more easily.
These changes will make self hosting super simple. You'll basically be able to run "docker run -d exceptionless`" and be done!
We know the dashboard is crucial for many things, and we want to make them more customizable to fit each user's individual cases. At the same time, we want to make sure we maximize business value. So, we plan on working on several features and functional aspects of dashboards, namely:
Creating pre-define dashboards that are more granular, such as device usage, breakdown by exception type, etc.
Allow users to create their own custom dashboards.
We really want to add new native clients to Exceptionless, and are including that as a major goal for 2017. Everything we are working on is moving in that direction, one way or the other, and we hope to make further announcements in this department soon!
We are, of course, always accepting pull requests as well!
What have we forgotten? We know everything isn't included on this list, but if we have missed something big that you can't live without, please let us know!
Otherwise, look for a new version announcement very soon, and get ready to upgrade and take advantage of several of these new upcoming changes.
In the first episode our all newweekly live streaming demo session, Blake talks about the recent push to migrate Exceptionless from Elasticsearch 1.x to Elasticsearch 5.x.
We were curious, so we thought we would put together some stats for 2016, along with and a recap of some of the notable changes and progression. Enjoy!
We changed the server licensing to use the Apache license. Now, every Exceptionless project is under the Apache license and there should be no more confusion on how it's licensed. Read more here.
The ability to track and automatically manage a users session was added, giving you visibility into how long and what a user did while they were using your product. This also answers the question, "what did my users do leading up to this exception?" Read more here.
We spent a lot of time this past year making massive performance and reliability improvements across the board. Tons of bugs got fixed along the way, and additions like bulk updates and deletes, along with optimized query generation got squeezed in there, too! One big factor here is that it has allowed us to move to Elasticsearch 5 much quicker because all of these improvements allowed us to do things more generically throughout. Much less code to update/change!
Elasticsearch 5
Like we mentioned above, we've done a ton of work to ensure we can move to Elasticsearch 5:
Insanely fast document indexing and searching
Dynamic aggregations (allowing us to do completely customize stats on any data)
Reindexing support
Daily indexes
Making it easier to self host
Getting one step closer to server being able to run cross-platform
We have been doing QA on the Elasticsearch 5 migration since the end of the year and it is currently still in progress - coming soon!
We've got big things coming up, like Exceptionless 5.0 with Elasticsearch 5, which should boost performance gains like 80% faster indexing, lower memory usage, and native reindexing support! No slowing down over here, so stay tuned and enjoy the ride.
And, as always, if you have anything you'd like to see in upcoming Exceptionless releases, please let us know!
Today we've got a great example of Exceptionless helping software developers identify major issues in their web applications affecting thousands of users.
In this case, the problem was major enough that the development team stopped using WPF and rewrote their entire UI layer!
Here's what Will Graham, CEVO, Inc. developer, had to say when we asked him a few Exceptionless case study questions.
What’s the number one customer-facing bug Exceptionless helped you squash? #
"The issue was so prevalent we completely ditched WPF and rewrote our UI layer in GDI+/WinForms" - Will Graham, CEVO, Inc
Will:
Definitely crashes related to .NET Framework installation corruptions on end-user machines. While Exceptionless didn't (and can't) immediately point and say, "here's how to fix it", it did shine light on an issue that was affecting approximately 5% of our 500,000 installed base. The issue was so prevalent we completely ditched WPF and rewrote our UI layer in GDI+/WinForms, but Exceptionless gave us insight to see the problem and how many users were being affected.
Were you surprised at the initial results of using Exceptionless for the first time? How many errors were you seeing? #
"Exceptionless has changed our internal development process and how we approach code. Proper error handling and visibility is now a first-class priority for us and Exceptionless makes it super easy." - Will Graham, CEVO, Inc
Will:
Yes and no. Error handling and tracking had always been more of an afterthought - like, something is wrong, we need to look into it now. We weren't aware of issues until after our users reported it and filtering out the noise to find the signal of real issues was a time consuming process. Running on the Azure stack, we were frustrated with the information provided in the default diagnostics logging.
When we added Exceptionless, we knew we had problems with our system, we just didn't know where and to what extent. To be honest, Exceptionless has changed our internal development process and how we approach code. Proper error handling and visibility is now a first-class priority for us and Exceptionless makes it super easy.
What is the number one internal bug you were able to track down with Exceptionless? #
Will:
The biggest bug would have to be intermittent failures on our Service Bus instance. Using the metrics provided by Exceptionless, we were able to spend time implementing fault tolerance in very focused areas of the code based on incidence rate and end-user impact.
If you had one feature you’d like us to add to Exceptionless, what would it be? #
Will:
I'm not a huge fan of the current dashboard. It's effective, but with the amount of data Exceptionless has at it's disposal, I feel there's more interesting (and actionable) information that can be shown.
Thanks for your time, Will! We love to hear from our users, and we'll definitely take that dashboard feedback and see what we can do there to make it more useful.
Do you have feedback on Exceptionless? Has it helped you find bugs and beat them into submission? Join us in the Exceptionless Discord and let us know!
That's right folks, Exceptionless 4.0 is here and we've got the full low-down on all the improvements, upgrades, and enhancements in this week's blog article, along with how to upgrade instructions for self hosters.
Case study time!