Appendix A. DIY analytics using ELK
This appendix covers
- Reviewing the overall architecture of an agile metrics collection and analytics system
- Setting up an ELK server
- Building a data-collection application using Grails
- Installing the data collector on the ELK server
In this appendix I’ll walk you through setting up a powerful analytics system using the ELK stack (EC, Logstash, and Kibana). You can download the basic stack from the EC website. You can also get the setup I’ve used for this book on GitHub at github.com/cwhd/measurementor.
Let’s start by looking at the high-level component diagram in figure A.1 to get an idea of the pieces you need to build. You’ll need to write some scripts to collect data, set up a database to store it, and put together an indexing, search, and visualization engine. Once you have all of these components, you’ll be ready to start building out connectors to external applications used on the development lifecycle to get data and analyze it. You’ll use the following technologies for these components:
- Data collection —Grails (grails.org/)
- Database —MongoDB (www.mongodb.org/)
- Data indexing and search —EC
- (www.elasticsearch.org/)
- Data visualization —Kibana (www.elasticsearch.org/overview/kibana/)
Figure A.2 shows how data flows through the system.
