Setup Logstash on DigitalOcean with a Rails App hosted on Heroku : Full SSL

Logstash is a part of the ELK (Elasticsearch — Kibana — Logstash) stack, from Elastic. Its a centralized log drain for all your logs from various sources.

I couldn’t find any nice tutorials on setting up Logstash on a DigitalOcean droplet, and sending logs to it from a Heroku Rails App, over TCP with SSL. Certificate creation also seems poorly documented online.

So let’s get to it.

Create A new DigitalOcean droplet with Ubuntu 18.04, and at least 2gb of ram, as logstash eats 1 gb.

Use this Github Repo to setup basic packages.

Follow DigitalOcean’s fantastic guide on setting up postfix with mailgun.

The tutorial is pretty good, provided you already have mailgun configured.

Logstash can be configured to use SSL certificates, when receiving TCP input.

I followed virtually a 100 different tutorials on setting up self-signed SSL certificates for use with Logstash. None of them worked.

I spent nearly 6–8 hours on all this and eventually realized that there is no simple tool to automate the process. (6–8 hours is a long time for me)

I created a Github Repository, that you can find below, that does all the heavy lifting for you. I would strongly suggest you use it, rather than tinker with OpenSSL on your own.

More importantly, I’ve included a simple sh script to check whether Logstash responds to your client over SSL. It took me a long time to figure out how to even test the SSL connection.

Full instructions on my Github Repo.

Rails Heroku Logger Setup

I won’t get into the details of setting up a Rails App on Heroku, you can follow their own tutorial online, its pretty simple.

The gems I used were:

gem “lograge”
gem ‘logstash-event’
gem ‘logstash-logger’

Configuring your Rails App can be a huge mess depending on which logger you plan to use.

I’m outlining my configuration, but it can get messy, primarily because I used Rails.logger.(debug/error) all throughout my application. So it became necessary to extend Rails.logger to output its stuff to Logstash in a json friendly format.

Create a module in your /lib directory:

This simplifies setting up the SSL Connection.

module SslModule    class Context

def self.certificates_path
Rails.root.join("config","certificates","certificates")
end
def self.get_context
ssl_context = OpenSSL::SSL::SSLContext.new()
ssl_context.cert = OpenSSL::X509::Certificate.new(File.open("# {certificates_path}/ssl_certificates/host/host.crt"))
ssl_context.ca_file = "#{certificates_path}/ssl_certificates/ca/ca.crt"
ssl_context.key = OpenSSL::PKey::RSA.new(File.open("#{certificates_path}/ssl_certificates/host/host.key"))
ssl_context.verify_mode = OpenSSL::SSL::VERIFY_PEER
ssl_context.ssl_version = :SSLv23
ssl_context
end
def self.get_machine_ip
ip = Socket.ip_address_list.detect{|intf| intf.ipv4_private?}
ip.ip_address
end
def self.get_logstash_ip
machine_ip = get_machine_ip
#puts "machine ip is :#{machine_ip}"
#puts "ip address is #{ENV['IP_ADDRESS']}"
return ENV["LOGSTASH_SERVER_IP_ADDRESS"] unless ENV["LOGSTASH_SERVER_IP_ADDRESS"].blank?
unless ENV["LOGSTASH_SERVER_IP_ADDRESSES"].blank?
ENV["LOGSTASH_SERVER_IP_ADDRESSES"].split(",").each do |ip|
if machine_ip == ip
return ip
end
end
end
return "0.0.0.0"
end
##should check eth for the ip address or hit 0.0.0.0
## if more than one ip address is specified.
## if no IP_ADDRESS is specified, defaults to eth / or 0.0.0.0
def self.logstash_logger
ls_ip = get_logstash_ip
#puts "ls ip is #{ls_ip}"
LogStashLogger.new(type: :tcp, host: get_logstash_ip, port: ENV["LOGSTASH_PORT"], ssl_context: get_context)
end
end
end

Create an initializer:

#logger.rb
Rails.logger = ::SslModule::Context.logstash_logger
Delayed::Worker.logger = Rails.logger

Add the following to your environment files:

#development.rb
Rails.application.configure do
config.use_logstash = false
config.lograge.enabled = false

config.lograge.logger = Rails.logger
config.lograge.formatter = Lograge::Formatters::Logstash.new
config.log_tags = [:uuid]
config.lograge.custom_payload do |controller|
{
host: controller.request.host,
user_id: controller.current_user.try(:id),
request_id: controller.request.request_id || RequestStore.store[:request_id],
params: controller.request.params.to_s
}
end
end

This should get everything working, and you should see the logs popping up in your logstash server.

Comment if you face any problems!

All the best, this is not easy.

Post-Graduate in Clinical Pathology, Lab Director - Pathofast, Computer Vision Enthusiast, Founder algorini.com