Setup Logstash on DigitalOcean with a Rails App hosted on Heroku : Full SSL
Logstash is a part of the ELK (Elasticsearch — Kibana — Logstash) stack, from Elastic. Its a centralized log drain for all your logs from various sources.
I couldn’t find any nice tutorials on setting up Logstash on a DigitalOcean droplet, and sending logs to it from a Heroku Rails App, over TCP with SSL. Certificate creation also seems poorly documented online.
So let’s get to it.
DigitalOcean Droplet Setup
Create A new DigitalOcean droplet with Ubuntu 18.04, and at least 2gb of ram, as logstash eats 1 gb.
Use this Github Repo to setup basic packages.
Postfix-Mailgun-Setup
Follow DigitalOcean’s fantastic guide on setting up postfix with mailgun.
The tutorial is pretty good, provided you already have mailgun configured.
Certificates/Logstash Setup & Installation:
Logstash can be configured to use SSL certificates, when receiving TCP input.
A word of caution!
I followed virtually a 100 different tutorials on setting up self-signed SSL certificates for use with Logstash. None of them worked.
I spent nearly 6–8 hours on all this and eventually realized that there is no simple tool to automate the process. (6–8 hours is a long time for me)
I created a Github Repository, that you can find below, that does all the heavy lifting for you. I would strongly suggest you use it, rather than tinker with OpenSSL on your own.
More importantly, I’ve included a simple sh script to check whether Logstash responds to your client over SSL. It took me a long time to figure out how to even test the SSL connection.
Full instructions on my Github Repo.
Rails Heroku Logger Setup
I won’t get into the details of setting up a Rails App on Heroku, you can follow their own tutorial online, its pretty simple.
Gems
The gems I used were:
gem “lograge”
gem ‘logstash-event’
gem ‘logstash-logger’
Configure Rails App For Heroku
Configuring your Rails App can be a huge mess depending on which logger you plan to use.
I’m outlining my configuration, but it can get messy, primarily because I used Rails.logger.(debug/error) all throughout my application. So it became necessary to extend Rails.logger to output its stuff to Logstash in a json friendly format.
Create a module in your /lib directory:
This simplifies setting up the SSL Connection.
module SslModule class Context
def self.certificates_path
Rails.root.join("config","certificates","certificates")
end def self.get_context
ssl_context = OpenSSL::SSL::SSLContext.new()
ssl_context.cert = OpenSSL::X509::Certificate.new(File.open("# {certificates_path}/ssl_certificates/host/host.crt"))
ssl_context.ca_file = "#{certificates_path}/ssl_certificates/ca/ca.crt"
ssl_context.key = OpenSSL::PKey::RSA.new(File.open("#{certificates_path}/ssl_certificates/host/host.key"))
ssl_context.verify_mode = OpenSSL::SSL::VERIFY_PEER
ssl_context.ssl_version = :SSLv23
ssl_context
end def self.get_machine_ip
ip = Socket.ip_address_list.detect{|intf| intf.ipv4_private?}
ip.ip_address
end def self.get_logstash_ip
machine_ip = get_machine_ip
#puts "machine ip is :#{machine_ip}"
#puts "ip address is #{ENV['IP_ADDRESS']}"
return ENV["LOGSTASH_SERVER_IP_ADDRESS"] unless ENV["LOGSTASH_SERVER_IP_ADDRESS"].blank? unless ENV["LOGSTASH_SERVER_IP_ADDRESSES"].blank?
ENV["LOGSTASH_SERVER_IP_ADDRESSES"].split(",").each do |ip|
if machine_ip == ip
return ip
end
end
end return "0.0.0.0"
end ##should check eth for the ip address or hit 0.0.0.0
## if more than one ip address is specified.
## if no IP_ADDRESS is specified, defaults to eth / or 0.0.0.0
def self.logstash_logger
ls_ip = get_logstash_ip
#puts "ls ip is #{ls_ip}"
LogStashLogger.new(type: :tcp, host: get_logstash_ip, port: ENV["LOGSTASH_PORT"], ssl_context: get_context)
end
end
end
Create an initializer:
#logger.rb
Rails.logger = ::SslModule::Context.logstash_logger
Delayed::Worker.logger = Rails.logger
Add the following to your environment files:
#development.rb
Rails.application.configure do config.use_logstash = false
config.lograge.enabled = false
config.lograge.logger = Rails.logger
config.lograge.formatter = Lograge::Formatters::Logstash.new config.log_tags = [:uuid]
config.lograge.custom_payload do |controller|
{
host: controller.request.host,
user_id: controller.current_user.try(:id),
request_id: controller.request.request_id || RequestStore.store[:request_id],
params: controller.request.params.to_s
}
endend
This should get everything working, and you should see the logs popping up in your logstash server.
Comment if you face any problems!
All the best, this is not easy.