Setup Logstash on DigitalOcean with a Rails App hosted on Heroku : Full SSL

Logstash is a part of the ELK (Elasticsearch — Kibana — Logstash) stack, from Elastic. Its a centralized log drain for all your logs from various sources.

I couldn’t find any nice tutorials on setting up Logstash on a DigitalOcean droplet, and sending logs to it from a Heroku Rails App, over TCP with SSL. Certificate creation also seems poorly documented online.

So let’s get to it.

DigitalOcean Droplet Setup

Create A new DigitalOcean droplet with Ubuntu 18.04, and at least 2gb of ram, as logstash eats 1 gb.

Use this Github Repo to setup basic packages.

Postfix-Mailgun-Setup

Follow DigitalOcean’s fantastic guide on setting up postfix with mailgun.

How to Set Up a Mail Relay with Postfix and Mailgun on Ubuntu 16.04 | DigitalOcean

The tutorial is pretty good, provided you already have mailgun configured.

Certificates/Logstash Setup & Installation:

Logstash can be configured to use SSL certificates, when receiving TCP input.

A word of caution!

I followed virtually a 100 different tutorials on setting up self-signed SSL certificates for use with Logstash. None of them worked.

I spent nearly 6–8 hours on all this and eventually realized that there is no simple tool to automate the process. (6–8 hours is a long time for me)

I created a Github Repository, that you can find below, that does all the heavy lifting for you. I would strongly suggest you use it, rather than tinker with OpenSSL on your own.

More importantly, I’ve included a simple sh script to check whether Logstash responds to your client over SSL. It took me a long time to figure out how to even test the SSL connection.

Full instructions on my Github Repo.

Rails Heroku Logger Setup

I won’t get into the details of setting up a Rails App on Heroku, you can follow their own tutorial online, its pretty simple.

Gems

The gems I used were:

gem “lograge”
gem ‘logstash-event’
gem ‘logstash-logger’

Configure Rails App For Heroku

Configuring your Rails App can be a huge mess depending on which logger you plan to use.

I’m outlining my configuration, but it can get messy, primarily because I used Rails.logger.(debug/error) all throughout my application. So it became necessary to extend Rails.logger to output its stuff to Logstash in a json friendly format.

Create a module in your /lib directory:

This simplifies setting up the SSL Connection.

module SslModule
    class Context
  
      def self.certificates_path
         Rails.root.join("config","certificates","certificates")
      end
      def self.get_context
        ssl_context = OpenSSL::SSL::SSLContext.new()
        ssl_context.cert =     OpenSSL::X509::Certificate.new(File.open("#  {certificates_path}/ssl_certificates/host/host.crt"))
        ssl_context.ca_file = "#{certificates_path}/ssl_certificates/ca/ca.crt"
        ssl_context.key = OpenSSL::PKey::RSA.new(File.open("#{certificates_path}/ssl_certificates/host/host.key"))
        ssl_context.verify_mode = OpenSSL::SSL::VERIFY_PEER
        ssl_context.ssl_version = :SSLv23
        ssl_context
     end
     def self.get_machine_ip
       ip = Socket.ip_address_list.detect{|intf| intf.ipv4_private?}
       ip.ip_address
     end
     def self.get_logstash_ip
       machine_ip = get_machine_ip
       #puts "machine ip is :#{machine_ip}"
       #puts "ip address is #{ENV['IP_ADDRESS']}"
       return ENV["LOGSTASH_SERVER_IP_ADDRESS"] unless  ENV["LOGSTASH_SERVER_IP_ADDRESS"].blank?
       unless ENV["LOGSTASH_SERVER_IP_ADDRESSES"].blank?
         ENV["LOGSTASH_SERVER_IP_ADDRESSES"].split(",").each do |ip|
         if machine_ip == ip
           return ip
         end
       end
     end
     return "0.0.0.0"
  end
  ##should check eth for the ip address or hit 0.0.0.0
  ## if more than one ip address is specified.
  ## if no IP_ADDRESS is specified, defaults to eth / or 0.0.0.0
  def self.logstash_logger
      ls_ip = get_logstash_ip
      #puts "ls ip is #{ls_ip}"
      LogStashLogger.new(type: :tcp, host: get_logstash_ip, port: ENV["LOGSTASH_PORT"], ssl_context: get_context)
  end
 end
end

Create an initializer:

#logger.rb
Rails.logger = ::SslModule::Context.logstash_logger
Delayed::Worker.logger = Rails.logger

Add the following to your environment files:

#development.rb
Rails.application.configure do
  config.use_logstash = false
  config.lograge.enabled = false
  
  config.lograge.logger = Rails.logger  
  config.lograge.formatter = Lograge::Formatters::Logstash.new
  config.log_tags = [:uuid]
  config.lograge.custom_payload do |controller|
    {
      host: controller.request.host,
      user_id: controller.current_user.try(:id),
      request_id: controller.request.request_id || RequestStore.store[:request_id],
      params: controller.request.params.to_s
    }
  end
end

This should get everything working, and you should see the logs popping up in your logstash server.

Comment if you face any problems!

All the best, this is not easy.